Infrared image processing method and device, equipment and medium

Abstract

The embodiment of the invention provides an infrared image processing method and device, equipment and a medium, and relates to the technical field of image processing, and the method comprises the steps: obtaining an initial infrared image of a first resolution and a visible light image of a second resolution, the first resolution being lower than the second resolution; a first weight of the initial infrared image and a second weight of the visible light image are determined by using the similarity between the initial infrared image and the visible light image, the similarity is in positive correlation with the first weight, and the sum value of the first weight and the second weight is a preset value; and fusing the initial infrared image and the visible light image by using the first weight and the second weight to obtain a first target infrared image with a second resolution. By applying the technical scheme provided by the embodiment of the invention, the resolution of the infrared image can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an infrared image processing method, device, equipment and medium. Background Art

[0002] At present, infrared thermal imaging technology is widely used in security, medical treatment, industrial detection and other fields. However, due to the hardware cost and resolution limit of infrared sensors, the resolution of infrared (IR) images (Image, I) collected by existing infrared thermal imaging equipment is low, and the imaging quality cannot meet the application requirements of high-precision detection, detail analysis, etc. How to improve the resolution of infrared images without increasing hardware costs is a problem that needs to be solved urgently. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide an infrared image processing method, device, equipment and medium to improve the resolution of infrared images. The specific technical solution is as follows:

[0004] In a first aspect, an embodiment of the present application provides an infrared image processing method, the method comprising:

[0005] Acquire an initial infrared image of a first resolution and a visible light image of a second resolution, wherein the first resolution is lower than the second resolution;

[0006] Extracting a first feature of each pixel in the initial infrared image and a second feature of each pixel in the visible light image;

[0007] Calculating the similarity between the initial infrared image and the visible light image at each pixel by using the first feature and the second feature of each pixel;

[0008] Determine a first weight of the initial infrared image and a second weight of the visible light image by using the similarity between the initial infrared image and the visible light image, wherein the similarity is positively correlated with the first weight, and a sum of the first weight and the second weight is a preset value;

[0009] Using the first weight and the second weight, the initial infrared image and the visible light image are fused to obtain a first target infrared image with the second resolution;

[0010] The step of determining a first weight of the initial infrared image and a second weight of the visible light image by using the similarity between the initial infrared image and the visible light image comprises:

[0011] Determine a first weight of the initial infrared image and a second weight of the visible light image at each pixel by using the similarity between the initial infrared image and the visible light image at each pixel;

[0012] The step of fusing the initial infrared image and the visible light image using the first weight and the second weight to obtain the first target infrared image of the second resolution includes:

[0013] The first weight of the initial infrared image at each pixel and the second weight of the visible light image are used to fuse each pixel in the initial infrared image and the visible light image to obtain the first target infrared image with the second resolution.

[0014] In some embodiments, the step of fusing each pixel in the initial infrared image and the visible light image using the first weight of the initial infrared image at each pixel and the second weight of the visible light image to obtain the first target infrared image of the second resolution includes:

[0015] The first feature of each pixel in the initial infrared image and the second feature of each pixel in the visible light image are weighted averaged by using the first weight and the second weight at each pixel to obtain the first target infrared image with the second resolution.

[0016] In some embodiments, the step of determining the first weight of the initial infrared image and the second weight of the visible light image by using the similarity between the initial infrared image and the visible light image further includes:

[0017] Determine a first weight of the initial infrared image and a second weight of the visible light image at the multiple scales by using similarities between the initial infrared image and the visible light image at the multiple scales;

[0018] The step of fusing the initial infrared image and the visible light image using the first weight and the second weight to obtain the first target infrared image of the second resolution also includes:

[0019] Using the first weight and the second weight at each scale, the initial infrared image and the visible light image are fused to obtain a target sub-image at each scale;

[0020] The target sub-images at the multiple scales are fused using the weights corresponding to the multiple scales to obtain the first target infrared image at the second resolution.

[0021] In some embodiments, after obtaining the infrared image of the first target with the second resolution, the method further includes:

[0022] The first target infrared image is subjected to a preset processing to obtain a second target infrared image, wherein the preset processing includes at least one of the following: performing denoising processing using a residual block and performing detail enhancement processing using an adversarial network.

[0023] In some embodiments, the method further comprises:

[0024] Acquire a first sample infrared image, a second sample infrared image, and a sample visible light image, wherein the resolution of the first sample infrared image is a third resolution, the resolution of the second sample infrared image and the sample visible light image are a fourth resolution, and the third resolution is higher than the fourth resolution;

[0025] Inputting the first sample infrared image and the sample visible light image into a preset network model to obtain a reconstructed infrared image, wherein the preset network model is used to determine a third weight of the first sample infrared image and a fourth weight of the sample visible light image by using the similarity between the first sample infrared image and the sample visible light image, wherein the similarity is positively correlated with the third weight, and the sum of the third weight and the fourth weight is the preset value; using the third weight and the fourth weight, fusing the first sample infrared image and the sample visible light image, and outputting the reconstructed infrared image;

[0026] acquiring a reconstruction loss and a structural similarity loss between the reconstructed infrared image and the second sample infrared image;

[0027] Determine whether the preset network model converges according to the reconstruction loss and the structural similarity loss.

[0028] In some embodiments, the preset network model includes an adversarial network, and the method further includes:

[0029] Acquire an adversarial loss between the reconstructed infrared image and the second sample infrared image;

[0030] The step of determining whether the preset network model converges according to the reconstruction loss and the structural similarity loss comprises:

[0031] Whether the preset network model converges is determined according to the sum of the reconstruction loss, the structural similarity loss and the adversarial loss.

[0032] In a second aspect, an embodiment of the present application provides an infrared image processing device, the device comprising:

[0033] An acquisition module, configured to acquire an initial infrared image of a first resolution and a visible light image of a second resolution, wherein the first resolution is lower than the second resolution;

[0034] A determination module, configured to extract a first feature of each pixel in the initial infrared image and a second feature of each pixel in the visible light image; using the first feature and the second feature of each pixel, calculate the similarity between the initial infrared image and the visible light image at each pixel; using the similarity between the initial infrared image and the visible light image, determine a first weight of the initial infrared image and a second weight of the visible light image, wherein the similarity is positively correlated with the first weight, and the sum of the first weight and the second weight is a preset value;

[0035] a fusion module, configured to fuse the initial infrared image and the visible light image using the first weight and the second weight to obtain a first target infrared image with the second resolution;

[0036] The determination module is specifically used to: determine a first weight of the initial infrared image and a second weight of the visible light image at each pixel point by using the similarity between the initial infrared image and the visible light image at each pixel point;

[0037] The fusion module is specifically used to: use the first weight of the initial infrared image at each pixel point and the second weight of the visible light image to fuse each pixel point in the initial infrared image and the visible light image to obtain the first target infrared image with the second resolution.

[0038] In some embodiments, the fusion module is specifically used to: use the first weight and the second weight at each pixel point to weighted average the first feature of each pixel point in the initial infrared image and the second feature of each pixel point in the visible light image to obtain the first target infrared image of the second resolution.

[0039] In some embodiments, the determination module is specifically used to: determine a first weight of the initial infrared image and a second weight of the visible light image at the multiple scales by using the similarity between the initial infrared image and the visible light image at the multiple scales;

[0040] The fusion module is specifically used to: use the first weight and the second weight at each scale to fuse the initial infrared image and the visible light image to obtain a target sub-image at each scale; use the weights corresponding to the multiple scales to fuse the target sub-images at the multiple scales to obtain the first target infrared image of the second resolution.

[0041] In some embodiments, the device also includes a processing module, which is used to: after obtaining the first target infrared image of the second resolution, perform preset processing on the first target infrared image to obtain a second target infrared image, and the preset processing includes at least one of the following: denoising processing using a residual block and detail enhancement processing using an adversarial network.

[0042] In some embodiments, the apparatus further comprises a training module, wherein the training module is configured to:

[0043] Acquire a first sample infrared image, a second sample infrared image, and a sample visible light image, wherein the resolution of the first sample infrared image is a third resolution, the resolution of the second sample infrared image and the sample visible light image are a fourth resolution, and the third resolution is higher than the fourth resolution;

[0044] Inputting the first sample infrared image and the sample visible light image into a preset network model to obtain a reconstructed infrared image, wherein the preset network model is used to determine a third weight of the first sample infrared image and a fourth weight of the sample visible light image by using the similarity between the first sample infrared image and the sample visible light image, wherein the similarity is positively correlated with the third weight, and the sum of the third weight and the fourth weight is the preset value; using the third weight and the fourth weight, fusing the first sample infrared image and the sample visible light image, and outputting the reconstructed infrared image;

[0045] acquiring a reconstruction loss and a structural similarity loss between the reconstructed infrared image and the second sample infrared image;

[0046] Determine whether the preset network model converges according to the reconstruction loss and the structural similarity loss.

[0047] In some embodiments, the preset network model includes an adversarial network, and the training module is further used to: obtain an adversarial loss between the reconstructed infrared image and the second sample infrared image;

[0048] The training module is specifically used to determine whether the preset network model converges according to the sum of the reconstruction loss, the structural similarity loss and the adversarial loss.

[0049] In a third aspect, an embodiment of the present application provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0050] Memory, used to store computer programs;

[0051] The processor is used to implement any method described in the first aspect when executing a program stored in the memory.

[0052] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, any method described in the first aspect is implemented.

[0053] An embodiment of the present application also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any method described in the first aspect of the above embodiments.

[0054] Beneficial effects of the embodiments of the present application:

[0055] In the technical solution provided in the embodiment of the present application, a high-resolution visible light image is used to guide the super-resolution reconstruction of a low-resolution infrared image, which can ensure that the temperature information of the infrared image is consistent with the high-frequency details of the visible light image. In addition, the similarity between the low-resolution infrared image and the high-resolution visible light image is used to dynamically determine the weight of the infrared image and the weight of the visible light image when reconstructing the infrared image, and the infrared image and the visible light image are fused using the dynamically determined weights to achieve super-resolution reconstruction. When the similarity between the visible light image and the infrared image is high, the weight is tilted toward the infrared image to retain more characteristics of the infrared image, and when the similarity is low, the weight is tilted toward the visible light image, which can effectively prevent the generation of artifacts while retaining key temperature and texture information, ensuring that the generated infrared image has high resolution and high authenticity, and improving the resolution of the infrared image.

[0056] Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0058] Figure 1 A schematic diagram of a first flow chart of an infrared image processing method provided in an embodiment of the present application;

[0059] Figure 2 A second flow chart of the infrared image processing method provided in the embodiment of the present application;

[0060] Figure 3 A third flow chart of the infrared image processing method provided in the embodiment of the present application;

[0061] Figure 4 A flowchart of a training method provided in an embodiment of the present application;

[0062] Figure 5 A schematic diagram of the structure of an infrared image processing device provided in an embodiment of the present application;

[0063] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0064] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.

[0065] At present, infrared thermal imaging technology is widely used in security, medical treatment, industrial detection and other fields. Infrared thermal imaging equipment detects infrared radiation on the surface of an object and converts it into a visible image. Since infrared thermal imaging technology can work in complete darkness, smoke or obstructed environments, and can directly measure temperature information, it has irreplaceable advantages in harsh environments. However, due to the hardware cost and resolution limitations of infrared sensors, the resolution of infrared images collected by existing infrared thermal imaging equipment is low, and the imaging quality is difficult to meet the application requirements of high-precision detection, detail analysis, etc. How to improve the resolution of infrared images without increasing hardware costs is a problem that needs to be solved urgently.

[0066] In the existing technology, super-resolution (SR) reconstruction (rec) can be performed based on interpolation, sparse representation, deep learning and other methods to improve image resolution. However, the interpolation-based method only relies on the low-resolution information of the image itself and cannot restore the high-frequency details in the image. The generated high-resolution (HR) images are usually blurred, especially in the edge and texture areas. The sparse representation-based method has high computational complexity, slow processing speed, and it is difficult to build a high-quality dictionary in some complex scenes. In addition, the deep learning-based method relies on large-scale high-quality data training and is unstable in new environments.

[0067] In order to solve the above problems, an embodiment of the present application provides an infrared image processing method, which can be applied to electronic devices such as computers, servers, and clusters. For the convenience of description, the electronic devices are used as the execution subjects in the following, and it does not serve as a limitation.

[0068] See also Figure 1 , Figure 1 A first flow chart of an infrared image processing method provided in an embodiment of the present application is provided. The infrared image processing method includes the following steps.

[0069] Step S11, acquiring an initial infrared image with a first resolution and a visible light image with a second resolution, wherein the first resolution is lower than the second resolution.

[0070] Step S12, using the similarity between the initial infrared image and the visible light image, determine a first weight of the initial infrared image and a second weight of the visible light image, the similarity is positively correlated with the first weight, and the sum of the first weight and the second weight is a preset value.

[0071] Step S13: using the first weight and the second weight, fuse the initial infrared image and the visible light image to obtain an infrared image of the first target with a second resolution.

[0072] In the technical solution provided in the embodiment of the present application, the electronic device uses a high-resolution visible light image to guide the super-resolution reconstruction of the low-resolution infrared image, which can ensure that the temperature information of the infrared image is consistent with the high-frequency details of the visible light image. In addition, the electronic device uses the similarity between the low-resolution infrared image and the high-resolution visible light image to dynamically determine the weight of the infrared image and the weight of the visible light image when reconstructing the infrared image, and uses the dynamically determined weights to fuse the infrared image and the visible light image to achieve super-resolution reconstruction. When the similarity between the visible light image and the infrared image is high, the weight is tilted toward the infrared image to retain more characteristics of the infrared image, and when the similarity is low, the weight is tilted toward the visible light image, which can effectively prevent the generation of artifacts while retaining key temperature and texture information, ensuring that the generated infrared image has high resolution and high authenticity, and improving the resolution of the infrared image.

[0073] In the above step S11, the electronic device collects infrared images and visible light (VIS) images of the same scene through infrared thermal imaging devices, cameras and other collection devices. The collected infrared images are low-resolution (LR) images, and the collected visible light images are high-resolution images. In the embodiment of the present application, the collection device can be a separate device or integrated in the electronic device, and the collection device is not limited here.

[0074] The electronic device can obtain the infrared image and the visible light image collected by the collection device. In one example, the electronic device can directly use the collected visible light image as the visible light image to be processed. , and the collected infrared image is used as the initial infrared image to be processed In another example, the electronic device may use the collected visible light image as the visible light image to be processed. , and preprocess the collected infrared image, such as performing preliminary noise filtering and normalization processing on the collected infrared image, and use the preprocessed infrared image as the initial infrared image to be processed The manner in which the electronic device acquires the initial infrared image and the visible light image is not limited herein.

[0075] In an embodiment of the present application, in order to achieve super-resolution reconstruction of the initial infrared image, the electronic device may first scale (resize) the initial infrared image, and perform subsequent steps S12 and S13 on the scaled initial infrared image. The scale (scale, s) of the scaled initial infrared image is the same as that of the visible light image, and the number of pixels of the scaled initial infrared image is the same as that of the visible light image, that is, the resolution of the scaled initial infrared image is the same as that of the visible light image. After the electronic device performs the scaling process, the scaled initial infrared image obtained is relatively blurred and of low quality, so it is necessary to perform super-resolution reconstruction on the scaled initial infrared image to improve clarity and texture details. The initial infrared images referred to subsequently are all scaled initial infrared images.

[0076] In the above step S12, the first weight is the weight of the initial infrared image, the second weight is the weight of the visible light image, and the sum of the first weight and the second weight is a preset value, which can be 1, 2, etc., and can be set specifically according to actual conditions. For the sake of ease of description, the following description will take the preset value of 1 as an example, and will not be limited here.

[0077] The similarity between the initial infrared image and the visible light image is positively correlated with the first weight and negatively correlated with the second weight, that is, the higher the similarity, the greater the first weight and the smaller the second weight; the lower the similarity, the smaller the first weight and the greater the second weight.

[0078] In one example, after the electronic device determines the similarity between the initial infrared image and the visible light image, it can directly use the determined similarity as the first weight of the initial infrared image, and calculate the difference between the preset value and the first weight as the second weight of the visible light image.

[0079] In another example, the electronic device can use the determined similarity as an independent variable, combine it with a preset positive correlation function, calculate the first weight, and calculate the difference between the preset value and the first weight to obtain the second weight. The preset positive correlation function can be a positive proportional function with a slope greater than 0, a square function, etc., which is not limited to this.

[0080] Similarly, the electronic device can calculate the second weight in combination with a preset negative correlation function, and calculate the difference between the preset value and the second weight to obtain the first weight. The preset negative correlation function can be a direct proportional function or an inverse proportional function with a slope less than 0.

[0081] The electronic device may also calculate the first weight and the second weight respectively in combination with the preset positive correlation function and the negative correlation function, in which case the sum of the preset positive correlation function and the negative correlation function is a preset value. The manner in which the electronic device determines the first weight and the second weight is not limited here.

[0082] In the above step S13, the electronic device uses the first weight of the initial infrared image and the second weight of the visible light image to fuse the initial infrared image with the visible light image, that is, the visible light image is used as a guide image, and the high-resolution information of the visible light image is used to guide the initial infrared image to perform super-resolution reconstruction. After that, the electronic device obtains a fused infrared image, which is the first target infrared image for super-resolution reconstruction. The resolution of the first target infrared image is the same as that of the visible light image, and is a higher quality and clearer infrared image.

[0083] In the embodiment of the present application, the electronic device can use the preset network model to perform super-resolution reconstruction to obtain the first target infrared image. The preset network model is a pre-trained guided super-resolution (Guided Super Resolution, GuidedSR) network model. The training process of the preset network model will be described in detail later and will not be described in detail here.

[0084] The electronic device can input the initial infrared image and the visible light image into the preset network model. The preset network model uses the similarity between the initial infrared image and the visible light image to determine the first weight and the second weight, and uses the first weight and the second weight to fuse the initial infrared image and the visible light image to obtain and output the first target infrared image. The process of super-resolution reconstruction by the preset network model can be expressed as:

[0085] Formula (1)

[0086] in, represents the infrared image of the first target; Indicates the preset network model; represents the initial infrared image; Represents a visible light image.

[0087] See also Figure 2 , Figure 2A second flow chart of the infrared image processing method provided in an embodiment of the present application is provided. The infrared image processing method may include the following steps.

[0088] Step S21, acquiring an initial infrared image of a first resolution and a visible light image of a second resolution, wherein the first resolution is lower than the second resolution. This is the same as the above step S11.

[0089] Step S22: extracting a first feature of each pixel in the initial infrared image and a second feature of each pixel in the visible light image.

[0090] Step S23, using the first feature and the second feature of each pixel, calculate the similarity between the initial infrared image and the visible light image at each pixel.

[0091] Step S24, using the similarity between the initial infrared image and the visible light image at each pixel, determine a first weight of the initial infrared image and a second weight of the visible light image at each pixel.

[0092] Step S25, using the first weight of the initial infrared image at each pixel and the second weight of the visible light image, fuse each pixel in the initial infrared image and the visible light image to obtain a first target infrared image of a second resolution.

[0093] In the technical solution provided in the embodiment of the present application, the electronic device can adaptively adjust the first weight and the second weight according to the first feature and the second feature of each pixel of the low-resolution infrared image and the high-resolution visible light image, and then fuse the infrared image and the visible light image for each pixel, thereby improving the fusion accuracy and making the fused pixel more accurate. While enhancing the image and improving the resolution, the key information of the infrared image (such as temperature information) is retained, ensuring the consistency of image enhancement and temperature information.

[0094] In the above step S22, the first feature is the image feature of the initial infrared image, and the second feature is the image feature of the visible light image. The electronic device extracts features from the initial infrared image and the visible light image respectively to obtain image features of the initial infrared image and the visible light image, that is, to obtain the first feature of each pixel in the initial infrared image and the second feature of each pixel in the visible light image.

[0095] In the embodiment of the present application, the number of pixels in the initial infrared image and the visible light image is the same, and any pixel can be represented as (x, y), where x represents the horizontal coordinate and y represents the vertical coordinate.

[0096] The electronic device can extract features using a feature extraction network, which can be a convolutional neural network, etc., and the feature extraction network is not limited here. The electronic device can input the initial infrared image and the visible light image into the feature extraction network corresponding to the image to extract features. The process of feature extraction by the feature extraction network can be expressed as:

[0097] Formula (2)

[0098] Formula (3)

[0099] in, A second feature representing a pixel in a visible light image; A feature extraction network representing a visible light image; represents a visible light image; Represents the first feature of the pixel in the initial infrared image; A feature extraction network representing the initial infrared image; represents the initial infrared image.

[0100] In the above step S23, after obtaining the first feature and the second feature, the electronic device calculates the similarity between the first feature and the second feature of each pixel point, and the similarity is the similarity between the initial infrared image and the visible light image at the pixel point.

[0101] In the embodiment of the present application, the electronic device may use a distance algorithm such as Euclidean distance, Canberra distance, and Manhattan distance to calculate the distance between the first feature and the second feature, and use the distance to represent the similarity. The smaller the distance, the higher the similarity; the larger the distance, the lower the similarity. The way in which the electronic device calculates the similarity and represents the similarity is not limited here.

[0102] In the above step S24, for each pixel, the electronic device determines the first weight and the second weight at the pixel using the similarity between the initial infrared image and the visible light image at the pixel. In the embodiment of the present application, the manner in which the electronic device determines the first weight and the second weight using the similarity can be specifically referred to the relevant description of the above step S12, which will not be described in detail here.

[0103] In the above step S25, for each pixel point, the electronic device uses the first weight and the second weight at the pixel point to fuse the pixel point in the initial infrared image and the pixel point in the visible light image to obtain a fused pixel point. The electronic device obtains all the fused pixel points, and then obtains the first target infrared image composed of all the pixel points.

[0104] In some embodiments, the above step S25 can be implemented by the following steps: using the first weight and the second weight at each pixel point, weighted averaging the first feature of each pixel point in the initial infrared image and the second feature of each pixel point in the visible light image to obtain a first target infrared image of a second resolution.

[0105] For each pixel, the electronic device calculates the product of the first weight at the pixel and the first feature of the pixel in the initial infrared image (referred to as the first product), and the product of the second weight at the pixel and the second feature of the pixel in the visible light image (referred to as the second product), and accumulates the two products to obtain the fused feature value of the pixel. The electronic device obtains the fused features of all the pixels, that is, the features of the first target infrared image, and then obtains the first target infrared image. The electronic device can obtain the fused features more accurately by taking into account the relative importance of the initial infrared image and the visible light image through weighted averaging.

[0106] In the embodiment of the present application, the electronic device can use a guided filter (GF) to perform the above steps S23 to S25 for super-resolution reconstruction. The electronic device can input the first feature and the second feature of each pixel into the guided filter, which determines the first weight and the second weight, and fuses the first feature and the second feature of each pixel to obtain the fused feature of each pixel, and then obtain the feature of the first target infrared image, that is, obtain the first target infrared image. The process of super-resolution reconstruction by the guided filter can be expressed as:

[0107] Formula (4)

[0108] in, Indicates the features of the pixels in the infrared image of the first target, and may also indicate the infrared image of the first target; represents the guided filter; Represents the first feature of the pixel in the initial infrared image; Represents the second feature of a pixel in a visible light image.

[0109] After the electronic device inputs the first feature and the second feature of each pixel into the guided filter, it calculates the similarity between the initial infrared image and the visible light image at each pixel. The similarity at the pixel (x, y) can be expressed as:

[0110] Formula (5)

[0111] Wherein, d represents the distance between the first feature and the second feature at the pixel point (x, y), which is used to represent the similarity at the pixel point (x, y). The larger d is, the smaller the similarity is, and the smaller d is, the greater the similarity is. (x, y) represents the second feature of the pixel point (x, y) in the visible light image; (x, y) represents the first feature of the pixel (x, y) in the initial infrared image; Represents a preset constant. In order to prevent a smaller constant with a denominator of zero, it can be set according to actual conditions and is not limited here.

[0112] Furthermore, the electronic device can determine the first weight and the second weight at each pixel point using the similarity at the pixel point (x, y). The second weight at the pixel point (x, y) can be expressed as:

[0113] Formula (6)

[0114] The first weight at the pixel point (x, y) can be expressed as:

[0115] Formula (7)

[0116] in, (x,y) represents the second weight of the visible light image at the pixel point (x,y); 1- (x,y) represents the first weight of the initial infrared image at the pixel point (x,y); (x, y) represents the second feature of the pixel point (x, y) in the visible light image; (x, y) represents the first feature of the pixel (x, y) in the initial infrared image; Indicates a preset constant.

[0117] The electronic device can use the first weight and the second weight at the pixel point (x, y) to weighted average the first feature of the pixel point (x, y) in the initial infrared image and the second feature of the pixel point (x, y) in the visible light image. The output of the guided filter at the pixel point (x, y) can be expressed as:

[0118] Formula (8)

[0119] in, (x, y) represents the feature of the pixel point (x, y) in the infrared image of the first target; (x, y) represents the second weight of the visible light image at the pixel point (x, y); (x, y) represents the second feature of the pixel (x, y) in the visible light image; 1- (x,y) represents the first weight of the initial infrared image at the pixel point (x,y); (x, y) represents the first feature of the pixel point (x, y) in the initial infrared image.

[0120] Formula (6) to Formula (8) indicate that the first weight and the second weight can be dynamically adjusted according to the difference in local features (i.e., the first feature and the second feature) between the initial infrared image and the visible light image at the pixel point (x, y). When the local feature similarity between the initial infrared image and the visible light image at the pixel point (x, y) is high, the second weight is small, the first weight is large, and the weight is inclined toward the initial infrared image; when the local feature similarity between the initial infrared image and the visible light image at the pixel point (x, y) is low, the second weight is large, the first weight is small, and the weight is inclined toward the visible light image.

[0121] In the embodiment of the present application, the electronic device may also calculate, for each pixel point, a first product of a first weight at the pixel point and a pixel value of the pixel point in the initial infrared image, and a second product of a second weight at the pixel point and a pixel value of the pixel point in the visible light image, and accumulate the two calculated products to obtain a fused pixel value of the pixel point, thereby obtaining a first target infrared image. The manner in which the electronic device fuses the pixels using the first weight and the second weight is not limited herein.

[0122] In the technical solution provided in the embodiment of the present application, for the initial infrared image and visible light image at the same scale and resolution, each pixel in the initial infrared image is relatively fuzzy and inaccurate. The electronic device uses the first weight and the second weight to fuse the features of the initial infrared image and the features of the visible light image for each pixel, accurately calculates the feature value of each pixel, and realizes super-resolution reconstruction.

[0123] See also Figure 3 , Figure 3 This is a third flow chart of the infrared image processing method provided in an embodiment of the present application. The infrared image processing method may include the following steps.

[0124] Step S31, acquiring an initial infrared image with a first resolution and a visible light image with a second resolution, wherein the first resolution is lower than the second resolution. This is the same as the above step S11.

[0125] Step S32, using the similarities between the initial infrared image and the visible light image at various scales, determine first weights of the initial infrared image and second weights of the visible light image at various scales.

[0126] Step S33, using the first weight and the second weight at each scale, the initial infrared image and the visible light image are fused to obtain a target sub-image at each scale.

[0127] Step S34: using weights corresponding to the various scales, fuse the target sub-images at the various scales to obtain an infrared image of the first target at a second resolution.

[0128] In the technical solution provided in the embodiment of the present application, the electronic device processes the initial infrared image and the visible light image at multiple scales, and fuses the target sub-images at multiple scales in an adaptive weighted manner, thereby ensuring detail processing at different scales, effectively utilizing the initial infrared image information and visible light image information at different scales, enhancing the detail performance and resolution of the fused infrared image, and making the super-resolution reconstruction effect more stable.

[0129] In the above step S32, the electronic device converts the initial infrared image and the visible light image into images at multiple scales. For each scale, the electronic device determines the first weight and the second weight at the scale by using the similarity between the initial infrared image and the visible light image at the scale. The specific manner in which the electronic device determines the first weight and the second weight by using the similarity can be found in the relevant description of the above step S12.

[0130] In an embodiment of the present application, the electronic device can extract image features of the initial infrared image and the visible light image at multiple scales, calculate the similarity between the first feature of the initial infrared image and the second feature of the visible light image at multiple scales, and use the calculated similarity as the similarity between the initial infrared image and the visible light image at multiple scales.

[0131] For example, the electronic device can convert the scales of the initial infrared image and visible light image of scale m×n into m×n, (m / 2)×(n / 2) and (m / 4)×(n / 4), respectively, to obtain initial infrared images and visible light images at three scales, and extract image features from the initial infrared images and visible light images at the three scales to obtain low-frequency image features, medium-frequency image features and high-frequency image features, and calculate the similarities at the three scales based on the extracted features.

[0132] In one example, the electronic device may extract image features of the initial infrared image and the visible light image based on pixel points, calculate similarity, and determine the first weight and the second weight.

[0133] After obtaining the initial infrared image and visible light image at multiple scales, the electronic device can extract the first feature of each pixel in the initial infrared image and the second feature of each pixel in the visible light image at each scale. The electronic device can use the feature extraction network for multiple scales to input the initial infrared image and visible light image at each scale into the feature extraction network corresponding to the image and scale, respectively, to perform feature extraction. The image features extracted by the feature extraction network at scale s can be expressed as:

[0134] Formula (9)

[0135] Formula (10)

[0136] Among them, s represents any scale; Represents the second feature of the pixel in the visible light image at scale s; represents the feature extraction network for visible light images of scale s; represents a visible light image at scale s; Represents the first feature of the pixel in the initial infrared image at scale s; represents the feature extraction network for the initial infrared image of scale s; Represents the initial infrared image at scale s.

[0137] For each scale, the electronic device can use the first feature and the second feature of each pixel at the scale to calculate the similarity of the initial infrared image and the visible light image at each pixel at the scale. The electronic device can extract the first feature and the second feature to calculate the similarity, as described above. Figure 2 The relevant description of some parts will not be elaborated here.

[0138] For each scale, the electronic device can use the similarity between the initial infrared image and the visible light image at each pixel point at the scale to determine the first weight and the second weight at the pixel point at the scale. The specific method for the electronic device to determine the first weight and the second weight can be found in the above Figure 2 The relevant description of some parts will not be elaborated here.

[0139] In the above step S33, for each scale, the electronic device uses the first weight and the second weight at the scale to fuse the initial infrared image and the visible light image at the scale to obtain a fused image at the scale. The fused image is the target sub-image at the scale. The specific way in which the electronic device fuses the initial infrared image and the visible light image can be found in the relevant description of the above step S13.

[0140] In one example, the electronic device may fuse the initial infrared image and the visible light image based on pixels.

[0141] After obtaining the first weight and the second weight at each pixel point at multiple scales, the electronic device can use the first weight and the second weight at each pixel point at the scale for each scale to fuse each pixel point in the initial infrared image and the visible light image to obtain a fused image at the scale.

[0142] In an embodiment of the present application, the electronic device can use a guided filter to obtain target sub-images at multiple scales. The electronic device can input the first feature and the second feature of each pixel at each scale into the guided filter corresponding to each scale. The guided filter corresponding to each scale determines the first weight and the second weight of each pixel at the scale, and fuses each pixel of the initial infrared image and the visible light image at the scale to obtain the fused feature value of each pixel at the scale, and obtain the feature of the target sub-image at the scale, that is, to obtain the target sub-image at the scale. The process of obtaining the target sub-image at scale s by the guided filter corresponding to scale s can be expressed as:

[0143] Formula (11)

[0144] Among them, s represents any scale; It represents the features of the pixels in the target sub-image at scale s, and can also represent the target sub-image at scale s; represents the guided filter at scale s; Represents the first feature of the pixel in the initial infrared image at scale s; The second feature of the pixel in the visible light image at scale s is represented. The electronic device obtains the fused image in detail, as shown in the above Figure 2 The relevant description of some parts will not be elaborated here.

[0145] In the above step S34, the weights corresponding to the multiple scales may be determined in the process of training the preset network model, or may be pre-set, and there is no limitation on this. The electronic device uses the determined weights corresponding to the multiple scales (i.e., the weighting coefficients of the multiple scales) to perform weighted summation on the target sub-images at the multiple scales, realize multi-scale image fusion, and obtain the first target infrared image. The process of fusing the target sub-images at different scales to obtain the first target infrared image can be expressed as:

[0146] Formula (12)

[0147] in, represents the features of the pixels in the infrared image of the first target, and can also represent the infrared image of the first target; s represents any scale; Represents the weight corresponding to scale s; Represents the features of the pixels in the target sub-image at scale s.

[0148] In some embodiments, after obtaining the first target infrared image, the electronic device may perform subsequent processing on the first target infrared image to obtain a second target infrared image that has undergone super-resolution reconstruction and subsequent processing. The above infrared image processing method may also include the following steps: performing a preset processing on the first target infrared image to obtain a second target infrared image, wherein the preset processing includes at least one of the following: performing denoising processing using a residual block and performing detail enhancement processing using an adversarial network.

[0149] In the embodiment of the present application, according to the different processing methods included in the preset processing, the electronic device can process the infrared image of the first target in the following three ways.

[0150] In case 1, the preset processing includes denoising processing using a residual block.

[0151] In the embodiment of the present application, the electronic device can use the residual block in the pre-trained residual network (Residual Network, ResNet) to remove the noise in the first target infrared image. The electronic device can input the features of the first target infrared image into the residual network based on the features of the first target infrared image obtained, and use the residual block to learn the noise distribution of the first target infrared image. The process of the residual block learning the noise distribution can be expressed as:

[0152] Formula (13)

[0153] in, represents the noise distribution in the infrared image of the first target; represents the residual block; Represents the features of the pixels in the infrared image of the first target.

[0154] After obtaining the noise distribution of the first target infrared image, the electronic device can remove the noise component from the features of the first target infrared image to obtain the features of the first target infrared image after denoising, that is, to obtain the first target infrared image after denoising. The process of removing the noise component can be expressed as:

[0155] Formula (14)

[0156] in, The feature representing the first target infrared image after denoising can represent the first target infrared image after denoising; Represents the features of the pixels in the infrared image of the first target; represents the noise distribution in the first target infrared image. The first target infrared image after denoising obtained by the electronic device is the second target infrared image.

[0157] In case 2, the preset processing includes using an adversarial network to perform detail enhancement processing.

[0158] In the embodiment of the present application, the electronic device can enhance the details of the first target infrared image by using the Generative Adversarial Networks (GAN) (referred to as the adversarial network) in the pre-trained detail enhancement module. The electronic device can input the features of the first target infrared image into the detail enhancement module according to the features of the first target infrared image obtained, and use the generator of the adversarial network in the detail enhancement module to generate the first target infrared image with enhanced details. The first target infrared image with enhanced details obtained by the electronic device is the second target infrared image.

[0159] In case three, the preset processing includes denoising using residual blocks and detail enhancement using adversarial networks.

[0160] In the embodiment of the present application, in order to suppress noise and enhance details at the same time, the electronic device can adopt a joint optimization strategy, first using the residual block in the pre-trained residual network to perform denoising on the first target infrared image to obtain the denoised first target infrared image, for details, please refer to the relevant description of situation 1. After the electronic device obtains the features of the denoised first target infrared image, it uses the adversarial network in the pre-trained detail enhancement module to perform detail enhancement on the features of the denoised first target infrared image to obtain the processed image as the second target infrared image. The process of detail enhancement by the detail enhancement module can be expressed as:

[0161] Formula (15)

[0162] in, represents the infrared image of the second target; Representation detail enhancement module; Represents the characteristics of the first target infrared image after denoising.

[0163] In an embodiment of the present application, the electronic device may first use the adversarial network in the pre-trained detail enhancement module to perform detail enhancement processing on the first target infrared image, and then use the residual block in the pre-trained residual network to denoise the first target infrared image after detail enhancement to obtain a second target infrared image, without limitation.

[0164] In the technical solution provided in the embodiment of the present application, the electronic device uses the residual block to suppress noise, and improves the visual quality of the reconstructed image by learning the noise distribution of the image; the detail enhancement module uses the adversarial network to enhance the detail information in the infrared image, further improves the edge clarity of the image and the resolution of the small target area, ensures that the infrared image has better visual details and quality, and effectively improves the clarity and reliability of the infrared image. In addition, the electronic device can also adopt a joint optimization strategy of noise suppression and detail enhancement to reduce noise while improving the visual quality of the image.

[0165] In the embodiment of the present application, the electronic device can use the preset network model to perform preset processing on the first target infrared image, that is, the preset network model can include a residual network and an adversarial network. The electronic device inputs the initial infrared image and the visible light image into the preset network model to obtain the first target infrared image, and uses the residual network and the adversarial network to perform denoising and detail enhancement processing on the first target external image, respectively, to obtain the second target infrared image.

[0166] In some embodiments, the electronic device may further train a preset network model before processing the initial infrared image. Figure 4 , Figure 4 A flow chart of a training method provided in an embodiment of the present application, the training method may include the following steps.

[0167] Step S41, obtaining a first sample infrared image, a second sample infrared image and a sample visible light image, wherein the resolution of the first sample infrared image is a third resolution, and the resolutions of the second sample infrared image and the sample visible light image are a fourth resolution, and the third resolution is higher than the fourth resolution.

[0168] Step S42, input the first sample infrared image and the sample visible light image into a preset network model to obtain a reconstructed infrared image, the preset network model is used to use the similarity between the first sample infrared image and the sample visible light image to determine the third weight of the first sample infrared image and the fourth weight of the sample visible light image, the similarity is positively correlated with the third weight, and the sum of the third weight and the fourth weight is a preset value; use the third weight and the fourth weight to fuse the first sample infrared image and the sample visible light image, and output the reconstructed infrared image.

[0169] Step S43: obtaining a reconstruction loss and a structural similarity loss between the reconstructed infrared image and the second sample infrared image.

[0170] Step S44, determining whether the preset network model has converged based on the reconstruction loss and the structural similarity loss. If so, executing step S45; if not, adjusting the parameters of the preset network model and returning to executing step S42.

[0171] Step S45, outputting the preset network model.

[0172] In the technical solution provided by the embodiment of the present application, since the electronic device uses visible light images for guidance in the process of super-resolution reconstruction of infrared images, the amount of infrared image data required for training the preset network model is reduced. In addition, the electronic device uses a joint loss function including reconstruction loss and structural similarity loss to optimize the preset network model to ensure that the reconstructed infrared image is optimal in terms of structural similarity and overall visual effect. In addition, the coefficients of each loss term and the parameters of the preset network model can be dynamically adjusted through training to achieve the best effect.

[0173] In the above step S41, the electronic device obtains a sample infrared image and a sample visible light image of the same scene. For the convenience of description, the sample infrared image and the sample visible light image of the same scene obtained are referred to as a sample set. The sample infrared image includes a low-resolution first sample infrared image and a high-resolution second sample infrared image, and the sample visible light image is a high-resolution visible light image.

[0174] In the embodiment of the present application, the electronic device can obtain the first sample infrared image, the second sample infrared image and the sample visible light image in multiple scenes, that is, obtain multiple sample sets. For the convenience of description, the following description will be given by taking one sample set for training as an example, which is not limiting.

[0175] In the embodiment of the present application, the electronic device may first scale the acquired first sample infrared image, and input the scaled infrared image into the preset network model. The scaled first sample infrared image is a blurred, low-quality infrared image. At this time, the scaled first sample infrared image has the same scale, the same number of pixels, and the same resolution as the sample visible light image. The first sample infrared images referred to subsequently are all scaled first sample infrared images.

[0176] In the embodiment of the present application, the first resolution and the third resolution may be the same or different; the second resolution and the fourth resolution may be the same or different, and there is no limitation on this.

[0177] In the above step S42, the electronic device inputs the first sample infrared image and the sample visible light image into a preset network model. The preset network model uses the first sample infrared image and the sample visible light image, takes the sample visible light image as a guide map, and uses the high-resolution information of the sample visible light image to guide the first sample infrared image to perform super-resolution reconstruction, and obtains and outputs a fused reconstructed infrared image.

[0178] In an embodiment of the present application, the preset network model can extract the third feature of each pixel in the first sample infrared image and the fourth feature of each pixel in the sample visible light image, and use the third feature and the fourth feature of each pixel to calculate the similarity between the first sample infrared image and the sample visible light image at each pixel, and use the similarity between the first sample infrared image and the sample visible light image at each pixel to determine the third weight of the first sample infrared image and the fourth weight of the sample visible light image at each pixel, and use the third weight of the first sample infrared image and the fourth weight of the sample visible light image at each pixel to fuse each pixel in the first sample infrared image and the sample visible light image to obtain a reconstructed infrared image.

[0179] The process of the electronic device using the preset network model to super-resolution reconstruct the first sample infrared image and the sample visible light image to obtain the reconstructed infrared image is the same as the process of super-resolution reconstructing the initial infrared image and the visible light image to obtain the first target infrared image or the second target infrared image. For details, please refer to the above Figure 1~Figure 3 The relevant description of the first target infrared image or the second target infrared image part is obtained.

[0180] In the above step S43, the electronic device uses the reconstructed infrared image and the second sample infrared image to obtain the loss (Loss, L) of the preset network model for super-resolution reconstruction of the first sample infrared image. The electronic device can calculate the reconstruction loss between the reconstructed infrared image and the second sample infrared image, as shown in formula (16).

[0181] Formula (16)

[0182] in, represents reconstruction losses; represents the reconstructed infrared image; represents the second sample infrared image.

[0183] The electronic device may calculate a structural similarity (SSIM) loss between the reconstructed infrared image and the second sample infrared image, as shown in formula (17).

[0184] Formula (17)

[0185] in, represents the structural similarity loss; represents the reconstructed infrared image; represents the second sample infrared image; SSIM(·) represents the calculated structural similarity.

[0186] In the above step S44, after obtaining the reconstruction loss and the structural similarity loss, the electronic device can determine the relationship between the reconstruction loss and the structural similarity loss and the threshold value to determine whether the preset network model converges.

[0187] In one example, the electronic device can use preset reconstruction weights and structural similarity weights to perform weighted summation of reconstruction loss and structural similarity loss, calculate the total loss, and determine whether the total loss is less than a preset first threshold to determine whether the preset network model converges.

[0188] When the total loss is less than the first threshold, the electronic device can determine that the preset network model has converged, execute step S45, output the preset network model, and perform infrared image processing based on the output preset network model; when the total loss is greater than or equal to the first threshold, the electronic device can determine that the preset network model has not converged, then adjust the parameters in the preset network model, such as the weights corresponding to multiple scales, and return to execute step S42, and continue to use other acquired sample sets to train the adjusted preset network model. The electronic device loops through steps S42 to S44 until it is determined that the preset network model has converged.

[0189] In another example, the electronic device may determine whether the reconstruction loss is less than a preset second threshold, and determine whether the structural similarity loss is less than a preset third threshold, to determine whether the preset network model converges.

[0190] When the reconstruction loss is less than the second threshold and the structural similarity loss is less than the third threshold, the electronic device can determine that the preset network model has converged; when the reconstruction loss is greater than or equal to the second threshold, or the structural similarity loss is greater than or equal to the third threshold, the electronic device can determine that the preset network model has not converged. The specific process can be found in the relevant description in the above example.

[0191] In another example, the electronic device may determine whether the preset network model converges by combining the reconstruction loss, the structural similarity loss, and the total loss.

[0192] When the total loss is less than a first threshold, the reconstruction loss is less than a second threshold, and the structural similarity loss is less than a third threshold, the electronic device can determine that the preset network model has converged; when the total loss is greater than or equal to the first threshold, the reconstruction loss is greater than or equal to the second threshold, or the structural similarity loss is greater than or equal to the third threshold, the electronic device can determine that the preset network model has not converged. For the specific process, please refer to the relevant description in the above example.

[0193] The values ​​of the first threshold, the second threshold and the third threshold are not limited here.

[0194] In some embodiments, the preset network model may include an adversarial network, and the electronic device may also obtain an adversarial loss between the reconstructed infrared image and the second sample infrared image, as shown in formula (18).

[0195] Formula (18)

[0196] in, It means fighting against loss; represents a second sample infrared image; represents the reconstructed infrared image, Express expectations.

[0197] In an example, the above step S44 can be implemented by the following steps: determining whether the preset network model converges according to the sum of the reconstruction loss, the structural similarity loss and the adversarial loss.

[0198] The electronic device can use the preset reconstruction weight, structural similarity weight and adversarial weight to perform weighted summation of the reconstruction loss, structural similarity loss and adversarial loss to calculate the total loss, as shown in formula (19).

[0199] Formula (19)

[0200] in, represents the total loss; represents the reconstruction weight; represents reconstruction losses; represents the adversarial weight; It means fighting against loss; represents the structural similarity weight; represents the structural similarity loss.

[0201] Furthermore, the electronic device determines whether the total loss is less than a preset fourth threshold value to determine whether the preset network model converges.

[0202] When the total loss is less than the fourth threshold, the electronic device can determine that the preset network model has converged, execute step S45, output the preset network model, and perform infrared image processing based on the output preset network model; when the total loss is greater than or equal to the fourth threshold, the electronic device can determine that the preset network model has not converged, then adjust the parameters in the preset network model, such as the weights corresponding to multiple scales and the parameters in the adversarial network, and return to execute step S42, and continue to use other acquired sample sets to train the adjusted preset network model. The electronic device loops through steps S42 to S44 until it is determined that the preset network model has converged.

[0203] In another example, the electronic device can determine whether the reconstruction loss is less than a preset fifth threshold, determine whether the structural similarity loss is less than a preset sixth threshold, and determine whether the adversarial loss is less than a preset seventh threshold to determine whether the preset network model converges.

[0204] When the reconstruction loss is less than the fifth threshold, the structural similarity loss is less than the sixth threshold, and the adversarial loss is less than the seventh threshold, the electronic device can determine that the preset network model has converged; when the reconstruction loss is greater than or equal to the fifth threshold, the structural similarity loss is greater than or equal to the sixth threshold, or the adversarial loss is greater than or equal to the seventh threshold, the electronic device can determine that the preset network model has not converged. The specific process can be found in the relevant description in the above example.

[0205] In another example, the electronic device may determine whether a preset network model converges by combining reconstruction loss, structural similarity loss, adversarial loss, and total loss.

[0206] When the total loss is less than the fourth threshold, the reconstruction loss is less than the fifth threshold, the structural similarity loss is less than the sixth threshold and the adversarial loss is less than the seventh threshold, the electronic device can determine that the preset network model has converged; when the total loss is greater than or equal to the fourth threshold, the reconstruction loss is greater than or equal to the fifth threshold, the structural similarity loss is greater than or equal to the sixth threshold, or the adversarial loss is greater than or equal to the seventh threshold, the electronic device can determine that the preset network model has not converged. The specific process can be found in the relevant description in the above example.

[0207] The values ​​of the fourth threshold, the fifth threshold, the sixth threshold and the seventh threshold are not limited here.

[0208] By applying the technical solution provided in the embodiments of the present application, the electronic device uses reconstruction loss, adversarial loss and structural similarity loss for joint optimization, so that the reconstructed infrared image reaches the optimal level in terms of structural similarity, edge clarity and overall visual effect.

[0209] In the technical solution provided in the embodiment of the present application, a visible light image is used as a guide to perform super-resolution reconstruction of a low-resolution infrared image using an adaptive guided filter and a convolutional neural network, that is, the infrared image is scaled, and the characteristic value of each pixel of the scaled infrared image is determined to achieve super-resolution reconstruction of the infrared image. In addition, the characteristic values ​​of pixels at multiple scales are determined to achieve multi-modal guided super-resolution reconstruction, ensuring that the temperature information of the infrared image is consistent with the high-frequency details of the visible light. Without increasing the hardware cost, the processed infrared image has high resolution and high detail expression, and solves the problems of high-frequency detail loss and noise interference in the prior art.

[0210] In addition, during the fusion process of infrared images and visible light images, the guided filter dynamically adjusts the weights of the local feature similarities of the infrared images and the visible light images to control the fusion ratio, effectively avoiding the problem of introducing artifacts during the visible light image guidance process while retaining key temperature and texture information to ensure that the generated image has high resolution and high authenticity.

[0211] At the same time, in the process of super-resolution reconstruction of infrared images using the technical solution provided in the embodiment of the present application, the calculation complexity is low and the processing speed is fast, so the super-resolution reconstruction of infrared images can be more convenient.

[0212] Corresponding to the above-mentioned infrared image processing method embodiment, the present application embodiment also provides an infrared image processing device, see Figure 5 , is a schematic diagram of a structure of an infrared image processing device provided in an embodiment of the present application. The infrared image processing device comprises:

[0213] An acquisition module 51 is used to acquire an initial infrared image with a first resolution and a visible light image with a second resolution, wherein the first resolution is lower than the second resolution;

[0214] A determination module 52 is used to extract a first feature of each pixel in the initial infrared image and a second feature of each pixel in the visible light image; using the first feature and the second feature of each pixel, calculate the similarity between the initial infrared image and the visible light image at each pixel; using the similarity between the initial infrared image and the visible light image, determine a first weight of the initial infrared image and a second weight of the visible light image, wherein the similarity is positively correlated with the first weight, and the sum of the first weight and the second weight is a preset value;

[0215] A fusion module 53, configured to fuse the initial infrared image and the visible light image using the first weight and the second weight to obtain an infrared image of the first target with a second resolution;

[0216] The determination module 52 is specifically used to: determine a first weight of the initial infrared image and a second weight of the visible light image at each pixel point by using the similarity between the initial infrared image and the visible light image at each pixel point;

[0217] The fusion module 53 is specifically used to fuse each pixel in the initial infrared image and the visible light image using the first weight of the initial infrared image at each pixel and the second weight of the visible light image to obtain a first target infrared image with a second resolution.

[0218] In the technical solution provided in the embodiment of the present application, a high-resolution visible light image is used to guide the super-resolution reconstruction of a low-resolution infrared image, which can ensure that the temperature information of the infrared image is consistent with the high-frequency details of the visible light image. In addition, the similarity between the low-resolution infrared image and the high-resolution visible light image is used to dynamically determine the weight of the infrared image and the weight of the visible light image when reconstructing the infrared image, and the infrared image and the visible light image are fused using the dynamically determined weights to achieve super-resolution reconstruction. When the similarity between the visible light image and the infrared image is high, the weight is tilted toward the infrared image to retain more characteristics of the infrared image, and when the similarity is low, the weight is tilted toward the visible light image, which can effectively prevent the generation of artifacts while retaining key temperature and texture information, ensuring that the generated infrared image has high resolution and high authenticity, and improving the resolution of the infrared image.

[0219] In some embodiments, the above-mentioned fusion module 53 can be specifically used to: use the first weight and the second weight at each pixel point to weighted average the first feature of each pixel point in the initial infrared image and the second feature of each pixel point in the visible light image to obtain a first target infrared image of a second resolution.

[0220] In some embodiments, the determination module 52 may be specifically used to: determine the first weight of the initial infrared image and the second weight of the visible light image at multiple scales by using the similarity between the initial infrared image and the visible light image at multiple scales;

[0221] The above-mentioned fusion module 53 can be specifically used to: use the first weight and the second weight at each scale to fuse the initial infrared image and the visible light image to obtain the target sub-image at each scale; use the weights corresponding to multiple scales to fuse the target sub-images at multiple scales to obtain the first target infrared image of the second resolution.

[0222] In some embodiments, the above-mentioned infrared image processing device may further include a processing module, which is used to: after obtaining the first target infrared image of the second resolution, perform preset processing on the first target infrared image to obtain a second target infrared image, and the preset processing includes at least one of the following: denoising processing using a residual block and detail enhancement processing using an adversarial network.

[0223] In some embodiments, the infrared image processing device may further include a training module, the training module being used to:

[0224] Acquire a first sample infrared image, a second sample infrared image, and a sample visible light image, wherein the resolution of the first sample infrared image is a third resolution, the resolution of the second sample infrared image and the sample visible light image is a fourth resolution, and the third resolution is higher than the fourth resolution;

[0225] Inputting the first sample infrared image and the sample visible light image into a preset network model to obtain a reconstructed infrared image, wherein the preset network model is used to determine a third weight of the first sample infrared image and a fourth weight of the sample visible light image by using the similarity between the first sample infrared image and the sample visible light image, wherein the similarity is positively correlated with the third weight, and the sum of the third weight and the fourth weight is a preset value; using the third weight and the fourth weight, fusing the first sample infrared image and the sample visible light image, and outputting a reconstructed infrared image;

[0226] Obtaining a reconstruction loss and a structural similarity loss between the reconstructed infrared image and the second sample infrared image;

[0227] Based on the reconstruction loss and structural similarity loss, determine whether the preset network model converges.

[0228] In some embodiments, the preset network model includes an adversarial network, and the above training module may also be used to: obtain an adversarial loss between the reconstructed infrared image and the second sample infrared image;

[0229] The above training module can be specifically used to determine whether the preset network model converges according to the sum of reconstruction loss, structural similarity loss and adversarial loss.

[0230] The present application also provides an electronic device, such as Figure 6 As shown, it includes a processor 61, a communication interface 62, a memory 63 and a communication bus 64, wherein the processor 61, the communication interface 62 and the memory 63 communicate with each other through the communication bus 64;

[0231] Memory 63, used for storing computer programs;

[0232] The processor 61 is used to implement the steps of any of the above infrared image processing methods when executing the program stored in the memory 63.

[0233] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0234] The communication interface is used for communication between the above electronic device and other devices.

[0235] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0236] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0237] In another embodiment provided in the present application, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned infrared image processing methods are implemented.

[0238] In another embodiment provided in the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any infrared image processing method in the above embodiments.

[0239] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

[0240] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0241] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, electronic device, computer-readable storage medium, and computer program product embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0242] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. An infrared image processing method, characterized in that: The method comprises: Acquire an initial infrared image of a first resolution and a visible light image of a second resolution, wherein the first resolution is lower than the second resolution; Extracting a first feature of each pixel in the initial infrared image and a second feature of each pixel in the visible light image; Calculating the similarity between the initial infrared image and the visible light image at each pixel by using the first feature and the second feature of each pixel; Determine a first weight of the initial infrared image and a second weight of the visible light image by using the similarity between the initial infrared image and the visible light image, wherein the similarity is positively correlated with the first weight, and a sum of the first weight and the second weight is a preset value; Using the first weight and the second weight, the initial infrared image and the visible light image are fused to obtain a first target infrared image with the second resolution; The step of determining a first weight of the initial infrared image and a second weight of the visible light image by using the similarity between the initial infrared image and the visible light image comprises: Determine a first weight of the initial infrared image and a second weight of the visible light image at each pixel by using the similarity between the initial infrared image and the visible light image at each pixel; The step of fusing the initial infrared image and the visible light image using the first weight and the second weight to obtain the first target infrared image of the second resolution includes: The first weight of the initial infrared image at each pixel and the second weight of the visible light image are used to fuse each pixel in the initial infrared image and the visible light image to obtain the first target infrared image with the second resolution.

2. The method according to claim 1, characterized in that The step of fusing each pixel point in the initial infrared image and the visible light image using the first weight of the initial infrared image at each pixel point and the second weight of the visible light image to obtain the first target infrared image of the second resolution comprises: The first feature of each pixel in the initial infrared image and the second feature of each pixel in the visible light image are weighted averaged by using the first weight and the second weight at each pixel to obtain the first target infrared image with the second resolution.

3. The method according to claim 1, characterized in that The step of determining the first weight of the initial infrared image and the second weight of the visible light image by using the similarity between the initial infrared image and the visible light image further includes: Determine a first weight of the initial infrared image and a second weight of the visible light image at the multiple scales by using similarities between the initial infrared image and the visible light image at the multiple scales; The step of fusing the initial infrared image and the visible light image using the first weight and the second weight to obtain the first target infrared image of the second resolution also includes: Using the first weight and the second weight at each scale, the initial infrared image and the visible light image are fused to obtain a target sub-image at each scale; The target sub-images at the multiple scales are fused using the weights corresponding to the multiple scales to obtain the first target infrared image at the second resolution.

4. The method according to claim 1, characterized in that After obtaining the first target infrared image with the second resolution, the method further includes: The first target infrared image is subjected to a preset processing to obtain a second target infrared image, wherein the preset processing includes at least one of the following: performing denoising processing using a residual block and performing detail enhancement processing using an adversarial network.

5. The method according to claim 1, characterized in that The method further comprises: Acquire a first sample infrared image, a second sample infrared image, and a sample visible light image, wherein the resolution of the first sample infrared image is a third resolution, the resolution of the second sample infrared image and the sample visible light image are a fourth resolution, and the third resolution is higher than the fourth resolution; Inputting the first sample infrared image and the sample visible light image into a preset network model to obtain a reconstructed infrared image, wherein the preset network model is used to determine a third weight of the first sample infrared image and a fourth weight of the sample visible light image by using the similarity between the first sample infrared image and the sample visible light image, wherein the similarity is positively correlated with the third weight, and the sum of the third weight and the fourth weight is the preset value; using the third weight and the fourth weight, fusing the first sample infrared image and the sample visible light image, and outputting the reconstructed infrared image; acquiring a reconstruction loss and a structural similarity loss between the reconstructed infrared image and the second sample infrared image; Determine whether the preset network model converges according to the reconstruction loss and the structural similarity loss.

6. The method according to claim 5, characterized in that The preset network model includes an adversarial network, and the method further includes: Acquire an adversarial loss between the reconstructed infrared image and the second sample infrared image; The step of determining whether the preset network model converges according to the reconstruction loss and the structural similarity loss comprises: Whether the preset network model converges is determined according to the sum of the reconstruction loss, the structural similarity loss and the adversarial loss.

7. An infrared image processing device, characterized in that: The device comprises: An acquisition module, configured to acquire an initial infrared image of a first resolution and a visible light image of a second resolution, wherein the first resolution is lower than the second resolution; A determination module, configured to extract a first feature of each pixel in the initial infrared image and a second feature of each pixel in the visible light image; using the first feature and the second feature of each pixel, calculate the similarity between the initial infrared image and the visible light image at each pixel; using the similarity between the initial infrared image and the visible light image, determine a first weight of the initial infrared image and a second weight of the visible light image, wherein the similarity is positively correlated with the first weight, and the sum of the first weight and the second weight is a preset value; a fusion module, configured to fuse the initial infrared image and the visible light image using the first weight and the second weight to obtain a first target infrared image with the second resolution; The determination module is specifically used to: determine a first weight of the initial infrared image and a second weight of the visible light image at each pixel point by using the similarity between the initial infrared image and the visible light image at each pixel point; The fusion module is specifically used to: use the first weight of the initial infrared image at each pixel point and the second weight of the visible light image to fuse each pixel point in the initial infrared image and the visible light image to obtain the first target infrared image with the second resolution.

8. The device according to claim 7, characterized in that The fusion module is specifically used to: use the first weight and the second weight at each pixel point to weighted average the first feature of each pixel point in the initial infrared image and the second feature of each pixel point in the visible light image to obtain the first target infrared image of the second resolution.

9. The device according to claim 7, characterized in that The determination module is specifically used to: determine the first weight of the initial infrared image and the second weight of the visible light image at the multiple scales by using the similarity between the initial infrared image and the visible light image at the multiple scales; The fusion module is specifically used to: use the first weight and the second weight at each scale to fuse the initial infrared image and the visible light image to obtain a target sub-image at each scale; use the weights corresponding to the multiple scales to fuse the target sub-images at the multiple scales to obtain the first target infrared image of the second resolution.

10. The device according to claim 7, characterized in that The device also includes a processing module, which is used to: after obtaining the first target infrared image of the second resolution, perform preset processing on the first target infrared image to obtain a second target infrared image, and the preset processing includes at least one of the following: denoising processing using a residual block and detail enhancement processing using an adversarial network.

11. The device according to claim 7, characterized in that The device further comprises a training module, wherein the training module is used to: Acquire a first sample infrared image, a second sample infrared image, and a sample visible light image, wherein the resolution of the first sample infrared image is a third resolution, the resolution of the second sample infrared image and the sample visible light image are a fourth resolution, and the third resolution is higher than the fourth resolution; Inputting the first sample infrared image and the sample visible light image into a preset network model to obtain a reconstructed infrared image, wherein the preset network model is used to determine a third weight of the first sample infrared image and a fourth weight of the sample visible light image by using the similarity between the first sample infrared image and the sample visible light image, wherein the similarity is positively correlated with the third weight, and the sum of the third weight and the fourth weight is the preset value; using the third weight and the fourth weight, fusing the first sample infrared image and the sample visible light image, and outputting the reconstructed infrared image; acquiring a reconstruction loss and a structural similarity loss between the reconstructed infrared image and the second sample infrared image; Determine whether the preset network model converges according to the reconstruction loss and the structural similarity loss.

12. The device according to claim 11, characterized in that The preset network model includes an adversarial network, and the training module is further used to: obtain the adversarial loss between the reconstructed infrared image and the second sample infrared image; The training module is specifically used to determine whether the preset network model converges according to the sum of the reconstruction loss, the structural similarity loss and the adversarial loss.

13. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; Memory, used to store computer programs; A processor, for implementing any of the methods described in claims 1-6 when executing a program stored in a memory.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images