Infrared Image Super-Resolution Reconstruction Method and System Based on Convolutional Neural Networks

Abstract

A method and system for super-resolution reconstruction of infrared images based on convolutional neural networks, relating to the field of electronic digital data processing, is disclosed. The method includes: inputting a low-resolution infrared image into a preset convolutional neural network to obtain multi-layer feature maps of the low-resolution infrared image; scaling the feature maps of different layers in the multi-layer feature maps according to a preset ratio and then stitching and fusing them to obtain a fused feature map; generating feature vectors from the fused feature map using global average pooling, and transforming the feature vectors using a fully connected layer to obtain attention weights; weighting the attention weights with the fused feature map to obtain an enhanced feature map; and performing upsampling reconstruction processing on the enhanced feature map using an upsampling structure, introducing a residual connection structure during the upsampling reconstruction process to generate a high-resolution infrared image. Implementing this method generates high-resolution infrared images with more detail.

Description

Technical Field

[0001] This application relates to the field of electronic digital data processing, and in particular to a method and system for super-resolution reconstruction of infrared images based on convolutional neural networks. Background Technology

[0002] With the widespread application of infrared imaging technology in military reconnaissance, security monitoring, and medical diagnosis, increasingly higher demands are being placed on the resolution and clarity of infrared images. However, due to the influence of factors such as imaging devices and atmospheric conditions, acquired infrared images often suffer from low resolution and blurred details, which severely restricts the practical application effectiveness of infrared imaging systems.

[0003] Related technologies apply convolutional neural networks to infrared image super-resolution reconstruction tasks. Early neural network methods, such as SRCNN (Super-Resolution Convolutional Neural Network), directly learn the mapping relationship between low-resolution and high-resolution images through a three-layer convolutional network.

[0004] However, these early neural network methods still have significant shortcomings when processing infrared images: the network structure is shallow, the receptive field is limited, and it is difficult to make full use of the global information of the image; they lack effective fusion of multi-scale features, and cannot simultaneously utilize shallow texture details and deep semantic information. Moreover, due to the single information transmission path, high-frequency details of the image are easily lost during the transmission process in the deep network. This leads to problems such as blurred edges and texture distortion in the reconstructed high-resolution image, making it difficult to accurately restore the detailed information in the image. The reconstruction effect is even worse when the image contains complex target scenes. Summary of the Invention

[0005] This application provides a method for super-resolution reconstruction of infrared images based on convolutional neural networks, which is used to generate high-resolution infrared images with more details.

[0006] Firstly, this application provides an infrared image super-resolution reconstruction method based on a convolutional neural network, applied to an infrared image super-resolution reconstruction system. The method includes: inputting a low-resolution infrared image into a preset convolutional neural network to obtain multi-layer feature maps of the low-resolution infrared image. The convolutional neural network is trained using a hybrid loss function containing adversarial and perceptual loss terms. The adversarial loss term is calculated based on the discrimination result between the generated image and the real image by the discriminator network, and the perceptual loss term is calculated based on the difference between the generated image and the real image in a high-dimensional feature space. The feature maps at different levels of the multi-layer feature maps are scaled according to a preset ratio and then stitched together to obtain a fused feature map. A feature vector is generated from the fused feature map using global average pooling, and attention weights are obtained by transforming the feature vector using a fully connected layer. The attention weights are weighted with the fused feature map to obtain an enhanced feature map. An upsampling reconstruction process is performed on the enhanced feature map using an upsampling structure, and a residual connection structure is introduced during the upsampling reconstruction process to generate a high-resolution infrared image.

[0007] In the above embodiments, a convolutional neural network is trained using a hybrid loss function combining adversarial and perceptual loss terms. This enables the network to learn the realistic features and high-dimensional semantic features of the image, and the extracted multi-layer feature maps contain comprehensive information from edge texture to high-level semantics. Through multi-scale feature fusion and attention-based weighting, feature channels that contribute significantly to reconstruction quality are highlighted. The upsampling structure, combined with residual connections, effectively preserves detail information while improving resolution, resulting in a high-resolution infrared image with rich detail and clear structure.

[0008] In conjunction with some embodiments of the first aspect, in some embodiments, the feature maps of different levels in the multi-layer feature map are scaled according to a preset ratio and then spliced ​​and fused to obtain a fused feature map. Specifically, this includes: outputting a shallow feature map from a first preset position of the convolutional neural network, outputting a middle-layer feature map from a second preset position, and outputting a deep feature map from a third preset position, wherein the layers of the first preset position, the second preset position, and the third preset position are progressively deeper, and the spatial resolution of the output feature maps is progressively reduced; obtaining the spatial size of the shallow feature map, performing upsampling and scaling on the middle-layer feature map to make the spatial size of the middle-layer feature map the same as that of the shallow feature map, to obtain a first scaled feature map; performing upsampling and scaling on the deep feature map to make the spatial size of the deep feature map the same as that of the shallow feature map, to obtain a second scaled feature map; splicing the shallow feature map, the first scaled feature map, and the second scaled feature map along the channel dimension to generate a spliced ​​feature map; inputting the spliced ​​feature map into a convolutional layer and performing a convolution operation on the spliced ​​feature map to obtain a fused feature map.

[0009] In the above embodiments, the spatial resolution of feature maps is progressively reduced from shallow to deep layers, ensuring the integrity of image information captured at different scales. By unifying the feature map size and stitching them together along the channel dimension, lossless fusion of multi-scale features is achieved. Finally, convolutional operations are performed to optimize and integrate the fused features, making full use of the information in the feature maps and enhancing the feature representation capabilities of subsequent processing.

[0010] In conjunction with some embodiments of the first aspect, in some embodiments, the upsampling reconstruction process of the enhanced feature map using an upsampling structure, and the introduction of a residual connection structure during the upsampling reconstruction process to generate a high-resolution infrared image, specifically includes: performing a first pixel rearrangement operation on the enhanced feature map to generate a first upsampling feature map, the spatial resolution of which is twice that of the enhanced feature map; extracting a residual feature map from a fourth preset position of the convolutional neural network, the spatial resolution of which is the same as that of the first upsampling feature map; performing an element-wise addition operation between the residual feature map and the first upsampling feature map to generate a residual fusion feature map; performing a second pixel rearrangement operation on the residual fusion feature map to generate a second upsampling feature map, the spatial resolution of which is twice that of the residual fusion feature map; inputting the second upsampling feature map into a convolutional layer, and performing a convolution operation on the second upsampling feature map to output a high-resolution infrared image.

[0011] In the above embodiments, multiple pixel rearrangement operations gradually improve the spatial resolution of the feature map, avoiding information loss caused by a single large-scale upsampling. The residual feature map is introduced and element-wise added to the upsampled feature map, supplementing the detailed information lost in the deep network. The features are integrated and mapped through a final convolution operation, ensuring the structural integrity and detail restoration of the reconstructed image.

[0012] In conjunction with some embodiments of the first aspect, in some embodiments, after the steps of upsampling and reconstructing the enhanced feature map using an upsampling structure and introducing a residual connection structure during the upsampling and reconstruction process to generate a high-resolution infrared image, the method further includes: calculating the gradient direction distribution data of the high-resolution infrared image, and determining the principal direction parameter based on the gradient direction distribution data; dividing the high-resolution infrared image into multiple sector regions in polar coordinate space based on the principal direction parameter; determining the corresponding mapping parameter set for each sector region according to the pixel intensity distribution of the sector region; performing a fusion operation on the mapping parameter set of each sector region at the region boundary, and converting the high-resolution infrared image into a three-channel color image through a pixel value conversion mapping function.

[0013] In the above embodiments, the gradient direction distribution of the infrared image is calculated and the principal direction parameter is determined, accurately capturing the directional information of the main structural features in the image. Fan-shaped regions are divided in polar coordinate space based on the principal direction, ensuring that the region division is consistent with the image's structural features. Mapping parameters are determined based on the pixel intensity distribution characteristics within the regions, preserving the original structural and detailed features when converting the infrared image to a color image.

[0014] In conjunction with some embodiments of the first aspect, in some embodiments, the step of determining the corresponding mapping parameter set for each sector region based on the pixel intensity distribution of the sector region specifically includes: calculating the average intensity of pixels within the sector region; calculating the intensity space entropy based on the difference between the pixel intensity within the sector region and the intensity of its neighboring pixels; mapping the average intensity to a hue parameter in the color space; mapping the intensity space entropy to a saturation parameter in the color space; and generating the corresponding mapping parameter set based on the hue parameter and saturation parameter combined with the original pixel intensity value.

[0015] In the above embodiments, the mean pixel intensity and intensity spatial entropy within the fan-shaped region are calculated to comprehensively characterize the overall brightness level and local variation characteristics of the pixels within the region. The mean intensity and spatial entropy are mapped to hue and saturation parameters, respectively, establishing an effective mapping relationship from infrared image grayscale features to color features, thus giving the converted color image stronger visual expressiveness.

[0016] In conjunction with some embodiments of the first aspect, in some embodiments, after the steps of upsampling and reconstructing the enhanced feature map using an upsampling structure and introducing a residual connection structure during the upsampling and reconstruction process to generate a high-resolution infrared image, the method further includes: dividing the high-resolution infrared image into multiple overlapping sub-regions according to a preset ratio; extracting texture feature parameters and contrast parameters for each overlapping sub-region, and calculating weight values ​​based on the texture feature parameters and contrast parameters; generating corresponding threshold parameters based on the weight values, and extracting target boundary features based on the threshold parameters; performing region merging and contour fitting operations on the target boundary features to generate target localization results.

[0017] In the above embodiments, the high-resolution infrared image is divided into overlapping sub-regions, ensuring the continuity and integrity of feature extraction. Weight values ​​are calculated based on texture features and contrast parameters, highlighting important regions with rich details in the image. The confidence-weighted gradient is obtained by multiplying the gradient magnitude by the weight values, enhancing the accuracy of target boundary detection.

[0018] In conjunction with some embodiments of the first aspect, in some embodiments, the step of generating a corresponding threshold parameter based on the weight value and extracting target boundary features based on the threshold parameter specifically includes: calculating the gradient magnitude of the high-resolution infrared image, multiplying the gradient magnitude with the weight value at the corresponding pixel position to obtain the confidence-weighted gradient; determining the threshold parameter based on the distribution characteristics of the confidence-weighted gradient; and segmenting the confidence-weighted gradient using the threshold parameter to extract target boundary features.

[0019] In the above embodiments, different upsampling scaling ratios and mapping parameters are applied to the target region and the background region respectively, highlighting the detailed features of important targets. Bilinear interpolation is used to achieve refined upsampling of the target region, while the background region is moderately compressed, optimizing the overall image clarity and detail levels. An adaptive threshold segmentation method is used to determine the target region, ensuring that the upsampling ratio is adapted to the image content, guaranteeing clear distinction between primary and secondary elements and the integrity of details in the reconstructed image.

[0020] In a second aspect, embodiments of this application provide an infrared image super-resolution reconstruction system, the infrared image super-resolution reconstruction system comprising: one or more processors and a memory; the memory is coupled to the one or more processors, the memory being used to store computer program code, the computer program code including computer instructions, the one or more processors calling the computer instructions to cause the infrared image super-resolution reconstruction system to perform the method as described in the first aspect and any possible implementation thereof.

[0021] Thirdly, embodiments of this application provide a computer program product containing instructions that, when the computer program product is run on an infrared image super-resolution reconstruction system, cause the infrared image super-resolution reconstruction system to perform the method described in the first aspect and any possible implementation thereof.

[0022] Fourthly, embodiments of this application provide a computer-readable storage medium including instructions that, when executed on an infrared image super-resolution reconstruction system, cause the infrared image super-resolution reconstruction system to perform the method described in the first aspect and any possible implementation thereof.

[0023] Understandably, the infrared image super-resolution reconstruction system provided in the second aspect, the computer program product provided in the third aspect, and the computer storage medium provided in the fourth aspect are all used to execute the methods provided in the embodiments of this application. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods, and will not be repeated here.

[0024] One or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages:

[0025] 1. This application trains a convolutional neural network using a hybrid loss function combining adversarial and perceptual loss terms, enabling the network to learn realistic features and high-dimensional semantic features of images. The extracted multi-layer feature maps contain comprehensive information from edge texture to high-level semantics. Through multi-scale feature fusion and attention mechanism weighting, feature channels that contribute significantly to reconstruction quality are highlighted. The upsampling structure, combined with residual connections, effectively preserves detail information while improving resolution, resulting in a high-resolution infrared image with rich detail and clear structure.

[0026] 2. This application extracts feature maps from shallow to deep layers, progressively reducing the spatial resolution to ensure the integrity of captured image information at different scales. By unifying the feature map size and stitching them together along the channel dimension, lossless fusion of multi-scale features is achieved. Finally, convolutional operations are used to optimize and integrate the fused features, making full use of the information in the feature maps and enhancing the feature representation capabilities of subsequent processing.

[0027] 3. This application progressively improves the spatial resolution of the feature map through multiple pixel rearrangement operations, avoiding the information loss caused by a single large-scale upsampling. Residual feature maps are introduced and element-wise added to the upsampled feature maps to supplement the detailed information lost in deep networks. The features are integrated and mapped through a final convolution operation, ensuring the structural integrity and detail restoration of the reconstructed image. Attached Figure Description

[0028] Figure 1 This is a flowchart illustrating an infrared image super-resolution reconstruction method based on a convolutional neural network in an embodiment of this application.

[0029] Figure 2 This is another flowchart illustrating the infrared image super-resolution reconstruction method based on convolutional neural networks in the embodiments of this application;

[0030] Figure 3 This is a schematic diagram of the physical device structure of an infrared image super-resolution reconstruction system in the embodiments of this application. Detailed Implementation

[0031] The terminology used in the following embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification of this application, the singular expressions “a,” “an,” “the,” “the,” and “this” are intended to include the plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this application refers to any or all possible combinations including one or more of the listed items.

[0032] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature, and in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0033] To facilitate understanding, the application scenarios of the embodiments of this application are described below.

[0034] In a security monitoring system for a large industrial park, 24 / 7 infrared monitoring of key equipment and important areas is required. Especially at night or in adverse weather conditions with low visibility, infrared monitoring becomes a crucial means of ensuring park security. However, due to hardware limitations of infrared imaging equipment and the influence of complex environmental factors, the acquired infrared images often suffer from insufficient resolution. For example, when the monitoring system needs to monitor the status of equipment 500 meters away, the acquired infrared image resolution is only 256×256 pixels, resulting in the inability to clearly display detailed features of the equipment (such as temperature anomalies at pipe connections, valve opening / closing status, etc.). Simultaneously, the numerous metal components within the industrial park generate reflections and scattering, resulting in significant noise in the infrared images, further degrading image quality. Therefore, effectively improving the resolution of infrared images and accurately restoring the detailed information of the monitored targets has become a critical issue that urgently needs to be addressed.

[0035] A factory uses a traditional image processing system based on interpolation algorithms to improve the resolution of infrared surveillance images. This system employs bicubic interpolation to enlarge a low-resolution 256×256 pixel infrared image to 512×512 pixels. During processing, the system first divides the input image into a grid, then calculates the value of the newly added pixel using polynomial interpolation based on the grayscale values ​​of adjacent pixels. For example, for four adjacent pixels in the input image, the system inserts a new pixel between them, and the interpolated pixel value is determined by the weighted average of the surrounding 16 pixels. However, this method relies excessively on mathematical operations on local pixel values ​​and fails to understand the semantic information and structural features of the image. When complex industrial equipment (such as multi-layered pipes or temperature gradients on equipment surfaces) appears in the monitored image, the interpolation algorithm is prone to severe blurring effects. Especially in edge regions and areas with rich texture details, the reconstructed image exhibits significant distortion and artifacts, making it impossible to accurately identify the operating status and potential faults of the equipment.

[0036] The infrared image super-resolution reconstruction system employing this invention has been applied in a chemical plant area. The system first inputs a 256×256 pixel low-resolution infrared image into a pre-defined convolutional neural network, extracting image features layer by layer through five convolutional layers of different depths. For example, the first convolutional layer (32 3×3 kernels) extracts basic features such as edges and textures, while the fifth convolutional layer (512 3×3 kernels) captures more complex semantic information. The system then uses multi-scale feature fusion technology to appropriately scale and stitch together feature maps from different levels, and highlights important features through a channel attention mechanism. In practical applications, when monitoring a reactor, the system can accurately reconstruct the temperature distribution details on the reactor surface, clearly displaying minute temperature anomalies. Through optimization of the residual learning structure and hybrid loss function, the reconstructed 512×512 pixel image not only maintains the original temperature information but also accurately restores the geometric structural features of the reactor, enabling operators to promptly detect equipment anomalies. Actual tests show that the high-resolution images reconstructed by this system are significantly superior to traditional methods in terms of detail fidelity and edge sharpness, providing more reliable data support for the condition monitoring of industrial equipment.

[0037] To facilitate understanding, the method provided in this implementation will be described in detail below, using the above scenario as an example. Please refer to [link / reference]. Figure 1 This is a flowchart illustrating an infrared image super-resolution reconstruction method based on a convolutional neural network in an embodiment of this application.

[0038] S101. Input the low-resolution infrared image into a preset convolutional neural network to obtain a multi-layer feature map of the low-resolution infrared image. The convolutional neural network is trained by using a hybrid loss function that includes an adversarial loss term and a perceptual loss term. The adversarial loss term is calculated based on the discriminator network's distinction between the generated image and the real image, and the perceptual loss term is calculated based on the difference between the generated image and the real image in the high-dimensional feature space.

[0039] Among them, low-resolution infrared image refers to the original image acquired by infrared imaging equipment, whose resolution is lower than the target resolution; pre-designed convolutional neural network refers to a deep learning model for image feature extraction that has been pre-designed and trained; multi-layer feature map refers to the image feature representation extracted by different layers of convolutional neural network; adversarial loss term refers to the loss metric used to evaluate the realism of generated image; perceptual loss term refers to the loss metric used to measure the difference between generated image and real image in high-level semantic features; discriminator network refers to the neural network model used to distinguish between generated image and real image.

[0040] This step is performed in the initial stage of infrared image super-resolution reconstruction to extract multi-scale feature information from the input image. Specifically, the input low-resolution infrared image is first normalized preprocessed, scaling pixel values ​​to the range of 0-1. The preprocessed image is then fed into a pre-defined convolutional neural network containing five convolutional layers, each using 32, 64, 128, 256, and 512 3×3 convolutional kernels for feature extraction, respectively. During training, the network employs a hybrid loss function to optimize parameters, where adversarial loss evaluates the realism of the generated image through a discriminator, and perceptual loss calculates feature space differences using a pre-trained VGG network. This design enables the network to extract multi-level feature representations of the image.

[0041] In some embodiments, low-resolution infrared image feature extraction can be achieved in several ways: Optionally, a cascaded convolutional structure can be used, combining successive convolutional layers and pooling layers for feature extraction. Each convolutional layer uses a different number and size of convolutional kernels, combined with the ReLU activation function, to obtain multiple feature maps of different scales. Optionally, a parallel convolutional structure can be used, simultaneously feeding the input image into multiple convolutional branches with different receptive field sizes. Each branch extracts features independently, and the feature maps extracted by each branch are then fused. It is understood that other network structures can also be used for feature extraction, and this is not limited here.

[0042] S102. After scaling the feature maps of different levels in the multi-layer feature map according to a preset ratio, the feature maps are spliced ​​and fused to obtain a fused feature map.

[0043] Among them, feature maps of different levels represent feature representations with different receptive fields and semantic levels extracted by each layer of the convolutional neural network; preset ratio refers to the feature map scaling factor determined in advance according to the network structure; feature map splicing and fusion refers to the operation of connecting multiple feature maps in the channel dimension; fused feature map represents the comprehensive feature representation obtained after splicing and fusion processing.

[0044] This step is performed after multi-layer feature extraction to integrate feature information from different levels. Specifically, the size information of each layer's feature map is first obtained, using the spatial size of the first-layer feature map as a reference. Bilinear interpolation is then used to upsample or downsample the feature maps of other layers to ensure their spatial size matches the reference size. The processed feature maps are then concatenated along the channel dimension to obtain a preliminary fused feature map. Finally, a 1×1 convolutional layer is used to perform channel dimensionality reduction and feature fusion on the concatenated feature map, generating the final fused feature map.

[0045] In some embodiments, feature map fusion can be achieved in several ways: Optionally, a pyramid feature fusion method can be used, fusing feature maps of different scales layer by layer in descending order, with each layer considering the feature information of the previous layer, and passing detailed information through skip connections, finally obtaining a fused feature map containing multi-scale information; Optionally, an attention mechanism can be used to assist feature fusion, assigning different weights to feature maps at different levels, with important features receiving higher weights, and obtaining a more effective feature representation through weighted fusion. It is understood that other feature fusion methods can also be used, and are not limited here.

[0046] In some embodiments, this step specifically includes:

[0047] The convolutional neural network outputs shallow feature maps from a first preset position, mid-level feature maps from a second preset position, and deep feature maps from a third preset position, wherein the layers at the first, second, and third preset positions increase in depth, and the spatial resolution of the output feature maps decreases sequentially. The spatial dimensions of the shallow feature maps are obtained, and the mid-level feature maps are upsampled and scaled to make their spatial dimensions the same as the shallow feature maps, resulting in a first scaled feature map. The deep feature maps are also upsampled and scaled to make their spatial dimensions the same as the shallow feature maps, resulting in a second scaled feature map. The shallow feature maps, the first scaled feature map, and the second scaled feature map are concatenated along the channel dimension to generate a concatenated feature map. This concatenated feature map is then input into a convolutional layer, and a convolution operation is performed on it to obtain a fused feature map.

[0048] First, we will define, explain, and provide examples of the core technical entities involved in this step. A feature map, conceptually, refers to the output of each convolutional operation in a convolutional neural network; it is the feature representation of the input data after being extracted by the convolutional kernel. Specifically, feature maps at different depths in the network carry information at different levels: shallow feature maps, typically the outputs of the first few layers (such as layers 1-2), retain a large amount of spatial details, such as low-level information like image edges, corners, textures, and colors, but have weak semantic meaning; mid-level feature maps, located in the middle of the network, combine shallow features and begin to form more complex structures such as local parts of objects; deep feature maps, from the back end of the network, have low spatial resolution but, after multiple layers of abstraction, contain the richest global semantic information, reflecting what kind of objects exist in the image. For example, for a vehicle infrared image with a resolution of 256x256 pixels, the shallow feature map output after the second convolution layer might be 128x128 pixels, clearly outlining the vehicle's contours; while the deep feature map output after the fifth convolution layer might only be 32x32 pixels, with its activation regions concentrated on the overall position of the vehicle. Upsampling scaling is an operation that increases the spatial resolution (i.e., width and height) of a feature map, aiming to unify feature maps of different sizes to the same size for subsequent fusion. Common implementations include bilinear interpolation, nearest neighbor interpolation, or learning-based transposed convolution. For example, bilinear interpolation calculates the value of a new pixel by finding the floating-point coordinates of the target pixel in the source image and using a weighted average of the values ​​of its four nearest neighbors. Channel concatenation is a feature fusion operation that stacks multiple feature maps of the same size but from different sources along the channel direction (i.e., the depth direction) to form a "thicker" feature map. This operation only increases the number of channels without changing the spatial size. For example, concatenating a 32x32x64 feature map with a 32x32x128 feature map will result in a 32x32x192 feature map. Finally, the fused feature map is the final feature representation containing multi-level information generated after concatenation and subsequent convolution processing.

[0049] The detailed implementation of this technique aims to construct an information-rich multi-scale feature representation, providing a solid foundation for subsequent super-resolution reconstruction. The process begins by extracting features from a pre-defined convolutional neural network containing five convolutional layers. Specifically, a shallow feature map is output from a first pre-defined location (e.g., after the second convolutional layer), which has high spatial resolution and preserves rich image details. A mid-level feature map is then output from a second pre-defined location (e.g., after the fourth convolutional layer), which has lower resolution but more abstract feature representation. Finally, a deep feature map is output from a third pre-defined location (e.g., after the fifth convolutional layer), which has the lowest spatial resolution but contains the highest-order semantic information. Subsequently, the remaining feature maps are upsampled based on the shallow feature map with the largest spatial size. The specific calculation method is as follows: If the shallow feature map size is H×W, the middle layer is (H / 2)×(W / 2), and the deep layer is (H / 4)×(W / 4), then the middle layer feature map needs to be upsampled by 2 times, and the deep layer feature map needs to be upsampled by 4 times, so that their sizes all reach H×W, thereby generating the first scaled feature map and the second scaled feature map. This upsampling process uses a bilinear interpolation algorithm to ensure the smoothness of the features during the amplification process. After size alignment, the original shallow feature map, the first scaled feature map, and the second scaled feature map are concatenated along the channel dimension. If the number of channels for the three are C1, C2, and C3 respectively, then the size of the concatenated feature map is H×W×(C1+C2+C3), which physically aggregates all feature information from different network depths. Finally, in order to effectively fuse this information and control computational complexity, the concatenated feature map is input into a 1×1 convolutional layer. The core of this convolution operation is to perform weighted summation and nonlinear transformation on the feature vectors of the (C1+C2+C3) dimensions at each spatial location, learn how to optimally combine these features from different levels, and reduce the number of channels to a preset value (such as 256), and finally output a fused feature map that contains both low-level details and high-level semantics.

[0050] S103. Generate feature vectors from the fused feature map using global average pooling, and use a fully connected layer to transform the feature vectors to obtain attention weights. Weight the attention weights with the fused feature map to obtain the enhanced feature map.

[0051] Global average pooling refers to the operation of calculating the average value of the feature map in the spatial dimension; the feature vector represents the one-dimensional feature representation obtained after pooling; the fully connected layer refers to the densely connected neural network layer used for feature transformation; the attention weights represent the weighting coefficients used to highlight important features; and the enhanced feature map refers to the enhanced feature representation obtained after weighting by the attention mechanism.

[0052] This step is performed after feature fusion to highlight key information in the feature map. Specifically, first, global average pooling is performed on the fused feature map, compressing the two-dimensional feature map of each channel into a single value, forming a feature vector. This feature vector is then input into a two-layer fully connected network. The first layer uses the ReLU activation function to introduce a non-linear transformation, and the second layer uses the Sigmoid function to normalize the output to the 0-1 range, obtaining the attention weights. Finally, the attention weights are multiplied element-wise with each channel of the original fused feature map to achieve adaptive weighted feature enhancement.

[0053] In some embodiments, feature enhancement can be achieved in several ways: Optionally, a dual attention mechanism of channel and spatial dimensions can be employed. First, the attention weights in the channel dimension are calculated, then the attention map in the spatial dimension is calculated, and finally the results of the two attention mechanisms are combined to obtain a more comprehensive feature enhancement effect. Optionally, a multi-head attention mechanism can be used, where the feature vector is divided into multiple subspaces, attention weights are calculated separately for each subspace, and then the multiple attention results are fused to improve the robustness of feature enhancement. It is understood that other feature enhancement methods can also be used, and no limitation is made here.

[0054] S104. Upsampling reconstruction is performed on the enhanced feature map using an upsampling structure, and a residual connection structure is introduced during the upsampling reconstruction process to generate a high-resolution infrared image.

[0055] Among them, the upsampling structure represents the network module used to improve the resolution of the feature map; the upsampling reconstruction process refers to the process of converting the low-resolution feature map into a high-resolution feature map; the residual connection structure represents the design of adding skip connections to the network; and the high-resolution infrared image refers to the target resolution image obtained by the final reconstruction.

[0056] This step is performed after feature enhancement to generate the final high-resolution image. Specifically, firstly, the enhanced feature map is upsampled using a pixel rearrangement operation, doubling its spatial resolution. Then, residual feature maps are extracted from the shallow layers of the network and added element-wise to the upsampled feature map to supplement detail information. Next, a second pixel rearrangement operation is performed, again doubling the spatial resolution of the feature map. Finally, a convolutional layer converts the feature map into a three-channel high-resolution infrared image output.

[0057] In some embodiments, image reconstruction can be achieved in several ways: Optionally, a progressive upsampling strategy can be adopted, using multiple small-scale upsampling modules connected in series, each module equipped with residual connections, to gradually increase the image resolution and ensure the stability of the reconstruction quality; Optionally, a densely connected upsampling network can be used, incorporating dense connections of multiple layers of features during the upsampling process to fully utilize the feature information of each layer and improve the detail restoration capability of the reconstructed image. It is understood that other image reconstruction methods can also be used, and are not limited here.

[0058] In some embodiments, this step specifically includes:

[0059] A first pixel rearrangement operation is performed on the enhanced feature map to generate a first upsampled feature map, whose spatial resolution is twice that of the enhanced feature map. A residual feature map is extracted from the fourth preset position of the convolutional neural network; this residual feature map has the same spatial resolution as the first upsampled feature map. The residual feature map and the first upsampled feature map are then added element-wise to generate a residual fusion feature map. A second pixel rearrangement operation is performed on the residual fusion feature map to generate a second upsampled feature map, whose spatial resolution is twice that of the residual fusion feature map. The second upsampled feature map is then input into a convolutional layer, and a convolution operation is performed on it to output a high-resolution infrared image.

[0060] First, the key technical entities in this upsampling reconstruction module are defined, explained, and illustrated with examples. An enhanced feature map refers to the feature representation obtained after feature extraction, multi-scale fusion, and weighting by an attention mechanism. It densely contains the most important information for reconstructing a high-resolution image. Pixel shuffle operation is an efficient image upsampling technique. Its core idea is not to create new pixels, but to rearrange the information in the channel dimension of the feature map into the spatial dimension (height and width). Specifically, for an upsampling factor of r, this operation will reconstruct an input feature map of size H×W×(C×r²) into an output feature map of size (r×H)×(r×W)×C, thereby achieving resolution improvement without any convolution operations and effectively avoiding the blurring effect caused by interpolation. For example, to perform a 2x upsampling (r=2), a feature map of size 16×16×128 will become a 32×32×32 feature map after pixel shuffle. Residual feature maps are feature maps extracted from shallower layers in the backbone network (i.e., the source of "skip connections"). They are rich in high-frequency detail information that might be lost in deeper networks, such as fine textures and sharp edges. Element-wise addition is a matrix operation that requires two input feature maps to have exactly the same dimensions. During the operation, the values ​​at the same positions (i, j, k) of the two feature maps are directly added to generate a new feature map. This operation is crucial for residual learning, directly fusing deep semantic information with shallow detail information. Finally, the high-resolution infrared image is the final output of the entire process, an image that significantly surpasses the original input in both spatial resolution and detail sharpness.

[0061] The specific execution flow of this upsampling reconstruction module is a phased, refined image generation process that incorporates residual learning, aiming to gradually transform abstract feature information into clear image details. The input to this process is the enhanced feature map generated in the previous steps, assuming its size is H×W×C. First, a pixel rearrangement operation is performed on this feature map to achieve a 2x upsampling. This operation requires the number of input channels C to be a multiple of 4, rearranging the H×W×C feature map into a first upsampled feature map with a size of (2H)×(2W)×(C / 4). Simultaneously, to compensate for details that may be lost during feature extraction by the deep network, the system extracts a residual feature map from a pre-defined, shallower location in the convolutional neural network (i.e., the fourth pre-defined location). This extraction location is carefully designed so that its output feature map has a spatial resolution of exactly (2H)×(2W), perfectly matching the size of the first upsampled feature map. Next, this residual feature map, rich in high-frequency details, is added element-wise to the first upsampled feature map. This addition operation effectively fuses the macroscopic structure generated by the deep network with the fine texture preserved by the shallow network, generating a residual fused feature map. The size of this feature map remains (2H)×(2W)×(C / 4), but its content combines the advantages of both features. Subsequently, to further improve the resolution, a second pixel rearrangement operation is performed on this residual fused feature map, and it is upsampled by a factor of 2, increasing its size from (2H)×(2W)×(C / 4) to (4H)×(4W)×(C / 16), resulting in a second upsampled feature map. At this point, the spatial resolution of the feature map has reached its final goal. The final step is to input this high-resolution second upsampled feature map into a final convolutional layer (typically using a 3×3 kernel). This convolutional layer maps the high-dimensional feature space back to the image space, performs final fine-tuning and integration of the features, and outputs an image with a specified number of channels (e.g., 1 channel for a monochrome infrared image). Through this series of rigorous steps, a high-resolution infrared image with rich detail and sharp edges is finally generated.

[0062] Furthermore, this invention does not simply superimpose the functions of multiple steps such as super-resolution, pseudo-colorization, and target localization, but proposes a holistic solution with inherent technological connections and synergistic effects. The super-resolution reconstruction module proposed in this invention, through multi-scale feature fusion and residual learning, not only improves the physical resolution of the image, but more importantly, it preserves and optimizes specific image statistical characteristics crucial for subsequent intelligent processing (such as pseudo-colorization and target localization) during the reconstruction process. For example, the reconstructed edge gradient distribution more closely resembles the real scene, and artifacts of high-frequency textures are effectively suppressed. This creates unique, high-quality input conditions for subsequent pseudo-colorization based on the principal gradient direction and target localization based on weighted gradients, enhancing the performance of subsequent processing modules and achieving an overall technological improvement of "1+1>2".

[0063] The following provides a more detailed description of the process of the method provided in this implementation. Please refer to [link / reference]. Figure 2 This is another flowchart illustrating the infrared image super-resolution reconstruction method based on convolutional neural networks in this application embodiment.

[0064] To fully utilize the high-resolution infrared images with optimized gradient distribution characteristics generated by the aforementioned super-resolution reconstruction method and further enhance their visual interpretability, this method also includes:

[0065] S201. Upsampling reconstruction is performed on the enhanced feature map using an upsampling structure, and a residual connection structure is introduced during the upsampling reconstruction process to generate a high-resolution infrared image.

[0066] Upsampling structures refer to combinations of network modules used to improve the spatial resolution of feature maps, including pixel rearrangement layers and convolutional layers; enhanced feature map representations are feature representations weighted by attention mechanisms; residual connection structures refer to skip connection pathways added outside the backbone network; high-resolution infrared images refer to the target resolution images obtained after reconstruction.

[0067] After feature enhancement, image reconstruction is performed using a two-stage upsampling structure. The first stage uses pixel rearrangement to double the spatial resolution of the feature map. Specifically, the C channels are rearranged into an H×W×(C / 4) form, increasing the spatial size to 2H×2W. Simultaneously, residual feature maps are extracted from the shallow network and added pixel-by-pixel to the result of the first upsampling, supplementing image details. The second stage performs pixel rearrangement again, further increasing the feature map resolution by a factor of two. Finally, a 3×3 convolutional layer converts the feature map into an output image. The number of output channels in the convolutional layer is set to 3, corresponding to the number of channels in the final high-resolution infrared image. It is worth noting that, thanks to the optimization of the image edge gradient distribution in step S201, the principal direction parameters calculated here more accurately reflect the structural orientation of the main target in the image, avoiding misjudgments caused by noise or artifacts.

[0068] It is particularly important to emphasize that the high-resolution infrared image generated through the super-resolution reconstruction process described in S201, which includes multi-scale fusion, attention mechanisms, and residual learning, is not merely an increase in the number of pixels. Compared to images generated by traditional interpolation methods or other simple CNN models, the images generated by this invention possess the following specific technical features that are highly advantageous for subsequent processing: 1. Preservation of gradient distribution: Because residual connections effectively supplement high-frequency details, the edge gradient directions of objects in the image are more clearly defined, and the interference of noise on the gradient direction is significantly reduced. This provides a high-quality data foundation for the accurate calculation of the "principal direction parameter" in S202. 2. Realism of texture and contrast: Multi-scale feature fusion ensures that texture information at different levels is taken into account, avoiding the texture "smoothing" or "patterning" problems that may occur with single-scale models. This allows the "texture feature parameters" and "contrast parameters" extracted in S209 to more realistically reflect the scene content.

[0069] S202. Calculate the gradient direction distribution data of the high-resolution infrared image, and determine the main direction parameters based on the gradient direction distribution data.

[0070] Gradient direction distribution data refers to the statistical information of gradient changes in an image in different directions; the principal direction parameter represents the directional angle of the main structural features in the image. The gradient direction is calculated using the Sobel operator, which contains two 3×3 convolution kernels, one horizontal and one vertical, used to calculate the gradients in the x and y directions, respectively.

[0071] The reconstructed high-resolution infrared image is first subjected to gradient calculation. The Sobel operator is used to calculate the gradient values ​​Gx and Gy in the horizontal and vertical directions, respectively. Then, the gradient direction angle of each pixel is calculated using the arctangent function arctan(Gy / Gx). An orientation histogram is constructed from the calculated gradient direction angles, and the cumulative sum of gradient magnitudes in each directional interval within the 0-360 degree range is calculated. By identifying the peak positions in the orientation histogram, the principal orientation parameter of the image is determined. This parameter reflects the orientation of the most significant structural features in the image, providing a benchmark for subsequent region segmentation.

[0072] S203. Divide the high-resolution infrared image into multiple sector regions in polar coordinate space based on the principal direction parameter.

[0073] Polar coordinate space refers to a coordinate system that uses polar radius and polar angle to represent the position of a point; a sector represents a region in polar coordinate space bounded by two rays and a circular arc with the image center as the pole.

[0074] Image region segmentation is performed based on the acquired principal direction parameters. First, the image is transformed from Cartesian coordinates to polar coordinates, using the image center as the pole. The polar radius *r* and polar angle *θ* of each pixel relative to the pole are calculated. Then, using the principal direction parameters as the starting direction, the 360-degree angular range is divided into several equal sector regions, each with an equal angular range. For example, when divided into 8 sector regions, each region has an angular range of 45 degrees. This principal direction-based region segmentation method better adapts to the structural features of the image, facilitating subsequent adaptive processing of different regions.

[0075] S204. Calculate the average intensity of pixels within the sector region, and calculate the intensity spatial entropy based on the difference between the intensity of pixels within the sector region and the intensity of their neighboring pixels.

[0076] The intensity mean refers to the average grayscale value of all pixels within a sector; the neighborhood pixel intensity refers to the grayscale value of pixels within a specific range surrounding the target pixel; the intensity spatial entropy represents a measure of the uncertainty in the pixel intensity distribution within the region. For a single pixel, its neighborhood is typically selected as a 3×3 or 5×5 window region centered on that pixel.

[0077] Intensity statistics are calculated for each sector. First, all pixels within the sector are traversed, their grayscale values ​​are summed and divided by the total number of pixels to obtain the mean intensity of the sector. Then, for each pixel within the sector, the intensity difference between it and its neighboring pixels is calculated. Specifically, a 5×5 window is taken around the pixel, and the variance of the intensity difference between pixels within the window and the center pixel is calculated. Based on the calculated intensity differences, the intensity spatial entropy is calculated using the Shannon entropy formula: H = -∑(pi*log2(pi)), where pi represents the probability of the i-th intensity difference value occurring. This entropy value reflects the complexity of pixel intensity changes within the sector.

[0078] S205. Map the mean intensity value to the hue parameter of the color space, and map the intensity space entropy to the saturation parameter of the color space.

[0079] Hue parameters refer to the components in the HSV color space that represent the basic attributes of color; saturation parameters represent the purity or vividness of color. In the HSV color space, hue values ​​range from 0 to 360 degrees, and saturation values ​​range from 0 to 1.

[0080] Perform an intensity-to-color parameter mapping transformation. The mean intensity of the sector is linearly mapped to the hue parameter space using the formula: H = (Imean / 255) * 360, where Imean is the mean intensity. For the intensity space entropy, an exponential mapping function is used to convert it into a saturation parameter: S = 1 - exp(-α * Entropy), where α is an adjustable scaling factor set to 2.0, and Entropy is the calculated intensity space entropy. This mapping method ensures that regions with rich intensity variations have high color saturation.

[0081] S206. Based on the hue parameter and saturation parameter and combined with the original pixel intensity value, generate the corresponding mapping parameter group.

[0082] A mapping parameter set refers to a set of parameters used to convert a grayscale image into a color image, containing three components: hue, saturation, and brightness. The brightness component is obtained directly from the normalized grayscale values ​​of the original pixels.

[0083] Generate complete color mapping parameters. Combine the hue parameter H and saturation parameter S obtained in the previous steps with the original pixel intensity value V to form the mapping parameter set (H, S, V) for the HSV color space. The lightness component V is normalized by dividing the original pixel intensity value by 255. For pixels at the boundary of a sector, bilinear interpolation is used to smoothly transition the parameter values ​​of adjacent regions, avoiding abrupt changes. Specifically, for points within a range of d pixels from the boundary, their parameter values ​​are weighted averages of the parameters of two adjacent sector regions according to distance: P = w1P1 + w2P2, where w1 and w2 are weighting coefficients calculated based on distance, and P1 and P2 are the parameter values ​​of adjacent regions.

[0084] S207. Perform a fusion operation on the mapping parameter group of each sector at the region boundary, and convert the high-resolution infrared image into a three-channel color image through the pixel value conversion mapping function.

[0085] Blending operation refers to the calculation process of smoothly transitioning parameters between adjacent regions at the boundary of regions; pixel value conversion mapping function represents the calculation method of converting HSV color space parameters to RGB color space; three-channel color image refers to an image containing three color components: red, green, and blue.

[0086] Parameter fusion is performed at the region boundary. First, the boundary region is defined as a strip of 10 pixels on each side of the boundary. Within the boundary region, a distance-weighted fusion method is used. For a point d pixels away from the boundary, its HSV parameter value is calculated using P=(1-w)P1+wP2, where w=d / 20 is the weighting coefficient, and P1 and P2 are the parameter values ​​of two adjacent sector regions, respectively. After parameter fusion, a standard HSV-to-RGB color space conversion is performed. Specifically, different conversion formulas are used when H falls into different intervals. For example, when H is in the interval [0, 60), R=V, G=V*(1-S*(1-f)), B=V*(1-S), where f is the decimal part of H / 60. Finally, an RGB three-channel color image is obtained.

[0087] To achieve more accurate and robust target localization, the high-resolution infrared image generated by the aforementioned super-resolution reconstruction method, which features high-fidelity texture and high signal-to-noise ratio edges, further includes the following:

[0088] S208. Divide the high-resolution infrared image into multiple overlapping sub-regions according to a preset ratio.

[0089] Overlapping sub-regions refer to partially overlapping local image patches in an image; the preset ratio refers to the setting parameters for the size and degree of overlap of the sub-regions. The size of the sub-region is usually set to 1 / 4 to 1 / 8 of the original image, and the overlap rate of adjacent sub-regions is set to 50%.

[0090] The high-resolution image is divided into overlapping sub-regions. Let the image size be H×W, and select a sub-region size of h×w (h=H / 4, w=W / 4). Overlapping sub-regions are generated by sliding the sub-regions in the horizontal and vertical directions with steps of h / 2 and w / 2, respectively. For example, for a 512×512 image, if the sub-region size is set to 128×128, the sliding steps in the horizontal and vertical directions are 64, generating a total of 49 overlapping sub-regions. This division method ensures complete image coverage and sufficient information overlap between adjacent regions, which is beneficial for the continuity of subsequent feature extraction.

[0091] S209. Extract the texture feature parameters and contrast parameters of each overlapping sub-region, and calculate the weight value based on the texture feature parameters and contrast parameters.

[0092] Texture feature parameters are statistical measures that describe the local structural patterns of an image; contrast parameters are metrics that measure changes in the brightness of an image; and weight values ​​are regional importance coefficients calculated based on the feature parameters.

[0093] Feature extraction and weight calculation are performed for each overlapping sub-region. First, texture features are extracted using the Gray-Level Co-occurrence Matrix (GLCM) method. The calculation process includes constructing a 0-degree GLCM and calculating statistics such as energy, entropy, and correlation based on this matrix. The contrast parameter is obtained by dividing the difference between the maximum and minimum pixel values ​​within the region by their sum. Then, the texture features and contrast parameter are combined to calculate the weight value: w = αT + βC, where T is the normalized texture feature value, C is the contrast parameter, and α and β are weight coefficients, set to 0.6 and 0.4, respectively. This weight value reflects the richness of image detail in the sub-region.

[0094] S210. Calculate the gradient magnitude of the high-resolution infrared image, and multiply the gradient magnitude with the weight value at the corresponding pixel position to obtain the confidence-weighted gradient.

[0095] Gradient magnitude refers to the magnitude of the gradient vector at each pixel location in an image; confidence-weighted gradient represents the gradient value after weight adjustment. Gradient magnitude is calculated from the gradient components in the horizontal and vertical directions: G = sqrt(Gx^2 + Gy^2).

[0096] The gradient calculation and weighting process is performed. First, the Sobel operator is used to calculate the horizontal and vertical gradients of the image. The Sobel operator contains two 3×3 convolutional kernels: horizontal [-1, 0, 1; -2, 0, 2; -1, 0, 1] and vertical [-1, -2, -1; 0, 0, 0; 1, 2, 1]. For each pixel location, the gradient values ​​Gx and Gy in both directions are calculated, and then the gradient magnitude G is calculated. The resulting gradient magnitude map is then multiplied at the pixel level with the weight values ​​calculated in the previous step: Gw(x, y) = G(x, y) * w(x, y), where (x, y) represents the pixel coordinates, and w(x, y) is the weight value corresponding to that location. This weighting method highlights the edge features of important regions while suppressing the noise influence of unimportant regions.

[0097] S211. Determine the threshold parameter based on the distribution characteristics of the confidence-weighted gradient.

[0098] Distribution characteristics refer to the statistical distribution features of the confidence-weighted gradient values; the threshold parameter represents the decision value used to distinguish the target boundary from the background. Distribution characteristics mainly include statistical measures such as histogram shape, mean, and standard deviation.

[0099] Threshold determination is based on weighted gradient distribution. First, a histogram of the confidence-weighted gradient is plotted, dividing the gradient value range into 256 equal intervals, and the number of pixels in each interval is calculated. Then, the OTSU algorithm is used to automatically determine the optimal threshold, which maximizes the inter-class variance. The specific calculation process is as follows: All possible thresholds are iterated, the image is divided into foreground and background classes, and the means μ1 and μ2 and variances σ1 and σ2 of the two classes are calculated. The threshold that maximizes the inter-class variance σB = ω1(μ1-μT)^2 + ω2(μ2-μT)^2 is selected, where ω1 and ω2 are the pixel ratios of the two classes, and μT is the overall mean.

[0100] S212. Use the threshold parameter to segment the confidence-weighted gradient and extract the target boundary features.

[0101] Segmentation refers to dividing an image into two parts, the target and the background, based on a threshold. Target boundary features represent a set of feature points describing the target's contour. Boundary features typically include the position coordinates and orientation information of the boundary points.

[0102] Threshold-based boundary extraction is performed. First, the confidence-weighted gradient map is binarized, marking pixels greater than the threshold as boundary points (value 255) and pixels less than the threshold as background points (value 0). Then, the binarized result is thinned using a morphological thinning algorithm to iteratively remove boundary points until a single-pixel-width boundary line is obtained. Specific steps include: scanning the image using a 3×3 structuring element, determining whether a pixel is a boundary point based on the pattern of its 8-neighborhood. For each boundary point, the connectivity of its 8-neighborhood is calculated, retaining points necessary to maintain boundary connectivity and removing redundant points. The final boundary features include the position sequence of boundary points and the normal direction at each point.

[0103] S213. Perform region merging and contour fitting operations on the target boundary features to generate the target localization result.

[0104] Region merging refers to combining adjacent boundary feature points into continuous boundary segments; contour fitting operation means using a mathematical model to smoothly fit the boundary segments; target localization result refers to the final determined target region contour and location information.

[0105] Boundary post-processing and target localization are performed. First, connectivity analysis is conducted on boundary feature points, and adjacent boundary points are grouped into initial boundary segments using the 8-connectivity criterion. For each boundary segment, its length, curvature, and other geometric features are calculated. If the endpoint distance between two boundary segments is less than a preset threshold (usually 5 pixels) and the directional difference is less than 30 degrees, these two boundary segments are merged. If the length of the merged boundary segment is less than a minimum length threshold (set to 20 pixels), it is discarded to remove noise. Then, B-spline curve fitting is performed on the retained boundary segments. Specifically, control points are uniformly sampled on the boundary point sequence, and a smooth curve is constructed using a 3rd-order B-spline basis function. The fitting process uses the least squares criterion, iteratively optimizing the control point positions to minimize the cumulative error between the fitted curve and the original boundary points. Finally, based on the fitted closed contour, the target's center position, area, orientation, and other feature parameters are calculated to form a complete target localization result. The localization result typically includes the following information: the vertex coordinate sequence of the target contour, the coordinates of the target center, the target's principal direction angle, and the target's minimum bounding rectangle parameters (center coordinates, length, width, and rotation angle). These parameters can be used for subsequent target tracking and behavior analysis.

[0106] In practical applications, contour fitting also considers the following aspects:

[0107] For complex targets, a piecewise fitting strategy is adopted, setting segment points where the curvature changes significantly. Each segment uses different fitting parameters, and a penalty term is introduced to control the smoothness of the fitted curve and avoid overfitting. The fitting results are adaptively adjusted, and the density of control points is adjusted according to local features. When there are local gaps in the contour, the geometric constraints of adjacent segments are used to complete the fit, and the image grayscale information is combined to fine-tune the fitting results to ensure that the contour coincides with the actual edge to the greatest extent.

[0108] In addition to geometric features, the final target localization output also includes reliability evaluation metrics such as contour completeness and fitting accuracy, providing a reference for subsequent processing. These localization results can be directly used for high-level vision tasks such as target detection and tracking, or they can serve as the basis for feature extraction, supporting more complex scene understanding and behavior analysis.

[0109] The infrared image super-resolution reconstruction system in the embodiments of this invention is described below from the perspective of hardware processing. Please refer to [link to relevant documentation]. Figure 3 This is a schematic diagram of the physical device structure of an infrared image super-resolution reconstruction system in the embodiments of this application.

[0110] It should be noted that, Figure 3 The structure of the infrared image super-resolution reconstruction system shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0111] like Figure 3 As shown, the infrared image super-resolution reconstruction system includes a Central Processing Unit (CPU) 301, which can perform various appropriate actions and processes based on a program stored in Read-Only Memory (ROM) 302 or a program loaded from storage section 308 into Random Access Memory (RAM) 303, such as performing the methods described in the above embodiments. The RAM 303 also stores various programs and data required for system operation. The CPU 301, ROM 302, and RAM 303 are interconnected via a bus 304. An Input / Output (I / O) interface 305 is also connected to the bus 304.

[0112] The following components are connected to I / O interface 305: input section 306 including audio input devices, push-button switches, etc.; output section 307 including a liquid crystal display (LCD) and audio output devices, indicator lights, etc.; storage section 308 including a hard disk, etc.; and communication section 309 including a network interface card such as a LAN (Local Area Network) card, modem, etc. Communication section 309 performs communication processing via a network such as the Internet. Drive 310 is also connected to I / O interface 305 as needed. Removable media 311, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 310 as needed so that computer programs read from them can be installed into storage section 308 as needed.

[0113] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing computer programs for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 309, and / or installed from removable medium 311. When the computer program is executed by central processing unit (CPU) 301, it performs the various functions defined in the present invention.

[0114] It should be noted that specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0115] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. Each block in a flowchart or block diagram may represent a module, program segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those shown in the drawings.

[0116] Specifically, the infrared image super-resolution reconstruction system of this embodiment includes a processor and a memory. The memory stores a computer program. When the computer program is executed by the processor, it implements the infrared image super-resolution reconstruction method based on convolutional neural networks provided in the above embodiment.

[0117] In another aspect, the present invention also provides a computer-readable storage medium, which may be included in the infrared image super-resolution reconstruction system described in the above embodiments; or it may exist independently and not assembled into the infrared image super-resolution reconstruction system. The storage medium carries one or more computer programs, which, when executed by a processor of the infrared image super-resolution reconstruction system, cause the infrared image super-resolution reconstruction system to implement the convolutional neural network-based infrared image super-resolution reconstruction method provided in the above embodiments.

[0118] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0119] As used in the above embodiments, depending on the context, the term "when..." can be interpreted as meaning "if...", "after...", "in response to determining...", or "in response to detecting...". Similarly, depending on the context, the phrase "when determining..." or "if (the stated condition or event) is interpreted as meaning "if determining...", "in response to determining...", "when (the stated condition or event) is detected", or "in response to detecting (the stated condition or event)".

[0120] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

Claims

1. An infrared image super-resolution reconstruction method based on a convolutional neural network, characterized in that, The method, applied to an infrared image super-resolution reconstruction system, includes: A low-resolution infrared image is input into a preset convolutional neural network to obtain a multi-layer feature map of the low-resolution infrared image. The convolutional neural network is trained using a hybrid loss function that includes an adversarial loss term and a perceptual loss term. The adversarial loss term is calculated based on the discriminator network's distinction between the generated image and the real image, and the perceptual loss term is calculated based on the difference between the generated image and the real image in the high-dimensional feature space. The feature maps of different levels in the multi-layer feature map are scaled according to a preset ratio and then stitched together to obtain a fused feature map. The fused feature map is processed by global average pooling to generate a feature vector, and the feature vector is transformed by a fully connected layer to obtain attention weights. The attention weights are then weighted with the fused feature map to obtain an enhanced feature map. The enhanced feature map is upsampled and reconstructed using an upsampling structure, and a residual connection structure is introduced during the upsampling and reconstruction process to generate a high-resolution infrared image. Calculate the gradient direction distribution data of the high-resolution infrared image, and determine the main direction parameters based on the gradient direction distribution data; The high-resolution infrared image is divided into multiple sector regions in polar coordinate space based on the principal direction parameter; For each of the sector regions, a corresponding mapping parameter set is determined based on the pixel intensity distribution of the sector region; The mapping parameter groups of each sector region are fused at the region boundary, and the high-resolution infrared image is converted into a three-channel color image through a pixel value conversion mapping function.

2. The method according to claim 1, characterized in that, The step of scaling and then merging the feature maps of different levels in the multi-layer feature map according to a preset ratio to obtain a fused feature map specifically includes: The convolutional neural network outputs a shallow feature map from a first preset position, a middle feature map from a second preset position, and a deep feature map from a third preset position, wherein the layers of the first preset position, the second preset position, and the third preset position are progressively deeper, and the spatial resolution of the output feature maps is progressively lower. The spatial dimensions of the shallow feature map are obtained, and upsampling scaling is performed on the middle feature map to make the spatial dimensions of the middle feature map the same as those of the shallow feature map, thus obtaining a first scaled feature map; the upsampling scaling is performed on the deep feature map to make the spatial dimensions of the deep feature map the same as those of the shallow feature map, thus obtaining a second scaled feature map. The shallow feature map, the first scaled feature map, and the second scaled feature map are spliced ​​along the channel dimension to generate a spliced ​​feature map. The stitched feature map is input into a convolutional layer, and a convolution operation is performed on the stitched feature map to obtain the fused feature map.

3. The method according to claim 1, characterized in that, The process of upsampling and reconstructing the enhanced feature map using an upsampling structure, and introducing a residual connection structure during the upsampling and reconstruction process to generate a high-resolution infrared image, specifically includes: Perform a first pixel rearrangement operation on the enhanced feature map to generate a first upsampled feature map, the spatial resolution of the first upsampled feature map being twice that of the enhanced feature map; A residual feature map is extracted from the fourth preset position of the convolutional neural network. The spatial resolution of the residual feature map is the same as that of the first upsampled feature map. The residual feature map and the first upsampled feature map are added element by element to generate a residual fusion feature map. A second pixel rearrangement operation is performed on the residual fusion feature map to generate a second upsampled feature map, the spatial resolution of which is twice that of the residual fusion feature map; The second upsampled feature map is input into the convolutional layer, and a convolution operation is performed on the second upsampled feature map to output the high-resolution infrared image.

4. The method according to claim 1, characterized in that, The step of determining the corresponding mapping parameter set for each of the sector regions based on the pixel intensity distribution of the sector region specifically includes: Calculate the average intensity of pixels within the sector region, and calculate the intensity spatial entropy based on the difference between the intensity of pixels within the sector region and the intensity of their neighboring pixels; The intensity mean is mapped to the hue parameter of the color space, and the intensity space entropy is mapped to the saturation parameter of the color space. Based on the hue and saturation parameters and combined with the original pixel intensity values, a corresponding mapping parameter set is generated.

5. The method according to claim 1, characterized in that, After the steps of upsampling and reconstructing the enhanced feature map using an upsampling structure and introducing a residual connection structure during the upsampling and reconstruction process to generate a high-resolution infrared image, the method further includes: The high-resolution infrared image is divided into multiple overlapping sub-regions according to a preset ratio; Extract the texture feature parameters and contrast parameters of each overlapping sub-region, and calculate the weight value based on the texture feature parameters and contrast parameters; The corresponding threshold parameters are generated based on the weight values, and the target boundary features are extracted based on the threshold parameters. Perform region merging and contour fitting operations on the target boundary features to generate the target localization result.

6. The method according to claim 5, characterized in that, The step of generating a corresponding threshold parameter based on the weight value and extracting target boundary features based on the threshold parameter specifically includes: Calculate the gradient magnitude of the high-resolution infrared image, and multiply the gradient magnitude with the weight value at the corresponding pixel position to obtain the confidence-weighted gradient. The threshold parameter is determined based on the distribution characteristics of the confidence-weighted gradient. The confidence-weighted gradient is segmented using the threshold parameter to extract target boundary features.

7. An infrared image super-resolution reconstruction system, characterized in that, The infrared image super-resolution reconstruction system includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code including computer instructions, and the one or more processors call the computer instructions to cause the infrared image super-resolution reconstruction system to perform the method as described in any one of claims 1-6.

8. A computer-readable storage medium comprising instructions, characterized in that, When the instruction is run on the infrared image super-resolution reconstruction system, the infrared image super-resolution reconstruction system performs the method as described in any one of claims 1-6.

9. A computer program product, characterized in that, When the computer program product is run on the infrared image super-resolution reconstruction system, the infrared image super-resolution reconstruction system performs the method as described in any one of claims 1-6.

Images