Thermal imaging super-resolution method, device and system based on image reconstruction

Abstract

This invention relates to the field of image processing technology, specifically to a thermal imaging super-resolution method, apparatus, and system based on image reconstruction. The technical solution of this invention extracts image frames from infrared images captured by a mobile imaging terminal, analyzes the temperature gradient distribution and foreground region distribution of each image frame, combines the effective frame coefficient of each image frame with the thermal energy fluctuation characteristics of each convolutional feature map after convolution processing, obtains the importance of thermal information of each foreground region mapped to the convolutional feature map, calculates corresponding reference weights within each target region based on the spatial relationship between pixels and the temperature gradient changes, fills the corresponding pixels to be expanded according to the reference weights and the pixel values ​​of neighboring pixels, and combines all image frames in chronological order when the reconstruction resolution of each image frame meets the target resolution, outputting a super-resolution thermal imaging video. This achieves high-quality reconstruction of infrared images under relatively limited computing resources.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and specifically to a thermal imaging super-resolution method, apparatus, and system based on image reconstruction. Background Technology

[0002] Infrared images can be captured using thermal imaging devices. However, existing mobile thermal imaging devices are limited by hardware, resulting in low image resolution and blurred details, which is not conducive to detailed analysis. General image super-resolution techniques suffer from computational waste or poor reconstruction results in key areas when reconstructing infrared images. For example, when reconstructing infrared images using Super-Resolution Convolutional Neural Networks (SRCNN), they are trained on visible light images, which do not conform to the physical characteristics of thermal radiation, making it difficult to recover the unique blur caused by thermal diffusion. Summary of the Invention

[0003] To address the technical problem of improving the quality of infrared image reconstruction under limited computing resources, the present invention aims to provide a thermal imaging super-resolution method, apparatus, and system based on image reconstruction. The specific technical solution adopted is as follows: In a first aspect, embodiments of the present invention provide a thermal imaging super-resolution method based on image reconstruction, the method comprising: Extract image frames from infrared images captured by a mobile imaging terminal, and obtain the effective frame coefficient of each image frame based on the temperature gradient distribution and foreground region distribution of each image frame; Based on the effective frame coefficient of each image frame and the thermal energy fluctuation characteristics of each convolutional feature map after convolution processing, the importance of thermal information of each foreground region mapped to the convolutional feature map is obtained. The foreground region with thermal information importance greater than the importance threshold is identified as the target region. Based on the positional relationship and temperature gradient between each pixel in each target region, the reference weights of the pixel to be expanded and its neighboring pixels are calculated. The corresponding pixels to be expanded are filled according to the reference weight and the pixel value of the neighboring pixels. When the reconstruction resolution of each image frame meets the target resolution, all image frames are combined in chronological order and the super-resolution thermal imaging video is output.

[0004] In one optional embodiment, the effective frame coefficient of the corresponding image frame is obtained based on the temperature gradient distribution and foreground region distribution of each image frame, including: The number of pixels for each temperature gradient in the current image frame is counted, and the gradient feature coefficients representing the temperature gradient distribution characteristics of the current image frame are calculated based on the statistical results. Perform foreground segmentation on the current image frame to obtain the target area of ​​the largest connected component and the total number of foreground regions in all foreground regions; Based on the target area and the total number of foreground regions, obtain the foreground feature coefficients of the current image frame; The effective frame coefficients of the current image frame are obtained based on the gradient feature coefficients and the foreground feature coefficients.

[0005] In one optional embodiment, the number of pixels for each temperature gradient in the current image frame is counted, and gradient feature coefficients representing the temperature gradient distribution characteristics of the current image frame are calculated based on the statistical results, including: Based on the statistical results of the number of pixels of each temperature gradient in the current image frame, the number of target pixels contained in the mode of gradient magnitude is obtained. The mode of gradient magnitude is the temperature gradient that contains the most pixels among all temperature gradient magnitudes. The gradient feature coefficients of the current image frame are obtained by comparing the number of target pixels with the total number of pixels in the current image frame.

[0006] In one optional embodiment, the importance of thermal information of each foreground region mapped to the convolutional feature map is obtained based on the frame effectiveness coefficient of each image frame and the thermal fluctuation characteristics of each convolutional feature map after convolution processing, including: When the effective coefficient of the current image frame is determined to be greater than the preset effective threshold, the current image frame is input into the super-resolution convolutional neural network model to obtain the convolutional feature maps of the current image frame. The pixel values ​​of the corresponding foreground regions in each convolutional feature map of the current image frame are statistically analyzed, and the cumulative feature index representing the heat source image details of each foreground region in the current image frame is calculated. Based on the effective frame coefficient and feature accumulation index of the current image frame, the thermal information importance of each foreground region of the current image frame mapped to the convolutional feature map is obtained.

[0007] In one optional embodiment, the pixel values ​​of the corresponding foreground regions in each convolutional feature map of the current image frame are statistically analyzed, and a feature accumulation index characterizing the heat source image details of each foreground region in the current image frame is calculated, including: Based on the statistical results of pixel values ​​of each foreground region in each convolutional feature map of the current image frame, obtain the individual pixel value, the first average pixel value, and the pixel standard deviation of each foreground region; The fluctuation intensity of each pixel is obtained based on the first pixel value difference and pixel standard deviation of each pixel in the foreground region. The first pixel value difference is the difference between the individual pixel value and the first average pixel value of that pixel. The cumulative feature index of the corresponding foreground region is obtained by summing the cumulative fluctuation intensity of all pixels in each foreground region.

[0008] In one optional embodiment, the reference weights of the pixel to be expanded and its neighboring pixels are calculated based on the positional relationships and temperature gradients between pixels within each target region, including: The information correlation degree is evaluated based on the relative positional relationship of each pixel to be expanded in its target area, so as to obtain the information dependence of the pixel to be expanded on its neighboring pixels. The second pixel value difference of the corresponding pixel to be expanded is obtained based on the difference between the maximum pixel value of each pixel to be expanded in its target region and the target pixel value of the reference target pixel. Based on the difference in the second pixel value of each pixel to be expanded and the second average pixel value of its neighboring region, the degree of information change of the corresponding pixel to be expanded compared with the neighboring pixels is obtained when it is expanded. Based on the information dependency and information variability of each pixel to be expanded, the reference weights of the corresponding pixel to be expanded and its neighboring pixels are obtained.

[0009] In one optional embodiment, information correlation is evaluated based on the relative positional relationship of each pixel to be expanded within its target region to obtain the information dependence of the pixel to be expanded on its neighboring pixels, including: Based on the current coordinates of the pixel to be expanded within the target area and the center coordinates of the area, the distance correlation between the pixel to be expanded and the target area is obtained. Based on the distance correlation of the current pixel to be expanded and the minimum edge distance, the information dependence of the current pixel to be expanded on its neighboring pixels is obtained. The minimum edge distance is the minimum distance from the center coordinates and the current coordinates to the edge of the target area.

[0010] In one optional embodiment, filling the corresponding pixels to be expanded according to the reference weight and the pixel values ​​of neighboring pixels includes: Normalize the calculation of all reference weights of the neighboring pixels of the pixel to be expanded to obtain the expansion weight of each reference pixel in the neighborhood of the pixel to be expanded. Based on the expansion weight and pixel value of each reference pixel in the neighborhood of the pixel to be expanded, obtain the pixel assignment of the pixel to be expanded, and fill the pixel to be expanded accordingly based on the pixel assignment.

[0011] Secondly, embodiments of the present invention also provide a thermal imaging super-resolution device based on image reconstruction. The device corresponds to any super-resolution method described in the first aspect, and includes: The extraction module is used to extract image frames of infrared images captured by the mobile shooting terminal, and obtain the effective frame coefficient of the corresponding image frame based on the temperature gradient distribution and foreground region distribution of each image frame. The processing module is used to obtain the importance of thermal information of each foreground region mapped to the convolutional feature map based on the effective frame coefficient of each image frame and the thermal energy fluctuation characteristics of each convolutional feature map after convolution processing. The calculation module is used to determine the foreground region where the importance of thermal information is greater than the importance threshold as the target region, and to calculate the reference weight between the pixel to be expanded and the neighboring pixels based on the positional relationship and temperature gradient between each pixel in each target region. The fill output module is used to fill the corresponding pixels to be expanded according to the reference weight and the pixel value of the neighboring pixels. When the reconstruction resolution of each image frame meets the target resolution, all image frames are combined in chronological order and the super-resolution thermal imaging video is output.

[0012] Thirdly, embodiments of the present invention also provide a thermal imaging super-resolution system based on image reconstruction, the system comprising: The processor and the memory connected to the processor; The memory is used to store computer programs, and the processor is used to execute the computer programs to implement the super-resolution method of any of the first aspects.

[0013] The present invention has the following beneficial effects: The technical solution of this invention extracts image frames from infrared images captured by a mobile imaging terminal. Based on the temperature gradient distribution and foreground region distribution of each image frame, it obtains the frame effectiveness coefficient of that image frame. The temperature gradient distribution reflects the degree of change in the edges and texture details of heat sources in the image, while the foreground region distribution characterizes the spatial proportion of effective thermal targets in the image. Therefore, based on the frame effectiveness coefficient, image frames containing effective thermal information and with relatively clear details can be identified. Based on the frame effectiveness coefficient of each image frame and the thermal fluctuation characteristics of each convolutional feature map after convolution processing, since different convolutional kernels can respond to different thermal textures and structural features in the image frame during convolution processing, the effective coefficients of each image frame can be obtained. The importance of thermal information in the foreground region is mapped to the convolutional feature map. Foreground regions with thermal information importance greater than the importance threshold are identified as target regions. Target regions contain more significant heat source structures and temperature change features. Within each target region, the degree of information correlation between the pixel to be expanded and its neighboring pixels is analyzed based on the spatial relationship between pixels and temperature gradient changes, and corresponding reference weights are calculated accordingly. The corresponding pixels to be expanded are filled according to the reference weights and the pixel values ​​of neighboring pixels to achieve a gradual increase in image resolution. When the reconstruction resolution of each image frame meets the target resolution, all image frames are combined in chronological order, and a super-resolution thermal imaging video is output.

[0014] The technical solution of this invention, based on the effectiveness screening of image frames, performs reconstruction processing on key regions containing rich thermal information, and utilizes thermal energy fluctuation features in the convolutional feature map to assist in determining the target region, reducing unnecessary computational overhead. Simultaneously, by combining the pixel positional relationship and temperature gradient within the target region for weighted filling, the ability to restore image thermal source details is effectively improved. Therefore, high-quality reconstruction of infrared images can be achieved under the condition of relatively limited computing resources in mobile thermal imaging devices. Attached Figure Description

[0015] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 A flowchart of a thermal imaging super-resolution method based on image reconstruction provided in one embodiment of the present invention; Figure 2 This is a flowchart for calculating the importance of thermal information provided in one embodiment of the present invention; Figure 3 This is an extended image schematic diagram provided for one embodiment of the present invention; Figure 4 A flowchart for calculating reference weights provided in one embodiment of the present invention; Figure 5 This is a schematic diagram illustrating the weighted filling of pixels to be expanded according to an embodiment of the present invention; Figure 6 This is a schematic diagram of the structure of a thermal imaging super-resolution device based on image reconstruction, provided in one embodiment of the present invention. Detailed Implementation

[0017] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of a thermal imaging super-resolution method, apparatus, and system based on image reconstruction proposed according to the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0018] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0019] For infrared images acquired by mobile imaging terminals, due to limited computing resources and low sensor resolution, directly performing high-complexity super-resolution processing makes it difficult to achieve ideal reconstruction quality while ensuring real-time performance. If uniform processing of the entire image is used, without distinguishing between key targets and background, it leads to wasted computation or poor reconstruction results in key areas. Therefore, a reasonable trade-off needs to be struck between computational efficiency and image detail recovery capabilities. The following, with reference to the accompanying drawings, details a specific solution for a thermal imaging super-resolution method, apparatus, and system based on image reconstruction provided by this invention.

[0020] Please see Figure 1 , Figure 1 A flowchart of a thermal imaging super-resolution method based on image reconstruction according to an embodiment of the present invention is shown. This method can be applied to the processor of a mobile imaging terminal or configured to run on a processing terminal connected to the imaging terminal, as long as it can perform super-resolution reconstruction of infrared images. No specific limitations are placed on the type of terminal or device. The super-resolution method includes: S11: Extract the image frames of the infrared images captured by the mobile shooting terminal, and obtain the effective frame coefficient of the corresponding image frame based on the temperature gradient distribution and foreground area distribution of each image frame.

[0021] Specifically, in nature, objects above absolute zero actively emit infrared radiation. The higher the object's temperature, the greater the energy of the emitted infrared radiation. Therefore, infrared thermal imaging technology converts invisible thermal radiation into infrared thermal images that can be observed by the human eye. Based on the surface temperature distribution revealed by the intensity of thermal radiation, it determines the difference in radiation intensity between the target and the background, thus revealing features that cannot be shown in flat lighting images.

[0022] Infrared images are raw images captured by mobile imaging terminals. They can be used to capture short-wave infrared images, which offer superior imaging quality in strong light and high-temperature environments. Therefore, by deploying mobile infrared devices, setting initial shooting parameters, and initiating shooting, continuous infrared image frames are obtained. Infrared images are composed of multiple image frames combined according to shooting time. Due to the hardware limitations of the imaging sensor, the resolution of infrared images cannot clearly display the infrared image details of the photographed object. Therefore, image reconstruction is necessary to generate super-resolution thermal imaging videos, enabling technicians to perform more intuitive analysis of detailed features. Image frame extraction from infrared images can be done by extracting partial image frames based on preset intervals; alternatively, all image frames can be extracted sequentially, with each image frame's timestamp marked and stored.

[0023] Analyzing the temperature gradient distribution within an image frame reveals the degree of change in heat source edges and thermal texture details. Simultaneously, foreground segmentation of the image frame yields foreground image distribution features, reflecting the spatial distribution of effective thermal targets within the image. A comprehensive evaluation of these temperature gradient and foreground region distribution features yields a frame effectiveness coefficient, characterizing the effectiveness of the current image frame's information, providing a quantitative reference for subsequent image reconstruction processing.

[0024] For example, step S11 includes sub-steps S11-1 to S11-4, which are described in detail below: S11-1: The number of pixels at each temperature gradient in the current image frame is counted, and the gradient feature coefficients representing the temperature gradient distribution characteristics of the current image frame are calculated based on the statistical results. It can be understood that thermal imaging targets the thermal radiation distribution information in the environment, which is difficult to capture with ordinary optical images. Therefore, it is necessary to evaluate the thermal characteristics exhibited by continuous frame images. Mobile shooting terminals, being portable devices, have relatively lower basic performance compared to fixed devices. Therefore, the information distribution in the image frame is evaluated to determine the inherent infrared characteristics of the image. The pixels in the current image frame exhibit different temperature gradients. By counting the number of pixels at each temperature gradient, the number of pixels at each temperature gradient can be obtained. Based on the distribution characteristics of the number of pixels, the temperature gradient distribution characteristics of the image frame can be analyzed. Quantifying this characteristic yields the gradient feature coefficients.

[0025] Specifically, sub-step S11-1 includes: The first step is to obtain the number of target pixels contained in the mode of the gradient magnitude based on the statistical results of the number of pixels for each temperature gradient in the current image frame. The mode of the gradient magnitude is the temperature gradient that contains the most pixels among all temperature gradient magnitudes. The gradient t of temperature T corresponding to each pixel in the current image frame f is calculated, and the mode of the gradient magnitude in the frame is taken. The number of pixels included is the number of target pixels. This is used to describe the overall situation of thermal environment foreground and background differentiation in the current image frame.

[0026] The second step is to obtain the gradient feature coefficients of the current image frame based on the ratio of the number of target pixels to the total number of pixels in the current image frame. (The last sentence appears to be incomplete and possibly refers to setting the target pixel count to something else entirely.) The number of all pixels in the current image frame In contrast, the ratio characterizes the overall background differentiation in the current image frame; that is, the more data points the mode of the gradient magnitude contains, the more obvious the distinction between the foreground and background in the overall image frame. Further analysis combines the foreground and background differentiation in the current image frame with information from the foreground region to more effectively evaluate the overall target characteristics of the current image frame.

[0027] S11-2: Perform foreground segmentation on the current image frame to obtain the target area of ​​the largest connected component and the total number of foreground regions in all foreground regions. Foreground segmentation of the current image frame f can be performed using the OTSU (Otsu thresholding method). For the segmented foreground portion, all connected components are extracted. The area of ​​the largest connected component in the target area is denoted as the target area. There are multiple foreground regions in the foreground portion, and the total number of foreground regions in each image frame is equal to the total number of foreground regions. .

[0028] S11-3: Obtain the foreground feature coefficient of the current image frame based on the target area and the total number of foreground regions. The foreground feature coefficient can be derived from the ratio of the target area to the total number of foreground regions. This parameter reflects whether the targets in the foreground of the current image frame are concentrated. The larger the foreground feature coefficient, the more accurate the super-resolution is in targeting the target; conversely, the targets in the foreground are scattered.

[0029] S11-4: Obtain the effective frame coefficients of the current image frame based on the gradient feature coefficients and foreground feature coefficients. The effective frame coefficients of the current image frame f are denoted as... Then it can be done through the formula: Calculate the effective frame coefficient The coefficient 0.001 in the formula is to prevent calculation errors where the denominator is 0. Based on this, the effective frame coefficient of each image frame can be calculated, and the global target distribution of the infrared image can be obtained, providing a basis for judging the performance overhead of the super-resolution process.

[0030] At this point, the effective frame coefficients for each image frame have been obtained based on the above steps, and we proceed to step S12.

[0031] S12: Based on the effective frame coefficient of each image frame and the thermal energy fluctuation characteristics of each convolutional feature map after convolution processing, obtain the thermal information importance of each foreground region mapped to the convolutional feature map.

[0032] Specifically, the frame effectiveness coefficients of each image frame can be used for image filtering. The filtering results can be image frames or partial regions of each image frame. Convolution processing is then performed on image frames that meet the criteria to extract the corresponding convolutional feature maps (or convolutional signal maps). Since different convolutional kernels in a convolutional neural network can extract image features of different scales and types, the convolutional feature maps can reflect multi-level information about the heat source structure and thermal energy changes in the image frame. Based on this, statistical analysis is performed on the pixel features of the corresponding foreground regions in each convolutional feature map. Combined with the frame effectiveness coefficients of the image frames, the degree of thermal energy fluctuation contained in each foreground region is comprehensively evaluated to obtain the thermal information importance of each foreground region in the convolutional feature map. This parameter is used to characterize the information value of each region in the image detail reconstruction process. The greater the thermal information importance, the greater the information value of the image frame; conversely, the lower the importance, the lower the information value.

[0033] For example, please refer to Figure 2 Step S12 includes sub-steps S12-1 to S12-3, which are described in detail below: S12-1: When the effective coefficient of the current image frame is determined to be greater than the preset effective threshold, it indicates that the super-resolution effectiveness of different regions in the image frame needs to be evaluated. Combined with the content distribution under the target of different image frames, the super-resolution requirement of different regions of the image is evaluated. Then, the current image frame is input into the super-resolution convolutional neural network model (SRCNN) to obtain the convolutional feature maps of the current image frame.

[0034] It is understandable that different convolutional feature maps represent different thermal features of the current image frame. For example, when there are three convolutional feature maps for each image frame, they can represent temperature gradient features, thermal texture features, and thermal region boundary features. Multiple convolutional feature maps can be used to comprehensively evaluate the reconstruction value of the foreground region. The effective threshold can be set based on the actual situation. For example, the effective coefficients of all image frames can be normalized so that the values ​​of all effective coefficients are within the range of 0-1, and the effective threshold can be configured as 0.6. Super-resolution technology itself addresses the issue of blurred and poorly sharp details in images captured by mobile shooting terminals due to equipment performance limitations and heat conduction before the target. Therefore, frame effectiveness provides weights for global information processing when performing super-resolution processing on frames based on super-resolution convolutional neural networks.

[0035] S12-2: Statistically analyze the pixel values ​​of the corresponding foreground regions in each convolutional feature map of the current image frame, and calculate the cumulative feature index representing the heat source image details of each foreground region in the current image frame. Since there are multiple corresponding foreground regions in the convolutional feature map, for each extracted foreground region q, mark the range of pixel coordinates within that region to define the thermal influence range of the region, avoiding the inability to determine the region range during deep feature extraction.

[0036] Furthermore, sub-step S12-2 includes: The first step is to obtain the individual pixel value, first average pixel value, and pixel standard deviation of each foreground region based on the statistical results of pixel values ​​in each convolutional feature map of the current image frame. After convolution processing, the current image frame yields multiple convolutional feature maps. Each convolutional feature map has multiple corresponding regions mapped to the foreground region. For a single corresponding region Y, the total number of pixels in the current region is calculated... By traversing each pixel, the signal value of its position can be extracted and recorded as a single pixel value. The first average pixel value for a single region Y is calculated by averaging all individual pixel values ​​within that region. Similarly, the standard deviation of pixels in a single region Y is calculated by performing a standard deviation calculation on all individual pixel values. .

[0037] The second step involves obtaining the fluctuation intensity of each pixel based on the first pixel value difference and pixel standard deviation of each pixel in the foreground region. The first pixel value difference is the difference between a single pixel value and the first average pixel value of that pixel. The ratio of this difference to the pixel standard deviation is determined as the fluctuation intensity of the corresponding pixel. By analyzing all the fluctuation intensities in region Y, the fluctuation changes in that region can be obtained. The stronger the fluctuation, the more heat source targets exist in the region, and the more image details are considered in the reconstruction of the current region. Furthermore, the frame validity of the original image frame in which the region is located is used as the global weight of the image in which the region is located to optimize the current target.

[0038] The third step is to obtain the cumulative feature index for each foreground region based on the cumulative sum of the fluctuation intensities of all pixels in each foreground region. Taking the region Y mapped from the foreground region to the convolutional feature map as an example, this can be achieved by using the cumulative sum of the fluctuation intensities of all pixels. The cumulative feature index is derived, and setting the coefficient to 0.001 also prevents calculation errors where the denominator is 0. This indicates taking the absolute value.

[0039] S12-3: Based on the effective frame coefficient and feature accumulation index of the current image frame, obtain the thermal information importance of each foreground region mapped to the convolutional feature map in the current image frame. Based on the formula: The importance of thermal information of the foreground region mapped to the corresponding region Y of the convolutional feature map is calculated. The meanings of all characters in the formula are the same as described above. Based on this, the importance of thermal information during the reconstruction process is calculated for all foreground regions corresponding to each convolutional feature map in the current image frame.

[0040] At this point, the importance of the thermal information of the foreground region mapped to the convolutional feature map has been obtained based on the above method, and we proceed to step S13.

[0041] S13: Determine the foreground region where the thermal information importance is greater than the importance threshold as the target region, and calculate the reference weight between the pixel to be expanded and the neighboring pixels based on the positional relationship and temperature gradient between each pixel in each target region.

[0042] Specifically, in practical applications, mobile shooting terminals have limited computing power and processing capacity, making it impossible to perform traversal super-resolution of the entire image frame. Therefore, it is necessary to determine the necessity of super-resolution of target regions within the image frame and evaluate the resulting changes in content effectiveness to obtain super-resolution content that meets the application requirements of mobile shooting terminals. Based on a comparison of thermal information importance with a preset importance threshold, regions with higher thermal information importance are selected from all foreground regions as target regions. Since these regions typically contain more significant heat source structures and temperature change features, they have higher priority in image resolution reconstruction. It should be noted that corresponding importance thresholds can be configured for different types of convolutional feature maps to accurately select the target regions that need to be reconstructed.

[0043] Furthermore, within each target region, based on the spatial relationship between pixels and the temperature gradient change, the degree of information correlation between the pixel to be expanded and its neighboring pixels is analyzed, and a corresponding reference weight is calculated accordingly. This reference weight is used to characterize the influence of neighboring pixels on the pixel to be expanded, thereby providing a quantitative reference for pixel value reconstruction. In the technical solution of this embodiment, pixel value reconstruction of an image frame involves adding pixels to that frame, so as to... Figure 3 As shown in the example, pixel y is the original pixel, and its surrounding pixels... For expanded pixels. Extract the foreground region Y corresponding to the coordinates in the image frame, and extract a single target point y in this foreground region. Using a 3x3 window centered on this target point as the influence range of the target point, the effect is applied to one of the points. Evaluate the relationship between the current target point y and the expansion point. The reference weight k. The image frame is expanded in different rounds. Each time the pixels are expanded, the distance between pixels in the current image frame is increased by 1, meaning the distance between any two adjacent pixels is 2.

[0044] It should be noted that in the process of super-resolution reconstruction of the foreground region, in order to ensure that the thermal imaging edge information of the target object is clearer and to reduce the information distortion caused by the increase of pixels per unit area, it is necessary to take the distribution information of the target region as an information distribution consideration in the reconstruction process, and thus guide its distribution when expanding image details based on the pixel value distribution.

[0045] For example, please refer to Figure 4 Step S13 includes sub-steps S13-1 to S13-4, which are described in detail below: S13-1: Evaluate the information correlation degree based on the relative positional relationship of each pixel to be expanded in its target region to obtain the information dependence of the pixel to be expanded on its neighboring pixels. Since super-resolution reconstruction of an image frame is based on the original image, it is necessary to perform correlation analysis based on the pixel distribution characteristics of the pixel to be expanded in its target region and quantify the information dependence degree.

[0046] Furthermore, the calculation of information dependency can begin by determining the distance correlation between the current coordinates of the pixel to be expanded and the center coordinates of its respective target region. For example, for the current pixel to be expanded... The distance between the location in the target region and the center point yc of the region is used as the distance correlation degree. This parameter describes the actual location of the pixel to be expanded within the target region, as well as its distance relative to the center of the target region. That is, the closer to the center, the more likely it is to originate from a theoretical heat source at the region center. Then, based on the distance correlation and minimum edge distance of the current pixel to be expanded, the information dependency of the current pixel to be expanded on its neighboring pixels is obtained. The minimum edge distance is the minimum distance from the center coordinates and the current coordinates to the edge of its target region. The data is extracted from the center point yc, passing through the current point... The minimum distance to any edge point yb of the current target region Y, i.e., the minimum edge distance. As a result of the effective distribution describing the current region, its ratio is thus determined. The parameter is defined as information dependency, which represents the degree to which the current pixel y depends on the surrounding information when it is expanded by integrating surrounding information.

[0047] S13-2: Based on the difference between the maximum pixel value of each pixel to be expanded in its respective target region and the target pixel value of the reference target pixel, obtain the second pixel value difference for the corresponding pixel to be expanded. (The sentence is incomplete and ends abruptly.) By combining the dependence on surrounding information with the amount of change in surrounding information, the effectiveness of the current pixel y can be accurately evaluated, thus extracting the current augmentation point. For a given target point y, the maximum pixel value T passing through the point with the maximum gradient in that target region. max The difference between the target pixel value and the current target pixel value y is used to obtain the second pixel value difference of the pixel to be expanded. For ease of subsequent calculation, the absolute value of the two is used as the second pixel value difference. In practical applications, the absolute value of the difference between the first gradient magnitude of the pixel to be expanded at the maximum gradient point in its target region and the second gradient magnitude of the reference target pixel can also be used as the second pixel value difference, which can characterize the difference between the two.

[0048] S13-3: Based on the difference in the second pixel value of each pixel to be expanded and the second average pixel value of its neighboring region, obtain the degree of information change of the corresponding pixel to be expanded compared to its neighboring pixels during expansion. The neighboring region of each pixel to be expanded can be configured based on actual needs. For example, a 3x3 window centered on the reference target pixel y can be used to extract the average pixel value of 9 pixels existing in the 3x3 window of the target point y in the original image frame, which is recorded as the second average pixel value. The ratio of the difference in the second pixel value to the second average pixel value is used as the degree of change in the surrounding information, i.e., the degree of information change, thereby characterizing whether the information at the current target pixel y position helps to maintain the integrity of details after super-resolution.

[0049] S13-4: Based on the information dependency and information variability of each pixel to be expanded, obtain the reference weights between the corresponding pixel to be expanded and its neighboring pixels. Calculate the neighboring pixels of the target point y. The reference weight is denoted as According to the formula: Calculate reference weights The coefficient 0.001 is used to prevent calculation errors due to a denominator of 0. Based on this, a reference weight is calculated for all pixels to be expanded within a 3x3 range of their reference target pixels. It should be noted that 3x3 is only an example; the neighborhood range is not limited in size, and the larger the range, the more target points are extracted.

[0050] At this point, the reference weights of each pixel to be expanded and its neighboring pixels have been calculated based on the above method, and we proceed to step S14.

[0051] S14: Fill the corresponding pixels to be expanded according to the reference weight and the pixel value of the neighboring pixels. When the reconstruction resolution of each image frame meets the target resolution, combine all image frames in chronological order and output the super-resolution thermal imaging video.

[0052] Specifically, based on the calculated reference weights and the pixel values ​​of neighboring pixels, the pixels to be expanded are weighted and filled to gradually improve the image resolution of the image frames. Once the reconstructed resolution of each image frame reaches the preset target resolution, all image frames are combined according to the original time sequence to generate the corresponding super-resolution thermal imaging video output. Through the above processing, effective restoration of infrared image detail information can be achieved while ensuring computational efficiency.

[0053] For example, step S14 includes sub-steps S14-1 to S14-2, which are described in detail below: S14-1: Normalize the calculation of all reference weights for the neighboring pixels of the pixel to be expanded, and obtain the expansion weight for each reference pixel in the neighborhood of the pixel to be expanded. This applies to a single pixel to be expanded. This requires calculating the pixel value for the current round. For a single pixel to be expanded... In the current round, the final pixel value is obtained by weighting its K nearest known high-confidence source pixels. Please refer to [link to relevant documentation]. Figure 5 The center pixel in the image is the expanded pixel to be filled. The remaining pixels are known reference pixels. The neighborhood of the pixel to be expanded can be configured based on actual needs; 16 pixels are preset for high-precision reconstruction, and 8 pixels are preset for low-precision reconstruction. During local reconstruction, the grayscale transition of the newly added pixels is ensured to be natural and smooth, reducing the block effect caused by too few reference points; by adaptively fusing information from multiple source points locally, the noise suppression of the reconstructed area is enhanced; high-frequency details and edge directions are represented according to the local structure (reflected by weights), so that the texture generated in the expanded area maintains coherence and structural consistency with the surrounding original image content, thereby improving the visual realism of the local reconstruction.

[0054] For the current target point y, there are pixels to be expanded. The reference weight k is normalized. , The normalization function ensures that the normalized data falls within a preset range, such as 0-1. Extract all target points y around the current expanded pixel, and denot the normalized sum of the reference weights of all target points relative to the current expanded point as... This ensures that the expansion points can accurately connect to the expansion information of expansion points at different locations, and further utilizes the pixel values ​​of each target point. The weighted values ​​are used to obtain the allocated pixel values ​​of the expansion points, and finally, the values ​​are obtained by traversing the range. For each target pixel, the pixel value of the expanded point is assigned.

[0055] S14-2: Based on the expansion weight and pixel value of each reference pixel in the neighborhood of the pixel to be expanded, obtain the pixel assignment for the pixel to be expanded, and fill the corresponding pixel to be expanded based on the pixel assignment. (The last part, "the pixel to be expanded...", appears to be incomplete and requires further context.) Pixel values ​​are assigned based on the formula: Calculate the number of pixels to be expanded Pixel assignment Still with Figure 5 As shown in the example, the pixel values ​​of 8 target pixels are known in the image, with reference to the weighted normalized sum value. The weight distribution of each target pixel is 0.4, 0.1, 0.3, 0.05, 0.05, 0.05, 0.02, and 0.03, and the pixel values ​​of each target pixel are 5, 1, 2, 2, 2, 1, 0.5, and 0.5. 5×0.4+1×0.1+2×0.3+2×0.05+2×0.05+1×0.05+0.5×0.02+0.5×0.03, finally yielding the expanded pixel points. Pixel assignment The value is 2.975. It should be noted that in practical applications, the number of pixels to be expanded... If the pixel values ​​of some surrounding pixels are known, then pixel value assignment calculations are performed only based on the pixels with known pixel values.

[0056] When filling in the pixels to be expanded, the calculated pixel values ​​need to be filled into the corresponding positions. Then, according to the target resolution requirements, the calculation is continuously iterated to complete the super-resolution processing of the image. Finally, the super-resolution image is output in the form of consecutive frames. All expanded pixels in the image frame are repeatedly assigned pixel values ​​until each expanded pixel has obtained an initial assigned pixel value. The calculated expanded pixel values ​​are then filled into the corresponding pixel units according to their coordinate positions in the high-resolution image grid. This process is repeated for all pixel unit positions to complete the filling of the target value.

[0057] Repeat the above steps for multiple iterations. Each iteration adjusts the resolution target to gradually optimize pixel values ​​and enhance image details. During iteration, the resolution is gradually increased until the preset target resolution is met, at which point the super-resolution processing stops. After all image frames have been processed, the results are sorted chronologically and output as a continuous super-resolution image sequence or video stream. The output frame rate is set during output, and a temporal smoothing algorithm is further applied to reduce inter-frame jitter, ultimately achieving high-quality thermal imaging super-resolution video output.

[0058] Based on the same technical concept as the super-resolution method, this embodiment of the invention also provides a thermal imaging super-resolution device based on image reconstruction. This device corresponds to any of the aforementioned thermal imaging super-resolution methods based on image reconstruction. Please refer to [link to relevant documentation]. Figure 6 , Figure 6 This is a schematic diagram of a thermal imaging super-resolution device based on image reconstruction. The thermal imaging super-resolution device based on image reconstruction includes an extraction module 61, a processing module 62, a determination calculation module 63, and a filling output module 64.

[0059] The extraction module 61 is used to extract image frames of infrared images captured by the mobile shooting terminal, and obtain the effective frame coefficient of the corresponding image frame based on the temperature gradient distribution and foreground region distribution of each image frame.

[0060] The processing module 62 is used to obtain the importance of thermal information of each foreground region mapped to the convolutional feature map based on the effective frame coefficient of each image frame and the thermal energy fluctuation characteristics of each convolutional feature map after convolution processing.

[0061] The calculation module 63 is used to determine the foreground region where the importance of thermal information is greater than the importance threshold as the target region, and calculates the reference weight between the pixel to be expanded and the neighboring pixels based on the positional relationship and temperature gradient between each pixel in each target region.

[0062] The fill output module 64 is used to fill the corresponding pixels to be expanded according to the reference weight and the pixel value of the neighboring pixels. When the reconstruction resolution of each image frame meets the target resolution, all image frames are combined in chronological order and the super-resolution thermal imaging video is output.

[0063] Based on the same technical concept as the super-resolution method, this embodiment of the invention also provides a thermal imaging super-resolution system based on image reconstruction. The system includes a processor and a memory connected to the processor. The memory is used to store a computer program, and the processor is used to execute the computer program to implement any of the above-described thermal imaging super-resolution methods based on image reconstruction.

[0064] It should be noted that the order of the above embodiments of the present invention is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0065] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

Claims

1. A thermal imaging super-resolution method based on image reconstruction, characterized in that, The method includes: Extract image frames from infrared images captured by a mobile imaging terminal, and obtain the effective frame coefficient of each image frame based on the temperature gradient distribution and foreground region distribution of each image frame; Based on the effective frame coefficient of each image frame and the thermal energy fluctuation characteristics of each convolutional feature map after convolution processing, the importance of thermal information of each foreground region mapped to the convolutional feature map is obtained. The foreground region with thermal information importance greater than the importance threshold is determined as the target region, and the reference weight between the pixel to be expanded and the neighboring pixel is calculated based on the positional relationship and temperature gradient between each pixel in each target region. The corresponding pixels to be expanded are filled according to the reference weight and the pixel value of the neighboring pixels. When the reconstruction resolution of each image frame meets the target resolution, all image frames are combined in chronological order and a super-resolution thermal imaging video is output. The process of obtaining the frame effectiveness coefficient of a corresponding image frame based on the temperature gradient distribution and foreground region distribution of each image frame includes: counting the number of pixels for each temperature gradient in the current image frame and calculating the gradient feature coefficient representing the temperature gradient distribution characteristics of the current image frame based on the statistical results; performing foreground segmentation on the current image frame to obtain the target area of ​​the largest connected component and the total number of foreground regions in all foreground regions; obtaining the foreground feature coefficient of the current image frame based on the target area and the total number of foreground regions; and obtaining the frame effectiveness coefficient of the current image frame based on the gradient feature coefficient and the foreground feature coefficient. The process involves obtaining the thermal information importance of each foreground region mapped to the convolutional feature map based on the effective frame coefficient of each image frame and the thermal fluctuation characteristics of each convolutional feature map after convolution processing. This includes: when the effective frame coefficient of the current image frame is greater than a preset effective threshold, inputting the current image frame into a super-resolution convolutional neural network model to obtain each convolutional feature map of the current image frame; statistically analyzing the pixel values ​​of the corresponding foreground regions in each convolutional feature map of the current image frame and calculating the feature accumulation index representing the thermal source image details of each foreground region in the current image frame; and obtaining the thermal information importance of each foreground region mapped to the convolutional feature map of the current image frame based on the effective frame coefficient of the current image frame and the feature accumulation index. Specifically, based on the positional relationship and temperature gradient between pixels within each target region, the reference weights of the pixel to be expanded and its neighboring pixels are calculated. This includes: evaluating the information correlation based on the relative positional relationship of each pixel to be expanded within its target region to obtain the information dependence of the pixel to be expanded on its neighboring pixels; obtaining the second pixel value difference of the corresponding pixel to be expanded based on the difference between the maximum pixel value of each pixel to be expanded in its target region and the target pixel value of the referenced target pixel; obtaining the information change degree of the corresponding pixel to be expanded compared to its neighboring pixels when expanded based on the second pixel value difference of each pixel to be expanded and the second average pixel value of its neighboring region; and obtaining the reference weights of the corresponding pixel to be expanded and its neighboring pixels based on the information dependence and information change degree of each pixel to be expanded. Specifically, the information correlation degree is evaluated based on the relative positional relationship of each pixel to be expanded in its target region to obtain the information dependence of the pixel to be expanded on its neighboring pixels. This includes: obtaining the distance correlation degree of the pixel to be expanded in its target region based on the current coordinates of the pixel to be expanded in the target region and the center coordinates of the region; and obtaining the information dependence degree of the pixel to be expanded on its neighboring pixels based on the distance correlation degree of the pixel to be expanded and the minimum edge distance, wherein the minimum edge distance is the minimum distance from the center coordinates and the current coordinates to the edge of the target region.

2. The thermal imaging super-resolution method based on image reconstruction according to claim 1, characterized in that, The number of pixels for each temperature gradient in the current image frame is counted, and the gradient feature coefficients representing the temperature gradient distribution characteristics of the current image frame are calculated based on the statistical results, including: Based on the statistical results of the number of pixels at each temperature gradient in the current image frame, the number of target pixels contained in the mode of gradient magnitude is obtained. The mode of gradient magnitude is the temperature gradient that contains the most pixels among all temperature gradient magnitudes. The gradient feature coefficients of the current image frame are obtained based on the ratio of the number of target pixels to the total number of pixels in the current image frame.

3. The thermal imaging super-resolution method based on image reconstruction according to claim 1, characterized in that, The pixel values ​​of the corresponding foreground regions in each convolutional feature map of the current image frame are statistically analyzed, and the cumulative feature index representing the heat source image details of each foreground region in the current image frame is calculated, including: Based on the statistical results of pixel values ​​of each foreground region in each convolutional feature map of the current image frame, the individual pixel value, the first average pixel value, and the pixel standard deviation of each foreground region are obtained. The fluctuation intensity of each pixel is obtained based on the first pixel value difference and pixel standard deviation of each pixel in the foreground region. The first pixel value difference is the difference between the single pixel value of the pixel and the first average pixel value. The cumulative feature index of the corresponding foreground region is obtained by summing the cumulative fluctuation intensity of all pixels in each foreground region.

4. The thermal imaging super-resolution method based on image reconstruction according to claim 1, characterized in that, Filling the corresponding pixels to be expanded according to the reference weight and the pixel values ​​of the neighboring pixels includes: Normalize the calculation of all reference weights of the neighboring pixels of the pixel to be expanded to obtain the expansion weight of each reference pixel in the neighborhood of the pixel to be expanded. Based on the expansion weight and pixel value of each reference pixel in the neighborhood to which the pixel to be expanded belongs, the pixel assignment of the pixel to be expanded is obtained, and the pixel to be expanded is filled accordingly based on the pixel assignment.

5. A thermal imaging super-resolution device based on image reconstruction, characterized in that, The device is the device corresponding to any one of the super-resolution methods described in claims 1-4, and the device includes: The extraction module is used to extract image frames of infrared images captured by the mobile shooting terminal, and obtain the effective frame coefficient of the corresponding image frame based on the temperature gradient distribution and foreground region distribution of each image frame. The processing module is used to obtain the importance of thermal information of each foreground region mapped to the convolutional feature map based on the effective frame coefficient of each image frame and the thermal energy fluctuation characteristics of each convolutional feature map after convolution processing. The calculation module is used to determine the foreground region where the importance of the thermal information is greater than the importance threshold as the target region, and to calculate the reference weight between the pixel to be expanded and the neighboring pixels based on the positional relationship and temperature gradient between each pixel in each target region. The filling output module is used to fill the corresponding pixels to be expanded according to the reference weight and the pixel value of the neighboring pixels. When the reconstruction resolution of each image frame meets the target resolution, all image frames are combined in chronological order and the super-resolution thermal imaging video is output.

6. A thermal imaging super-resolution system based on image reconstruction, characterized in that, The system includes: A processor and a memory connected to the processor; The memory is used to store a computer program, and the processor is used to execute the computer program to implement the super-resolution method according to any one of claims 1-4.

Images