Infrared image super-resolution reconstruction method, device, equipment and medium

Abstract

This application discloses an infrared image super-resolution reconstruction method, apparatus, device, and medium, relating to the field of image processing technology. It improves reconstruction stability and image quality by actively controlling half-pixel-level subpixel displacement to acquire multiple frames of infrared images and combining subpixel registration, a forward degradation model, and a dual-constraint objective function for iterative reconstruction. The method includes: controlling the relative position between the infrared imaging system and the target scene to generate half-pixel-level subpixel displacement, obtaining multiple frames of original infrared images; performing subpixel registration on the multiple frames of original infrared images to obtain the subpixel displacement amount of each frame of original infrared images relative to a reference frame of infrared images; establishing a forward degradation model based on the subpixel displacement amount, using the forward degradation model to characterize the mapping relationship between each frame of original infrared images and the high-resolution image to be reconstructed; constructing an objective function based on the forward degradation model; and solving for the high-resolution image using an iterative optimization algorithm based on the objective function.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method, apparatus, device and medium for infrared image super-resolution reconstruction. Background Technology

[0002] Infrared imaging systems are widely used in space remote sensing, target detection, security monitoring, industrial inspection, gas imaging, night vision sensing, and thermal target identification. Compared to visible light imaging, infrared imaging can acquire thermal radiation information of targets under low light, nighttime, smoke, or complex background conditions, and is therefore widely used in these engineering fields. However, the spatial resolution of infrared imaging systems is typically limited by factors such as detector pixel size, focal length, optical aperture, system modulation transfer function, signal-to-noise ratio, non-uniform noise, and manufacturing cost. For mid-wave and long-wave infrared detectors, the detector pixel size is usually significantly larger than that of visible light complementary metal-oxide-semiconductor (CMOS) or charge-coupled devices (CCDs), resulting in a relatively low inherent resolution of infrared images. Once the system focal length and detector pixel size are determined, the spatial sampling interval of a single frame image is also determined, making it impossible to obtain higher resolution information from an infrared image using a single frame. Directly using smaller pixels or larger array detectors to improve resolution would significantly increase detector cost, cooling burden, readout circuit complexity, and system size. Increasing the focal length to improve spatial resolution would reduce the field of view, which is not conducive to large-area infrared observation.

[0003] Against this backdrop, infrared image super-resolution reconstruction technology has emerged. This technology does not rely on hardware upgrades but instead uses software algorithms to reconstruct images with higher spatial resolution by utilizing sub-pixel displacement differences between multiple frames of the same scene, without changing the physical pixel size of the detector. However, while existing infrared multi-frame super-resolution methods employ multi-frame image reconstruction strategies, random sub-pixel displacements cannot guarantee complete coverage of the half-pixel sampling phase. This results in the reconstruction algorithm failing to obtain sufficient spatial frequency information, leading to instability and quality degradation in the reconstruction results. Consequently, infrared image super-resolution reconstruction technology struggles to meet the practical requirements of high-precision imaging. Summary of the Invention

[0004] In view of this, this application provides an infrared image super-resolution reconstruction method, apparatus, device and medium. The main purpose is to solve the technical problem that existing random subpixel displacements cannot guarantee the complete coverage of half-pixel sampling phase, which leads to the reconstruction algorithm being unable to obtain sufficient spatial frequency information, thereby causing instability and quality degradation of the reconstruction results.

[0005] In a first aspect, this application provides a method for super-resolution reconstruction of infrared images, the method comprising: By controlling the relative position between the infrared imaging system and the target scene to generate a half-pixel-level subpixel displacement, the same target scene is imaged under multiple different sampling phases, resulting in multiple frames of original infrared images. Subpixel registration is performed on the multiple frames of original infrared images to obtain the subpixel displacement of each frame of original infrared image relative to the reference frame of infrared image. A forward degradation model is established based on the subpixel displacement, and the forward degradation model is used to characterize the mapping relationship between each frame of the original infrared image and the high-resolution image to be reconstructed. An objective function is constructed based on the forward degradation model, and the objective function includes a data fidelity term and a regularization term that constrain the image reconstruction process. The objective function is solved by an iterative optimization algorithm to obtain a high-resolution image.

[0006] Secondly, this application provides an infrared image super-resolution reconstruction apparatus, the apparatus comprising: The control unit is used to control the relative position between the infrared imaging system and the target scene to produce a half-pixel-level subpixel displacement, so that the same target scene is imaged under multiple different sampling phases to obtain multiple frames of original infrared images. The registration unit is used to perform subpixel registration on the multiple frames of original infrared images to obtain the subpixel displacement of each frame of original infrared image relative to the reference frame infrared image. The establishment unit is used to establish a forward degradation model based on the sub-pixel displacement, and the forward degradation model is used to characterize the mapping relationship between each frame of original infrared image and the high-resolution image to be reconstructed; A construction unit is used to construct an objective function based on the forward degradation model, the objective function including a data fidelity term and a regularization term that constrain the image reconstruction process; The solving unit is used to solve the objective function through an iterative optimization algorithm to obtain a high-resolution image.

[0007] Thirdly, this application provides an infrared image super-resolution reconstruction device, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor. When the processor executes the program, it implements the above-described infrared image super-resolution reconstruction method.

[0008] Fourthly, this application provides a storage medium storing a computer program thereon, which, when executed by a processor, implements the above-described infrared image super-resolution reconstruction method.

[0009] By employing the above technical solution, this application provides an infrared image super-resolution reconstruction method, apparatus, device, and medium. Compared with existing technologies that utilize sub-pixel displacement differences between multiple frames of the same scene for infrared image super-resolution reconstruction, this application generates half-pixel-level sub-pixel displacement by controlling the relative position between the infrared imaging system and the target scene, enabling the same target scene to be imaged at multiple different sampling phases, obtaining multiple frames of original infrared images. Sub-pixel registration is performed on these multiple frames of original infrared images to obtain the sub-pixel displacement of each frame relative to a reference frame. A forward degradation model is established based on the sub-pixel displacement, characterizing the mapping relationship between each frame of original infrared images and the high-resolution image to be reconstructed. An objective function is constructed based on the forward degradation model, including a data fidelity term and a regularization term constraining the image reconstruction process. Based on the objective function, an iterative optimization algorithm is used to solve for the high-resolution image. The entire process actively controls the half-pixel-level sub-pixel displacement to acquire multiple frames of infrared images, effectively covering different sampling phases and obtaining sufficient spatial frequency information. By combining subpixel registration, forward degradation model, and objective function containing data fidelity and regularization terms for iterative reconstruction, the instability caused by displacement randomness in traditional methods is effectively overcome. This fully explores high-frequency detail information in multiple frames of images, significantly improves the stability and reliability of the reconstruction process, optimizes the imaging quality of infrared images, and meets the practical application requirements of high-precision infrared imaging.

[0010] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0011] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a schematic flowchart of an infrared image super-resolution reconstruction method in one embodiment of this application; Figure 2 This is a schematic diagram of the process of achieving subpixel displacement and grid mapping with a 2x resolution improvement using four-frame mutual compensation sampling in one embodiment of this application; Figure 3 yes Figure 1 A flowchart illustrating a specific implementation method for step 102; Figure 4 yes Figure 1 A flowchart illustrating a specific implementation method for step 103; Figure 5 This is a schematic diagram of an iterative optimization process based on an objective function in one embodiment of this application; Figure 6 This is a flowchart illustrating an infrared image super-resolution reconstruction method in another embodiment of this application; Figure 7 This is a flowchart illustrating an infrared image super-resolution reconstruction method in another embodiment of this application; Figure 8 This is a flowchart illustrating the process of infrared image super-resolution reconstruction quality evaluation and parameter optimization in one embodiment of this application; Figure 9 This is a schematic diagram of the structure of an infrared image super-resolution reconstruction system according to an embodiment of this application; Figure 10 This is a schematic diagram of the structure of an infrared image super-resolution reconstruction device in one embodiment of this application; Figure 11 This is a schematic diagram of the device structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0012] The present application will be described in detail below with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in the embodiments of the present application can be combined with each other.

[0013] In related technologies, infrared image super-resolution reconstruction does not rely on hardware upgrades but rather on software algorithms. It reconstructs images with higher spatial resolution by utilizing sub-pixel displacement differences between multiple frames of the same scene without changing the physical pixel size of the detector. However, while existing infrared multi-frame super-resolution methods employ multi-frame image reconstruction strategies, random sub-pixel displacements cannot guarantee complete coverage of the half-pixel sampling phase. This results in the reconstruction algorithm failing to obtain sufficient spatial frequency information, leading to instability and quality degradation in the reconstruction results. Consequently, infrared image super-resolution reconstruction technology struggles to meet the practical requirements of high-precision imaging.

[0014] To address this problem, this embodiment provides a method for super-resolution reconstruction of infrared images, such as... Figure 1 As shown, the method includes the following steps: 101. Control the relative position between the infrared imaging system and the target scene to generate a half-pixel level subpixel displacement, so that the same target scene is imaged under multiple different sampling phases, and multiple frames of original infrared images are obtained.

[0015] In this embodiment, by controlling the relative position between the infrared imaging system and the target scene, half-pixel-level subpixel displacement is generated, ensuring that each displacement is half a pixel. This allows for precise sampling phase differences between multiple frames, resulting in the acquisition of multiple original infrared images. This method effectively overcomes the limitation of single-frame infrared image resolution, improving the spatial resolution and detail representation of the final reconstructed image.

[0016] Specifically, half-pixel-level subpixel displacement can be achieved through several methods. The first method is to fix the target scene on a high-precision micro-displacement platform, keeping the infrared imaging system stationary, and then precisely move the platform in the horizontal and vertical directions to cause a half-pixel displacement on the detector. The second method is to use a micro-displacement platform to directly support the infrared detector, controlling the detector to move relative to the image plane with nanometer-level precision, thereby generating the required subpixel displacement. The third method is to install a fast-reflecting mirror or a piezoelectric deflector in the optical path, and drive the mirror to produce a tiny angle deflection by sending a voltage signal through the control system, changing the incident angle of the beam and thus causing a half-pixel displacement on the image plane. The fourth method is to set a scanning mirror or other beam deflection mechanism at the front end of the optical path, and change the field of view position by precisely controlling the deflection angle of the beam. Each deflection strictly corresponds to a half-pixel displacement, and ultimately, multiple high-quality original images for super-resolution reconstruction can be acquired.

[0017] 102. Perform subpixel registration on the multiple frames of original infrared images to obtain the subpixel displacement of each frame of original infrared image relative to the reference frame of infrared image.

[0018] As one implementation method, subpixel registration can employ a registration method using calibration target images. This method uses a high-precision calibration target as the test target, which has a known geometric structure and precise dimensional parameters. The calibration target is placed at the center of the infrared imaging system's field of view, and multiple frames of calibration target images are acquired simultaneously through half-pixel-level subpixel displacement. Utilizing the known geometric features of the calibration target, image processing algorithms are used to accurately calculate the positional change of the calibration target in each frame of the original infrared image, thereby obtaining the precise subpixel displacement.

[0019] As another implementation method, subpixel registration can employ a feature-point-based registration approach. This method first selects one frame from multiple original infrared images as a reference frame, typically choosing the first frame or the frame with the best quality as the reference baseline. Then, a scale-invariant feature transform algorithm or an accelerated robust feature extraction algorithm is used to extract feature points from the reference frame and the remaining frames. A feature point matching algorithm is used to establish the correspondence between feature points in the reference frame and each frame, initially obtaining an integer-pixel level displacement estimate. Based on this, subpixel interpolation techniques are used to precisely locate the feature points at the subpixel level, thereby obtaining the precise subpixel level displacement.

[0020] As another implementation method, subpixel registration can be based on phase correlation registration. This method utilizes the translation property of Fourier transform to convert multiple frames of original infrared images into the frequency domain for analysis. By calculating the cross-power spectrum between the reference frame and each frame, the translation information between the images can be obtained. The phase correlation method has high registration accuracy and robustness, and is particularly suitable for image types with relatively little texture, such as infrared images. After obtaining the displacement estimate at the integer pixel level, subpixel interpolation techniques such as parabolic fitting or Gaussian fitting are further used to accurately locate the phase correlation peak at the subpixel level, thereby obtaining the precise subpixel displacement.

[0021] Understandably, the accuracy of subpixel registration directly impacts the quality of subsequent super-resolution reconstruction. In practical applications, subpixel registration accuracy is typically required to reach one-tenth of a pixel or even higher. To improve registration accuracy, a multi-scale registration strategy can be employed, starting with coarse registration at a low resolution scale and then gradually increasing the resolution for fine registration. Furthermore, a hybrid registration strategy can be used, combining the advantages of multiple registration methods to further enhance the accuracy and robustness of subpixel registration. Precise subpixel registration ensures that subsequent super-resolution reconstruction algorithms can fully utilize complementary information from multiple frames, achieving high-quality resolution enhancement.

[0022] 103. Establish a forward degradation model based on the subpixel displacement, and characterize the mapping relationship between the original infrared image and the high-resolution image to be reconstructed in each frame through the forward degradation model.

[0023] In this embodiment, the forward degradation model is the core mathematical model in the infrared image super-resolution reconstruction algorithm. This model is used to accurately describe the complete process by which a high-resolution image is transformed into a low-resolution image after undergoing a series of physical degradation processes. In actual infrared imaging systems, the high-resolution image to be reconstructed first undergoes multiple degradation stages such as optical blurring, geometric transformation, downsampling, and noise contamination, ultimately forming the original infrared images captured by the system. By establishing an accurate forward degradation model, the intrinsic mapping relationship between each frame of the original infrared image and the high-resolution image to be reconstructed can be accurately characterized.

[0024] When establishing a forward degradation model based on subpixel displacement, the subpixel displacement of each original infrared image frame relative to the reference frame reflects the actual spatial position change of that frame relative to the high-resolution scene during acquisition. These displacements are incorporated into the forward degradation model in the form of a coordinate transformation matrix to describe the geometric mapping relationship from the high-resolution image to each low-resolution image frame. Correspondingly, the subpixel displacements are converted into translation components of the affine transformation matrix through the forward degradation model, accurately characterizing the spatial correspondence between images.

[0025] Meanwhile, establishing a forward degradation model also requires considering the point spread function of the infrared imaging system, which describes the blurring characteristics of the optical system. The point spread function is typically characterized by a Gaussian function or a more complex physical model, and its parameters can be obtained through system calibration or theoretical calculations. In the forward degradation model, the point spread function acts on the high-resolution image in the form of a convolution operation, simulating the blurring effect of the optical system on the image. By combining the point spread function with sub-pixel displacement, the forward degradation model can comprehensively describe the intermediate state of a high-resolution image after optical blurring and geometric transformation.

[0026] Downsampling, another crucial component of the forward degradation model, simulates the physical process by which a high-resolution image is sampled by a detector to form a low-resolution image. In the forward degradation model, downsampling is typically implemented using methods such as average pooling or nearest-neighbor interpolation, with the sampling interval corresponding to the detector's pixel size. By combining downsampling with the preceding blurring and geometric transformations, the forward degradation model can comprehensively describe the entire degradation chain from high-resolution to low-resolution images.

[0027] Through the organic combination of the above steps, the forward degradation model ultimately forms a complete mathematical expression that accurately describes the mapping relationship between the high-resolution image to be reconstructed and each frame of the original infrared image. In practical applications, the forward degradation model can be represented as a matrix equation, where the high-resolution image is the unknown variable, each frame of the original infrared image is the observation data, and the subpixel displacement, point spread function, downsampling matrix, and noise parameters are the known parameters. By solving this equation, accurate reconstruction from multiple frames of low-resolution images to a high-resolution image can be achieved.

[0028] 104. Construct an objective function based on the aforementioned forward degradation model.

[0029] In this embodiment, the objective function integrates the physical constraints of the forward degradation model and prior image knowledge, providing a mathematical optimization framework for high-resolution image reconstruction. Specifically, when constructing the objective function, the characteristics of infrared images and practical application requirements need to be fully considered to ensure that the reconstruction results not only conform to the physical imaging laws but also possess good visual quality and practical value. The objective function typically consists of two parts: a data fidelity term and a regularization term. These two parts are balanced by appropriate weighting coefficients, jointly guiding the optimization direction of the reconstruction process.

[0030] The data fidelity term is a fundamental component of the objective function. Its main function is to ensure that the reconstructed high-resolution image, after undergoing the forward degradation model, maintains a high degree of consistency with the actual acquired multiple frames of original infrared images. The data fidelity term is constructed directly based on the forward degradation model, measuring the fidelity by calculating the difference between the reconstructed image after the degradation process and the observed image. In practice, the data fidelity term typically employs the least squares criterion, which calculates the sum of the squared errors between each frame of the original infrared image and the corresponding degraded reconstructed image.

[0031] The regularization term is another important component of the objective function. Its main function is to introduce prior knowledge about the image and address the ill-conditioned nature of the super-resolution reconstruction problem. Since reconstructing a high-resolution image from a low-resolution image is an underdetermined problem, relying solely on the data fidelity term cannot yield a unique and reasonable solution; therefore, additional constraints must be introduced through the regularization term. The design of the regularization term needs to fully consider the characteristics of infrared images, such as edge sparsity, the continuity of smooth regions, and the regularity of texture structures. By designing a reasonable regularization term, noise amplification and artifact generation during the reconstruction process can be effectively suppressed, improving the quality and practicality of the reconstructed image.

[0032] Correspondingly, the final form of the objective function is a weighted combination of data fidelity terms and regularization terms, and the choice of weight coefficients has a significant impact on the reconstruction results. The weight coefficients need to be adjusted according to the actual application scenario and image characteristics. In practical applications, an adaptive weighting strategy can be adopted to dynamically adjust the regularization intensity based on the local features of the image, enabling the reconstruction process to better adapt to the complex structure of the image. By constructing a reasonable objective function, the infrared super-resolution reconstruction algorithm can effectively suppress noise and artifacts while preserving image details, obtaining high-quality reconstruction results.

[0033] 105. Solve the objective function using an iterative optimization algorithm to obtain a high-resolution image.

[0034] In practical applications, since the objective function is usually a complex, nonlinear function, an analytical solution cannot be obtained directly. Therefore, iterative optimization algorithms must be used for numerical solutions. The iterative optimization process starts with an initial high-resolution image estimate and continuously updates the image estimate by repeatedly calculating the gradient or search direction of the objective function until the convergence condition is met.

[0035] When solving for the objective function, gradient descent, conjugate gradient, ADMM algorithm, primal-dual algorithm, or other iterative optimization algorithms can be used. Taking gradient descent as an example, this algorithm first calculates the gradient of the objective function with respect to the high-resolution image, and then updates the image estimate along the negative gradient direction. In each iteration, the algorithm calculates the gradients of the data fidelity term and the regularization term based on the current image estimate, weights them together to obtain the total gradient, and then updates the image according to the preset step size parameter.

[0036] The infrared image super-resolution reconstruction method provided in this application, compared with the existing technology that utilizes the sub-pixel displacement difference between multiple frames of the same scene to achieve infrared image super-resolution reconstruction, generates half-pixel-level sub-pixel displacement by controlling the relative position between the infrared imaging system and the target scene, enabling the same target scene to be imaged at multiple different sampling phases, obtaining multiple frames of original infrared images; sub-pixel registration is performed on the multiple frames of original infrared images to obtain the sub-pixel displacement amount of each frame of original infrared images relative to the reference frame of infrared images; a forward degradation model is established based on the sub-pixel displacement amount, which characterizes the mapping relationship between each frame of original infrared images and the high-resolution image to be reconstructed; an objective function is constructed based on the forward degradation model, which includes a data fidelity term and a regularization term constraining the image reconstruction process; and a high-resolution image is obtained by solving the objective function through an iterative optimization algorithm. The entire process actively controls the half-pixel-level sub-pixel displacement to acquire multiple frames of infrared images, effectively covering different sampling phases and obtaining sufficient spatial frequency information. By combining subpixel registration, forward degradation model, and objective function containing data fidelity and regularization terms for iterative reconstruction, the instability caused by displacement randomness in traditional methods is effectively overcome. This fully explores high-frequency detail information in multiple frames of images, significantly improves the stability and reliability of the reconstruction process, optimizes the imaging quality of infrared images, and meets the practical application requirements of high-precision infrared imaging.

[0037] In practical applications, by introducing half-pixel-level subpixel displacement, the system can oversample the scene through minute spatial changes across multiple frames without altering the detector hardware. This allows for the subsequent reconstruction algorithm to recover high-frequency detail information exceeding the resolution of a single frame. Specifically, half-pixel-level subpixel displacement can be achieved through at least one of the following methods: In one implementation method, the target is moved so that the image of the target formed by the optical system produces a half-pixel subpixel displacement on the focal plane of the infrared detector. This process involves fixing the target scene on a micro-displacement platform while keeping the infrared imaging system stationary. During image acquisition, the control system first positions the target scene at an initial reference position, and the infrared imaging system performs the first imaging of the target, obtaining the first frame of raw infrared image. Subsequently, the micro-displacement platform drives the target scene to move precisely in the horizontal direction, causing the target image to shift by half a pixel on the detector. After the displacement is complete, the infrared imaging system performs a second imaging of the same target scene from the new position. This process is repeated, controlling the micro-displacement platform to produce precise half-pixel displacements in the horizontal and vertical directions according to a preset sequence, ultimately obtaining multiple frames of raw infrared images covering multiple different sampling phases.

[0038] As another implementation, the infrared detector can be moved to achieve a half-pixel sub-pixel displacement relative to the image plane of the optical system. In this process, the infrared detector is mounted on a micro-displacement platform capable of nanometer-level precision position control in both horizontal and vertical directions. During image acquisition, the micro-displacement platform positions the detector at an initial reference position for the first image formation. Subsequently, according to instructions from the control system, the micro-displacement platform precisely moves the detector horizontally by half a pixel. The detector then images the same target scene again from the new position. By cyclically executing detector displacement and image acquisition operations, half-pixel sub-pixel displacements are generated in multiple directions according to a preset displacement sequence.

[0039] In another implementation, the beam propagation direction can be changed by a beam deflection mechanism, causing a half-pixel sub-pixel displacement on the focal plane of the infrared detector to be produced in the image of the target object formed by the optical system. In this process, a fast-reflecting mirror or piezoelectric deflecting mirror is precisely installed in the optical path of the infrared imaging system as the beam deflection mechanism. This deflection mechanism can achieve micro-radian-level angle control accuracy in both the horizontal and vertical directions. During image acquisition, the control system sends a precise control voltage signal to the fast-reflecting mirror drive circuit according to a preset sub-pixel displacement mode, causing the mirror to deflect slightly, thereby changing the incident angle of the beam. According to the principle of optical imaging, a small change in the beam angle is linearly mapped to a lateral displacement of the image point on the focal plane. By precisely controlling the deflection angle, a half-pixel sub-pixel displacement can be produced in the image of the target on the detector's focal plane.

[0040] For 2x super-resolution reconstruction, the goal is to double the image resolution both horizontally and vertically. This means inserting an extra pixel within each pixel of the existing low-resolution pixel grid, forming a 2×2 high-resolution pixel grid. According to the Nyquist sampling theorem, to reconstruct a high-resolution image, sampling information must be obtained at all possible locations within the low-resolution sampling grid. This sampling pattern ensures that each high-resolution pixel location has sufficient information for accurate interpolation or reconstruction when reconstructing the high-resolution image, thus achieving a 2x resolution increase.

[0041] For details on the process of using four-frame mutual compensation sampling to achieve sub-pixel displacement and mesh mapping with a 2x resolution improvement, please refer to [link to documentation]. Figure 2 As shown, the process begins with the original low-resolution pixel grid as a base. A second phase image is obtained by applying a half-pixel subpixel displacement in the horizontal direction, and a third phase image is obtained by applying a half-pixel subpixel displacement in the vertical direction. Then, a fourth phase image is obtained by applying half-pixel subpixel displacements in both the horizontal and vertical directions, thus forming four sampling positions with different subpixel offsets. Subsequently, these four sets of sampling points are superimposed and mapped onto a high-resolution pixel grid with an equivalent size halved, so that each high-resolution pixel unit contains sampling information from different original frames, ultimately achieving a reconstruction of the original low-resolution image at twice the spatial resolution.

[0042] Combination Figure 2 As shown, let the pixel size of the original infrared image be... The goal is to obtain a high-resolution image with twice the sampling density. Therefore, the equivalent cell size of the high-resolution grid is:

[0043] It should be noted that, Figure 2 The unit is a low-resolution pixel. To ensure that multiple frames of low-resolution images cover different sampling phases of the high-resolution grid, the system preferably acquires four frames: The first frame is the reference frame, and the sampling phase is:

[0044] The second frame is shifted horizontally by half a low-resolution pixel, and the sampling phase is:

[0045] The third frame moves vertically by half a low-resolution pixel, and the sampling phase is:

[0046] The fourth frame moves half a low-resolution pixel simultaneously in both the horizontal and vertical directions, with the sampling phase being:

[0047] The four images mentioned above collectively cover four sub-pixel locations within a grid of images at twice the resolution. Unlike ordinary bilinear or cubic interpolation, this method obtains complementary observation information on the high-resolution grid through actual acquisition. In practical applications, when the micro-displacement platform or beam deflection mechanism is exposed during motion, the position of the target image on the detector continuously changes, causing motion blur and resulting in the loss of high-frequency information. Furthermore, if image acquisition is performed before the displacement has stabilized, the actual displacement may deviate from the preset half-pixel value, leading to inaccurate sub-pixel offset relationships between multiple frames and undermining the theoretical basis of oversampling. Therefore, it is essential to ensure that each frame of the original infrared image strictly corresponds to its preset sub-pixel micro-displacement state, and that camera exposure is only triggered after the displacement has fully stabilized, thereby guaranteeing precise and controllable spatial sampling position for each frame.

[0048] Correspondingly, when generating half-pixel-level subpixel micro-displacement, the timing synchronization between the subpixel micro-displacement and the infrared camera exposure is coordinated so that each frame of the original infrared image corresponds one-to-one with its corresponding subpixel micro-displacement state, and the infrared camera performs image acquisition after the subpixel micro-displacement stabilizes.

[0049] In one embodiment, the control system first sends a half-pixel-level displacement command to the micro-displacement platform, driving the platform to move along a preset trajectory. The high-precision position sensor built into the micro-displacement platform monitors the displacement status in real time and feeds the position data back to the main controller. The main controller continuously compares the actual position with the target position. When the detected displacement reaches the half-pixel target value and the position fluctuation is less than a preset threshold, it determines that the displacement has stabilized. At this point, the main controller immediately sends an exposure trigger signal to the infrared camera, which then initiates exposure acquisition upon receiving the signal. Through this closed-loop control based on position feedback, it ensures that each frame of infrared image is acquired after the displacement has completely stabilized, achieving a precise correspondence between the sub-pixel micro-displacement state and the infrared image frame.

[0050] In another embodiment, the micro-displacement controller is equipped with a dedicated hardware synchronization output port. When the micro-displacement platform completes a half-pixel-level displacement and remains stable, the controller automatically outputs a synchronization pulse signal from this port. This pulse signal is directly transmitted to the external trigger input port of the infrared camera via hardware wiring. The infrared camera is configured in external trigger mode, and once the rising edge of the synchronization pulse is detected, the exposure process is immediately initiated. This hardware-level synchronization method requires no software intervention, has extremely low timing delay and jitter, and can complete the transition from displacement stabilization to image acquisition within microseconds, ensuring precise timing coordination between sub-pixel micro-displacement and infrared camera exposure.

[0051] Due to factors such as mechanical errors, temperature drift, and vibration interference in real-world systems, the actual displacement generated by the micro-displacement platform may deviate from the preset value. Directly using the preset displacement for image reconstruction would lead to inaccurate sampling positions, severely impacting the quality of the super-resolution image. Furthermore, infrared cameras may be affected by external environmental factors during acquisition, resulting in slight jitter or drift, further exacerbating displacement errors. Therefore, precise sub-pixel registration of multiple frames of raw infrared images is essential to obtain the true sub-pixel displacement of each frame relative to the reference frame, providing accurate spatial location information for subsequent high-precision image reconstruction. Sub-pixel registration can effectively compensate for system errors, improving the accuracy and reliability of super-resolution reconstruction. Specifically, for example... Figure 3 As shown, step 102 above includes the following steps: 201. Select one frame from the multiple original infrared images as a reference frame infrared image.

[0052] 202. Based on at least one of the system's preset displacement control command, the pre-acquired calibration target image, or the image registration calculation result, obtain the sub-pixel displacement of each of the remaining original infrared images relative to the reference frame infrared image.

[0053] The image registration calculation results are based on at least one of phase correlation, feature matching, subpixel edge localization, or spot centroid method.

[0054] In one implementation, the system acquires subpixel displacement by combining preset displacement control commands with calibration target images. First, an intermediate frame or the frame with the best quality is selected from multiple original infrared images as a reference frame. Then, based on the displacement control commands executed by the micro-displacement platform, the theoretical displacement of each frame is initially determined, i.e., the subpixel displacement of each of the remaining original infrared frames relative to the reference frame. To further improve accuracy, calibration target images can be pre-acquired during the equipment installation and commissioning phase. By analyzing the imaging characteristics of the calibration target under different displacement states, a displacement error compensation model is established. During actual imaging, the theoretical displacement is input into the error compensation model to obtain the corrected subpixel displacement. This method combines theoretical control and actual calibration, ensuring both the accuracy of the displacement and improving the robustness of the system.

[0055] In another implementation, the system employs a phase correlation method for image registration calculation. After selecting one frame of the original infrared image as the reference frame, Fourier transforms are performed on each of the remaining frames with the reference frame to calculate the phase difference between the two frames in the frequency domain. The phase difference is then converted into a cross-correlation function in the spatial domain using an inverse Fourier transform. The peak position of the cross-correlation function is located, and this peak position corresponds to the sub-pixel displacement between the two frames. The phase correlation method has advantages such as high computational efficiency and strong noise resistance, making it particularly suitable for scenarios with low signal-to-noise ratios, such as infrared images. Using this method, the system can directly extract precise sub-pixel displacement information from the image content without relying on external calibration or control commands, thus improving the system's adaptability.

[0056] In another implementation, the system employs a combination of feature matching and sub-pixel edge localization. First, feature point detection is performed on the reference frame and the frame to be registered, extracting salient features such as corners and edges. Then, a feature descriptor matching algorithm is used to establish the feature correspondence between the two frames. After obtaining a coarse matching result, sub-pixel-level precise localization is performed on the matched edge features. By fitting edge curves or analyzing edge gradient distribution, the edge positions are located to sub-pixel accuracy. Finally, based on the sub-pixel displacements of multiple pairs of matched feature points, the least squares method is used to fit and obtain the global sub-pixel displacement of the entire frame. This method fully utilizes the structural information of the image, has high registration accuracy and reliability, and is particularly suitable for infrared scenes with rich texture features.

[0057] In practical applications, the forward degradation model describes the physical mechanism of reconstructing a high-resolution image into a low-resolution observation image. To avoid problems such as noise amplification, artifacts, or over-smoothing in the reconstruction results, regularization constraints need to be introduced into the objective function to ensure data consistency while applying prior knowledge to guide the reconstruction results. Specifically, such as... Figure 4 As shown, step 103 above includes the following steps: 301. Based on the forward degradation model, calculate the data fidelity terms between the degraded high-resolution image to be reconstructed and the original infrared images of each frame.

[0058] 302. Construct regularization terms to impose smoothness and edge preservation constraints on the reconstruction results.

[0059] 303. Combine the data fidelity term and the regularization term with preset weights to form the objective function.

[0060] In this embodiment, the forward degradation model is used to describe the complete mechanism by which a high-resolution image is transformed into a low-resolution image after undergoing a series of physical processes. Specifically, the forward degradation model can be expressed as the following formula:

[0061] in, For the first Frame infrared image; The high-resolution image to be reconstructed; This is a downsampling operator for converting from a high-resolution grid to a low-resolution grid. A fuzzy operator formed by the combined action of an infrared optical system and a detector; For the first Geometric transformation operator caused by subpixel displacement in a frame of infrared image; Noise items include readout noise, thermal noise, fixed pattern noise, and environmental noise.

[0062] In one implementation, the data fidelity term is calculated by first simulating degradation of the high-resolution image to be reconstructed based on a forward degradation model. Specifically, the image to be reconstructed is spatially transformed according to the sub-pixel displacements corresponding to each frame of the original infrared image. Then, a systematic point spread function is applied for convolutional blurring, followed by downsampling to match the resolution of the original infrared image. Finally, a noise model is added. The pixel differences between the degraded simulated image and the corresponding original infrared image are calculated, typically using squared error or absolute error as the metric. The differences across all frames are summed to obtain the overall data fidelity term. This data fidelity term ensures that the reconstructed result, after degradation, approximates the actual observed multiple frames of original infrared images to the greatest extent possible, reflecting the fidelity of the reconstruction process to the observed data.

[0063] In one implementation, the regularization term employs a strategy combining total variation regularization (TVGF) and edge-preserving smoothing. TVGF encourages piecewise smoothness in the reconstruction by calculating the sum of the absolute values ​​of the gradients in the reconstructed image, effectively suppressing noise and artifacts. Simultaneously, to protect important edge structures from over-smoothing, an edge-preserving weight factor is introduced. This factor dynamically adjusts the smoothing intensity based on the magnitude of local gradients in the image. In edge regions with larger gradients, the weight factor is smaller, reducing smoothing to maintain edge sharpness; in smoothing regions with smaller gradients, the weight factor is larger, enhancing the smoothing effect to suppress noise. Through this adaptive regularization strategy, the reconstruction maintains overall smoothness while effectively preserving image details and edge information.

[0064] Accordingly, the objective function constructed based on the forward degradation model can be expressed by the following formula:

[0065]

[0066] in, The objective function consists of two parts: a data fidelity term and a regularization term. This is a downsampling operator for converting from a high-resolution grid to a low-resolution grid. A fuzzy operator formed by the combined action of an infrared optical system and a detector; For the first Geometric transformation operator caused by subpixel displacement in a frame of infrared image; For the high-resolution image to be reconstructed; For the first Frame infrared image; This constitutes the data fidelity term in the objective function; For total variation regularization weights; The final reconstructed high-resolution image; For all possible image variables Searching for the energy function The specific image that takes the minimum value.

[0067] The regularization term that constitutes the objective function can be expressed by the following formula:

[0068] in, The first in the infrared image line, number The grayscale value of the corresponding pixel in the column; For the first infrared image line, number The grayscale value of the corresponding pixel in the column; For the first infrared image line, number The grayscale value of the corresponding pixel in the column; This is a stable term used to avoid a denominator of zero.

[0069] Accordingly, when solving the objective function, gradient descent, conjugate gradient, ADMM algorithm, primal-dual algorithm, or other iterative optimization algorithms can be used. Taking gradient descent as an example, the process of solving the objective function can be represented by the following formula:

[0070] in, This represents the number of iterations. For the first The high-resolution image output by the next iteration; For the first The high-resolution image output by the next iteration; This is the iteration step size; This is a downsampling operator for converting from a high-resolution grid to a low-resolution grid. A fuzzy operator formed by the combined action of an infrared optical system and a detector; For the first Geometric transformation operator caused by subpixel displacement in a frame of infrared image; For the first The conjugate operator of the degradation process corresponding to a frame of infrared image, i.e., the original degradation operator. transpose; For total variation regularization weights; For regularization terms in The gradient at that point.

[0071] In the implementation process, a high-resolution image is first initialized, followed by an iterative loop. In each iteration, the gradient of the objective function is calculated based on the current estimated image. This gradient consists of two parts: first, the gradient corresponding to the data fidelity term, which is the weighted sum of the degradation residuals of each frame after backpropagation through the transposed degradation operator; second, the sub-gradient corresponding to the regularization term, which is usually obtained through finite difference approximation or smoothing. Then, the gradient is multiplied by the iteration step size and subtracted from the current estimate to obtain a new image estimate. The iteration step size can be preset to a fixed value or dynamically adjusted according to the local curvature of the objective function or the line search strategy to improve the convergence speed and stability. The iteration process continues until the convergence condition is met, for example, the change in the objective function value between two adjacent iterations is less than a threshold, or the preset maximum number of iterations is reached. The final output image is the optimal high-resolution reconstruction result.

[0072] For details regarding the super-resolution reconstruction algorithm process, see the iterative optimization process based on the objective function. Figure 5 As shown, the process begins with an input low-resolution image sequence, actual displacement parameters, a blur kernel, and a downsampling matrix, followed by initialization of the high-resolution image. Next, an objective function containing data fidelity and regularization terms is constructed, and the optimal solution is obtained iteratively. Each iteration sequentially performs forward projection / residual calculation, backpropagation gradient calculation, regularization gradient calculation, and variable updates. After the update, it is determined whether convergence has been achieved or the maximum number of iterations has been reached. If the conditions are met, the final high-resolution reconstructed image is output; otherwise, iteration continues. The entire process embodies a systematic reconstruction process that starts from observed data and gradually approximates the real high-resolution scene through a model-driven optimization strategy.

[0073] To objectively and quantitatively evaluate the quality of reconstructed images, particularly their detail restoration capability and spatial resolution performance, this invention introduces a modulation transfer function (MTF) evaluation system. The MTF is a recognized core indicator in optical imaging and image processing, characterizing a system's ability to transfer details at different spatial frequencies. It intuitively reflects the extent to which the reconstruction result recovers high-frequency information from the original scene, avoiding incomplete evaluation methods that rely solely on subjective vision or simple pixel errors. Furthermore, as... Figure 6As shown, after step 105 above, the method further includes the following steps: 401. The modulation transfer function of the high-resolution image is evaluated using the hypotenuse method to obtain the modulation transfer function curve.

[0074] 402. Based on the modulation transfer function curve, determine the modulation transfer function value at a preset spatial frequency as an evaluation index.

[0075] 403. Use the evaluation index to evaluate the spatial frequency response characteristics of image reconstruction.

[0076] This invention employs the modulation transfer function (MTF) to evaluate whether super-resolution reconstruction truly improves spatial resolution. The MTF curve is obtained by differentiating the edge spread function to obtain the line spread function, and then performing a Fourier transform on the line spread function. Specifically, it can be measured using the hypotenuse method. The hypotenuse method involves the following steps: acquiring an image of a target with an inclined edge using an infrared camera; cropping the region of interest (ROI) in the edge area; resampling the pixel grayscale based on the edge angle to obtain the edge spread function; differentiating the edge spread function to obtain the line spread function; and performing a Fourier transform on the line spread function to obtain the MTF curve.

[0077] The above line spread function can be expressed by the following formula:

[0078] Wherein, the line spread function is equal to the derivative of the edge spread function with respect to the spatial variables; These are the spatial coordinates, i.e., the unknown variables of the edge normal direction; This is the edge expansion function.

[0079] The modulation transfer function curve described above can be represented by the following formula:

[0080] The modulation transfer function is equal to the magnitude of the Fourier transform of the line spread function divided by the magnitude at zero frequency. Spatial frequency; For line expansion functions; The amplitude of the complex response, i.e., the amplitude spectrum, reflects the energy transfer intensity of the system at different spatial frequencies; This is the amplitude at spatial frequency zero, i.e., the DC component, corresponding to total energy or average brightness, used for normalization.

[0081] Specifically, the following modulation transfer function metrics can be used to evaluate super-resolution performance: 1. Modulation transfer function This represents the spatial frequency corresponding to when the modulation transfer function drops to 0.5. The higher the value, the stronger the ability to express medium contrast details in the image.

[0082] 2. Modulation transfer function This represents the spatial frequency corresponding to the modulation transfer function dropping to 0.1. It is often used to describe the resolvable high-frequency boundaries of a system.

[0083] 3. Modulation transfer function at the Nyquist frequency; Nyquist frequency in low-resolution images. It can be expressed by the following formula:

[0084] After doubling the super-resolution, the equivalent pixel size is half the original value, corresponding to the Nyquist frequency of the high-resolution grid. It can be expressed by the following formula:

[0085] If super-resolution reconstruction is effective, the modulation transfer function of the reconstructed image in the high-frequency band should be higher than that of the ordinary interpolated image, especially near the original low-resolution Nyquist frequency and the high-resolution target frequency, and should exhibit a better spatial frequency response.

[0086] 4. Modulation transfer function boost rate, defined at a specified frequency. place The improvement rate can be expressed by the following formula:

[0087] in, To reconstruct images at high frequencies place , For low-resolution or interpolated images at the same frequency .

[0088] This invention improves the effective sampling density of an image through half-pixel and sub-pixel complementary sampling, and also improves the modulation transfer function curve and modulation transfer function. Modulation transfer function Or, at a specified spatial frequency, it exhibits a spatial frequency response superior to the original low-resolution image and ordinary interpolated images.

[0089] Furthermore, an evaluation index is used to comprehensively assess the spatial frequency response characteristics of image reconstruction. This index not only verifies the effectiveness of the reconstruction method and proves that it can effectively improve the high-frequency information content of the image, but also forms a complete and quantifiable closed loop for high-resolution image reconstruction and evaluation.

[0090] In practical applications, infrared image super-resolution reconstruction requires consideration not only of edge sharpness but also of noise amplification. While total variational regularization can suppress noise, improper parameter settings can lead to staircase effects or loss of detail. Therefore, this invention uses the signal-to-noise ratio (SNR) to evaluate noise changes before and after reconstruction. Furthermore, as... Figure 7 As shown, after step 105 above, the method further includes the following steps: 501. Select a uniform region in the high-resolution image as the region of interest, calculate the gray mean and noise standard deviation of the region of interest, and obtain the signal-to-noise ratio index.

[0091] 502. When there are target areas and background areas in the high-resolution image, calculate the gray-scale mean and noise standard deviation of the target area and background area respectively, and calculate the contrast-to-noise ratio index based on the combination of the gray-scale mean difference and the noise standard deviation.

[0092] 503. Using the signal-to-noise ratio (SNR) and contrast-to-noise ratio (CNR) metrics, evaluate the amplification or suppression of noise during image reconstruction.

[0093] For uniform regions, a region of interest (ROI) without obvious edges and textures can be selected, and the mean and standard deviation can be calculated. The mean is the average gray level of all pixels within the ROI, and the standard deviation represents the degree to which the pixel gray levels deviate from the mean, reflecting the local noise level. The signal-to-noise ratio (SNR) is defined as the ratio of the mean to the standard deviation; a higher SNR indicates less noise in smooth regions and higher image quality. The SNR metric is obtained by calculating the mean and standard deviation. It can be expressed by the following formula:

[0094] in, The grayscale value of the region of interest; represents the grayscale standard deviation of the region of interest.

[0095] For infrared target detection scenarios, the contrast-to-noise ratio (SNR) between the target and the background can be calculated. The mean grayscale value of the target region represents the average brightness of pixels within the target, while the mean grayscale value of the background region represents the average brightness of surrounding background pixels. The difference between these two values ​​reflects the contrast between the target and the background. The standard deviations of noise in the target region and the background region characterize the degree of noise fluctuation within the target and background, respectively. The contrast-to-noise ratio is calculated by dividing the grayscale difference between the target and the background by the joint measure of their noise standard deviations, thus comprehensively reflecting the target's detectability in the presence of noise interference. Specifically, the contrast-to-noise ratio between the target and the background is calculated. It can be expressed by the following formula:

[0096] in, The average gray level of the target area; The average gray level of the background area; The noise standard deviation of the target area; The noise standard deviation of the background area.

[0097] It should be noted that the embodiments of the present invention do not solely rely on subjective image sharpness as the evaluation criterion for super-resolution performance. Instead, they employ a combination of modulation transfer function (MTF) and signal-to-noise ratio (SNR) metrics. The MTF metrics evaluate the improvement in spatial frequency response, while the SNR metrics evaluate noise amplification or suppression. This combined evaluation avoids spurious resolution improvements caused solely by sharpening algorithms. For example, while some sharpening methods can enhance edge steepness and make images appear sharper, they do not truly restore high-frequency information and may even amplify noise, leading to a significant decrease in SNR. The combined evaluation system proposed in this invention effectively identifies such pseudo-super-resolution phenomena, ensuring that the reconstruction results maintain good noise control performance while improving resolution, thereby achieving realistic, reliable, and usable super-resolution imaging.

[0098] For the detailed process of infrared image super-resolution reconstruction quality evaluation and parameter optimization, please refer to [link to relevant documentation]. Figure 8 As shown, Figure 8 This paper comprehensively demonstrates the closed-loop mechanism for evaluating reconstruction results and adjusting algorithm parameters accordingly. The process begins with the input of the reconstructed image to be evaluated and is divided into two parallel evaluation branches: Branch A evaluates the modulation transfer function (MTF), which sequentially selects the region of interest (ROI) on the hypotenuse, calculates the edge spread function, derives the line spread function, obtains the MTF curve through Fourier transform, and extracts the MTF from it. Modulation transfer function Nyquist frequency The evaluation process includes key metrics such as the specified frequency upscaling rate; Branch B evaluates the signal-to-noise ratio (SNR), selecting the region of interest (ROI) in the target background area and calculating the mean gray level and noise standard deviation of that region to obtain the SNR or contrast-to-noise ratio (CNR) value; the results from both branches are merged into C for joint judgment. If the modulation transfer function (MJF) is improved and the SNR is acceptable, the reconstruction is considered effective; if the MJF is improved but the SNR is significantly reduced, it may be due to over-sharpening or noise amplification; if the SNR is improved but the MJF is not significantly improved, the resolution is insufficient; finally, in D, the evaluation results are fed back to the reconstruction process to adjust the coverage accuracy of the displacement parameters, reset the total variation regularization coefficients, modify the number of iterations, and re-estimate the fuzzy kernel and displacement parameters, thus forming an evaluation-driven parameter adaptive optimization closed loop.

[0099] Furthermore, as a specific implementation of the above method, embodiments of this application provide an infrared image super-resolution reconstruction system, such as... Figure 9As shown, the system includes an infrared imaging module, a subpixel micro-displacement control module, a synchronization triggering module, an image acquisition and storage module, a displacement calibration module, a forward model construction module, a super-resolution reconstruction module, and an image quality evaluation module.

[0100] The infrared imaging module is used to perform infrared imaging of the target scene, obtaining multiple frames of raw infrared images. Specifically, it can employ a mid-wave infrared camera, a long-wave infrared camera, or other infrared detector imaging components. The pixel size of the infrared detector can be 12. 15 Other sizes are also possible. The infrared imaging module images the target scene through a lens or optical system and outputs a raw infrared image.

[0101] The subpixel micro-displacement control module is used to control the infrared imaging system, target, detector, or beam direction to produce half-pixel-level subpixel micro-displacement, enabling the same target scene to be imaged under multiple different sampling phases.

[0102] When using 2x super-resolution reconstruction, it is preferable to acquire four low-resolution images, with ideal sampling phases of (0,0), (0.5,0), (0,0.5), and (0.5,0.5). The displacement unit is the low-resolution image pixels.

[0103] If the pixel size of the infrared detector is Then the subpixel displacement at the half-pixel level is:

[0104] For example, when the pixel size is 12 At that time, the subpixel displacement at the half-pixel level is 6. When the pixel size is 15 At that time, the subpixel displacement at the half-pixel level is 7.5. .

[0105] When a fast-reflection mirror is used to generate image shift, if the effective focal length of the system is... The mechanical rotation angle of the mirror is Since the angle of reflected light changes by twice the angle of the mirror rotation, the subpixel displacement at the half-pixel level is approximately:

[0106] Therefore, to achieve sub-pixel displacement at the half-pixel level, the mechanical rotation angle required for the fast-reflection mirror is:

[0107] The above formula can be applied to convert half-pixel-level subpixel displacement into fast-reflection mirror angle control quantities.

[0108] The synchronization trigger module coordinates the subpixel micro-displacement control module and the infrared camera's exposure timing to ensure synchronization between subpixel micro-displacement, camera exposure, and image acquisition. Specifically, the system first controls the micro-displacement mechanism to move to the target position, and after the micro-displacement mechanism stabilizes, it outputs an exposure trigger signal, enabling the infrared camera to acquire images while in a stable displacement state. This method reduces motion blur and exposure blur during the process.

[0109] The image acquisition and storage module is used to acquire and store multiple frames of raw infrared images and their corresponding displacement parameters, exposure parameters and timestamp information.

[0110] The displacement calibration module is used to determine the subpixel displacement between each frame of infrared images. Due to factors such as fast-reflection mirror nonlinearity, platform hysteresis, mechanical vibration, synchronization errors, and image noise in actual systems, the theoretical half-pixel displacement may not equal the actual image displacement. Therefore, the displacement calibration module can obtain the subpixel displacement of each original image frame relative to the reference frame based on at least one of the following: preset displacement control commands, pre-acquired calibration target images, or image registration calculation results.

[0111] The forward model building module is used to establish a forward degradation model that reflects the mapping relationship between multiple frames of original infrared images and the high-resolution image to be reconstructed, based on the blurring characteristics of the infrared optical system, the detector sampling characteristics, the sub-pixel displacement relationship, and the noise characteristics.

[0112] The super-resolution reconstruction module is used to reconstruct high-resolution infrared images based on a forward degradation model, through an iterative inversion algorithm and total variation regularization constraints.

[0113] The image quality evaluation module is used to calculate the modulation transfer function index and signal-to-noise ratio index of the low-resolution image before reconstruction and the high-resolution image after reconstruction, and evaluate the super-resolution reconstruction effect based on the modulation transfer function boost and the change in signal-to-noise ratio.

[0114] Furthermore, as a specific implementation of the above method, embodiments of this application provide an infrared image super-resolution reconstruction device, such as... Figure 10 As shown, the device includes: a control unit 61, a registration unit 62, an establishment unit 63, a construction unit 64, and a solution unit 65.

[0115] The control unit 61 is used to control the relative position between the infrared imaging system and the target scene to generate a half-pixel-level subpixel displacement, so that the same target scene is imaged under multiple different sampling phases to obtain multiple frames of original infrared images. The registration unit 62 is used to perform subpixel registration on the multiple frames of original infrared images to obtain the subpixel displacement of each frame of original infrared image relative to the reference frame infrared image. Establishment unit 63 is used to establish a forward degradation model based on the sub-pixel displacement, and the forward degradation model is used to characterize the mapping relationship between each frame of original infrared image and the high-resolution image to be reconstructed; Construction unit 64 is used to construct an objective function based on the forward degradation model, the objective function including a data fidelity term and a regularization term that constrain the image reconstruction process; The solving unit 65 is used to solve the objective function through an iterative optimization algorithm to obtain a high-resolution image.

[0116] The infrared image super-resolution reconstruction device provided in this invention, compared with the existing technology that utilizes the sub-pixel displacement difference between multiple frames of the same scene to achieve infrared image super-resolution reconstruction, generates half-pixel-level sub-pixel displacement by controlling the relative position between the infrared imaging system and the target scene, enabling the same target scene to be imaged at multiple different sampling phases, obtaining multiple frames of original infrared images; sub-pixel registration is performed on the multiple frames of original infrared images to obtain the sub-pixel displacement amount of each frame of original infrared images relative to the reference frame of infrared images; a forward degradation model is established based on the sub-pixel displacement amount, which characterizes the mapping relationship between each frame of original infrared images and the high-resolution image to be reconstructed; an objective function is constructed based on the forward degradation model, which includes a data fidelity term and a regularization term constraining the image reconstruction process; and a high-resolution image is obtained by solving the objective function through an iterative optimization algorithm. The entire process actively controls the half-pixel-level sub-pixel displacement to acquire multiple frames of infrared images, effectively covering different sampling phases and obtaining sufficient spatial frequency information. By combining subpixel registration, forward degradation model, and objective function containing data fidelity and regularization terms for iterative reconstruction, the instability caused by displacement randomness in traditional methods is effectively overcome. This fully explores high-frequency detail information in multiple frames of images, significantly improves the stability and reliability of the reconstruction process, optimizes the imaging quality of infrared images, and meets the practical application requirements of high-precision infrared imaging.

[0117] In specific application scenarios, the half-pixel-level subpixel displacement in the control unit is achieved through at least one of the following methods: By moving the target under test, the image of the target under test formed by the optical system produces a half-pixel-level subpixel displacement on the focal plane of the infrared detector. By moving the infrared detector, a subpixel displacement of half a pixel is generated relative to the image plane of the optical system. By changing the direction of beam propagation through a beam deflection mechanism, the image of the target object formed by the optical system is shifted at the level of half a pixel on the focal plane of the infrared detector.

[0118] In specific application scenarios, the device further includes: The synchronization unit is used to coordinate the timing synchronization between the subpixel micro-displacement and the infrared camera exposure when a half-pixel-level subpixel micro-displacement is generated, so that each frame of the original infrared image corresponds one-to-one with its corresponding subpixel micro-displacement state, and the infrared camera performs image acquisition after the subpixel micro-displacement stabilizes.

[0119] In specific application scenarios, the registration unit is specifically used for: One frame is selected from the multiple original infrared images as a reference frame infrared image; Based on at least one of the system's preset displacement control command, pre-acquired calibration target image, or image registration calculation result, the sub-pixel displacement of each of the remaining original infrared images relative to the reference frame infrared image is obtained; the image registration calculation result is based on at least one of phase correlation, feature matching, sub-pixel edge localization, or spot centroid method.

[0120] In specific application scenarios, the building unit is specifically used for: The data fidelity terms between the degraded high-resolution image to be reconstructed and the original infrared images of each frame are calculated based on the forward degradation model. A regularization term is constructed to impose smoothness and edge preservation constraints on the reconstruction results; The data fidelity term and the regularization term are combined according to preset weights to form the objective function.

[0121] In specific application scenarios, the device further includes: The evaluation unit is used to evaluate the modulation transfer function of the high-resolution image by using the hypotenuse method after the objective function is solved by the iterative optimization algorithm to obtain the modulation transfer function curve; the modulation transfer function curve is obtained by taking the derivative of the edge spread function to obtain the line spread function, and then performing Fourier transform on the line spread function. The determining unit is used to determine the modulation transfer function value at a preset spatial frequency as an evaluation index based on the modulation transfer function curve. The first evaluation unit is used to evaluate the spatial frequency response characteristics of image reconstruction using the evaluation index.

[0122] In specific application scenarios, the device also includes: The selection unit is used to evaluate the unit, which, after solving the objective function through an iterative optimization algorithm to obtain a high-resolution image, selects a uniform region in the high-resolution image as a region of interest, calculates the gray-level mean and noise standard deviation of the region of interest, and obtains the signal-to-noise ratio index. The calculation unit is used to calculate the gray-level mean and noise standard deviation of the target region and the background region respectively when there are target regions and background regions in the high-resolution image, and to calculate the contrast-to-noise ratio index based on the combination of the gray-level mean difference and the noise standard deviation. The second evaluation unit is used to evaluate the amplification or suppression of noise during image reconstruction using the signal-to-noise ratio index and the contrast-to-noise ratio index.

[0123] Based on the above-described infrared image super-resolution reconstruction method, this application embodiment also provides a storage medium storing a computer program that, when executed by a processor, implements the above-described infrared image super-resolution reconstruction method.

[0124] Based on this understanding, the technical solution of this application can be embodied in the form of a software product. The software product can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, or portable hard drive), and includes several instructions to cause a computer device (such as a personal computer, server, or network device) to execute the methods described in the various implementation scenarios of this application.

[0125] Based on the above-described infrared image super-resolution reconstruction method, in order to achieve the above objectives, this application also provides a physical device for infrared image super-resolution reconstruction, specifically a computer, smartphone, tablet computer, smartwatch, server, or network device, etc. The physical device includes a storage medium and a processor; the storage medium is used to store a computer program; the processor is used to execute the computer program to implement the above-described infrared image super-resolution reconstruction method.

[0126] Optionally, the physical device may also include a user interface, a network interface, a camera, radio frequency (RF) circuitry, sensors, audio circuitry, a Wi-Fi module, etc. The user interface may include a display screen, input units such as a keyboard, etc., and optional user interfaces may also include USB interfaces, card reader interfaces, etc. The network interface may optionally include standard wired interfaces, wireless interfaces (such as Wi-Fi interfaces), etc.

[0127] In an exemplary embodiment, see Figure 11 The aforementioned physical device includes a communication bus, a processor, a memory, and a communication interface. It may also include an input / output interface and a display device. The various functional units can communicate with each other via the bus. The memory stores a computer program, and the processor executes the program stored in the memory to perform the infrared image super-resolution reconstruction method described in the above embodiments.

[0128] Those skilled in the art will understand that the physical device structure for infrared image super-resolution reconstruction provided in this embodiment does not constitute a limitation on the physical device, and may include more or fewer components, or combine certain components, or have different component arrangements.

[0129] The storage medium may also include an operating system and a network communication module. The operating system is a program that manages the hardware and software resources of the physical device for infrared image super-resolution reconstruction, supporting the operation of information processing programs and other software and / or programs. The network communication module is used to enable communication between the various components within the storage medium, as well as communication with other hardware and software in the information processing physical device.

[0130] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented using software plus necessary general-purpose hardware platforms, or it can be implemented in hardware. By applying the technical solution of this application, compared with existing methods, this application actively controls half-pixel-level subpixel displacement to acquire multiple frames of infrared images, effectively covering different sampling phases and obtaining sufficient spatial frequency information. Iterative reconstruction, combining subpixel registration, a forward degradation model, and an objective function containing data fidelity and regularization terms, effectively overcomes the reconstruction instability problem caused by displacement randomness in traditional methods, fully exploits high-frequency detail information in multiple frames of images, significantly improves the stability and reliability of the reconstruction process, and optimizes the imaging quality of infrared images while meeting the practical application requirements of high-precision infrared imaging.

[0131] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing this application. Those skilled in the art will understand that the modules in the apparatus of the embodiment can be distributed within the apparatus of the embodiment as described, or can be modified to be located in one or more apparatuses different from this embodiment. The modules of the above-described embodiment can be combined into one module, or further divided into multiple sub-modules.

[0132] The serial numbers in this application are for descriptive purposes only and do not represent the superiority or inferiority of any particular implementation scenario. The above disclosures are merely a few specific implementation scenarios of this application; however, this application is not limited thereto, and any variations conceived by those skilled in the art should fall within the protection scope of this application.

Claims

1. A method for super-resolution reconstruction of infrared images, characterized in that, include: By controlling the relative position between the infrared imaging system and the target scene to generate a half-pixel-level subpixel displacement, the same target scene is imaged under multiple different sampling phases, resulting in multiple frames of original infrared images. Subpixel registration is performed on the multiple frames of original infrared images to obtain the subpixel displacement of each frame of original infrared image relative to the reference frame of infrared image. A forward degradation model is established based on the subpixel displacement, and the forward degradation model is used to characterize the mapping relationship between each frame of the original infrared image and the high-resolution image to be reconstructed. An objective function is constructed based on the forward degradation model, and the objective function includes a data fidelity term and a regularization term that constrain the image reconstruction process. The objective function is solved by an iterative optimization algorithm to obtain a high-resolution image.

2. The infrared image super-resolution reconstruction method according to claim 1, characterized in that, The half-pixel level sub-pixel displacement is achieved through at least one of the following methods: By moving the target under test, the image of the target under test formed by the optical system produces a half-pixel-level subpixel displacement on the focal plane of the infrared detector. By moving the infrared detector, a subpixel displacement of half a pixel is generated relative to the image plane of the optical system. By changing the direction of beam propagation through a beam deflection mechanism, the image of the target object formed by the optical system is shifted at the level of half a pixel on the focal plane of the infrared detector.

3. The infrared image super-resolution reconstruction method according to claim 1, characterized in that, The method further includes: When generating half-pixel-level subpixel micro-displacement, the timing synchronization between the subpixel micro-displacement and the infrared camera exposure is coordinated so that each frame of the original infrared image corresponds one-to-one with its corresponding subpixel micro-displacement state, and the infrared camera performs image acquisition after the subpixel micro-displacement stabilizes.

4. The infrared image super-resolution reconstruction method according to claim 1, characterized in that, The subpixel registration of the multiple original infrared images to obtain the subpixel displacement of each original infrared image relative to the reference infrared image includes: One frame is selected from the multiple original infrared images as a reference frame infrared image; Based on at least one of the system's preset displacement control command, pre-acquired calibration target image, or image registration calculation result, the sub-pixel displacement of each of the remaining original infrared images relative to the reference frame infrared image is obtained; the image registration calculation result is based on at least one of phase correlation, feature matching, sub-pixel edge localization, or spot centroid method.

5. The infrared image super-resolution reconstruction method according to claim 1, characterized in that, The construction of the objective function based on the forward degradation model includes: The data fidelity terms between the degraded high-resolution image to be reconstructed and the original infrared images of each frame are calculated based on the forward degradation model. A regularization term is constructed to impose smoothness and edge preservation constraints on the reconstruction results; The data fidelity term and the regularization term are combined according to preset weights to form the objective function.

6. The infrared image super-resolution reconstruction method according to any one of claims 1-5, characterized in that, After solving the objective function using an iterative optimization algorithm to obtain the high-resolution image, the method further includes: The modulation transfer function (MTF) of the high-resolution image is evaluated using the hypotenuse method to obtain the MTF curve. The MTF curve is then used to obtain the line spread function by differentiating the edge spread function, and the line spread function is then calculated by performing a Fourier transform on the line spread function. Based on the modulation transfer function curve, the modulation transfer function value at a preset spatial frequency is determined as an evaluation index. The spatial frequency response characteristics of image reconstruction are evaluated using the aforementioned evaluation metrics.

7. The infrared image super-resolution reconstruction method according to any one of claims 1-5, characterized in that, After solving the objective function using an iterative optimization algorithm to obtain the high-resolution image, the method further includes: A uniform region in the high-resolution image is selected as the region of interest, and the gray-level mean and noise standard deviation of the region of interest are calculated to obtain the signal-to-noise ratio index. When a target region and a background region exist in the high-resolution image, the gray-level mean and noise standard deviation of the target region and the background region are calculated respectively, and the contrast-to-noise ratio index is calculated based on the combination of the gray-level mean difference and the noise standard deviation. The signal-to-noise ratio (SNR) and contrast-to-noise ratio (CNR) metrics are used to evaluate the amplification or suppression of noise during image reconstruction.

8. An infrared image super-resolution reconstruction device, characterized in that, include: The control unit is used to control the relative position between the infrared imaging system and the target scene to produce a half-pixel-level subpixel displacement, so that the same target scene is imaged under multiple different sampling phases to obtain multiple frames of original infrared images. The registration unit is used to perform subpixel registration on the multiple frames of original infrared images to obtain the subpixel displacement of each frame of original infrared image relative to the reference frame infrared image. The establishment unit is used to establish a forward degradation model based on the sub-pixel displacement, and the forward degradation model is used to characterize the mapping relationship between each frame of original infrared image and the high-resolution image to be reconstructed; A construction unit is used to construct an objective function based on the forward degradation model, the objective function including a data fidelity term and a regularization term that constrain the image reconstruction process; The solving unit is used to solve the objective function through an iterative optimization algorithm to obtain a high-resolution image.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images