Image restoration method and device based on time delay integration mismatch control, storage medium and equipment

Abstract

The application discloses an image restoration method and device based on time delay integration mismatch control, a storage medium and equipment, relates to the technical field of image processing, and mainly aims to solve the problem of poor accuracy of the existing restored image. The method comprises the following steps: acquiring original image data collected by energy-limited optical imaging and a mismatch ratio, wherein the mismatch ratio comprises a row transfer period multiple of a programmable timing controller and a detector integration order; performing convolution processing on the original image data based on the mismatch ratio to obtain intermediate image data, and performing down-sampling reading on the intermediate image data to obtain encoded degraded observation image data; performing up-sampling processing on the encoded degraded observation image data to obtain up-sampling observation image data, and performing decoding restoration processing on the up-sampling observation image data to obtain restored image data.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image restoration method, apparatus, storage medium, and device based on time delay integral mismatch control. Background Technology

[0002] Time-delay integration (TDI) is a key technology for improving the signal-to-noise ratio in pushbroom imaging systems and plays a crucial role in image restoration. Traditional TDI detectors require strict matching between the line transfer frequency and the ground projection velocity; any mismatch is considered an error source that needs to be eliminated. Currently, existing image restoration methods using TDI primarily minimize mismatch through closed-loop attitude control or compensate for image degradation caused by mismatch through post-processing deblurring algorithms. However, existing image restoration methods typically employ scalar regularization parameters, which cannot effectively handle the periodic optical transfer function zero-point problem unique to box kernels. For example, at the zero-point frequency, standard Wiener filtering either leads to catastrophic noise amplification due to insufficient regularization or complete signal smoothing due to excessive regularization. Furthermore, the blurring algorithm treats the blur kernel as an unknown perturbation, requiring joint estimation of the kernel function and latent image, which makes the kernel estimation error highly sensitive and the restoration quality unstable. Summary of the Invention

[0003] In view of this, this application provides an image restoration method, apparatus, storage medium and device based on time delay integral mismatch control, the main purpose of which is to solve the problem of poor accuracy of existing restored images.

[0004] According to one aspect of this application, an image restoration method based on time-delay integral mismatch control is provided, comprising: Acquire raw image data and mismatch ratios for energy-constrained optical imaging, wherein the mismatch ratios include the line transfer cycle multiple of the programmable timing controller and the detector integration stage number; Based on the mismatch ratio, the original image data is convolved to obtain intermediate image data, and the intermediate image data is downsampled and read to obtain coded degradation observation image data. The coded degraded observation image data is upsampled to obtain upsampled observation image data, and the upsampled observation image data is decoded and restored to obtain restored image data.

[0005] Further, the step of performing convolution processing on the original image data based on the mismatch ratio to obtain intermediate image data, and then downsampling the intermediate image data to obtain coded degradation observation image data includes: A frequency domain transfer function is constructed based on the mismatch ratio, and the frequency domain transfer function is transformed to obtain a spatial domain convolution kernel; The spatial convolution kernel is convolved with the original image data to obtain intermediate image data; The intermediate image data is subjected to track downsampling processing to read the coded degraded observation image data. The extraction interval of the track downsampling processing is configured based on the mismatch ratio.

[0006] Furthermore, after downsampling the intermediate image data to obtain coded degraded observation image data, the method further includes: The coded degraded observation image data is superimposed with target noise, and an upsampling process is performed on the coded degraded observation image data with superimposed target noise. The target noise includes shot noise and readout noise.

[0007] Further, the decoding and restoration processing of the upsampled observed image data to obtain the restored image data includes: The upsampled observed image data is decoded and restored using a pre-trained encoder-decoder network to obtain the restored image data; or, The upsampled observed image data is decoded and restored using frequency-adaptive Wiener deconvolution to obtain restored image data.

[0008] Furthermore, before performing decoding and restoration processing on the upsampled observed image data based on the encoder-decoder network that has completed model training to obtain the restored image data, the method further includes: Obtain image training samples, which include clear image samples and degraded image samples; Construct an encoder-decoder network to be trained. The encoder-decoder network includes an input layer, an encoding layer containing multiple encoders, a decoding layer containing multiple decoders, and an output layer. The number of encoders corresponds to the number of decoders. The encoder-decoder network is trained using the image training samples to obtain a trained encoder-decoder network.

[0009] Furthermore, before acquiring the raw image data and mismatch ratio obtained from energy-limited optical imaging, the method further includes: Obtain the data collection task parameters; The discrimination benefit threshold is calculated based on the collected task parameters, and the candidate mismatch ratio is first screened based on the discrimination benefit threshold. Based on engineering indicators, a second screening is performed on the candidate mismatch ratios after the first screening, and the candidate mismatch ratios after the second screening are scored by indicators. The overall evaluation value is determined based on the indicator weights and the indicator scores, and the mismatch ratio is determined from the candidate mismatch ratios based on the overall evaluation value.

[0010] Furthermore, before acquiring the data acquisition task parameters, the method further includes: Multiple candidate mismatch ratios with non-uniform coding are assigned to time delay integral detectors at each level; After determining the mismatch ratio from the candidate mismatch ratios based on the total evaluation value, the method further includes: The mismatch ratio is loaded into the programmable timing controller so that the programmable timing controller drives the time delay integration detector to perform optical imaging.

[0011] According to another aspect of this application, an image restoration apparatus based on time-delay integral mismatch control is provided, comprising: The acquisition module is used to acquire raw image data and mismatch ratios acquired for energy-constrained optical imaging. The mismatch ratios include the line transfer cycle multiple of the programmable timing controller and the number of detector integration stages. The processing module is used to perform convolution processing on the original image data based on the mismatch ratio to obtain intermediate image data, and to downsample and read the intermediate image data to obtain coded degradation observation image data; The restoration module is used to upsample the coded degraded observation image data to obtain upsampled observation image data, and to decode and restore the upsampled observation image data to obtain restored image data.

[0012] Further, the processing module is specifically used to construct a frequency domain transfer function based on the mismatch ratio, and to transform the frequency domain transfer function to obtain a spatial domain convolution kernel; to perform convolution processing on the spatial domain convolution kernel and the original image data to obtain intermediate image data; to perform track-side downsampling processing on the intermediate image data, and to read the coded degraded observation image data, wherein the extraction interval of the track-side downsampling processing is configured based on the mismatch ratio.

[0013] Furthermore, the device also includes: The overlay module is used to overlay target noise onto the coded degraded observation image data, so as to perform an upsampling process based on the coded degraded observation image data with the target noise overlaid, wherein the target noise includes shot noise and readout noise.

[0014] Furthermore, the restoration module is specifically used to perform decoding and restoration processing on the upsampled observation image data based on the encoder-decoder network that has completed model training, to obtain restored image data; or, to perform decoding and restoration processing on the upsampled observation image data based on frequency-adaptive Wiener deconvolution, to obtain restored image data.

[0015] Furthermore, the device also includes: The training module is used to acquire image training samples, which include clear image samples and degraded image samples; Construct an encoder-decoder network to be trained. The encoder-decoder network includes an input layer, an encoding layer containing multiple encoders, a decoding layer containing multiple decoders, and an output layer. The number of encoders corresponds to the number of decoders. The encoder-decoder network is trained using the image training samples to obtain a trained encoder-decoder network.

[0016] Furthermore, the device also includes: A filtering module is used to acquire data collection task parameters; calculate a discrimination benefit threshold based on the data collection task parameters, and perform a first filtering on candidate mismatch ratios based on the discrimination benefit threshold; perform a second filtering on the candidate mismatch ratios after the first filtering based on engineering indicators, and score the candidate mismatch ratios after the second filtering based on indicators; determine a total evaluation value based on indicator weights and indicator scores, and determine a mismatch ratio from the candidate mismatch ratios based on the total evaluation value.

[0017] Furthermore, the device also includes: The allocation module is used to allocate multiple candidate mismatch ratios in a non-uniform coding manner to time delay integral detectors at each level; A loading module is used to load the mismatch ratio into the programmable timing controller so that the programmable timing controller drives the time delay integration detector to perform optical imaging.

[0018] According to another aspect of this application, a storage medium is provided that stores at least one executable instruction, which causes a processor to perform operations corresponding to the image restoration method based on time delay integral mismatch control described above.

[0019] According to another aspect of this application, a computer device is provided, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other through the communication bus; The memory is used to store at least one executable instruction, which causes the processor to perform the operation corresponding to the image restoration method based on time delay integral mismatch control described above.

[0020] By employing the above technical solutions, the technical solutions provided in the embodiments of this application have at least the following advantages: This application provides an image restoration method, apparatus, storage medium, and device based on time-delay integral mismatch control. Compared with the prior art, the embodiments of this application acquire raw image data and mismatch ratios from energy-constrained optical imaging, wherein the mismatch ratios include the row transfer period multiple of the programmable timing controller and the detector integration stage; convolution processing is performed on the raw image data based on the mismatch ratio to obtain intermediate image data, and the intermediate image data is downsampled to obtain coded degraded observation image data; the coded degraded observation image data is upsampled to obtain upsampled observation image data, and the upsampled observation image data is decoded and restored to obtain restored image data. Through time-integral encoding with controlled mismatch, the optical transfer function zero-point problem is effectively avoided, and sensitivity is not increased by increasing the optical aperture, greatly improving the robustness and accuracy of image restoration.

[0021] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are given below. Attached Figure Description

[0022] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 A flowchart of an image restoration method based on time-delay integral mismatch control provided in an embodiment of this application is shown; Figure 2 This illustration shows a schematic diagram of image quality variation curves under different mismatch ratios provided in an embodiment of this application. Figure 3 This illustration shows a schematic diagram of image quality variation curves under different aperture reduction factors provided in an embodiment of this application; Figure 4 This illustration shows a network structure diagram provided in an embodiment of this application; Figure 5 This illustration shows another network structure provided by an embodiment of this application; Figure 6 This paper shows a block diagram of an image restoration device based on time delay integral mismatch control according to an embodiment of this application. Figure 7 A schematic diagram of the structure of a computer device provided in an embodiment of this application is shown. Detailed Implementation

[0023] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0024] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0025] This application provides an image restoration method based on time-delay integral mismatch control, such as... Figure 1 As shown, the method includes: 101. Acquire raw image data and mismatch ratios for energy-constrained optical imaging.

[0026] In this embodiment, the executing entity for image restoration can be either an imaging system server or a spaceborne server, to acquire the original image data to be restored in real time. Optical imaging refers to a method of capturing images of a ground scene using optical components via a detector, and outputting the original image data to be restored, i.e., the degraded image. Furthermore, the mismatch ratio includes the line transition period multiple of the programmable timing controller and the detector integration stage. In some embodiments, this can be configured based on different restoration requirements; this embodiment does not impose specific limitations.

[0027] It should be noted that the image restoration in this application embodiment is applicable to scenarios such as images captured by energy-constrained optical remote sensing cameras in aerospace. The detector can be, but is not limited to, TDI CCD or TDI CMOS, area array CMOS, area array CCD, linear array CCD, and TDI CMOS detectors, all of which are suitable for the image restoration method in this application embodiment. A K-level time delay integration detector is preferred, but this application embodiment does not impose specific limitations. Furthermore, the applicable scenarios in this application embodiment include, but are not limited to, aerospace remote sensors and industrial inspection fields, and are not specifically limited here.

[0028] In another embodiment of this application, for further definition and explanation, before the steps of acquiring the raw image data and mismatch ratio acquired for energy-limited optical imaging, the method further includes: Obtain the data collection task parameters; The discrimination benefit threshold is calculated based on the collected task parameters, and the candidate mismatch ratio is first screened based on the discrimination benefit threshold. Based on engineering indicators, a second screening is performed on the candidate mismatch ratios after the first screening, and the candidate mismatch ratios after the second screening are scored by indicators. The overall evaluation value is determined based on the indicator weights and the indicator scores, and the mismatch ratio is determined from the candidate mismatch ratios based on the overall evaluation value.

[0029] To improve the accuracy of image restoration by utilizing time-integrated encoding with controlled mismatch to enhance the accuracy of mismatch ratio selection, the current execution unit pre-acquires acquisition task parameters. These parameters include the detector single-stage signal rate s0, readout noise σr, and scene radiance. When calculating the discrimination benefit threshold, considering factors such as aperture reduction factor α, device line frequency upper limit f_max, ground speed, and sampling distance, the following formula can be used: Λ1=α²s0K / σr².

[0030] At this point, after obtaining the discriminant benefit threshold, the candidate mismatch ratios are first screened to eliminate mismatch ratio configurations that do not meet the lower limit of SNR, the lower limit of R_SNR, the lower limit of spectral support, and the line frequency constraint f0 / M≤f_max. For example, if Λ is less than or equal to 1, the readout noise dominates the domain, and there is excess signal-to-noise ratio gain; if Λ is greater than 1, the shot noise dominates the domain.

[0031] In a specific example, the first screening is used as a coarse screen, and the constraints that can be set are: output signal-to-noise ratio ≥ minimum SNR limit of the task; excess gain R_SNR preset gain limit, that is, when the gain is insufficient, the corresponding mismatch ratio is abandoned; spectrum support limit constraint, as a kernel support length constraint, to prevent excessive fuzziness; hardware line frequency constraint is expressed as: f0 / M≤f_max, where f0 is the nominal base line frequency, in order to screen out candidate mismatch ratios that meet the above conditions.

[0032] Furthermore, a second screening is conducted on the candidate mismatch ratios after the first screening based on engineering indicators. At this time, the engineering indicators are hard indicators, including but not limited to spectrum support ratio, Gram condition number, zero depth, along-track Nyquist MTF, line frequency requirement and data rate. At the same time, the threshold of each engineering indicator is obtained, and an initial screening is conducted according to the threshold comparison method to eliminate (K, M) candidate combinations that do not meet the hard constraints.

[0033] In one specific embodiment, after the above engineering indicators undergo a second screening, the candidate mismatch ratio is scored using corresponding indicator weights pre-configured. For example, the proportion of effective spectrum above -20dB within the output bandwidth; the higher the value, the better the preservation of high-frequency information in the image and the fewer restoration artifacts, so the proportion of spectrum support is assigned the highest weight. The Gram condition number characterizes the ill-conditioning of the image inverse problem; the smaller the value, the more stable the reconstruction and the less sensitive to noise, so its weight is configured as the second largest weight, less than the highest weight. Zero-point depth is the attenuation depth at the zero point in the dual Dirichlet kernel frequency domain, used to affect the aliasing suppression capability, and its weight is configured as the third largest weight, less than the second largest weight. Along the track, Nyquist MTF refers to the Nyquist frequency modulation transfer function, directly representing the ultimate resolution potential. Line frequency requirement and data rate respectively represent the uncertainties introduced by engineering, therefore, their corresponding weights decrease sequentially. In addition, the weights of various indicators can be configured according to the focus of the task. For example, in remote sensing and mapping tasks, the weights of spectrum support ratio and Nyquist MTF are tended to be increased. In weak light detection tasks, the weights of closed excess gain R_SNR are increased and the weights of Gram condition number constraints are relaxed. This application does not make specific limitations.

[0034] In addition, during the scoring process, all indicators are normalized and scalarized, and a comprehensive score is performed and ranked. The globally optimal set of candidate mismatch ratios is output as the mismatch ratio for the imaging process. Figure 2 The image quality variation curves are shown under different mismatch ratios.

[0035] It should be noted that when the mismatch ratio factor M > 1, it means that the line transfer frequency is reduced to 1 / M of the matching frequency. This allows the photogenerated charges at each ground point to be merged across M nominal line cycles within the charge domain before the readout noise event, extending the equivalent integration time by a factor of M and increasing the photon collection by a factor of M. Since merging occurs before readout, each output line only incurs readout noise once. In this embodiment, under readout noise-dominated operating conditions, the signal-to-noise ratio gain increases with... Linear accumulation greatly increases returns.

[0036] In another embodiment of this application, for further definition and explanation, before the step of obtaining the acquisition task parameters, the method further includes: Multiple candidate mismatch ratios with non-uniform coding are assigned to time delay integral detectors at each level; After determining the mismatch ratio from the candidate mismatch ratios based on the total evaluation value, the method further includes: The mismatch ratio is loaded into the programmable timing controller so that the programmable timing controller drives the time delay integration detector to perform optical imaging.

[0037] To select the optimal mismatch ratio, multiple candidate mismatch ratios with non-uniform coding are pre-assigned to each stage of the time-delay integration detector. In this case, the integration period of each stage of the time-delay integration detector is independently programmed as T_r = M_r·T0 (r = 0,…,K). 1) The mismatch ratio (M, K) is upgraded from a constant to a programmable sequence, which makes the synthesis kernel a weighted sum of variable-width integral windows. The side lobes of the transfer function can be shaped and the first zero can be pushed away by the sequence configuration, so as to flexibly balance the integral gain and recoverability.

[0038] 102. Based on the mismatch ratio, perform convolution processing on the original image data to obtain intermediate image data, and downsample the intermediate image data to obtain coded degradation observation image data.

[0039] In this embodiment, the mismatch ratio includes the line transfer period multiple M of the programmable timing controller and the detector integration level K. The line transfer period of the programmable timing controller can be set to M times the nominal period. When performing convolution processing, the blur kernel can be determined by the level K and the multiple M, such as obtaining a dual Dirichlet kernel, and then performing convolution calculation to obtain intermediate image data. This embodiment does not make specific limitations.

[0040] It should be noted that, due to the slowing down of the line transfer clock, one line is output every M nominal line cycles. Therefore, the intermediate image data is downsampled and read to obtain coded degraded observation image data. Preferably, the output sampling interval along the track is M times the ground sampling distance, that is, M times the decimation along the track, to obtain coded degraded observation image data.

[0041] In another embodiment of this application, for further definition and explanation, the step of convolving the original image data based on the mismatch ratio to obtain intermediate image data, and downsampling the intermediate image data to obtain coded degradation observation image data includes: A frequency domain transfer function is constructed based on the mismatch ratio, and the frequency domain transfer function is transformed to obtain a spatial domain convolution kernel; The spatial convolution kernel is convolved with the original image data to obtain intermediate image data; The intermediate image data is subjected to track downsampling processing to read the coded degraded observation image data. The extraction interval of the track downsampling processing is configured based on the mismatch ratio.

[0042] To form a forward coding model describing the scene-to-observation mapping relationship and thus encode the image to be restored, specifically, a frequency domain transfer function is first constructed based on the mismatch ratio, where the detector integration level K and the decimation factor M along the track are determined, which is also the line transfer period factor. In a specific embodiment, the programmable timing controller sets the line transfer period to M times the nominal period T0 = GSD / v_g, where M > 1, i.e., the line transfer frequency f_t = f_match / M, where v_g is the ground speed, GSD is the ground sampling distance, and f_match = v_g / GSD is the matching line frequency.

[0043] In this embodiment of the application, a mismatch ratio is constructed based on these two parameters, and a frequency domain transfer function is generated, which can be expressed as: H(f) = D_M(f)·D_K((M 1)f) / (KM); Where D_M(f) is the detector pixel photoelectric charge integral, D_K((M 1) f) represents the superposition of charge transfers corresponding to the K-level detector, and the core support length is L = K(M). 1) +1 pixel, the pixel integral zero family zeros are located at f=k / M and the inter-level comb zero family f=k / (K(M) 1)), all are determined by (K, M), D_m(f) = sin(πmf) / sin(πf), preferably a Dirichlet kernel. Then, the frequency domain transfer function H(f) is transformed to obtain the spatial domain convolution kernel h_{K,M}.

[0044] It should be noted that after obtaining the spatial convolution kernel, convolution processing is performed on the original image data with the spatial convolution kernel to obtain intermediate image data. That is, the original image data x is spatially convolved with the convolution kernel h_{K,M} to generate high-density intermediate sampled image data. Then, this intermediate image data is subjected to track downsampling processing to read the coded degraded observation image data. At this time, track downsampling is represented as ↓_M, that is, M times decimation sampling along the track. In engineering implementation, this is achieved by slowing down the line transfer clock, outputting only one line every M nominal line periods, and changing the sampling interval from nominal GSD to M·GSD. This yields the downsampled noiseless observation basis signal, i.e., the coded degraded observation image data. The decimation interval of the track downsampling processing is configured based on the mismatch ratio. At the same time, the output is decimated after convolution, which meets the hardware timing requirements of the detector to first integrate and transfer, and then read out at an opportune time, avoiding complete distortion of the physical model.

[0045] In another embodiment of this application, for further definition and explanation, after downsampling the intermediate image data to obtain coded degraded observation image data, the method further includes: Target noise is superimposed on the coded degraded observation image data, and an upsampling process is performed on the coded degraded observation image data with the superimposed target noise.

[0046] To avoid the impact of noise on image restoration, after obtaining the coded degraded observation image data, the current execution terminal can also superimpose target noise onto the coded degraded observation image data. The target noise η includes shot noise and readout noise, where readout noise occurs only once per output line. The noise variance table is: σn²=α²·s0·K·M· +σr².

[0047] Finally, the obtained coded degraded observation image data can be represented as: y=↓_M(h_{K,M} x)+η.

[0048] It should be noted that in the embodiments of this application, the ratio of shot noise to readout noise can be predefined, expressed as: Λ=α²·s0·K·M· / σr², where s0 is the number of electrons in a single-level unit radiance signal, and σr is the readout noise. Let α be the average radiance of the scene, and α be the aperture reduction factor. For example... Figure 3 The image quality variation curves shown are for different aperture reduction factors, if and only if Λ When reading out the noise-dominant domain, there is a net benefit: reducing α or decreasing scene brightness. This shifts Λ to the left into the net benefit region. All parameters in the embodiments of this application can be calculated during the design phase using radiation budgeting. Actual measurements have verified that the closed-loop error does not exceed 0.19%, meaning that without changing a single optical lens, the sensitivity that could only be achieved by increasing the aperture can be obtained.

[0049] 103. Upsample the coded degraded observation image data to obtain upsampled observation image data, and decode and restore the upsampled observation image data to obtain restored image data.

[0050] In this embodiment, to ensure the alignment of the encoded image, the encoded degraded observation image data is upsampled, such as by bilinear interpolation sampling or transposed convolution, to align the N / M sampled along the track to the input dimension N×N of the target grid, i.e., the encoded degraded observation image data. This embodiment does not impose specific limitations on this. When decoding and restoring the upsampled observation image data to obtain the restored image data, the current execution end can use an encoding / decoding network constructed by a neural network, including but not limited to decoding / encoding networks constructed by NAFNet-small, U-Net, Restormer, and Transformer architectures. It can also use analytical decoding such as frequency-adaptive Wiener deconvolution, i.e., Wiener filtering with frequency-varying regularization parameters constructed according to the optical transfer function amplitude, applying the strongest regularization at the zero-point frequency. This embodiment does not impose specific limitations on this. After decoding and restoration, the restored image data, i.e., the full-resolution restored image, is output. At this point, through restoration processing, range inversion, denoising, null inference, and M-fold along-track super-resolution can be completed in one go, greatly improving the refinement level of image restoration.

[0051] In this embodiment, the original image data is convolved using a mismatch ratio to obtain intermediate image data, which is then downsampled and read to achieve the purpose of encoding. This can be understood as encoding using Temporal Integration Coding (TIC) in the fields of computational imaging or aerospace push-broom imaging, and then upsampling the encoded degraded observation image data to achieve the purpose of decoding and restoring the upsampled observation image data.

[0052] In another embodiment of this application, for further definition and explanation, the step of decoding and restoring the upsampled observation image data to obtain restored image data includes: The upsampled observed image data is decoded and restored using a pre-trained encoder-decoder network to obtain the restored image data; or, The upsampled observed image data is decoded and restored using frequency-adaptive Wiener deconvolution to obtain restored image data.

[0053] In order to achieve the purpose of restoring degraded images, decoding restoration can be used to improve the accuracy of image restoration. In the embodiments of this application, encoding-decoding networks or deconvolution decoding methods can be used for decoding restoration.

[0054] In one embodiment, the upsampled observation image data can be decoded and restored based on the encoder-decoder network that has completed model training to obtain restored image data. The encoder-decoder network can be a lightweight backbone network or a model unfolding network, such as the U-Net network. By constructing a lightweight network, it can achieve the purpose of millions of parameters and single forward inference, meeting the real-time processing needs of computing resource-constrained platforms such as spaceborne and UAV platforms.

[0055] In one embodiment, the upsampled observed image data can be decoded and restored based on frequency-adaptive Wiener deconvolution to obtain restored image data, i.e., the original scene estimate is reconstructed from the upsampled observed image data y. Here, frequency-adaptive Wiener deconvolution refers to Wiener filtering with frequency-variable regularization parameters constructed according to the optical transfer function amplitude. Then, upsampling interpolation preprocessing, frequency domain modeling, adaptive Wiener gain construction, frequency domain correction, and inverse transform steps are performed to obtain the final restored image data. At this point, the restored image can be subjected to the strongest regularization at the zero frequency to avoid the appearance of artifacts.

[0056] In another embodiment of this application, for further definition and explanation, before the step of decoding and restoring the upsampled observed image data based on the encoder-decoder network that has completed model training to obtain the restored image data, the method further includes: Obtain image training samples; Construct the encoder-decoder network to be trained; The encoder-decoder network is trained using the image training samples to obtain a trained encoder-decoder network.

[0057] To achieve the goal of image decoding based on physical guidance, the current execution end acquires image training samples before decoding and restoring the upsampled observation image data using an encoder-decoder network. These training samples include both clear and degraded image samples. For example, the training samples may include synthetic samples from a publicly available set of clear remote sensing images, covering 60 configuration grids within the feasible domain. K, M, and α are configured as follows: K∈{2,4,8}, M∈{1.5,2.0,2.5,3.0,4.0}, and α∈{0.3,0.5,0.7,1.0}, respectively. Approximately 25% of the samples may also be digitally merged degraded samples; however, this embodiment does not impose specific limitations. The training data faithfully reproduces the accurate physical model, suppressing pseudo-details from purely data-driven approaches. The network structure itself does not introduce any physical branches, the number of parameters remains in the millions, and inference can be performed in a single forward pass, providing a feasible path for real-time on-board restoration.

[0058] Furthermore, when constructing the encoder-decoder network, it can be based on NAFNet-small, U-Net, Restormer, or Transformer architectures. The encoder-decoder network includes an input layer, an encoding layer containing multiple encoders, a decoding layer containing multiple decoders, and an output layer, such as... Figure 4 The neural network decoding structure shown has a backbone network of NAFNet-small (~3.1M) / U-Net (~7.7M) parameters, enabling real-time reconstruction of spaceborne / UAV systems with millions of parameters per forward pass. Simultaneously, physical priors are encoded in the training data distribution, the network architecture does not introduce physical branches, and it is universally applicable to any (K,M,α) within the feasible domain. A single forward pass simultaneously completes value range inversion, denoising, null space inference, and M× along-track super-resolution. Figure 5 The frequency-adaptive Wiener deconvolutional network structure shown is parsed and decoded without training, and is compatible with... Figure 4 The two alternative restoration methods shown in the neural network decoding can construct a frequency-varying regularization parameter λ(f) based on the OTF amplitude: applying the strongest regularization at the zero-point frequency avoids noise amplification and artifacts caused by the zero point of the box kernel period. Simultaneously, deblurring, denoising, and M× along-track super-resolution can be completed in one step by following the steps of upsampling interpolation, frequency domain modeling, Wiener gain, frequency domain correction, and inverse transform. Wherein, the box kernel H(f) = D_M(f)·D_K((M 1)f) / (KM), the family of zeros f=k / M and f=k / (K(M) 1)).

[0059] In one specific embodiment, during model training, the loss function is either L1 loss or L2 loss, along with perceptual loss or a weighted combination thereof, the optimizer is AdamW, and the learning rate is 10. -4Cosine annealing was used to optimize the parameters, supplemented by random rotation and flip augmentation, and real satellite image slices were mixed in to reduce the domain difference.

[0060] In a specific example, under the reference configuration of K=2, M=2, and α=0.5, in this embodiment of the application, preferably, the analytical Wiener decoding is 8.50dB, the learned decoding based on the encoder-decoder network is 23.78dB, and the prior contribution of the learned network increases by 15.28dB. In the equal-rate, same-backbone fairness comparison with digital merging, the encoder-decoder network gain of this embodiment of the application is 1.0dB net gain, and this gain comes entirely from charge domain merging physics rather than network differences. Meanwhile, the generalization loss for configurations not seen during training is only 0.101dB, and the various indicators for zero-sample transfer across sensors for real satellite imagery are significantly higher than the sensor benchmark, ensuring engineering feasibility.

[0061] In an example of a scenario with ultra-low orbit pushbroom and line frequency axis dominance, at an ultra-low orbit of 250–350 km, the ground speed is high. When M=1, the required line frequency for matching is f0=14.6kHz, which exceeds the device's line frequency limit f_max=5kHz. Based on the line frequency constraint, M_min=3 is obtained. After sorting by inherent indicators and weighted scalarization, the recommended operating points are K=2 and M=4, with a line frequency margin of 4 times, R_SNR=1.844, and spectral support of 1.0. The detector operates at a comfortable line frequency, and the measured end-to-end net gain ΔPSNR=+1.41dB. At this point, α=0.5, the noise parameter is ±20%, and the image is restored 25 times under mismatch, and it is stable after 25 attempts.

[0062] In a scenario example with extreme aperture high-resolution frame push and equivalent aperture axis dominance, the GSD is 0.3–0.5m, the aperture is compressed to just no MTF margin according to diffraction cutoff, and the single exposure is 16–26μs. It is stable in the readout noise domain in dark scenes such as dawn / dusk, high latitude, vegetation, and water bodies, with Λ<1. After constraint selection with SNR≥5, R_SNR≥1.2, and spectral support≥0.95, the unique mismatch ratio is selected as K=2 and M=4. The readout-limited SNR is improved from 3.6 to 14.5, and the equivalent aperture is increased by about 2 times. At this time, the camera mass, size, and cost remain unchanged.

[0063] This application provides an image restoration method based on time-delay integral mismatch control. It acquires raw image data and a mismatch ratio from energy-constrained optical imaging, where the mismatch ratio includes the row transfer period multiple of the programmable timing controller and the detector integration stage. Based on the mismatch ratio, the raw image data is convolved to obtain intermediate image data, which is then downsampled to obtain coded degraded observation image data. The coded degraded observation image data is then upsampled to obtain upsampled observation image data, which is then decoded to obtain restored image data. By using time-integral encoding with controlled mismatch, the method effectively avoids the optical transfer function zero-point problem, eliminating the need to increase the optical aperture to improve sensitivity, and significantly improving the robustness and accuracy of image restoration.

[0064] Furthermore, as a response to the above Figure 1 The implementation of the method shown in this application provides an image restoration device based on time-delay integral mismatch control, such as... Figure 6 As shown, the device includes: The acquisition module 21 is used to acquire raw image data and mismatch ratio acquired for energy-limited optical imaging, wherein the mismatch ratio includes the line transfer cycle multiple of the programmable timing controller and the detector integration stage number; Processing module 22 is used to perform convolution processing on the original image data based on the mismatch ratio to obtain intermediate image data, and to downsample and read the intermediate image data to obtain coded degradation observation image data; The restoration module 23 is used to perform upsampling processing on the coded degraded observation image data to obtain upsampled observation image data, and to perform decoding and restoration processing on the upsampled observation image data to obtain restored image data.

[0065] Further, the processing module is specifically used to construct a frequency domain transfer function based on the mismatch ratio, and to transform the frequency domain transfer function to obtain a spatial domain convolution kernel; to perform convolution processing on the spatial domain convolution kernel and the original image data to obtain intermediate image data; to perform track-side downsampling processing on the intermediate image data, and to read the coded degraded observation image data, wherein the extraction interval of the track-side downsampling processing is configured based on the mismatch ratio.

[0066] Furthermore, the device also includes: The overlay module is used to overlay target noise onto the coded degraded observation image data, so as to perform an upsampling process based on the coded degraded observation image data with the target noise overlaid, wherein the target noise includes shot noise and readout noise.

[0067] Furthermore, the restoration module is specifically used to perform decoding and restoration processing on the upsampled observation image data based on the encoder-decoder network that has completed model training, to obtain restored image data; or, to perform decoding and restoration processing on the upsampled observation image data based on frequency-adaptive Wiener deconvolution, to obtain restored image data.

[0068] Furthermore, the device also includes: The training module is used to acquire image training samples, which include clear image samples and degraded image samples. Construct an encoder-decoder network to be trained. The encoder-decoder network includes an input layer, an encoding layer containing multiple encoders, a decoding layer containing multiple decoders, and an output layer. The number of encoders corresponds to the number of decoders. The encoder-decoder network is trained using the image training samples to obtain a trained encoder-decoder network.

[0069] Furthermore, the device also includes: A filtering module is used to acquire data collection task parameters; calculate a discrimination benefit threshold based on the data collection task parameters, and perform a first filtering on candidate mismatch ratios based on the discrimination benefit threshold; perform a second filtering on the candidate mismatch ratios after the first filtering based on engineering indicators, and score the candidate mismatch ratios after the second filtering based on indicators; determine a total evaluation value based on indicator weights and indicator scores, and determine a mismatch ratio from the candidate mismatch ratios based on the total evaluation value.

[0070] Furthermore, the device also includes: The allocation module is used to allocate multiple candidate mismatch ratios in a non-uniform coding manner to time delay integral detectors at each level; A loading module is used to load the mismatch ratio into the programmable timing controller so that the programmable timing controller drives the time delay integration detector to perform optical imaging.

[0071] This application provides an image restoration device based on time-delay integral mismatch control. It acquires raw image data and a mismatch ratio from energy-constrained optical imaging, where the mismatch ratio includes the row transfer period multiple of the programmable timing controller and the detector integration stage. Based on the mismatch ratio, the raw image data is convolved to obtain intermediate image data, which is then downsampled to obtain coded degraded observation image data. The coded degraded observation image data is then upsampled to obtain upsampled observation image data, which is then decoded to obtain restored image data. Through controlled mismatch time-integral encoding, the optical transfer function zero-point problem is effectively avoided, eliminating the need to increase the optical aperture to improve sensitivity and significantly improving the robustness and accuracy of image restoration.

[0072] According to one embodiment of this application, a storage medium is provided, the storage medium storing at least one executable instruction, the computer-executable instruction being able to execute the image restoration method based on time delay integral mismatch control in any of the above method embodiments.

[0073] Figure 7 The diagram shows a structural schematic of a computer device according to one embodiment of the present application. The specific embodiments of the present application do not limit the specific implementation of the computer device.

[0074] like Figure 7 As shown, the computer device may include: a processor 302, a communications interface 304, a memory 306, and a communications bus 308.

[0075] The processor 302, communication interface 304, and memory 306 communicate with each other via communication bus 308.

[0076] Communication interface 304 is used to communicate with other network elements such as clients or other servers.

[0077] The processor 302 is used to execute program 310, specifically to execute the relevant steps in the above-described embodiment of the image restoration method based on time delay integral mismatch control.

[0078] Specifically, program 310 may include program code that includes computer operation instructions.

[0079] Processor 302 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application. The computer device includes one or more processors, which may be processors of the same type, such as one or more CPUs; or they may be processors of different types, such as one or more CPUs and one or more ASICs.

[0080] Memory 306 is used to store program 310. Memory 306 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0081] Specifically, program 310 can be used to cause processor 302 to perform the following operations: Acquire raw image data and mismatch ratios for energy-constrained optical imaging, wherein the mismatch ratios include the line transfer cycle multiple of the programmable timing controller and the detector integration stage number; Based on the mismatch ratio, the original image data is convolved to obtain intermediate image data, and the intermediate image data is downsampled and read to obtain coded degradation observation image data. The coded degraded observation image data is upsampled to obtain upsampled observation image data, and the upsampled observation image data is decoded and restored to obtain restored image data.

[0082] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0083] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. An image restoration method based on time-delay integral mismatch control, characterized in that, include: Acquire raw image data and mismatch ratios for energy-constrained optical imaging, wherein the mismatch ratios include the line transfer cycle multiple of the programmable timing controller and the detector integration stage number; Based on the mismatch ratio, the original image data is convolved to obtain intermediate image data, and the intermediate image data is downsampled and read to obtain coded degradation observation image data. The coded degraded observation image data is upsampled to obtain upsampled observation image data, and the upsampled observation image data is decoded and restored to obtain restored image data.

2. The method according to claim 1, characterized in that, The process of convolving the original image data based on the mismatch ratio to obtain intermediate image data, and then downsampling the intermediate image data to obtain coded degradation observation image data includes: A frequency domain transfer function is constructed based on the mismatch ratio, and the frequency domain transfer function is transformed to obtain a spatial domain convolution kernel; The spatial convolution kernel is convolved with the original image data to obtain intermediate image data; The intermediate image data is subjected to track downsampling processing to read the coded degraded observation image data. The extraction interval of the track downsampling processing is configured based on the mismatch ratio.

3. The method according to claim 2, characterized in that, After downsampling the intermediate image data to obtain coded degraded observation image data, the method further includes: The coded degraded observation image data is superimposed with target noise, and an upsampling process is performed on the coded degraded observation image data with superimposed target noise. The target noise includes shot noise and readout noise.

4. The method according to claim 1, characterized in that, The decoding and restoration process of the upsampled observation image data to obtain the restored image data includes: The upsampled observed image data is decoded and restored using a pre-trained encoder-decoder network to obtain the restored image data; or, The upsampled observed image data is decoded and restored using frequency-adaptive Wiener deconvolution to obtain restored image data.

5. The method according to claim 4, characterized in that, Before the method performs decoding and restoration processing on the upsampled observed image data based on the pre-trained encoder-decoder network to obtain the restored image data, the method further includes: Obtain image training samples, which include clear image samples and degraded image samples; Construct an encoder-decoder network to be trained. The encoder-decoder network includes an input layer, an encoding layer containing multiple encoders, a decoding layer containing multiple decoders, and an output layer. The number of encoders corresponds to the number of decoders. The encoder-decoder network is trained using the image training samples to obtain a trained encoder-decoder network.

6. The method according to claim 1, characterized in that, Before acquiring the raw image data and mismatch ratio obtained from energy-limited optical imaging, the method further includes: Obtain the data collection task parameters; The discrimination benefit threshold is calculated based on the collected task parameters, and the candidate mismatch ratio is first screened based on the discrimination benefit threshold. Based on engineering indicators, a second screening is performed on the candidate mismatch ratios after the first screening, and the candidate mismatch ratios after the second screening are scored by indicators. The overall evaluation value is determined based on the indicator weights and the indicator scores, and the mismatch ratio is determined from the candidate mismatch ratios based on the overall evaluation value.

7. The method according to claim 6, characterized in that, Before acquiring the data acquisition task parameters, the method further includes: Multiple candidate mismatch ratios with non-uniform coding are assigned to time delay integral detectors at each level; After determining the mismatch ratio from the candidate mismatch ratios based on the total evaluation value, the method further includes: The mismatch ratio is loaded into the programmable timing controller so that the programmable timing controller drives the time delay integration detector to perform optical imaging.

8. An image restoration device based on time-delay integral mismatch control, characterized in that, include: The acquisition module is used to acquire raw image data and mismatch ratios acquired for energy-limited optical imaging, wherein the mismatch ratios include the line transfer cycle multiple of the programmable timing controller and the detector integration stage number; The processing module is used to perform convolution processing on the original image data based on the mismatch ratio to obtain intermediate image data, and to downsample and read the intermediate image data to obtain coded degradation observation image data; The restoration module is used to upsample the coded degraded observation image data to obtain upsampled observation image data, and to decode and restore the upsampled observation image data to obtain restored image data.

9. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method of claim 1.

10. A computer device, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method of claim 1.

Images