Method and device for recovering sequence images of aerodynamic optical effects and electronic device

By constructing a comprehensive evaluation model for no-reference image quality and a consistency coordination mechanism, key frames of different qualities are selected and directional update quantities are generated, thus solving the image degradation problem caused by aero-optical effects and achieving efficient and stable image restoration results.

CN123071196APending Publication Date: 2026-11-10WUHAN INST OF TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611559912.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-10-10
Publication Date
2026-11-10

AI Technical Summary

Technical Problem

Existing technologies for dealing with image degradation caused by aero-optical effects suffer from problems such as reliance on large-scale training data, high computational load, long inference time, and insufficient information utilization and large errors in multi-frame image processing, resulting in poor image restoration quality and low efficiency.

Method used

By constructing a comprehensive evaluation model for the quality of no-reference images, three types of keyframes—low, medium, and high quality—are selected, a directional update amount for a unified blur kernel is generated, and a consistency coordination mechanism is introduced to optimize the iterative update process of the blur kernel, thereby improving the quality and efficiency of image restoration.

Benefits of technology

With less reliance on training data, it effectively improves the restoration quality and overall processing efficiency of aero-optical effect sequence images, avoids interference from invalid frames, and enhances the utilization efficiency of useful information and the stability of fuzzy kernel solution.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123071196A_ABST
    Figure CN123071196A_ABST
Patent Text Reader

Abstract

The application provides a kind of aerodynamic optical effect sequence image restoration method, device and electronic equipment, belong to image processing technical field.Its method includes: obtaining the same target scene sequence degradation image affected by aerodynamic optical effect;Build a no-reference image quality comprehensive evaluation model, the degradation characteristics of each frame are characterized, and the invalid frame is removed to obtain three types of key frames of low quality, medium quality and high quality;According to the degradation observation characteristics corresponding to the three types of key frames respectively, three types of directional update quantity on the support domain range, main structure and boundary details of the uniform blur kernel are generated, and the iteration update of the blur kernel is jointly acted, to build a uniform blur kernel fast solution model;The latent clear image and the uniform blur kernel are jointly optimized and solved, and the response difference between the three types of update quantity is adjusted through the consistency coordination mechanism, to obtain the restoration result.Using the application can effectively improve the restoration quality and processing efficiency of aerodynamic optical effect sequence image.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and specifically to a method, apparatus, and electronic device for restoring images based on aero-optical effects sequences. Background Technology

[0002] Aero-optical effect correction is an important research direction in the field of image processing. When high-speed vehicles (such as missiles and hypersonic vehicles) fly at high speeds in the atmosphere, their optical imaging windows interact violently with the incoming airflow, causing wavefront distortion, energy diffusion, and pixel shift in the imaging beam. Ultimately, this results in degradation phenomena in the acquired target sequence images, such as random blurring, intensity distribution diffusion, peak reduction, and inter-frame geometric jitter. Effectively suppressing image degradation caused by aero-optical effects and restoring clear original target images has significant engineering application value for military and civilian scenarios such as precision guidance, target recognition, remote sensing monitoring, and astronomical observation.

[0003] For the problem of image restoration in sequences affected by aero-optical effects, existing technologies mainly fall into two categories. The first category is based on deep learning methods, which directly learn the mapping from the degraded image to the sharp image by constructing a convolutional neural network or Transformer model. This type of method relies on large-scale, high-quality training data, and suffers from high computational cost and long inference time. The second category is based on physical modeling optimization methods, which model the image degradation process as a convolution between a blur kernel and the sharp image, and perform iterative restoration through joint optimization. When processing multi-frame sequences, this type of method typically applies the same processing strategy to all frames or performs overall joint optimization. Since the degree of degradation varies among frames in a sequence, noise and distortion in severely degraded frames introduce errors, while detailed information in higher-quality frames is difficult to fully utilize. Furthermore, in solving for the unified blur kernel, existing methods typically employ an ergonomic search strategy, and the information from multiple frames may exhibit directional inconsistencies during joint optimization.

[0004] Therefore, how to improve the restoration quality and efficiency of aero-optical effect sequence images without relying on large-scale training data is a technical problem that urgently needs to be solved in the field of aero-optical effect image restoration. Summary of the Invention

[0005] In view of this, it is necessary to provide a method, apparatus and electronic device for restoring sequential images using aero-optical effects, in order to solve the technical problems of poor image restoration quality and low restoration efficiency caused by insufficient utilization of information on different degradation states of sequential images and multi-frame constraint conflicts in existing methods.

[0006] To address the aforementioned technical problems, in a first aspect, the present invention provides a method for restoring a sequence of images related to aero-optical effects, comprising: Acquire a sequence of degraded images of the same target scene affected by aero-optical effects; A comprehensive evaluation model for the quality of no-reference images is constructed. Degradation features are characterized for each frame in the sequence of degraded images. Invalid frames are removed based on the degradation observation features, and three categories of key frames, namely low quality, medium quality and high quality, are selected. Based on the degradation observation features corresponding to the three types of keyframes, three types of directional update quantities of the unified fuzzy kernel are generated in the support domain range, main structure and boundary details. The three types of directional update quantities are applied together to the iterative update of the unified fuzzy kernel to construct a fast solution model for the unified fuzzy kernel. Under the unified fuzzy kernel fast solution model, the potential sharp image and the unified fuzzy kernel are jointly optimized and solved. The response differences between the three types of directional update quantities are adjusted through a consistency coordination mechanism, so that the unified fuzzy kernel converges stably and the restoration result is obtained.

[0007] In one possible implementation, acquiring a sequence of degraded images of the same target scene affected by aero-optical effects includes: Acquire initial sequence degraded images of the same target scene affected by atmospheric turbulence, wherein the initial sequence degraded images contain random blurring, inter-frame micro-displacement and imaging noise caused by turbulence; For each frame of the initial sequence of degraded images, a low-intensity two-dimensional Gaussian filter is used for denoising to suppress random noise while preserving turbulent degradation characteristics. Using the first frame in the initial sequence of degraded images as the reference frame, the denoised images of each frame are registered using a subpixel-level sparse optical flow method, and pixel-level spatial alignment is achieved through feature point matching and bilinear interpolation to obtain the spatially aligned sequence of degraded images.

[0008] In one possible implementation, the construction of the referenceless image quality comprehensive evaluation model includes: Calculate the gradient energy index, gray-level local variance index, and Laplacian variance index for each frame in the sequence of degraded images; The gradient energy index, gray-level local variance index, and Laplacian variance index of each frame image are normalized to obtain the corresponding normalized feature components. Based on the difference between the three normalized feature components and their mean values ​​of each frame image, the adaptive adjustment parameters of each feature component are determined. By employing a nonlinear geometric mean mapping, the adaptive adjustment parameters are fused with the corresponding normalized feature components to obtain the comprehensive quality evaluation value of each frame image.

[0009] In one possible implementation, the process of removing invalid frames based on degraded observation features and filtering out three categories of keyframes—low-quality, medium-quality, and high-quality—includes: Obtain a quality threshold coefficient, and determine and remove frames whose comprehensive quality evaluation value is less than the quality threshold coefficient as invalid frames. Based on the maximum value of the comprehensive quality evaluation value in the effective frame set and the quality threshold coefficient, the upper bound of the low quality interval and the lower bound of the high quality interval are determined, and the effective frame set is divided into a low quality frame candidate set, a medium quality frame candidate set and a high quality frame candidate set. The frame with the smallest overall quality evaluation value is selected from the low-quality frame candidate set as the low-quality keyframe, the frame with the largest overall quality evaluation value is selected from the high-quality frame candidate set as the high-quality keyframe, and the frame with the overall quality evaluation value closest to the center value of the medium-quality interval is selected from the medium-quality frame candidate set as the medium-quality keyframe.

[0010] In one possible implementation, the step of generating three types of directional update quantities for the unified blur kernel in terms of the support domain range, main structure, and boundary details, based on the degradation observation features corresponding to the three types of keyframes, includes: Based on the low-quality keyframes, the support domain determination threshold is determined according to the peak response of the unified fuzzy kernel in the current iteration, an effective support domain mask is constructed, and a support domain range update is generated. The support domain range update only applies to the position indicated by the effective support domain mask. Based on the medium-quality keyframes, the main structure determination threshold is determined according to the peak response of the unified fuzzy kernel in the current iteration, the main structure region mask is constructed, and the main structure update amount is generated. The main structure update amount is used to stabilize the main lobe structure morphology of the unified fuzzy kernel. Based on the high-quality keyframes, a boundary detail region mask is constructed. The boundary detail region mask is the difference between the effective support domain mask and the main structure region mask. A boundary detail update amount is generated, which is used to correct the edge morphology and high-frequency response characteristics of the unified blur kernel.

[0011] In one possible implementation, adjusting the response differences among the three types of directional update quantities through a consistency coordination mechanism includes: Calculate the average of the directional update amount over the support domain, the directional update amount over the main structure, and the directional update amount over the boundary details, and use this as the overall optimization trend; Calculate the degree of deviation of each type of targeted update quantity from the overall optimization trend based on the degree of directional and magnitude deviation of each type of targeted update quantity. Based on the degree of deviation, an exponential mapping function is used to calculate the consistency coordination weight of each type of directional update, so that the directional update with a greater degree of deviation receives a smaller coordination weight. The corresponding directional update amount is weighted using the consistency coordination weight, and then applied to the iterative update of the unified fuzzy kernel.

[0012] In one possible implementation, the joint optimization solution of the potentially sharp image and the unified blur kernel includes: The high-quality keyframes are used as the initial values ​​for the potential sharp image, and the unit impulse function is used as the initial value for the unified blur kernel. In each iteration, the current unified blur kernel is fixed, and the frequency domain deconvolution method is used to solve for the potential sharp image; The potentially clear image obtained by fixing the solution is used to update the unified fuzzy kernel according to the unified fuzzy kernel fast solution model and consistency coordination mechanism; The updated unified fuzzy kernel is projected onto the feasible region that satisfies the nonnegativity and normalization constraints.

[0013] In one possible implementation, achieving stable convergence of the unified fuzzy kernel to obtain the restoration result includes: After each iteration, the changes in the unified blur kernel, the changes in the potentially sharp image, and the changes in the joint optimization objective function are calculated. When the changes in the unified blur kernel, the potential sharp image, and the joint optimization objective function are all less than the corresponding convergence thresholds, the iteration is determined to be converged, the iteration is terminated, and the current potential sharp image is output as the restoration result.

[0014] On the other hand, the present invention also provides a device for restoring a sequence of images with aero-optical effects, comprising: The sequence acquisition module is used to acquire a sequence of degraded images of the same target scene affected by aero-optical effects; The quality assessment module is used to construct a comprehensive quality evaluation model for no-reference images, characterize the degradation features of each frame in the sequence of degraded images, remove invalid frames based on the degradation observation features, and select three categories of key frames: low quality, medium quality, and high quality. The model building module is used to generate three types of directional update quantities of the unified fuzzy kernel in the support domain range, main structure and boundary details according to the degradation observation characteristics corresponding to the three types of keyframes respectively. The three types of directional update quantities are used together to iteratively update the unified fuzzy kernel to build a fast solution model of the unified fuzzy kernel. The image restoration module is used to jointly optimize the potential sharp image and the unified fuzzy kernel under the unified fuzzy kernel fast solution model, and adjust the response differences among the three types of directional update quantities through a consistency coordination mechanism, so that the unified fuzzy kernel converges stably and the restoration result is obtained.

[0015] Thirdly, the present invention also provides an electronic device, including an image acquisition unit, a memory, and a processor, wherein, An image acquisition device is used to acquire initial sequence degraded images of the same target scene affected by aero-optical effects; The memory is used to store programs; The processor, coupled to the memory, is used to execute the program stored in the memory to implement the steps in the aero-optical effect sequence image restoration method described in any of the above implementations.

[0016] The beneficial effects of this invention are as follows: The aero-optical effect sequence image restoration method provided by this invention constructs a comprehensive evaluation model for the quality of no-reference images, actively selecting three types of keyframes—low, medium, and high quality—from the degraded sequence, avoiding interference from invalid frames on subsequent estimations, and improving the utilization efficiency of useful information; based on the degradation characteristics corresponding to keyframes of different quality, three types of directional update quantities—support domain range, main structure, and boundary details—are generated for a unified fuzzy kernel, respectively, allowing the solution of the fuzzy kernel to change from blind search to targeted step-by-step correction; the three types of update quantities are iterated together and a consistency coordination mechanism is introduced to adjust the response differences between them, making the update process more stable and less prone to divergence, thereby improving the stability of solution convergence; finally, with less reliance on training data, it effectively improves the restoration quality and overall processing efficiency of aero-optical effect sequence images. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A schematic flowchart of an embodiment of the aero-optical effect sequence image restoration method provided by the present invention; Figure 2 For the present invention Figure 1 A schematic diagram of an embodiment of S101; Figure 3 For the present invention Figure 1 A schematic diagram of an embodiment of S102; Figure 4 For the present invention Figure 1 A schematic diagram of another embodiment of S102; Figure 5 For the present invention Figure 1 A schematic diagram of an embodiment of S103; Figure 6 For the present invention Figure 1 A schematic diagram of an embodiment of S104; Figure 7 For the present invention Figure 1 A schematic diagram of another embodiment of S104; Figure 8 A schematic diagram of an embodiment of the aero-optical effect sequence image restoration device provided by the present invention; Figure 9 A schematic diagram of an embodiment of the electronic device provided by the present invention. Detailed Implementation

[0019] In the description of the embodiments of the present invention, unless otherwise stated, "multiple" means two or more. "And / or" describes the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0020] The terms "first," "second," etc., used in the embodiments of this invention are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a technical feature defined with "first" or "second" may explicitly or implicitly include at least one of that feature.

[0021] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0022] This invention provides a method, apparatus, and electronic device for restoring a sequence of images based on aero-optical effects. The technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0023] Figure 1 This is a schematic flowchart of an embodiment of the aero-optical effect sequence image restoration method provided by the present invention, as shown below. Figure 1 As shown, the method for restoring aero-optical effect sequence images includes: S101. Acquire a sequence of degraded images of the same target scene affected by aero-optical effects.

[0024] Specifically, this step acquires consecutive frame images of the same target scene affected by atmospheric turbulence, collected by the optical imaging system of a high-speed aircraft.

[0025] As an example, this embodiment uses a sequence of 20 frames of 24-bit BMP format color turbulence degradation images with an image size of 256×256 pixels for illustration.

[0026] It should be understood that this invention is also applicable to other image formats such as JPG and PNG, as well as image sequences with different frame numbers. The degraded image sequence contains random blurring caused by atmospheric turbulence, micro-shifts between frames, and weak imaging noise, but no significant motion blur or occlusion interference.

[0027] S102. Construct a comprehensive evaluation model for the quality of no-reference images, characterize the degradation features of each frame in the sequence of degraded images, remove invalid frames based on the degradation observation features, and select three types of key frames: low quality, medium quality, and high quality.

[0028] This step first evaluates the quality of each frame in the sequence, calculates multiple indicators that reflect image edge diffusion, local structural changes and high-frequency detail attenuation, and fuses them to obtain a comprehensive quality evaluation value.

[0029] Based on this evaluation value, severely degraded invalid frames are removed. Then, from the remaining valid frames, three representative images of the most severely degraded, moderately degraded, and least degraded types are selected and referred to as low-quality keyframes, medium-quality keyframes, and high-quality keyframes, respectively.

[0030] The specific evaluation and screening methods will be detailed in subsequent embodiments.

[0031] S103. Based on the degradation observation features corresponding to the three types of keyframes, generate three types of directional update quantities for the unified fuzzy kernel in terms of the support domain range, main structure and boundary details. Apply the three types of directional update quantities together to the iterative update of the unified fuzzy kernel to construct a fast solution model for the unified fuzzy kernel.

[0032] It should be noted that aero-optical blur kernels typically have characteristics such as spatial diffusion, main lobe concentration, and edge gradation.

[0033] Low-quality keyframes exhibit the most severe blur diffusion and are suitable for constraining the support domain range of the blur kernel; medium-quality keyframes show relatively stable structures and are suitable for maintaining the main lobe shape of the blur kernel; high-quality keyframes preserve the most complete details and are suitable for correcting the edges and high-frequency response of the blur kernel.

[0034] Based on this observation, this embodiment generates three types of directional update quantities, transforming the solution of the fuzzy kernel from the traditional large-scale blind search to a rapid convergence process of "first defining the range, then stabilizing the structure, and finally refining the details".

[0035] S104. Under the unified fuzzy kernel fast solution model, the potential clear image and the unified fuzzy kernel are jointly optimized and solved. The response difference between the three types of directional update quantities is adjusted through the consistency coordination mechanism, so that the unified fuzzy kernel converges stably and the restoration result is obtained.

[0036] When the three types of update quantities work together, some update quantities may have inconsistent directions or imbalances in intensity.

[0037] This embodiment introduces a consistency coordination mechanism to dynamically analyze the deviation of each update quantity from the overall optimization trend and adjust their weight accordingly, making the update process smoother and ultimately obtaining a stable unified fuzz kernel and a clear restored image. The specific solution and coordination process will be described in subsequent embodiments.

[0038] This embodiment achieves differentiated utilization of information on different degradation states in sequential images by screening three types of keyframes and assigning them different fuzzy kernel constraint tasks. At the same time, it adopts a directional update and consistency coordination strategy, which effectively improves the accuracy of fuzzy kernel estimation and the convergence stability of the solution, thereby improving the final image restoration quality.

[0039] In some embodiments of the present invention, such as Figure 2 As shown, step S101 acquires a sequence of degraded images of the same target scene affected by aero-optical effects, including: S201. Acquire initial sequence degraded images of the same target scene affected by atmospheric turbulence. The initial sequence degraded images contain random blurring, inter-frame micro-displacement and imaging noise caused by turbulence. S202. For each frame of the initial sequence of degraded images, a low-intensity two-dimensional Gaussian filter is used for denoising to suppress random noise while preserving turbulent degradation characteristics. S203. Using the first frame in the initial sequence degradation image as the reference frame, the image after denoising of each frame is registered using the sub-pixel level sparse optical flow method, and pixel-level spatial alignment is achieved through feature point matching and bilinear interpolation to obtain a spatially aligned sequence degradation image.

[0040] In step S201, the initial sequence contains random blurring caused by turbulence, inter-frame micro-displacement, and imaging noise.

[0041] To suppress noise while preserving turbulence degradation characteristics, step S202 applies a low-intensity two-dimensional Gaussian filter to each frame of the image for denoising. The standard deviation of the two-dimensional Gaussian kernel is typically between 0.5 and 1.5; in this embodiment, a value of [value missing] is preferred. σ =1.0, Gaussian kernel size is 3×3 pixels.

[0042] After denoising is completed, step S203 uses the first frame in the sequence as the reference frame and employs a sub-pixel level sparse optical flow method to register each frame.

[0043] Specifically, the Shi-Tomasi algorithm is first used to extract feature points from each frame of the image, with a feature point detection threshold set to 0.01 and a non-maximum suppression window of 5×5 pixels. Then, the Lucas-Kanade optical flow method is used to calculate the displacement of each feature point between the frame to be registered and the reference frame, with the optical flow window size set to 5×5 pixels, a maximum of 20 iterations, and a convergence threshold set according to the actual application requirements. For pixels in non-feature point regions, bilinear interpolation is used to spatially interpolate the optical flow field, thereby achieving sub-pixel-level pixel resampling across the entire image range to obtain a spatially aligned sequence of degraded images.

[0044] As an example, assuming the original sequence has 20 frames, with the first frame as a reference, the above registration operation is performed on the second to 20th frames respectively, and finally a spatially corresponding image sequence of 20 frames is obtained, which provides a spatial consistency basis for subsequent key frame screening and fuzzy kernel estimation.

[0045] This embodiment preserves the inherent turbulence degradation features of the image through low-intensity denoising, avoiding excessive smoothing; it eliminates inter-frame jitter through sub-pixel level registration, so that pixels at the same spatial location correspond to the same target point in different frames, thereby ensuring the accuracy of subsequent joint optimization solutions.

[0046] In some embodiments of the present invention, such as Figure 3 As shown, step S102 constructs a comprehensive evaluation model for the quality of a referenceless image, including: S301. Calculate the gradient energy index, gray-level local variance index, and Laplacian variance index of each frame in the sequence of degraded images. S302. Normalize the gradient energy index, gray-level local variance index and Laplacian variance index of each frame image to obtain the corresponding normalized feature components. S303. Based on the difference between the three normalized feature components and their mean values ​​of each frame image, determine the adaptive adjustment parameter for each feature component. S304. Using nonlinear geometric mean mapping, the adaptive adjustment parameters are fused with the corresponding normalized feature components to obtain the comprehensive quality evaluation value of each frame image.

[0047] The specific process of constructing the referenceless image quality comprehensive evaluation model in step S102 is as follows.

[0048] First, the gradient energy index of each frame in the sequence of degraded images is calculated.

[0049] Gradient energy is used to characterize the richness of image edge and texture information. In this embodiment, the Sobel operator is used to extract the gradient response in the horizontal and vertical directions, and the sum of the gradient magnitudes of all pixels is used as the gradient energy value.

[0050] A comprehensive evaluation function for the quality of a referenceless image is constructed for the registered image sequence. The gradient energy index is used to characterize the richness of image edge and texture information, and its calculation formula is as follows:

[0051] in, Indicates the first Gradient energy values ​​of frame-registered images, Indicates the first Frame-registered image at pixel location The grayscale value at that location; and The Sobel operator is respectively in direction and Convolution kernel in the direction; Represents a two-dimensional convolution operation; H and W These represent the height and width of the image, respectively. Among them, the Sobel operator is x direction and y The convolution kernels in each direction are 3×3 matrices, with the following specific values: .

[0052] Secondly, the local variance index of grayscale is calculated using the following formula. This metric is used to characterize the degree of grayscale variation in local areas of an image:

[0053] in, In pixels The variance of grayscale within a local window centered on the target. and These represent the height and width of the image, respectively.

[0054] Among them, the gray-level variance within the local window The calculation formula is: ( , )

[0055] in, and These represent the number of rows and columns of the local window, respectively. , , Represented by pixels local window centered Internal average gray value.

[0056] The specific method is as follows: For each pixel, take a 5×5 pixel local window centered on it, calculate the gray-level variance of all pixels within the window, and then average the local variance of all pixels in the whole image to obtain the gray-level local variance index of the whole frame image.

[0057] Next, calculate the Laplacian variance index. This index measures the high-frequency detail information of the image. Further, the Laplacian variance index, used to measure the high-frequency detail information of the image, is expressed as:

[0058] in: Indicates the first k The Laplacian variance index of frame-registered images; Indicates the first k Frame image at pixel position The response value obtained by the Laplace operator; This represents the mean of the Laplacian response of the image frame; Wherein, the mean of the Laplace response The calculation formula is:

[0059] in, The Laplace operator is used, and preferably, in this embodiment, a 3×3 Laplace operator is used.

[0060] After obtaining the three indicators, they are normalized to eliminate dimensional differences. The normalization method is a linear mapping, which involves subtracting the minimum value in the sequence from each indicator and then dividing by the difference between the maximum and minimum values.

[0061] To reflect that the contribution of each feature to the comprehensive evaluation result under different degradation conditions is not fixed but dynamically changes with the feature response relationship, this embodiment introduces an adaptive adjustment mechanism based on feature response consistency on the basis of normalization.

[0062] First, define the... k Mean values ​​of the three normalized feature components of a frame image:

[0063] Further define the consistency response quantity for the three types of features:

[0064]

[0065]

[0066] in, To prevent the use of tiny constants with a denominator of zero, this embodiment takes... .

[0067] Based on this, adaptive adjustment parameters for the three feature components are defined:

[0068]

[0069]

[0070] Therefore, when the three types of features respond relatively consistently in the same image, the corresponding consistency response is large, and the three types of features jointly participate in the comprehensive evaluation. When a certain feature deviates significantly from the other features, the weight of that feature is automatically reduced, thereby suppressing the interference of a single abnormal feature on the evaluation result.

[0071] Based on this, a comprehensive evaluation function is constructed, the expression of which is: =

[0072] in, Indicates the first k The overall quality evaluation value of the image after frame registration; , , They represent the first k The adaptive adjustment parameters corresponding to gradient energy, local variance, and Laplacian variance in the frame image, and the exponential... Used to perform normalized geometric mean calculation on three types of feature components.

[0073] The above fusion adopts a nonlinear geometric mean mapping method with response consistency adjustment capability to uniformly characterize image edge diffusion, local structure stability and high-frequency detail attenuation characteristics; when any feature component decreases or a feature response is significantly inconsistent with the other features, the comprehensive evaluation value decreases accordingly.

[0074] It should be noted that the comprehensive evaluation function is not only used for image quality evaluation, but also serves as an observational expression function for the characteristics of the blur kernel. The lower the comprehensive evaluation value, the stronger the spatial diffusion effect of the blur kernel, the more obvious the edge blurring, and the more severe the loss of high-frequency details. The higher the comprehensive evaluation value, the weaker the diffusion effect of the blur kernel, the more stable the structure, and the more sufficient the detail information is preserved.

[0075] This embodiment, through multi-index fusion and adaptive weighting, can robustly evaluate image quality under different degradation levels, avoiding the problem that single indexes are easily affected by noise or specific scene biases, and providing a reliable quantitative basis for subsequent keyframe selection.

[0076] In some embodiments of the present invention, such as Figure 4 As shown, step S102 removes invalid frames based on degraded observation characteristics, resulting in three categories of keyframes: low-quality, medium-quality, and high-quality, including: S401. Obtain the quality threshold coefficient, and determine and remove frames whose comprehensive quality evaluation value is less than the quality threshold coefficient as invalid frames. S402. Based on the maximum value of the comprehensive quality evaluation value and the quality threshold coefficient in the effective frame set, determine the upper bound of the low quality interval and the lower bound of the high quality interval, and divide the effective frame set into a low quality frame candidate set, a medium quality frame candidate set and a high quality frame candidate set. S403. Select the frame with the smallest comprehensive quality evaluation value from the low-quality frame candidate set as the low-quality key frame, select the frame with the largest comprehensive quality evaluation value from the high-quality frame candidate set as the high-quality key frame, and select the frame with the comprehensive quality evaluation value closest to the center value of the medium-quality interval from the medium-quality frame candidate set as the medium-quality key frame.

[0077] In this embodiment, the specific process of removing invalid frames and screening three types of key frames based on the degradation observation characteristics in step S102 is as follows.

[0078] A quality threshold coefficient is obtained. Frames with a comprehensive quality evaluation value less than this threshold are determined as invalid frames and directly discarded, without participating in subsequent processing. The remaining valid frames form a valid frame set.

[0079] In one embodiment, the quality threshold coefficient is calculated using the following formula: ,because After normalization and fusion, the value range is [0,1]. As a preferred method, this embodiment takes... =0.20.

[0080] Based on the aforementioned thresholds, the valid frames are divided into three candidate sets. Next, representative frames are selected from each candidate set as keyframes. For the low-quality candidate set, the frame with the smallest overall evaluation value (i.e., the most severely degraded frame) is selected as the low-quality keyframe. For the high-quality candidate set, the frame with the largest overall evaluation value (i.e., the best quality frame) is selected as the high-quality keyframe. For the medium-quality candidate set, the center value of the medium-quality interval is first calculated, and then the frame with the overall evaluation value closest to that center value is selected as the medium-quality keyframe.

[0081] If a certain frame's overall evaluation value If the frame is invalid, it is considered an invalid frame.

[0082] Based on this, and using the comprehensive evaluation function and its corresponding degraded observation feature components, for the effective frame set Perform degradation stratification.

[0083] A smaller value indicates a stronger spatial diffusion effect of the blur kernel, more obvious edge blurring, and more severe loss of high-frequency details; The larger the value, the weaker the diffusion effect of the fuzzy kernel, the more stable the structure, and the more detailed information is preserved. Therefore, it can be used as a basis for... The value range is used to classify valid frames into three categories: low quality, medium quality, and high quality.

[0084] Specifically, define the upper bound of the low-quality interval. Lower bound of high quality range They are respectively:

[0085]

[0086] in, The threshold for removing invalid frames; This is the maximum comprehensive evaluation value among the valid frame set. Preferably, in this embodiment... =0.82, 0.406, =0.61.

[0087] Therefore, the set of valid frames is divided into three degradation intervals: Low-quality frame candidate set: denoted as:

[0088] Medium-quality frame candidate set:

[0089] High-quality frame candidate set:

[0090] Furthermore, to avoid selecting frames solely based on the comprehensive evaluation value, representative selection criteria are constructed for each of the three candidate sets.

[0091] For low-quality keyframes, the sample with the smallest overall evaluation value and the most significant edge diffusion feature is selected, denoted as:

[0092] For high-quality keyframes, the sample with the highest overall evaluation value and the most complete preservation of high-frequency details is selected and denoted as:

[0093] For medium-quality keyframes, to ensure they can represent relatively stable structural information under moderate degradation, samples whose comprehensive evaluation value is closest to the center value of the medium-quality interval are selected from the medium-quality candidate set, denoted as:

[0094]

[0095] in This represents the center evaluation value of the medium quality interval.

[0096] This example =0.508, thus constructing a keyframe set. .

[0097] This embodiment uses a hierarchical screening and representative selection strategy to ensure that low-quality keyframes can fully reflect the diffusion effect of the blur kernel, medium-quality keyframes can reflect a stable main structure, and high-quality keyframes can retain rich boundary details, providing differentiated observation basis for the generation of subsequent directional update quantities.

[0098] In some embodiments of the present invention, step S103 generates three types of directional update quantities for the unified fuzzy kernel in terms of the support domain range, main structure, and boundary details, based on the degradation observation features corresponding to the three types of keyframes, including: Based on low-quality keyframes, the support domain determination threshold is determined according to the peak response of the unified fuzzy kernel in the current iteration, an effective support domain mask is constructed, and a support domain range update is generated. The support domain range update only applies to the position indicated by the effective support domain mask. Based on medium-quality keyframes, the main structure determination threshold is determined according to the peak response of the unified fuzzy kernel in the current iteration, the main structure region mask is constructed, and the main structure update amount is generated. The main structure update amount is used to stabilize the main lobe structure morphology of the unified fuzzy kernel. Based on high-quality keyframes, a boundary detail region mask is constructed. The boundary detail region mask is the difference between the effective support domain mask and the main structure region mask. Boundary detail update amount is generated and used to correct the edge morphology and high-frequency response characteristics of the unified blur kernel.

[0099] In this embodiment, the specific method for generating the three types of directional update quantities in step S103 is as follows.

[0100] First, an effective support domain mask for the unified fuzzy kernel is defined. This mask is used to mark regions with larger values ​​in the fuzzy kernel, i.e., the diffusion range of concentrated energy. In each iteration, a support domain decision threshold is determined based on the peak response of the current fuzzy kernel. ,in The scaling factor for the support domain is set to 0.10 in this embodiment. The value in the fuzzy kernel is greater than or equal to... The positions are marked as 1, and the rest as 0, thus obtaining the effective support domain mask. M .

[0101] The support domain extent update is generated based on low-quality keyframes. This update is calculated only at the location indicated by the valid support domain mask, and the calculation formula is as follows: ,in To support the domain update step size (0.08 in this embodiment), This indicates element-wise multiplication. Represents the unified fuzzy kernel h The partial derivative operation is used to quickly lock the spatial diffusion range of the fuzzy kernel, reducing the variable space in subsequent solutions.

[0102] Secondly, a mask for the main structure region is defined. This mask corresponds to the main lobe region in the blur kernel where energy is highly concentrated. A threshold for determining the main structure is then set. ,in Set to 0.50. Values ​​in the fuzzy kernel are greater than or equal to... The position is marked as 1, and the rest are 0, thus obtaining the mask of the main structure region. S .

[0103] The main structure update is generated based on medium-quality keyframes. This update applies to the entire blur kernel, but mainly affects the main lobe region. Its calculation formula is as follows: ,in The structure update step size is set to 0.05 in this embodiment. This represents the gradient operator. This update amount is used to stabilize the main lobe structure of the blur kernel and suppress kernel center shift and structural distortion.

[0104] Next, define the boundary detail region mask. B = M SThat is, the region remaining after removing the main structure within the effective support domain, corresponding to the transition zone and edge part of the fuzzy kernel.

[0105] The boundary detail update amount is generated based on high-quality keyframes, and the calculation formula is as follows: ,in The update step size is set to 0.02 in this embodiment. ² represents the Laplacian operator. This update applies only to the boundary detail region and is used to correct the edge morphology and high-frequency response characteristics of the blur kernel.

[0106] It should be noted that the above update amounts are all based on the gradient of the joint optimization objective function with respect to the fuzzy kernel. L / h The specific form of this gradient is determined by the subsequent joint optimization objective function.

[0107] This embodiment decomposes the task of solving the fuzzy kernel into three relatively independent subtasks: support domain locking, main structure stabilization, and edge detail correction. Each subtask is driven by keyframes of different qualities, so that each subtask has targeted observation information as guidance, thereby significantly improving the accuracy and efficiency of fuzzy kernel estimation.

[0108] In some embodiments of the present invention, the mathematical form of the unified degradation model is further clarified before describing step S103, which generates the three types of directional update quantities. This embodiment is based on the selected set of keyframes. A unified degradation model is established as follows:

[0109] In the formula, For the potential clear image to be estimated, The unified fuzzy kernel to be solved. This represents the convolution operation in a two-dimensional discrete spatial domain. The remaining noise in each keyframe after preprocessing. Since the three keyframes originate from the same target scene and are affected by the same aero-optical blurring mechanism, they can be considered to share the same potentially sharp image. and the same unified fuzzy kernel The differences lie only in the degree of degradation and the observation state. This model maps observation information under different degradation states to the same parameter space, providing a common mathematical basis for subsequent fuzzy kernel estimation.

[0110] In some embodiments of the present invention, such as Figure 5 As shown, step S103 adjusts the response differences among the three types of directional update quantities through a consistency coordination mechanism, including: S501. Calculate the average value of the directional update amount on the support domain, the directional update amount on the main structure, and the directional update amount on the boundary details, as the overall optimization trend. S502. Calculate the degree of deviation of each type of directional update quantity from the overall optimization trend based on the degree of directional deviation and the degree of magnitude deviation. S503. Based on the degree of deviation, the consistency coordination weight of each type of directional update is calculated using an exponential mapping function, so that the directional update with a greater degree of deviation receives a smaller coordination weight. S504. Use consistency coordination weights to weight the corresponding directional update amounts, and then apply them to the iterative update of the unified fuzzy kernel.

[0111] Let the support domain update response, main structure update response, and detail update response be denoted as follows: .

[0112] First, calculate the average of the three update responses as the overall optimization trend: .

[0113] Next, for each type of update response, its deviation from the overall trend is calculated. The deviation considers both directional and magnitude differences, and the formula is as follows: ,in k ∈{ s , m , f}, γ 1 and γ 2 is the adjustment coefficient, which is set to 0.6 and 0.4 in this embodiment. The first term reflects the directional deviation, and the second term reflects the amplitude deviation.

[0114] After obtaining the degree of deviation, the consistency coordination weight for each type of update response is calculated using an exponential mapping function: ,in For sensitivity adjustment, this embodiment uses a value of 1.0. Indicates the first Consistency weights for each keyframe constraint; This represents the response intensity obtained by mapping the degree of deviation; This represents the normalization factor for the intensity of all keyframe responses. This formula ensures that responses with greater deviations receive smaller weights.

[0115] Finally, the calculated consistency coordination weights are used to weight the three types of update responses, and then the weighted update amounts are superimposed on the current fuzzy kernel.

[0116] This embodiment dynamically coordinates the contributions of the three types of update quantities, transforming their potential competitive relationship into a cooperative one. This effectively suppresses oscillations and divergences during the iteration process, and improves the stability and convergence speed of the fuzzy kernel solution.

[0117] In some embodiments of the present invention, such as Figure 6 As shown, step S104 performs joint optimization of the potentially sharp image and the unified blur kernel, including: S601. Use high-quality keyframes as the initial values ​​for the potential sharp image, and use the unit impulse function as the initial value for the unified blur kernel. S602. In each iteration, fix the current unified blur kernel and use the frequency domain deconvolution method to solve for the potential sharp image; S603. The potential clear image obtained by fixing the solution is updated according to the unified fuzzy kernel fast solution model and consistency coordination mechanism; S604. Project the updated unified fuzzy kernel onto the feasible region that satisfies the nonnegativity constraint and the normalization constraint.

[0118] In this embodiment, the specific iterative process of jointly optimizing the potential sharp image and the unified blur kernel in step S104 is as follows.

[0119] First, perform initialization: set high-quality keyframes. As initial values ​​for a potentially sharp image X (0) The unit impulse function δ(i,j) As the initial value of the unified fuzzy kernel h (0) The center position of the unit impulse function is set to 1, and the rest are set to 0. The size of the blur kernel is set to 21×21 pixels in this embodiment.

[0120] Furthermore, in the joint optimization solution process, this embodiment constructs the following objective function: = +

[0121] in, The fuzzy kernel constraint term is formed by the three types of directional update quantities. For potentially sharp images, To unify the regularization terms of the fuzzy kernel.

[0122] Specifically, the image regularization term adopts a second-order norm form to suppress noise and maintain the continuity of image structure: = 1 ( + ) In the formula 1 represents the image smoothing constraint weight. and These represent the gradients of the image in the horizontal and vertical directions, respectively. In this embodiment, The value of 1 is 0.01.

[0123] The fuzzy kernel regularization term also adopts the second-order norm form to suppress noise disturbances in the fuzzy kernel estimation process: In the formula μ For fuzzy kernel smoothing constraint weights, h This represents the fuzzy kernel gradient. In this embodiment, μ The value is 0.005.

[0124] After entering the iteration loop, at the... t In the next iteration, the current fuzzy kernel is first fixed. h (t) Solving for the latent sharp image This embodiment uses the frequency domain deconvolution method for solving the problem, that is, deconvolving the objective function with respect to frequency domain. X By differentiating and setting the derivative to zero, the analytical solution in the frequency domain is obtained. Then, a clear image in the spatial domain is obtained through inverse Fourier transform. This process utilizes the weighted information from three keyframes, where the consistency coordination weights... This refers to the weights calculated above.

[0125] It should be noted that this embodiment employs a frequency domain deconvolution method when solving for potentially sharp images with a fixed blur kernel. Specifically, by adjusting the joint optimization objective function with respect to... X Taking the derivative and setting it to zero, we obtain the analytical solution in the frequency domain:

[0126] in, Indicates Fourier transform, Indicates the inverse Fourier transform. This represents the frequency domain representation of the fuzzy kernel in the t-th iteration. For their conjugate, The square of the spectral amplitude, For consistency and coordination weights, This is the frequency domain representation of the keyframe image. The regularization parameter in frequency domain deconvolution (similar to the regularization term in image regularization) The value is consistent; in this embodiment, it is 0.01.

[0127] This formula utilizes weighted information from three keyframes, with the weights dynamically determined by a consistency coordination mechanism. Employing a frequency domain method can significantly improve computational efficiency and avoid the large computational burden of spatial domain iterative solutions.

[0128] Then, fix the potential sharp image X obtained just from the solution. (t+ ¹ ) The fuzzy kernel is updated based on the unified fuzzy kernel fast solution model and its consistency coordination mechanism to obtain the following results. The specific update formula adopts the aforementioned weighted directional update method. Based on the unified fuzzy kernel fast update model constructed in the previous steps and the above consistency coordination weights, the unified fuzzy kernel is iteratively updated, and its expression is: =

[0129] in: ( () denotes the projection operator, used to project the updated unified fuzzy kernel onto a feasible region that satisfies physical constraints. Specifically, the projection process includes truncating the negative values ​​of the updated fuzzy kernel to 0 and normalizing all kernel elements to satisfy nonnegativity and energy conservation constraints.

[0130] After the update is complete, the fuzz kernel is projected onto a feasible region Ω that satisfies the physical constraints. The feasible region requires that all elements of the fuzz kernel be non-negative and that the sum of all elements is 1 (energy conservation). The projection operation includes truncating negative values ​​to 0 and then dividing all elements by their sum. In this embodiment, the fuzz kernel size is 21×21, ensuring that the fuzz kernel has a clear physical meaning after projection.

[0131] Repeat the above iterative process until the convergence condition is met.

[0132] Furthermore, the feasible region The following constraints must be met: } in, For the fuzzy kernel at position The value of ; , The size of the fuzzy kernel is M = N = 21 in this embodiment, meaning the fuzzy kernel size is 21 × 21. This constraint ensures that the fuzzy kernel satisfies the physical interpretability and energy conservation properties.

[0133] This embodiment uses an alternating optimization strategy to decompose the complex joint estimation problem into two relatively simple sub-problems to be solved separately. It uses frequency domain deconvolution to improve the computational efficiency of solving for clear images and uses projection operations to ensure the physical rationality of the blur kernel, thereby accelerating the overall convergence speed while ensuring accuracy.

[0134] In some embodiments of the present invention, such as Figure 7 As shown, step S104 stabilizes and converges the unified fuzzy kernel to obtain the restored result, including: S701. After each iteration, calculate the change in the unified blur kernel, the change in the potential sharp image, and the change in the joint optimization objective function; S702. When the changes in the unified blur kernel, the potential sharp image, and the joint optimization objective function are all less than the corresponding convergence thresholds, the iteration is determined to be converged, the iteration is terminated, and the current potential sharp image is output as the restoration result.

[0135] In this embodiment, the specific method for determining the convergence of the iteration and outputting the restoration result in step S104 is as follows.

[0136] After each iteration, calculate: the change in the unified fuzzy kernel. Δh = ||h (t+ ¹ ) h (t) || 2. The amount of change in the potentially sharp image ΔX = ||X (t+ ¹ ) X (t) || 2. Change in the objective function .when ≤ , ≤ and If the iteration converges, the optimization is terminated; otherwise, the iteration count t is increased by 1, and the next iteration continues. To standardize the convergence threshold of fuzzy kernels, For potential sharp images, the convergence threshold is... This represents the convergence threshold of the joint optimization objective function, used to measure the degree of change in the objective function value between two adjacent iterations, thereby determining whether the overall optimization process has reached a stable state.

[0137] Preferably, in this embodiment, =1× , =1× , =1× Under the above parameter settings, in this embodiment, the convergence condition is met when iterating to the 12th round, resulting in the final unified fuzzy kernel. = and the corresponding potentially clear images, = .

[0138] In the test of this embodiment, a sequence of 20 turbulence degradation images of 256×256 pixels was used. With the parameters set above, by the 12th iteration, all three changes were less than the threshold, satisfying the convergence condition. At this point, the potentially clear image obtained from the last iteration was used. X (t+ ¹ ) This will be the final output of the restoration.

[0139] If convergence is not achieved after reaching the preset maximum number of iterations (e.g., 30), the process will be forcibly terminated and the current result will be output, while a non-convergence message will be recorded.

[0140] This embodiment can accurately determine whether the optimization process has reached a stable state by simultaneously monitoring changes in three aspects: the blur kernel, the sharp image, and the objective function. This avoids premature termination leading to insufficient restoration or excessive iteration causing computational waste, and ensures a balance between the quality of the output results and the efficiency of the algorithm.

[0141] To better implement the aerodynamic optical effect sequence image restoration method in this embodiment of the invention, based on the aerodynamic optical effect sequence image restoration method, correspondingly, as follows: Figure 8 As shown, this embodiment of the invention also provides a pneumatic optical effect sequence image restoration device, the pneumatic optical effect sequence image restoration device 800 comprising: The sequence acquisition module 801 is used to acquire a sequence of degraded images of the same target scene affected by aero-optical effects; The quality assessment module 802 is used to construct a comprehensive quality evaluation model for no-reference images, characterize the degradation features of each frame in the sequence of degraded images, remove invalid frames based on the degradation observation features, and select three types of key frames: low quality, medium quality, and high quality. The model building module 803 is used to generate three types of directional update quantities of the unified fuzzy kernel in the support domain range, main structure and boundary details according to the degradation observation characteristics corresponding to the three types of key frames, and to apply the three types of directional update quantities together to the iterative update of the unified fuzzy kernel to build a fast solution model of the unified fuzzy kernel. The image restoration module 804 is used to jointly optimize the potential clear image and the unified fuzzy kernel under the unified fuzzy kernel fast solution model, and adjust the response differences among the three types of directional update quantities through a consistency coordination mechanism, so that the unified fuzzy kernel converges stably and the restoration result is obtained.

[0142] The aero-optical effect sequence image restoration device 800 provided in the above embodiments can realize the technical solutions described in the above aero-optical effect sequence image restoration method embodiments. The specific implementation principles of each module or unit can be found in the corresponding content in the above aero-optical effect sequence image restoration method embodiments, which will not be repeated here.

[0143] like Figure 9 As shown, the present invention also provides an electronic device 900. The electronic device 900 includes a processor 901, a memory 902, a display 903, and an image acquisition device 904. Figure 9 Only some components of the electronic device 900 are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0144] In some embodiments, processor 901 may be a central processing unit (CPU), microprocessor, or other data processing chip, used to run program code stored in memory 902 or process data, such as the aero-optical effect sequence image restoration method of the present invention.

[0145] In some embodiments, processor 901 may be a single server or a group of servers. The server group may be centralized or distributed. In some embodiments, processor 901 may be local or remote. In some embodiments, processor 901 may be implemented on a cloud platform. In one embodiment, the cloud platform may include a private cloud, public cloud, hybrid cloud, community cloud, distributed cloud, internal cloud, multi-cloud, etc., or any combination thereof.

[0146] In some embodiments, memory 902 may be an internal storage unit of electronic device 900, such as a hard disk or memory of electronic device 900. In other embodiments, memory 902 may also be an external storage device of electronic device 900, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. equipped on electronic device 900.

[0147] Furthermore, the memory 902 may include both internal storage units of the electronic device 900 and external storage devices. The memory 902 is used to store application software and various types of data installed on the electronic device 900.

[0148] In some embodiments, display 903 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. Display 903 is used to display information from electronic device 900 and to display a visual user interface.

[0149] In some embodiments, the image acquisition device 904 can be any one or a combination of a high-speed camera, an infrared imager, a charge-coupled device (CCD) camera, or a complementary metal-oxide-semiconductor (CMOS) camera. The image acquisition device 904 is mounted at the optical imaging window of the high-speed aircraft and is used to acquire in real time initial sequence degraded images of the same target scene affected by atmospheric turbulence and aero-optical effects.

[0150] In some embodiments, the output of the image acquisition unit 904 is coupled to the input of the processor 901, transmitting the acquired initial sequence of degraded images to the processor 901 in the form of a digital signal. The initial sequence of degraded images includes random blurring caused by turbulence, inter-frame micro-displacement, and imaging noise.

[0151] In some embodiments, the image acquisition device 904 can be configured in a continuous acquisition mode, with the acquisition frame rate dynamically adjusted according to the aircraft speed and target distance to obtain a sufficient number of sequential images with degradation differences. For example, in a high-speed flight scenario, the acquisition frame rate can be set from 100 frames / second to 1000 frames / second.

[0152] In some embodiments, the image acquisition device 904 further includes an optical lens assembly, which may be equipped with a filter to suppress interference light in a specific wavelength band and improve the image signal-to-noise ratio.

[0153] In some embodiments, the image acquisition unit 904 and the processor 901 can be integrated on the same chip to form a system on chip (SoC) to meet the requirements of miniaturization, low power consumption and real-time processing.

[0154] In one embodiment, when the processor 901 executes the aero-optical effect sequence image restoration program in the memory 902, the following steps can be implemented: Acquire a sequence of degraded images of the same target scene affected by aero-optical effects; A comprehensive evaluation model for the quality of no-reference images is constructed. Degradation features are characterized for each frame in the sequence of degraded images. Invalid frames are removed based on the degradation observation features, and three categories of key frames, namely low quality, medium quality and high quality, are selected. Based on the degradation observation features corresponding to the three types of keyframes, three types of directional update quantities of the unified fuzzy kernel are generated in the support domain range, main structure and boundary details. The three types of directional update quantities are applied together to the iterative update of the unified fuzzy kernel to construct a fast solution model for the unified fuzzy kernel. Under the unified fuzzy kernel fast solution model, the potential sharp image and the unified fuzzy kernel are jointly optimized and solved. The response differences between the three types of directional update quantities are adjusted through a consistency coordination mechanism, so that the unified fuzzy kernel converges stably and the restoration result is obtained.

[0155] It should be understood that when the processor 901 executes the aero-optical effect sequence image restoration program in the memory 902, in addition to the functions mentioned above, it can also perform other functions, as can be found in the description of the corresponding method embodiments above.

[0156] Furthermore, the embodiments of the present invention do not specifically limit the type of the electronic device 900 mentioned. The electronic device 900 can be a mobile phone, tablet computer, personal digital assistant (PDA), wearable device, laptop computer, or other portable electronic device. Exemplary embodiments of portable electronic devices include, but are not limited to, portable electronic devices running iOS, Android, Microsoft, or other operating systems. The aforementioned portable electronic device can also be other portable electronic devices, such as a laptop computer with a touch-sensitive surface (e.g., a touch panel). It should also be understood that in some other embodiments of the present invention, the electronic device 900 may not be a portable electronic device, but rather a desktop computer with a touch-sensitive surface (e.g., a touch panel).

[0157] Accordingly, this application also provides a computer-readable storage medium for storing computer-readable programs or instructions. When the programs or instructions are executed by a processor, they can implement the steps or functions of the aero-optical effect sequence image restoration method provided in the above-described method embodiments.

[0158] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware (such as a processor, controller, etc.), and the computer program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0159] The above provides a detailed description of the aero-optical effect sequence image restoration method, apparatus, and electronic device provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for restoring a sequence of images based on aero-optical effects, characterized in that, include: Acquire a sequence of degraded images of the same target scene affected by aero-optical effects; A comprehensive evaluation model for the quality of no-reference images is constructed. Degradation features are characterized for each frame in the sequence of degraded images to obtain degradation observation features. Invalid frames are removed based on the degradation observation features, and three types of key frames, namely low quality, medium quality and high quality, are selected. Based on the degradation observation features corresponding to the three types of keyframes, three types of directional update quantities of the unified fuzzy kernel are generated in the support domain range, main structure and boundary details. The three types of directional update quantities are applied together to the iterative update of the unified fuzzy kernel to construct a fast solution model for the unified fuzzy kernel. Under the unified fuzzy kernel fast solution model, the potential sharp image and the unified fuzzy kernel are jointly optimized and solved. The response differences between the three types of directional update quantities are adjusted through a consistency coordination mechanism, so that the unified fuzzy kernel converges stably and the restoration result is obtained.

2. The method for restoring a sequence of aero-optical effects according to claim 1, characterized in that, The acquisition of a sequence of degraded images of the same target scene affected by aero-optical effects includes: Acquire initial sequence degraded images of the same target scene affected by atmospheric turbulence, wherein the initial sequence degraded images contain random blurring, inter-frame micro-displacement and imaging noise caused by turbulence; For each frame of the initial sequence of degraded images, a low-intensity two-dimensional Gaussian filter is used for denoising to suppress random noise while preserving turbulent degradation characteristics. Using the first frame in the initial sequence of degraded images as the reference frame, the denoised images of each frame are registered using a subpixel-level sparse optical flow method, and pixel-level spatial alignment is achieved through feature point matching and bilinear interpolation to obtain the spatially aligned sequence of degraded images.

3. The method for restoring a sequence of aero-optical effects according to claim 1, characterized in that, The construction of the referenceless image quality comprehensive evaluation model includes: Calculate the gradient energy index, gray-level local variance index, and Laplacian variance index for each frame in the sequence of degraded images; The gradient energy index, gray-level local variance index, and Laplacian variance index of each frame image are normalized to obtain the corresponding normalized feature components. Based on the difference between the three normalized feature components and their mean values ​​of each frame image, the adaptive adjustment parameters of each feature component are determined. By employing a nonlinear geometric mean mapping, the adaptive adjustment parameters are fused with the corresponding normalized feature components to obtain the comprehensive quality evaluation value of each frame image.

4. The method for restoring a sequence of aero-optical effects according to claim 3, characterized in that, The process involves removing invalid frames based on degraded observation characteristics, resulting in three categories of keyframes: low-quality, medium-quality, and high-quality. Obtain a quality threshold coefficient, and determine and remove frames whose comprehensive quality evaluation value is less than the quality threshold coefficient as invalid frames. Based on the maximum value of the comprehensive quality evaluation value in the effective frame set and the quality threshold coefficient, the upper bound of the low quality interval and the lower bound of the high quality interval are determined, and the effective frame set is divided into a low quality frame candidate set, a medium quality frame candidate set and a high quality frame candidate set. The frame with the smallest overall quality evaluation value is selected from the low-quality frame candidate set as the low-quality keyframe, the frame with the largest overall quality evaluation value is selected from the high-quality frame candidate set as the high-quality keyframe, and the frame with the overall quality evaluation value closest to the center value of the medium-quality interval is selected from the medium-quality frame candidate set as the medium-quality keyframe.

5. The method for restoring a sequence of aero-optical effects images according to any one of claims 1 to 4, characterized in that, The process involves generating three types of directional update quantities for the unified blur kernel in terms of the support domain range, main structure, and boundary details, based on the degradation observation features corresponding to the three types of keyframes. Based on the low-quality keyframes, the support domain determination threshold is determined according to the peak response of the unified fuzzy kernel in the current iteration, an effective support domain mask is constructed, and a support domain range update is generated. The support domain range update only applies to the position indicated by the effective support domain mask. Based on the medium-quality keyframes, the main structure determination threshold is determined according to the peak response of the unified fuzzy kernel in the current iteration, the main structure region mask is constructed, and the main structure update amount is generated. The main structure update amount is used to stabilize the main lobe structure morphology of the unified fuzzy kernel. Based on the high-quality keyframes, a boundary detail region mask is constructed. The boundary detail region mask is the difference between the effective support domain mask and the main structure region mask. A boundary detail update amount is generated, which is used to correct the edge morphology and high-frequency response characteristics of the unified blur kernel.

6. The method for restoring a sequence of aero-optical effects images according to any one of claims 1 to 4, characterized in that, The method of adjusting the response differences among the three types of directional update quantities through a consistency coordination mechanism includes: Calculate the average of the directional update amount over the support domain, the directional update amount over the main structure, and the directional update amount over the boundary details, and use this as the overall optimization trend; Calculate the degree of deviation of each type of targeted update quantity from the overall optimization trend based on the degree of directional and magnitude deviation of each type of targeted update quantity. Based on the degree of deviation, an exponential mapping function is used to calculate the consistency coordination weight of each type of directional update, so that the directional update with a greater degree of deviation receives a smaller coordination weight. The corresponding directional update amount is weighted using the consistency coordination weight, and then applied to the iterative update of the unified fuzzy kernel.

7. The method for restoring a sequence of aero-optical effects images according to any one of claims 1 to 4, characterized in that, The joint optimization solution of the potentially sharp image and the unified blur kernel includes: The high-quality keyframes are used as the initial values ​​for the potential sharp image, and the unit impulse function is used as the initial value for the unified blur kernel. In each iteration, the current unified blur kernel is fixed, and the frequency domain deconvolution method is used to solve for the potential sharp image; The potentially clear image obtained by fixing the solution is used to update the unified fuzzy kernel according to the unified fuzzy kernel fast solution model and consistency coordination mechanism; The updated unified fuzzy kernel is projected onto the feasible region that satisfies the nonnegativity and normalization constraints.

8. The method for restoring a sequence of aero-optical effects according to claim 7, characterized in that, The process of making the unified fuzzy kernel stable and converge to obtain the restoration result includes: After each iteration, the changes in the unified blur kernel, the changes in the potentially sharp image, and the changes in the joint optimization objective function are calculated. When the changes in the unified blur kernel, the potential sharp image, and the joint optimization objective function are all less than the corresponding convergence thresholds, the iteration is determined to be converged, the iteration is terminated, and the current potential sharp image is output as the restoration result.

9. A device for restoring a sequence of images using aero-optical effects, characterized in that, include: The sequence acquisition module is used to acquire a sequence of degraded images of the same target scene affected by aero-optical effects; The quality assessment module is used to construct a comprehensive quality evaluation model for no-reference images, characterize the degradation features of each frame in the sequence of degraded images, remove invalid frames based on the degradation observation features, and select three categories of key frames: low quality, medium quality, and high quality. The model building module is used to generate three types of directional update quantities of the unified fuzzy kernel in the support domain range, main structure and boundary details according to the degradation observation characteristics corresponding to the three types of keyframes respectively. The three types of directional update quantities are used together to iteratively update the unified fuzzy kernel to build a fast solution model of the unified fuzzy kernel. The image restoration module is used to jointly optimize the potential sharp image and the unified fuzzy kernel under the unified fuzzy kernel fast solution model, and adjust the response differences among the three types of directional update quantities through a consistency coordination mechanism, so that the unified fuzzy kernel converges stably and the restoration result is obtained.

10. An electronic device, characterized in that, Includes an image acquisition unit, memory, and processor, among which, An image acquisition device is used to acquire initial sequence degraded images of the same target scene affected by aero-optical effects; The memory is used to store programs; The processor, coupled to the memory, is used to execute the program stored in the memory to implement the steps in the aero-optical effect sequence image restoration method according to any one of claims 1 to 8.