Image restoration method and device and storage medium

Abstract

The invention discloses an image restoration method, an image restoration device and a storage medium, which are used for improving the structure restoration precision and the detail restoration capability of image restoration. The method comprises the following steps: acquiring multi-dimensional inertial data in real time; performing frequency domain analysis on the multi-dimensional inertial data by adopting sliding window short-time Fourier transform to obtain vibration intensity; if the vibration intensity does not exceed the preset threshold value, acquiring an image; calculating a definition index of the image; determining whether the image is a blurred image according to a preset definition standard and the definition index of the image; if the image is judged to be a blurred image, dividing the blurred image into a motion blurred image and a focusing blurred image; respectively constructing point spread function models of the motion blurred image and the focusing blurred image; performing deconvolution processing or depth reconstruction on the blurred image through a point spread function model to obtain a clear image; and recalculating the definition index of the clear image, and if the definition index does not exceed the definition threshold, triggering reacquisition or switching the repair model to execute secondary repair.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image acquisition and processing, and in particular to an image restoration method, device, and storage medium. Background Art

[0002] With the development of technology, image quality self-inspection and blur repair are gaining increasing attention. In the field of intelligent manufacturing, high-precision product inspection relies on clear image data to identify minor defects. If the image is blurred due to equipment vibration or conveyor belt sway, defective products may be missed, directly affecting production efficiency and product quality. In autonomous driving scenarios, if the road images captured by on-board cameras contain motion blur or focus deviation, it will interfere with key functions such as lane recognition and obstacle detection, posing a serious threat to driving safety.

[0003] However, the blur repair methods in the existing technology separate the acquisition and repair processes, making it impossible for the image acquisition stage to provide effective information support for subsequent repairs. During acquisition, the device neither perceives its own motion state nor records the environmental parameters at the moment of imaging, resulting in the repair stage having to blindly process the blurred image. Due to the lack of motion data during acquisition, it is difficult for the repair algorithm to determine whether the blur is due to motion blur caused by device vibration or focus blur caused by focusing errors, and it can only use a general model for trial repairs. In addition, this separation also makes the entire process lack closed-loop optimization capabilities. The acquisition stage cannot adjust parameters according to the repair requirements, and the repair stage cannot feedback acquisition defects.

[0004] Traditional blur restoration methods separate the acquisition and restoration processes, resulting in less targeted image restoration. Lacking motion and environmental data at the time of acquisition, the algorithm consumes significant computing resources to repeatedly try different restoration strategies, resulting in inefficient restoration. Misjudging the blur type can lead to oversharpening, artifact enhancement, or loss of detail, making the restoration unreliable or even worse than the original image. Furthermore, the system suffers from poor adaptability. The acquisition device cannot adjust parameters based on restoration requirements, and the restoration algorithm cannot guide hardware optimization, resulting in poor performance in complex scenarios. Summary of the Invention

[0005] The present application discloses an image restoration method, device and storage medium for improving the structural restoration accuracy and detail recovery capability of image restoration.

[0006] The first aspect of the present application discloses an image restoration method, comprising: Acquire multi-dimensional inertial data in real time through an inertial measurement unit, wherein the multi-dimensional inertial data includes a three-axis acceleration and angular velocity data stream; Performing frequency domain analysis on the multidimensional inertial data using a sliding window short-time Fourier transform to obtain vibration intensity; If the vibration intensity does not exceed a preset threshold, capturing an image; Calculating a clarity index of the image; determining whether the image is a blurred image according to a preset clarity standard and a clarity index of the image; If the image is determined to be a blurred image, the blurred image is divided into a motion blurred image and a focus blurred image; constructing point spread function models of the motion blurred image and the focus blurred image respectively; Performing deconvolution processing or depth reconstruction on the blurred image using the point spread function model to obtain a clear image; The clarity index of the clear image is recalculated, and if it does not exceed the clarity threshold, recapture is triggered or the restoration model is switched to perform secondary restoration.

[0007] Optionally, calculating the clarity index of the image includes: Gray-scaling the image and cropping the ROI region to obtain a sub-region image; Subtracting the sub-region image from the mean image pixel by pixel to obtain a residual image; Calculating a mean square error of the residual image, and using the mean square error as the clarity index; The calculating of the clarity index of the image further includes: Evaluate the structural similarity between the sub-region image and a reference image, and take the inverse of the evaluation value as the clarity index, wherein the reference image is generated by performing mean or Gaussian blurring on the current image; The calculating of the clarity index of the image further includes: The standard deviation of the grayscale values ​​of the pixels in the sub-area is counted, and the clarity index is calculated based on the standard deviation.

[0008] Optionally, constructing point spread function models of the motion blurred image and the focus blurred image respectively includes: For the motion blurred image, constructing a directional point spread function model; For the out-of-focus blurred image, a Gaussian point spread function model is constructed.

[0009] Optionally, constructing a directional point spread function model for the motion blur includes: intercepting the three-axis acceleration and angular velocity data streams of the motion blurred image during the exposure period by time alignment; Integrating the angular velocity data stream to obtain a rotation trajectory, and integrating the three-axis acceleration twice to obtain a displacement trajectory; Projecting the rotation trajectory and the displacement trajectory onto the plane of the motion blurred image to form a two-dimensional line segment trajectory; Discretizing the two-dimensional line segment trajectory into a two-dimensional distribution to form a directional convolution kernel as a point spread function model of the motion blurred image; The directional convolution kernel is normalized so that the total energy of the directional convolution kernel is 1.

[0010] Optionally, constructing a Gaussian point spread function model for the out-of-focus blurred image includes: Counting the edge width of the out-of-focus image and evaluating the attenuation degree of the edge width; Performing Fourier transform on the out-of-focus blurred image to analyze spectrum energy distribution; determining a degree of spectrum compression according to the spectrum energy distribution; estimating Gaussian kernel parameters based on the attenuation degree of the edge width and the spectrum compression degree; A Gaussian point spread function model is constructed based on the Gaussian kernel parameters.

[0011] Optionally, performing deconvolution processing or depth reconstruction on the blurred image using the point spread function model to obtain a clear image includes: Construct image degradation model; Performing Fourier transform on the image degradation model and converting spatial domain operations into frequency domain operations to obtain a degradation model in frequency domain form; Constructing a Wiener filter based on the parameters of the degradation model in frequency domain form; Applying the Wiener filter to the frequency domain representation of the blurred image to obtain a filtered frequency domain image; Performing an inverse Fourier transform on the filtered frequency domain image to obtain a deblurred clear image.

[0012] Optionally, the method further includes: If the vibration intensity exceeds a preset threshold, the acquisition of the image is delayed and the exposure parameters of the image are dynamically adjusted.

[0013] A second aspect of the present application provides an image restoration device, comprising: an acquisition unit, configured to acquire multi-dimensional inertial data in real time through an inertial measurement unit, wherein the multi-dimensional inertial data includes a three-axis acceleration and angular velocity data stream; an analysis unit, configured to perform frequency domain analysis on the multidimensional inertial data using a sliding window short-time Fourier transform to obtain vibration intensity; an acquisition unit, configured to acquire an image if the vibration intensity does not exceed a preset threshold; A calculation unit, configured to calculate a clarity index of the image; a determining unit, configured to determine whether the image is a blurred image according to a preset clarity standard and a clarity index of the image; a determination unit, configured to, if determining that the image is a blurred image, classify the blurred image into a motion blurred image and a focus blurred image; a construction unit, configured to respectively construct point spread function models of the motion blurred image and the focus blurred image; a processing unit, configured to perform deconvolution processing or depth reconstruction on the blurred image using the point spread function model to obtain a clear image; The restoration unit is used to recalculate the clarity index of the clear image, and if it does not exceed the clarity threshold, trigger re-capture or switch the restoration model to perform secondary restoration.

[0014] A third aspect of the present application provides an image restoration device, comprising: processor, memory, input and output units, and buses; The processor is connected to the memory, input and output units, and the bus; The memory stores a program, and the processor calls the program to execute the first aspect and any optional method of the first aspect.

[0015] In a fourth aspect, the present application provides a computer-readable storage medium on which a program is stored. When the program is executed on a computer, the program executes the first aspect and any optional method of the first aspect.

[0016] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages: The motion data provided by the inertial measurement unit in this application provides prior information, and there is no need for an algorithm to reversely deduce the motion state of the device. At the same time, the clear classification of blur types reduces the trial-and-error process of the repair strategy. The image acquisition and repair process are combined to greatly shorten the repair time and improve processing efficiency. After determining that the image is blurry, the blurred image is first classified into motion blur and focus blur, and then a point spread function model is constructed specifically. Deconvolution or depth reconstruction is performed by matching the point function diffusion model of the blur type, avoiding blind attempts of general models, significantly improving the repair accuracy, and reducing over-sharpening and artifact enhancement. The clarity index of the repaired clear image is recalculated, and if the threshold is not reached, re-acquisition is triggered or the repair model is switched for secondary repair. This closed-loop mechanism forms a complete link of "acquisition-repair-evaluation-optimization": re-acquisition can obtain a better original image, and secondary repair adapts to complex blurry scenes through model switching, ultimately ensuring that the output image meets the clarity threshold requirements, avoiding the impact of incomplete repair on subsequent applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0018] Figure 1 This is a schematic diagram of an embodiment of the image restoration method of the present application; Figure 2 A schematic diagram of an embodiment of a method for calculating the clarity index of an image according to the present application; Figure 3 A schematic diagram of an embodiment of a method for constructing a directional point spread function model for this application; Figure 4 A schematic diagram of an embodiment of a method for constructing a Gaussian point spread function model for this application; Figure 5 A schematic diagram of an embodiment of a method for obtaining a clear image according to the present application; Figure 6 A schematic diagram of an embodiment of the delayed image acquisition method of the present application; Figure 7 This is a schematic structural diagram of an embodiment of the image restoration device of the present application; Figure 8 This is a schematic structural diagram of another embodiment of the image restoration device of the present application. DETAILED DESCRIPTION

[0019] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.

[0020] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections thereof.

[0021] It will also be understood that the term "and / or" used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0022] As used in this specification and the appended claims, the term "if" can be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0023] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0024] References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present application. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0025] Based on this, the present application discloses an image restoration method, device and storage medium for improving the structural restoration accuracy and detail recovery capability of image restoration.

[0026] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0027] The method of the present application can be applied to a server, device, terminal or other device with logic processing capability, and the present application does not limit this. For the convenience of description, the following description is based on the example of the execution subject being a system.

[0028] See also Figure 1 , the present application provides an embodiment of an image restoration method, comprising: 101. Acquire multi-dimensional inertial data in real time through an inertial measurement unit, the multi-dimensional inertial data including three-axis acceleration and angular velocity data streams; 102. Use sliding window short-time Fourier transform to perform frequency domain analysis on multi-dimensional inertial data to obtain vibration intensity; The three-axis acceleration data collected by the inertial measurement unit is:

[0029] in, Represents the instantaneous acceleration values ​​along the three spatial dimensions of x, y, and z. The data sampling frequency is set to , which means a set of acceleration samples is acquired every 5 milliseconds. This frequency can be adjusted according to the system response requirements.

[0030] The system uses a sliding window mechanism to continuously update the latest N=64 frames of acceleration data and construct a three-dimensional sliding window sequence:

[0031] The time length covered by the sliding window is:

[0032] The formula for performing discrete Fourier transform on the acceleration signal of each axis is:

[0033] in, is the kth frequency component; the frequency range obtained by DFT is .

[0034] Calculate the power spectral density:

[0035] Sum the spectral power (estimate of vibration intensity) independently for each axis:

[0036] Among them, klow is usually set to remove the influence of low-frequency components (<1Hz); khigh is set to around 20Hz to cover the high-frequency vibration range.

[0037] Fusion of the power of the three axes:

[0038] in: They are the vibration frequency domain energy in the x, y, and z axes directions respectively.

[0039] 103. If the vibration intensity does not exceed the preset threshold, then capture the image; Set vibration intensity threshold , for example, the experience value can be set to (Parameter adjustment is required through experiments).

[0040] Judgment condition: If , it means that there is a high probability of high-frequency vibration in the next 100ms. The system performs delayed acquisition (such as delaying the shutter triggering by 100-200ms) or performs exposure parameter adjustment (shortening the exposure time and increasing the acquisition shutter speed); otherwise, it allows immediate image acquisition.

[0041] 104. Calculate the clarity index of the image; 105. Determine whether the image is a blurred image based on a preset clarity standard and the clarity index of the image; The clarity index is calculated based on the local statistical characteristics of the acquired image. Metrics such as mean square error, structural similarity, and grayscale standard deviation can be selected. For sub-ROIs of the image, the image clarity is quantified by comparing the residuals of the original image and the smoothed version, analyzing the degree of image structural similarity, and calculating the dispersion of grayscale value distribution. This quantifies the clarity index used for quality assessment.

[0042] A pre-set clarity criterion is used to compare the image clarity index calculated in step 104 with the clarity criterion. If the clarity index indicates loss of image detail and structural distortion, the image is considered blurry. Otherwise, the image is considered clear and can be directly output for use.

[0043] 106. If the image is determined to be a blurred image, the blurred image is classified into a motion blurred image and a focus blurred image; If the image is determined to be blurred, the image is further classified into the type of blur. The image blur characteristics are analyzed. If the blur is directional and device motion is detected based on the inertial measurement unit data, the image is determined to be motion blurred. If the blur has no obvious direction and the image exhibits a uniform defocus effect, the image is determined to be focus blurred.

[0044] 107. Construct point spread function models for motion blurred images and focus blurred images respectively; 108. Deconvolution or depth reconstruction of blurred images is performed through the point spread function model to obtain a clear image; 109. Recalculate the clarity index of the clear image. If it does not exceed the clarity threshold, trigger re-acquisition or switch the restoration model to perform secondary restoration.

[0045] For motion-blurred images, the motion data collected by the inertial measurement unit is combined to restore the motion trajectory, project it onto the image plane and discretize it, and construct a point spread function model that reflects the motion projection law. For focus-blurred images, the image edge attenuation and spectral distribution are analyzed, the Gaussian kernel parameters are estimated, and a point spread function model that simulates defocus diffusion is constructed as a mathematical model for image degradation and restoration:

[0046] in: represents a blurred image, Indicates a clear image (to be restored), represents the blur kernel, that is, the point spread function model, Represents image noise, * represents a two-dimensional convolution operation, the goal is to Back-calculating blur kernel , so that the blur kernel is as close as possible to the actual motion or out-of-focus diffusion behavior during image degradation.

[0047] The blurred image is repaired based on the point spread function model constructed in step 107. If the image is motion blurred, an algorithm such as Wiener deconvolution is used to inversely compensate for motion artifacts using the point spread function model. If the image is focus blurred, a Gaussian prior inversion method is combined to restore the image's clear structure. Alternatively, a deep reconstruction model can be used to guide the network learning of blur mapping based on the point spread function model, resulting in a clear image output.

[0048] For the restored clear image, repeat step 104 to recalculate the clarity index. The clarity index is compared with the clarity threshold. If it does not meet the threshold, indicating that details are still blurred and the structure is distorted, the acquisition module is notified to dynamically adjust the strategy, triggering the re-acquisition process or switching the restoration model to perform a second restoration to ensure the output of a high-quality, clear image.

[0049] The entire feedback mechanism is implemented according to the following process: Image input: Input is the repaired image , while preserving the original blurred image And the fuzzy type discrimination results.

[0050] Image quality re-evaluation: quality re-evaluation is performed through steps 102 to 104.

[0051] Result decision: : The image is acceptable and the output is clear; :The image is unqualified and backtracking feedback control is executed.

[0052] in: is the clarity index, It is the set clarity experience value.

[0053] The logic of feedback control is as follows: Delayed acquisition: Wait until the device is stable before acquiring data again (combined with the current state of the IMU); Adjust exposure time: reduce the risk of blur (shorter exposure reduces motion blur); Increase gain / brightness threshold: maintain image brightness at low exposure; Dynamic repair strategy adjustment: modify deconvolution parameters (number of iterations, regularization strength); Logging and system learning: The blur map, repair map, indicators and success / failure labels are combined into a dataset, which can be used for subsequent model retraining and feedback learning.

[0054] In this embodiment, the motion data provided by the inertial measurement unit in this application provides prior information, and there is no need for an algorithm to reversely deduce the motion state of the device. At the same time, the clear classification of blur types reduces the trial-and-error process of the repair strategy. The image acquisition and repair process are combined to greatly shorten the repair time and improve processing efficiency. After determining that the image is blurry, the blurred image is first classified into motion blur and focus blur, and then a point spread function model is specifically constructed. Deconvolution or depth reconstruction is performed by matching the point function diffusion model of the blur type, avoiding blind attempts of general models, significantly improving the repair accuracy, and reducing over-sharpening and artifact enhancement. The clarity index of the repaired clear image is recalculated, and if the threshold is not reached, re-acquisition is triggered or the repair model is switched for secondary repair. This closed-loop mechanism forms a complete link of "acquisition-repair-evaluation-optimization": re-acquisition can obtain a better original image, and secondary repair adapts to complex blurry scenes through model switching, ultimately ensuring that the output image meets the clarity threshold requirements, avoiding the impact of incomplete repair on subsequent applications.

[0055] See also Figure 2 The present application provides an embodiment of a method for calculating a clarity index of an image, comprising: 201. Grayscale the image and crop the ROI region to obtain a sub-region image; 202. Subtract the sub-region image from the mean image pixel by pixel to obtain a residual image; 203. Calculate the mean square error of the residual image and use the mean square error as a clarity index; If the acquired image is a color image, the color image is converted into a grayscale image, and then the image is cropped, the ROI area is extracted, and the ROI area is divided into multiple sub-areas.

[0056] The pixel-by-pixel difference between the image sub-region and the mean map is calculated, and the blur focus value is defined as the mean square error of the residual map. The expression is as follows:

[0057] in, is the original image pixel, is the mean filtered image pixel, and N is the number of pixels in the sub-region.

[0058] The calculated mean square error is used as the clarity index. The larger the mean square error, the richer the image details in the sub-region and the higher the clarity.

[0059] 204. Evaluate the structural similarity between the sub-region image and the reference image, take the inverse of the evaluation value as the clarity index, and generate the reference image by performing mean or Gaussian blurring on the current image; The structural similarity between the sub-region image and its reference image is evaluated to obtain the SSIM value, i.e., the evaluation value. The fuzziness is defined as the inverse of the evaluation value or 1-SSIM, and the expression is as follows:

[0060] in is the mean, is the standard deviation, is the covariance, C1 and C2 are stability constants. If the reference image is not provided, the reference image is generated by performing mean or Gaussian blur on the current image.

[0061] 205. Calculate the standard deviation of the grayscale values ​​of pixels in the sub-region, and calculate the clarity index through the standard deviation.

[0062] Based on the statistical distribution characteristics of image grayscale values, the standard deviation is used to measure the magnitude of local grayscale changes in the image. The richer the image details, the more dramatic the grayscale changes, and the larger the standard deviation, the clearer the image.

[0063] Its blur focus value, namely the clarity index, is defined as:

[0064] in, : represents the blur focus value (sharpness index) of an image or sub-region; N represents the total number of pixels in the image region; Indicates the The grayscale value of each pixel; Represents the mean grayscale value of all pixels in the image area, and the calculation formula is: .

[0065] When there are a lot of edges, details or textures in the image, the grayscale value varies greatly in space, resulting in a large standard deviation. When the image is blurred, details are lost, the grayscale tends to be smooth, and the standard deviation decreases. The standard deviation can be used to quickly detect whether the image has information loss caused by motion blur, defocus, etc.

[0066] In addition, the fuzziness indicators of all sub-regions can be aggregated and output by one of the following strategies: taking the maximum value, taking the minimum value, taking the average value or the median, and outputting the indicator array of all sub-regions for subsequent regional analysis.

[0067] In this embodiment, the grayscale is used to unify the calculation benchmark and the ROI area is cropped to focus on the key areas, so as to avoid the overall statistical characteristics of the image masking the local blur problem and make the clarity assessment more in line with the actual visual focus; the mean square error of the mean map and the residual map is used to accurately capture the difference between the local details of the image and the smoothed version, and is sensitive to the blur caused by the loss of details such as edges and textures; based on the inverse value of the structural similarity, the degree of blur is quantified from the image structure level, and the reference map is generated by the current image to adapt to the scene without external reference, making up for the deficiency of a single indicator; with the help of the grayscale standard deviation, the amplitude of the local grayscale change of the image is reflected, and the loss of details caused by motion blur, defocus, etc. can be effectively identified. The coordination of multiple dimensional indicators greatly improves the accuracy of blur judgment.

[0068] See also Figure 3 The present application provides an embodiment of a method for constructing a directional point spread function model, comprising: 301. Intercepting the three-axis acceleration and angular velocity data streams of the motion blurred image during the exposure period by time alignment; 302. Integrate the angular velocity data stream to obtain a rotation trajectory, and integrate the three-axis acceleration twice to obtain a displacement trajectory; 303. Project the rotation trajectory and the displacement trajectory onto the plane of the motion blurred image to form a two-dimensional line segment trajectory; 304. Discretize the two-dimensional line segment trajectory into a two-dimensional distribution, and form a directional convolution kernel as a point spread function model of the motion blurred image; 305. Normalize the directional convolution kernel so that the total energy of the directional convolution kernel is 1.

[0069] In step 301, the actual exposure timestamp of the image is obtained , intercept the data stream of multi-dimensional inertial data within this time period (the time resolution is generally 1ms or higher). First, integrate the angular velocity to obtain the rotation, and then integrate the acceleration to obtain the position change. The formula is as follows:

[0070] This formula is used to reconstruct the angular displacement and linear position change trajectory of an object from the raw measurement data (angular velocity and acceleration) of multi-dimensional inertial data.

[0071] in, :Indicates from the initial moment The cumulative rotation angle or attitude change at time t is calculated by the angular velocity Perform time integration to obtain; : represents the displacement trajectory (or position change vector), which is the double integral of acceleration, starting from acceleration Get the velocity, and then integrate the velocity to get the displacement.

[0072] In steps 302 to 304, the captured angular velocity data is time-integrated to calculate the camera's rotation angle trajectory during exposure. The acceleration data is integrated twice (first to obtain velocity, then displacement) to obtain the camera's linear position trajectory. The three-dimensional rotation trajectory and displacement trajectory are projected onto the image plane and converted into two-dimensional line segment trajectories reflecting the pixel motion path, representing the direction and range of motion blur. The two-dimensional line segment trajectory is discretized into a two-dimensional distribution at the pixel level, forming a directional convolution kernel extending along the direction of motion, i.e., the point spread function model (blur kernel) of motion blur:

[0073] This formula indicates that the blur kernel is formed by the projection of the camera's motion trajectory during the exposure time onto the image plane. In reality, the camera moves slightly during the exposure period, and its position trajectory is Projected into the image coordinate system, forming a two-dimensional curve trajectory , discretize this trajectory into the energy distribution on the two-dimensional image (which can be regarded as the access frequency of the pixel point), that is, obtain a physically interpretable two-dimensional blur kernel .

[0074] Where t represents the time variable; Represents a time period or time window, from the start time T0 to the end time T1; So the whole formula means: in the time interval Inside, trajectory point The set of determines the current image pixel position Blur kernel at .

[0075] Step 305 is to make the blur kernel It can be used as an image convolution kernel. It needs to be normalized so that its total energy is 1, so that the blurring process does not change the overall brightness of the image: .

[0076] In this embodiment, multidimensional inertial data from the exposure period is captured through time alignment to ensure precise temporal alignment between motion information and the blurred image, avoiding model distortion caused by temporal deviation. The multidimensional inertial data is integrated to obtain rotational and displacement trajectories, which are then transformed into two-dimensional motion paths through projection. This allows the blur kernel construction to be directly linked to the physical motion process, more closely resembling the actual blurring causes and improving the accuracy of the point spread function model. The trajectories are discretized into directional convolution kernels and normalized, preserving the directional characteristics of motion blur while ensuring stable image brightness during restoration. This provides precise kernel parameters for targeted deconvolution restoration, improving the restoration effect of motion-blurred images and enhancing the system's adaptability to dynamic scenes.

[0077] See also Figure 4 The present application provides an embodiment of a method for constructing a Gaussian point spread function model, comprising: 401. Counting the edge width of the focus blurred image and evaluating the attenuation degree of the edge width; 402. Perform Fourier transform on the out-of-focus blurred image to analyze the spectrum energy distribution; 403. Determine the degree of spectrum compression according to spectrum energy distribution; 404. Estimate Gaussian kernel parameters based on the attenuation degree of edge width and spectrum compression degree; 405. Construct a Gaussian point spread function model based on Gaussian kernel parameters.

[0078] In step 401 , for the focus blurred image, the image edges are extracted and the average edge width is calculated, and the attenuation degree is evaluated by the change in edge width (the wider the edge, the more severe the focus blur).

[0079] In step 402, Fourier transform is performed on the image to convert the spatial domain information into frequency domain representation to obtain spectrum energy distribution characteristics.

[0080] In step 403 , the concentration of the spectrum energy distribution counted in step 402 is analyzed. If the energy is mainly concentrated in the low-frequency area and the high-frequency energy accounts for a low proportion, the spectrum compression is high, corresponding to a deep focus blur.

[0081] In step 404, the standard deviation parameter (kernel parameter) of the Gaussian kernel is estimated by combining the edge attenuation degree (edge ​​width) and the spectrum compression degree:

[0082] By edge width or frequency domain energy half Diameter estimation, the more blurred the focus, the softer the image edge, the corresponding The bigger.

[0083] In step 405 , a Gaussian point spread function model is constructed based on the σ value estimated in step 404 . The Gaussian point spread function model simulates the physical process of pixels uniformly spreading to the surroundings in focus blur.

[0084] In this embodiment, the attenuation degree is evaluated by statistically analyzing the edge width, which is directly related to the visual characteristics of focus blur, providing an intuitive quantitative basis for the blur degree; the spectrum energy distribution is analyzed in combination with Fourier transform, and the essential characteristics of focus blur are captured from the frequency domain perspective, realizing the coordinated judgment of spatial domain and frequency domain characteristics, and improving the accuracy of blur degree assessment; the Gaussian kernel parameters are jointly estimated based on edge attenuation and spectrum compression, avoiding the error caused by a single feature and making the Gaussian kernel parameters more consistent with the diffusion law of real focus blur.

[0085] See also Figure 5 , the present application provides an embodiment of a method for obtaining a clear image, comprising: 501. Construct an image degradation model; 502. Perform Fourier transform on the image degradation model and convert the spatial domain operation into a frequency domain operation to obtain the degradation model in frequency domain form; 503. Construct Wiener filter based on the parameters of the degradation model in frequency domain form; 504. Applying a Wiener filter to the frequency domain representation of the blurred image to obtain a filtered frequency domain image; 505. Perform inverse Fourier transform on the filtered frequency domain image to obtain a deblurred clear image.

[0086] Wiener filtering is a classic frequency domain deblurring method, applicable to image degradation models with known blur kernels and additive white Gaussian noise. The image degradation model constructed is consistent with the model in step 107. The image degradation model in step 501 is Fourier transformed (i.e., frequency domain representation):

[0087] in, represents the blurred image spectrum, represents the blur kernel spectrum, represents the original spectrum, represents the noise spectrum.

[0088] Construct a Wiener filter, the frequency domain form is as follows:

[0089] in, represents the deconvolution filter; represents the blur kernel spectrum; express The complex conjugate of Represents the noise and image power spectral density (can be estimated); : Blurred image spectrum.

[0090] Apply a filter to the blurred image spectrum: ; Blurred image spectrum Perform inverse Fourier transform to obtain the restored image : .

[0091] In addition, the image can be deblurred and repaired based on the deep learning image deblurring neural network DeblurGANv2.

[0092] In this embodiment, by constructing an image degradation model, the degradation process of the blurred image is quantified into a mathematical expression, providing a clear theoretical basis for deblurring; the spatial domain convolution is converted into frequency domain multiplication to simplify the computational complexity and improve processing efficiency; a Wiener filter is constructed based on the frequency domain degradation model parameters, which can effectively compensate for the influence of the blur kernel while suppressing noise, balance the deblurring effect and the noise amplification problem, and avoid artifacts caused by over-repair; after filtering, the spatial domain clear image is restored through an inverse Fourier transform. The entire process is logically closed from modeling to restoration, and the repair effect is stable for scenes with known motion or focus blurred point spread functions, which enhances the practicality and reliability of the deblurring algorithm.

[0093] See also Figure 6 , the present application provides an embodiment of a delayed image acquisition method, comprising: 601. If the vibration intensity exceeds a preset threshold, delay image acquisition and dynamically adjust image exposure parameters.

[0094] This embodiment uses real-time three-axis acceleration and angular velocity data collected by the inertial measurement unit (IMU) and a sliding window short-time Fourier transform (STFT) analysis to predict vibration intensity within a preset timeframe. If the predicted vibration intensity exceeds a preset threshold, a pre-capture intervention mechanism is triggered. Firstly, image acquisition is delayed until device vibration subsides below the threshold, preventing motion blur caused by exposure during periods of intense vibration. Second, exposure parameters are dynamically adjusted, including shortening exposure time to reduce motion artifacts or appropriately increasing sensor gain under low-exposure conditions to balance image brightness and blur risk.

[0095] Through the coordinated control of delayed acquisition and parameter adjustment, the probability of image blur caused by equipment vibration is reduced from the source, the initial clarity of the acquired image is improved, and the pressure on subsequent repair links is reduced.

[0096] In this embodiment, by delaying acquisition when the vibration intensity exceeds a threshold, periods of severe device vibration are avoided, thereby reducing motion blur at the source. At the same time, the exposure parameters are dynamically adjusted to ensure image brightness while reducing the risk of blur. The active intervention mechanism reduces the generation of blurred images, reduces the subsequent repair pressure, improves the overall efficiency of the system, and enhances image clarity, stability, and acquisition quality.

[0097] See also Figure 7 , the present application provides an embodiment of an image restoration device, comprising: An acquisition unit 701 is configured to acquire multi-dimensional inertial data in real time through an inertial measurement unit, wherein the multi-dimensional inertial data includes a three-axis acceleration and angular velocity data stream; An analysis unit 702 is configured to perform frequency domain analysis on the multi-dimensional inertial data using a sliding window short-time Fourier transform to obtain vibration intensity; The acquisition unit 703 is configured to acquire an image if the vibration intensity does not exceed a preset threshold; Specifically, the system further includes a delay unit 704, configured to: If the vibration intensity exceeds a preset threshold, image acquisition is delayed and the image exposure parameters are dynamically adjusted.

[0098] A calculation unit 705 is used to calculate the clarity index of the image; Specifically, the computing unit 705 is further configured to: Grayscale the image and crop the ROI area to obtain the sub-region image; Subtract the sub-region image from the mean image pixel by pixel to obtain the residual image; Calculate the mean square error of the residual image and use the mean square error as the clarity index; Specifically, the computing unit 705 is further configured to: The sub-region image is evaluated for structural similarity with the reference image, and the inverse of the evaluation value is taken as the clarity index. The reference image is generated by performing mean or Gaussian blurring on the current image. Specifically, the computing unit 705 is further configured to: The standard deviation of the grayscale values ​​of pixels in the sub-region is counted, and the clarity index is calculated by the standard deviation.

[0099] a determination unit 706, configured to determine whether the image is a blurred image based on a preset clarity standard and an image clarity index; The determining unit 707 is configured to classify the blurred image into a motion blurred image and a focus blurred image if the image is determined to be a blurred image; A construction unit 708 is used to construct point spread function models of the motion blurred image and the focus blurred image respectively; Specifically, the construction unit 708 is further configured to: For motion blurred images, a directional point spread function model is constructed; For the out-of-focus blurred image, a Gaussian point spread function model is constructed.

[0100] Specifically, the construction unit 708 is further configured to: The three-axis acceleration and angular velocity data streams of the motion blurred image during the exposure period are intercepted by time alignment; The rotation trajectory is obtained by integrating the angular velocity data stream, and the displacement trajectory is obtained by integrating the three-axis acceleration twice; Projecting the rotation trajectory and the displacement trajectory onto the plane of the motion blurred image to form a two-dimensional line segment trajectory; Discretize the two-dimensional line segment trajectory into a two-dimensional distribution to form a directional convolution kernel as a point spread function model of the motion blurred image; The directional convolution kernel is normalized so that the total energy of the directional convolution kernel is 1.

[0101] Specifically, the construction unit 708 is further configured to: Count the edge width of the focus blurred image and evaluate the attenuation degree of the edge width; Perform Fourier transform on the out-of-focus image and analyze the spectral energy distribution; Determine the degree of spectrum compression based on spectrum energy distribution; Estimate Gaussian kernel parameters based on the attenuation degree of edge width and spectrum compression; A Gaussian point spread function model is constructed based on the Gaussian kernel parameters.

[0102] The processing unit 709 is configured to perform deconvolution processing or depth reconstruction on the blurred image using a point spread function model to obtain a clear image; Specifically, the processing unit 709 is further configured to: Construct image degradation model; Perform Fourier transform on the image degradation model and convert the spatial domain operation into frequency domain operation to obtain the degradation model in frequency domain form; Construct Wiener filter based on the parameters of the degradation model in frequency domain; Applying the Wiener filter to the frequency domain representation of the blurred image to obtain a filtered frequency domain image; The filtered frequency domain image is subjected to inverse Fourier transform to obtain a deblurred clear image.

[0103] The restoration unit 710 is configured to recalculate the clarity index of the clear image, and if the clarity index does not exceed the clarity threshold, trigger re-capture or switch the restoration model to perform secondary restoration.

[0104] Specific implementation method Figures 1 to 6 The embodiments are not described in detail here.

[0105] See also Figure 8 , the present application provides an image restoration device, comprising: Processor 801 , memory 802 , input / output unit 804 , and bus 803 .

[0106] The processor 801 is connected to the memory 802 , the input and output unit 804 , and the bus 803 .

[0107] The memory 802 stores a program, and the processor 801 calls the program to execute the following Figure 1 、 Figure 2 、 Figure 3 、 Figure 4 、 Figure 5 and Figure 6 The method in .

[0108] The present application provides a computer-readable storage medium, wherein a program is stored on the computer-readable storage medium, and when the program is executed on a computer, the program performs the following operations: Figure 1 、 Figure 2 、 Figure 3 、 Figure 4 、 Figure 5 and Figure 6 The method in .

[0109] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0110] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0111] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0112] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0113] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), disk or optical disk, and other media that can store program code.

Claims

1. An image restoration method, characterized in that: include: Acquire multi-dimensional inertial data in real time through an inertial measurement unit, wherein the multi-dimensional inertial data includes a three-axis acceleration and angular velocity data stream; Performing frequency domain analysis on the multidimensional inertial data using a sliding window short-time Fourier transform to obtain vibration intensity; If the vibration intensity does not exceed a preset threshold, capturing an image; Calculating a clarity index of the image; determining whether the image is a blurred image according to a preset clarity standard and a clarity index of the image; If the image is determined to be a blurred image, the blurred image is divided into a motion blurred image and a focus blurred image; constructing point spread function models of the motion blurred image and the focus blurred image respectively; Performing deconvolution processing or depth reconstruction on the blurred image using the point spread function model to obtain a clear image; The clarity index of the clear image is recalculated, and if it does not exceed the clarity threshold, recapture is triggered or the restoration model is switched to perform secondary restoration.

2. The image restoration method according to claim 1, wherein: Calculating the clarity index of the image includes: Gray-scaling the image and cropping the ROI region to obtain a sub-region image; Subtracting the sub-region image from the mean image pixel by pixel to obtain a residual image; Calculating a mean square error of the residual image, and using the mean square error as the clarity index; The calculating of the clarity index of the image further includes: Evaluate the structural similarity between the sub-region image and a reference image, and take the inverse of the evaluation value as the clarity index, wherein the reference image is generated by performing mean or Gaussian blurring on the current image; The calculating of the clarity index of the image further includes: The standard deviation of the grayscale values ​​of the pixels in the sub-area is counted, and the clarity index is calculated based on the standard deviation.

3. The image restoration method according to claim 1, wherein: The step of constructing point spread function models for the motion blurred image and the focus blurred image respectively includes: For the motion blurred image, constructing a directional point spread function model; For the out-of-focus blurred image, a Gaussian point spread function model is constructed.

4. The image restoration method according to claim 3, wherein: For the motion blur, constructing a directional point spread function model includes: intercepting the three-axis acceleration and angular velocity data streams of the motion blurred image during the exposure period by time alignment; Integrating the angular velocity data stream to obtain a rotation trajectory, and integrating the three-axis acceleration twice to obtain a displacement trajectory; Projecting the rotation trajectory and the displacement trajectory onto the plane of the motion blurred image to form a two-dimensional line segment trajectory; Discretizing the two-dimensional line segment trajectory into a two-dimensional distribution to form a directional convolution kernel as a point spread function model of the motion blurred image; The directional convolution kernel is normalized so that the total energy of the directional convolution kernel is 1.

5. The image restoration method according to claim 3, wherein: The step of constructing a Gaussian point spread function model for the out-of-focus blurred image includes: Counting the edge width of the out-of-focus image and evaluating the attenuation degree of the edge width; Performing Fourier transform on the out-of-focus blurred image to analyze spectrum energy distribution; determining a degree of spectrum compression according to the spectrum energy distribution; estimating Gaussian kernel parameters based on the attenuation degree of the edge width and the spectrum compression degree; A Gaussian point spread function model is constructed based on the Gaussian kernel parameters.

6. The image restoration method according to claim 1, wherein: The step of performing deconvolution processing or depth reconstruction on the blurred image by using the point spread function model to obtain a clear image includes: Construct image degradation model; Performing Fourier transform on the image degradation model and converting spatial domain operations into frequency domain operations to obtain a degradation model in frequency domain form; Constructing a Wiener filter based on the parameters of the degradation model in frequency domain form; Applying the Wiener filter to the frequency domain representation of the blurred image to obtain a filtered frequency domain image; Performing an inverse Fourier transform on the filtered frequency domain image to obtain a deblurred clear image.

7. The image restoration method according to claim 1, wherein: The method further comprises: If the vibration intensity exceeds a preset threshold, the acquisition of the image is delayed and the exposure parameters of the image are dynamically adjusted.

8. An image restoration device, characterized in that: include: an acquisition unit, configured to acquire multi-dimensional inertial data in real time through an inertial measurement unit, wherein the multi-dimensional inertial data includes a three-axis acceleration and angular velocity data stream; an analysis unit, configured to perform frequency domain analysis on the multidimensional inertial data using a sliding window short-time Fourier transform to obtain vibration intensity; an acquisition unit, configured to acquire an image if the vibration intensity does not exceed a preset threshold; A calculation unit, configured to calculate a clarity index of the image; a determining unit, configured to determine whether the image is a blurred image according to a preset clarity standard and a clarity index of the image; a determination unit, configured to, if determining that the image is a blurred image, classify the blurred image into a motion blurred image and a focus blurred image; a construction unit, configured to respectively construct point spread function models of the motion blurred image and the focus blurred image; a processing unit, configured to perform deconvolution processing or depth reconstruction on the blurred image using the point spread function model to obtain a clear image; The restoration unit is used to recalculate the clarity index of the clear image, and if it does not exceed the clarity threshold, trigger re-capture or switch the restoration model to perform secondary restoration.

9. An image restoration device, characterized in that: include: A processor, a memory, an input-output unit, and a bus, wherein the processor is connected to the memory, the input-output unit, and the bus, the memory stores a program, and the processor calls the program to execute the image restoration method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a program, and when the program is executed on a computer, the image restoration method according to any one of claims 1 to 7 is executed.

Images