An image inpainting method, device and storage medium

Abstract

The application discloses an image restoration method and device and a storage medium, and is used for improving the structure restoration precision and the detail recovery ability of the image restoration. The method comprises the following steps: acquiring multi-dimensional inertia data in real time; performing frequency domain analysis on the multi-dimensional inertia data by using a sliding window short-time Fourier transform to obtain a vibration intensity; if the vibration intensity does not exceed a preset threshold, collecting an image; calculating a definition index of the image; determining whether the image is a blurred image according to a preset definition standard and the definition index of the image; if it is determined that the image is a blurred image, dividing the blurred image into a motion blurred image and a focus blurred image; respectively constructing a point spread function model of the motion blurred image and the focus blurred image; performing deconvolution processing or deep reconstruction on the blurred image by using the point spread function model to obtain a clear image; recalculating the definition index of the clear image, and if the definition index does not exceed a definition threshold, triggering re-collection or switching a restoration model to perform secondary restoration.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the field of image acquisition and processing, and in particular to an image restoration method and device and a storage medium. BACKGROUND

[0002] With the development of science and technology, image quality self-checking and blur restoration are increasingly valued. In the field of intelligent manufacturing, high-precision product detection relies on clear image data to identify minor defects. Once the image is blurred due to device vibration or conveyor shaking, it may lead to missed detection of defective products, directly affecting production efficiency and product quality. In the automatic driving scenario, if the road image collected by the vehicle-mounted camera has motion blur or focus deviation, it will interfere with key functions such as lane line recognition and obstacle detection, posing a serious threat to driving safety.

[0003] However, the existing blur restoration method separates the acquisition and restoration processes, making it impossible for the image acquisition stage to provide effective information support for subsequent restoration. During acquisition, the device does not perceive its own motion state or record environmental parameters at the imaging moment, resulting in the restoration stage being able to only blindly process the blurred image. Due to the lack of motion data during acquisition, the restoration algorithm cannot determine whether the blur is caused by motion blur due to device vibration or focus blur due to focus failure, and can only use a general model for trial restoration. In addition, this separation also makes the entire process lack closed-loop optimization capability. The acquisition stage cannot adjust parameters according to restoration needs, and the restoration stage cannot feedback acquisition defects.

[0004] Traditional blur restoration methods separate the acquisition and restoration processes, resulting in insufficient restoration of images. Due to the lack of motion and environmental data during acquisition, the algorithm needs to consume a large amount of computing resources to repeatedly try different restoration strategies, resulting in low restoration efficiency; incorrect determination of the blur type may lead to over-sharpening, artifact enhancement, or loss of details, making the restoration result unreliable, and even worse than the original image. In addition, the system has poor adaptability. The acquisition device cannot adjust parameters according to restoration needs, and the restoration algorithm cannot guide hardware optimization, resulting in poor performance in complex scenarios. SUMMARY

[0005] The present application discloses an image restoration method, device and storage medium for improving the structural restoration accuracy and detail recovery capability of image restoration.

[0006] The first aspect of the present application discloses an image restoration method, comprising:

[0007] real-time acquisition of multi-dimensional inertial data by an inertial measurement unit, the multi-dimensional inertial data including three-axis acceleration and angular velocity data streams;

[0008] perform frequency domain analysis on the multi-dimensional inertial data by using a sliding window short-time Fourier transform to obtain a vibration intensity;

[0009] If the vibration intensity does not exceed a preset threshold, an image is collected;

[0010] An image sharpness index is calculated;

[0011] According to a preset sharpness standard and the image sharpness index, it is determined whether the image is a blurred image;

[0012] If it is determined that the image is a blurred image, the blurred image is divided into a motion blurred image and a focus blurred image;

[0013] Point spread function models of the motion blurred image and the focus blurred image are respectively constructed;

[0014] The blurred image is deconvolved or depth reconstructed by the point spread function models to obtain a clear image;

[0015] The sharpness index of the clear image is recalculated, and if it does not exceed a clear threshold, re-collection or switching of a repair model is triggered to perform secondary repair.

[0016] Optionally, the calculation of the image sharpness index comprises:

[0017] The image is grayed and ROI region cropped to obtain a sub-region image;

[0018] The sub-region image is subtracted from a mean image pixel by pixel to obtain a residual image;

[0019] The mean square error of the residual image is calculated, and the mean square error is taken as the sharpness index;

[0020] The calculation of the image sharpness index further comprises:

[0021] The sub-region image is structurally similar to a reference image, and the inverse of the evaluation value is taken as the sharpness index, and the reference image is generated by mean or Gaussian blur of the current image;

[0022] The calculation of the image sharpness index further comprises:

[0023] The standard deviation of the pixel gray value in the sub-region is calculated, and the sharpness index is calculated by the standard deviation.

[0024] Optionally, the construction of the point spread function models of the motion blurred image and the focus blurred image respectively comprises:

[0025] For the motion blurred image, a directional point spread function model is constructed;

[0026] constructing a Gaussian point spread function model for the focus blur image.

[0027] Optionally, the constructing a directional point spread function model for the motion blur includes:

[0028] intercepting a three-axis acceleration and angular velocity data stream of the motion blur image within an exposure period by time alignment;

[0029] integrating the angular velocity data stream to obtain a rotation trajectory and twice integrating the three-axis acceleration to obtain a displacement trajectory;

[0030] projecting the rotation trajectory and the displacement trajectory to a plane of the motion blur image to form a two-dimensional line segment trajectory;

[0031] discretizing the two-dimensional line segment trajectory into a two-dimensional distribution to form a directional convolution kernel as a point spread function model of the motion blur image;

[0032] performing normalization processing on the directional convolution kernel to make a total energy of the directional convolution kernel be 1.

[0033] Optionally, the constructing a Gaussian point spread function model for the focus blur image includes:

[0034] counting an edge width of the focus blur image and evaluating a decay degree of the edge width;

[0035] performing Fourier transform on the focus blur image to analyze a spectrum energy distribution;

[0036] judging a spectrum compression degree according to the spectrum energy distribution;

[0037] estimating a Gaussian kernel parameter based on the decay degree of the edge width and the spectrum compression degree;

[0038] constructing a Gaussian point spread function model based on the Gaussian kernel parameter.

[0039] Optionally, the performing deconvolution processing or deep reconstruction on the blur image based on the point spread function model to obtain a clear image includes:

[0040] constructing an image degradation model;

[0041] performing Fourier transform on the image degradation model and converting a spatial domain operation into a frequency domain operation to obtain a frequency domain form of the degradation model;

[0042] constructing a Wiener filter based on a parameter of the frequency domain form of the degradation model;

[0043] The Wiener filter is applied to the frequency domain representation of the blurred image to obtain the filtered frequency domain image;

[0044] Perform an inverse Fourier transform on the filtered frequency domain image to obtain a clear image after deblurring.

[0045] Optionally, the method further includes:

[0046] If the vibration intensity exceeds a preset threshold, the acquisition of the image is delayed and the exposure parameters of the image are dynamically adjusted.

[0047] A second aspect of this application provides an image restoration apparatus, comprising:

[0048] The acquisition unit is used to acquire multidimensional inertial data in real time through the inertial measurement unit. The multidimensional inertial data includes triaxial acceleration and angular velocity data streams.

[0049] The analysis unit is used to perform frequency domain analysis on the multidimensional inertial data using a sliding window short-time Fourier transform to obtain the vibration intensity;

[0050] The acquisition unit is used to acquire an image if the vibration intensity does not exceed a preset threshold.

[0051] A calculation unit is used to calculate the sharpness index of the image;

[0052] The determining unit is used to determine whether the image is a blurry image based on a preset sharpness standard and the sharpness index of the image;

[0053] The determination unit is used to classify the blurred image into a motion-blurred image and a focus-blurred image if the image is determined to be a blurred image.

[0054] A construction unit is used to construct point spread function models for the motion-blurred image and the focus-blurred image, respectively.

[0055] The processing unit is used to perform deconvolution processing or depth reconstruction on the blurred image using the point spread function model to obtain a clear image;

[0056] The repair unit is used to recalculate the sharpness index of the clear image. If the sharpness threshold is not exceeded, a re-acquisition or a switch repair model is triggered to perform a secondary repair.

[0057] A third aspect of this application provides an image restoration apparatus, comprising:

[0058] Processor, memory, input / output units, and bus;

[0059] The processor is connected to memory, input / output units, and a bus;

[0060] The memory stores a program, and the processor invokes the program to execute the method of the first aspect and any optional method of the first aspect.

[0061] The fourth aspect of the present application provides a computer readable storage medium, which stores a program, and the program executes the method of the first aspect and any optional method of the first aspect when executed on a computer.

[0062] From the above technical solutions, the embodiments of the present application have the following advantages:

[0063] The motion data provided by the inertial measurement unit in the present application provides prior information, without the need for algorithm to deduce the device motion state reversely; at the same time, the explicit classification of the blur type reduces the trial-and-error process of the repair strategy, the combination of image acquisition and repair process greatly shortens the repair time and improves the processing efficiency; after determining the image blur, the blurred image is first classified into motion blur and focus blur, and then a point spread function model is constructed accordingly, and the deconvolution or deep reconstruction is performed through matching the point function spread model of the blur type, which avoids the blind attempt of the general model, significantly improves the repair accuracy, and reduces the over-sharpening and artifact enhancement; the sharpness index is recalculated for the clear image after repair, and if the threshold is not reached, the reacquisition or switching of the repair model is triggered for secondary repair. This closed-loop mechanism forms a complete link of "acquisition-repair-evaluation-optimization": the reacquisition can obtain a better original image, and the secondary repair can adapt to complex blur scenes through model switching, so as to finally ensure that the output image meets the clear threshold requirement and avoids the impact on subsequent applications due to incomplete repair. BRIEF DESCRIPTION OF DRAWINGS

[0064] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0065] Figure 1 An embodiment schematic diagram of the image repair method of the present application;

[0066] Figure 2 An embodiment schematic diagram of the method for calculating the sharpness index of the image of the present application;

[0067] Figure 3 An embodiment schematic diagram of the method for constructing the directional point spread function model of the present application;

[0068] Figure 4 An embodiment schematic diagram of the method for constructing the Gaussian type point spread function model of the present application;

[0069] Figure 5 Figure 1 is a schematic diagram of an embodiment of a method for obtaining a clear image according to the present application;

[0070] Figure 6 Figure 2 is a schematic diagram of an embodiment of a method for delaying image acquisition according to the present application;

[0071] Figure 7 Figure 3 is a schematic diagram of an embodiment of an image restoration device according to the present application;

[0072] Figure 8 Figure 4 is a schematic diagram of another embodiment of an image restoration device according to the present application. DETAILED DESCRIPTION

[0073] In the following description, for purposes of explanation and not limitation, specific details are set forth, such as particular sequences of steps, techniques, etc., in order to provide a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application can be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known methods, devices, circuits, and

[0074] It is to be understood that the terminology "includes", "has", "holds", "contains" and / or "comprising", when used in this specification and in the following claims, indicates the presence of the described features, integers, steps, operations, elements, and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0075] It is also to be understood that the terminology "and / or" when used in this specification and in the following claims, refers to at least one of the items, or any combination of the items, and includes all possible combinations when used in the following claims.

[0076] As used in this specification and in the claims, the terms "if" and "when" can be interpreted to mean "upon" or "in response to a determination" or "in response to a detection" depending on the context. Similarly, the phrase "if it is determined" or "if [a described condition or event] is detected" can be interpreted to mean "upon determining" or "in response to determining" or "upon detecting [the described condition or event]" or "in response to detecting [the described condition or event]" depending on the context.

[0077] In addition, the terms "first", "second", "third", etc. as used in this specification and in the following claims are used only to distinguish between different features, and do not imply or suggest relative importance.

[0078] Reference within the specification of this application to "one embodiment" or "some embodiments" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the application. The appearances of the phrase "in one embodiment" or "in some embodiments" in various places within specification are not necessarily all referring to the same embodiment, although it can. The terms "including," "comprising," "having" and variations thereof are meant to encompass the items listed thereafter, but do not exclude other items from also being present. The terms "a" and "an" are meant to encompass both the singular and the plural, unless otherwise indicated.

[0079] Based on this, the application discloses an image repairing method and device and a storage medium, which are used for improving the structure restoring precision and the detail recovery capability of image repairing.

[0080] The technical solutions in the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only some of the embodiments of the application, but not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the application.

[0081] The method of the application can be applied to a server, a device, a terminal or other devices with logical processing capability, and the application is not limited in this regard. For the convenience of description, the following will be described taking a system as an execution subject.

[0082] Please refer to Figure 1 An embodiment of the image repairing method provided by the application comprises the following steps.

[0083] 101. Real-time acquisition of multi-dimensional inertial data through an inertial measurement unit, the multi-dimensional inertial data comprising three-axis acceleration and angular velocity data streams;

[0084] 102. Frequency domain analysis of the multi-dimensional inertial data through a sliding window short-time Fourier transform to obtain vibration intensity;

[0085] The three-axis acceleration data collected through the inertial measurement unit is as follows:

[0086]

[0087] wherein, represents the instantaneous acceleration value along the x, y and z spatial dimensions respectively. The data sampling frequency is set to , that is, a group of acceleration samples is acquired every 5 milliseconds. This frequency can be adjusted according to the system response requirement.

[0088] The system adopts a sliding window mechanism to continuously update the latest N=64 frames of acceleration data, and constructs a three-dimensional sliding window sequence:

[0089]

[0090] The time length covered by the sliding window is:

[0091]

[0092] The formula for performing discrete Fourier transform on each axis acceleration signal is:

[0093]

[0094] wherein, is the kth frequency component; the frequency range obtained by DFT is .

[0095] Calculate the power spectral density:

[0096]

[0097] Calculate the total sum of spectral power (vibration intensity estimate) for each axis independently:

[0098] wherein, klow is usually set to remove the influence of low frequency components (<1Hz); khigh is set to about 20Hz, covering the high frequency vibration range.

[0099] Fuse the power of the three axes:

[0100]

[0101] wherein: are the vibration frequency energies of the x, y, and z three-axis directions respectively.

[0102] 103, if the vibration intensity does not exceed the preset threshold, collect the image;

[0103] Set the vibration intensity threshold , for example, the empirical value can be set to (necessary through experimental parameter tuning).

[0104] Judgment condition: if , it means that there is a high probability of high frequency vibration in the future 100ms, and the system performs delayed collection (such as delaying the trigger shutter by 100~200ms) or performs exposure parameter adjustment (shortening the exposure time and improving the acquisition shutter speed); otherwise, the image is allowed to be collected immediately.

[0105] 104, calculate the sharpness index of the image;

[0106] 105. determining whether the image is a blurred image according to a preset definition standard and a definition index of the image;

[0107] For the acquired image, a definition index is calculated according to local statistical characteristics. The mean square error, structural similarity, and gray standard deviation can be selected as indexes. For the image sub-ROI region, the image definition degree is quantified by comparing the residual of the original image and the smoothed version, analyzing the structural similarity of the image, and statistically analyzing the dispersion of the gray value distribution, so as to obtain the definition index for quality evaluation.

[0108] A definition determination standard is preset, and the definition index of the image calculated in step 104 is compared with the definition determination standard. If the definition index reflects the loss of image details and structural distortion, the image is determined to be a blurred image; otherwise, the image is determined to be a clear image and can be directly output for use.

[0109] 106. If the image is determined to be a blurred image, the blurred image is divided into a motion blurred image and a focus blurred image;

[0110] If the image is determined to be blurred, the type of blur is further distinguished. The blur characteristics of the image are analyzed. If the blur is directional and the device motion is detected by combining the inertial measurement unit data, the image is determined to be a motion blurred image. If the blur has no obvious direction and presents a uniform defocusing effect as a whole, the image is determined to be a focus blurred image.

[0111] 107. Point spread function models of the motion blurred image and the focus blurred image are respectively constructed;

[0112] 108. The blurred image is deconvolved or reconstructed in depth by using the point spread function model to obtain a clear image;

[0113] 109. The definition index of the clear image is recalculated. If the definition index does not exceed the definition threshold, reacquisition or switching of the repair model is triggered to perform secondary repair.

[0114] For the motion blurred image, the motion trajectory is restored by combining the motion data collected by the inertial measurement unit, projected to the image plane and discretized, and the point spread function model reflecting the motion projection law is constructed. For the focus blurred image, the image edge attenuation and spectral distribution are analyzed, the Gaussian kernel parameters are estimated, and the point spread function model simulating the defocusing diffusion is constructed as a mathematical model for image degradation and repair.

[0115]

[0116] wherein: represents the blurred image, represents the clear image (to be restored), represents the blur kernel, i.e., the point spread function model, denotes image noise, * denotes a two-dimensional convolution operation, and the goal is to deconvolve from the blur kernel so that the blur kernel is as close as possible to the actual motion or defocus spread behavior in the image degradation process.

[0117] According to the point spread function model constructed in step 107, the blurred image is repaired. If it is motion blur, Wiener deconvolution algorithm and the like are used to compensate for the motion smear by using the point spread function model in reverse; if it is focus blur, the clear structure of the image is restored by combining the Gaussian prior inversion method. In addition, a deep reconstruction model can also be enabled to guide the network to learn the blur mapping based on the point spread function model, and output a clear image.

[0118] For the clear image obtained by repairing, the clarity index is recalculated in the manner of step 104. The clarity index is compared with the clarity threshold value. If it does not meet the standard, it indicates that there is still detail blur and structural distortion, which reversely informs the acquisition module to dynamically adjust the strategy, triggers the reacquisition process or switches the repair model to perform secondary repair, to ensure that a high-quality clear image is output.

[0119] The entire feedback mechanism is implemented according to the following process:

[0120] Image input: the input is the repaired image , while retaining the original blurred image and the blur type discrimination result.

[0121] Image quality reevaluation: quality reevaluation is performed through steps 102 to 104.

[0122] Result decision: : the image is acceptable, and a clear image is output;

[0123] : the image is unqualified, and the backtracking feedback control is performed.

[0124] Wherein: is the clarity index, is the set clarity empirical value.

[0125] The logic of feedback control is specifically:

[0126] Delayed acquisition: wait for the device to stabilize and then acquire again (combined with the current state of IMU);

[0127] Adjust the exposure time: reduce the blur risk (short exposure reduces motion blur);

[0128] Increase the gain / luminance threshold: maintain the image brightness in the case of low exposure;

[0129] Dynamic repair strategy adjustment: modify the deconvolution parameters (number of iterations, regularization strength);

[0130] Log and system learning: fuzzy graph, repair graph, index and success or failure annotation form a dataset that can be used for subsequent model retraining and feedback learning.

[0131] In this embodiment, the motion data provided by the inertial measurement unit in this application provides prior information, and the algorithm does not need to deduce the device motion state; at the same time, the clear classification of the fuzzy type reduces the trial-and-error process of the repair strategy, and the combination of image acquisition and repair process greatly shortens the repair time and improves the processing efficiency; after determining the image blur, the blurred image is classified into motion blur and focus blur, and then the point spread function model is constructed, and the deconvolution or deep reconstruction is performed through matching the point function spread model of the fuzzy type, which avoids the blind attempt of the general model, significantly improves the repair accuracy, and reduces the over-sharpening and artifact enhancement; the sharpness index of the repaired clear image is recalculated, and if it does not reach the threshold, the reacquisition or switching repair model secondary repair is triggered. This closed-loop mechanism forms a complete link of "acquisition-repair-evaluation-optimization": reacquisition can obtain better original image, secondary repair can adapt to complex blur scene through model switching, and finally ensure that the output image meets the clear threshold requirement, avoiding the influence of incomplete repair on subsequent application.

[0132] Please refer to Figure 2 The application provides an embodiment of a method for calculating the sharpness index of an image, comprising:

[0133] 201, grayscale the image and crop the ROI region to obtain a sub-region image;

[0134] 202, subtract the sub-region image from the mean image pixel by pixel to obtain a residual image;

[0135] 203, calculate the mean square error of the residual image, and take the mean square error as the sharpness index;

[0136] If the acquired image is a color image, the color image is converted into a grayscale image, then the image is cropped, the ROI region is extracted, and the ROI region is divided into multiple sub-regions.

[0137] Calculate the pixel-by-pixel difference value of the image sub-region and the mean image, define the blur focus value as the mean square error of the residual image, and the expression is as follows:

[0138]

[0139] Wherein, is the pixel of the original image, is the pixel of the mean filter image, and N is the number of sub-region pixels.

[0140] The calculated mean square error is taken as the definition index, and the greater the mean square error, the more details the sub-region image has and the higher the definition is.

[0141] 204、The sub-region image is evaluated for structural similarity with a reference image, and the inverse of the evaluation value is taken as the definition index, and the reference image is generated by mean or Gaussian blur of the current image;

[0142] The sub-region image is evaluated for structural similarity with a reference image, and the SSIM value, i.e. the evaluation value, is obtained, and the blur degree is defined as the inverse of the evaluation value or 1-SSIM, and the expression is as follows:

[0143]

[0144] Among them is the mean, is the standard deviation, is the covariance, and C1 and C2 are stability constants. If the reference image is not provided, the reference image is generated by mean or Gaussian blur of the current image.

[0145] 205、The standard deviation of the pixel gray value in the sub-region is calculated, and the definition index is obtained by the standard deviation.

[0146] Based on the statistical distribution characteristics of the image gray value, the standard deviation is used to measure the local gray change amplitude of the image. The more details the image has, the more intense the gray change is, and the greater the standard deviation is, indicating that the image is clearer.

[0147] The blur degree focus value, i.e. the definition index, is defined as:

[0148]

[0149] Among them, : indicates the blur degree focus value (definition index) of the image or sub-region; N indicates the total number of pixels in the image region; indicates the gray value of the i-th pixel;

[0150] indicates the mean value of the gray values of all pixels in the image region, and the calculation formula is: .

[0151] When there are a large number of edges, details or textures in the image, the gray value changes sharply in space, resulting in a high standard deviation When the image is blurred, the details are lost, and the gray value tends to be smooth, and the standard deviation decreases. The standard deviation can be used to quickly detect whether the image has information loss due to motion blur, defocus, etc.

[0152] ​In addition, the blur indicators of all sub-regions can be aggregated and output by one of the following strategies: taking the maximum value, taking the minimum value, taking the average value or median, or outputting an indicator array of all sub-regions for subsequent regional analysis.

[0153] In the embodiment, by gray-scale unified calculation reference, ROI region cropping focuses on key regions, avoiding that the overall statistical characteristics of the image mask the local blur problem, so that the clarity evaluation is more in line with the actual visual focus; the mean square error of the mean image and the residual image is used to accurately capture the difference between the image local details and the smooth version, sensitive to blur caused by edge, texture and other detail loss; based on the structural similarity inverse value, the blur degree is quantified from the image structure layer, the reference image is generated through the current image, suitable for external reference scene, and the single indicator is compensated; with the help of gray standard deviation, the image local gray scale change amplitude is reflected, the detail loss caused by motion blur, defocus and the like can be effectively identified, and the multi-dimensional indicator cooperates to greatly improve the blur judgment accuracy.

[0154] Please refer to Figure 3 The application provides an embodiment of a method for constructing a directional point spread function model, comprising:

[0155] 301. The three-axis acceleration and angular velocity data streams of the motion blurred image within the exposure period are intercepted by time alignment;

[0156] 302. The angular velocity data stream is integrated to obtain a rotation trajectory, and the three-axis acceleration is integrated twice to obtain a displacement trajectory;

[0157] 303. The rotation trajectory and the displacement trajectory are projected onto the plane of the motion blurred image to form a two-dimensional line segment trajectory;

[0158] 304. The two-dimensional line segment trajectory is discretized into a two-dimensional distribution to form a directional convolution kernel as a point spread function model of the motion blurred image;

[0159] 305. The directional convolution kernel is normalized so that the total energy of the directional convolution kernel is 1.

[0160] In step 301, the actual exposure timestamp of the image is obtained The data stream of multi-dimensional inertial data within the time period (the time resolution is generally 1ms or higher) is intercepted. The angular velocity is first integrated to obtain the rotation amount, and then the acceleration is integrated to obtain the position change, as follows:

[0161]

[0162] The formula is used to reconstruct the angular displacement and linear position change trajectory of the object from the original measurement data (angular velocity and acceleration) of the multi-dimensional inertial data.

[0163] in, : Indicates from the initial moment The cumulative rotation angle or attitude change up to time t, expressed as an angular velocity. Obtained by time integration;

[0164] : Represents the displacement trajectory (or position change vector), which is the double integral of acceleration, starting from acceleration. We obtain the velocity, and then integrate the velocity to obtain the displacement.

[0165] In steps 302 to 304, the captured angular velocity data is integrated over time to calculate the camera's rotational angle change trajectory during the exposure; the acceleration data is integrated twice (first velocity, then displacement) to obtain the camera's linear position change trajectory. The three-dimensional rotational and displacement trajectories are projected onto the image plane, transforming them into two-dimensional line segment trajectories reflecting the pixel motion paths, thus representing the direction and extent of motion blur. These two-dimensional line segment trajectories are discretized into pixel-level two-dimensional distributions, forming a directional convolution kernel extending along the motion direction, i.e., the point spread function model (blur kernel) for motion blur.

[0166]

[0167] This formula states that the blur kernel is formed by the projection of the camera's motion trajectory during the exposure time onto the image plane. In reality, the camera undergoes minute movements during the exposure, and its positional trajectory... Projected onto the image coordinate system, it forms a two-dimensional curved trajectory. Discretizing this trajectory into an energy distribution on a two-dimensional image (which can be viewed as the access frequency of pixels) yields a physically interpretable two-dimensional fuzzy kernel. .

[0168] Where t represents the time variable;

[0169] This represents a time period or time window, from the start time T0 to the end time T1.

[0170] Therefore, the entire formula represents: within the time interval Inside, trajectory point The set determines the current image pixel position. Fuzzy kernel at the location .

[0171] Step 305 is to make the blur kernel It can be used as an image convolution kernel, but it needs to be normalized so that its total energy is 1, so that blurring will not change the overall brightness of the image.

[0172] .

[0173] In this embodiment, the multi-dimensional inertial data of the exposure period is intercepted by time alignment, ensuring the time accurate matching of motion information and blurred image, avoiding model distortion caused by time deviation. The rotation trajectory and displacement trajectory are obtained by integrating the multi-dimensional inertial data, combined with projection to convert into a two-dimensional motion path, so that the blur kernel construction is directly related to the physical motion process, more in line with the real blur cause, and the accuracy of the point spread function model is improved. Discretizing the trajectory into a directional convolution kernel and normalizing it not only retains the directional features of motion blur, but also ensures the stability of image brightness during repair, provides accurate kernel parameters for targeted deconvolution repair, improves the repair effect of motion blurred images, and enhances the adaptive ability of the system to dynamic scenes.

[0174] Please refer to Figure 4 An embodiment of a method for constructing a Gaussian point spread function model is provided, comprising:

[0175] 401. Statistically determine the edge width of the in-focus blurred image, and evaluate the attenuation degree of the edge width;

[0176] 402. Perform Fourier transform on the in-focus blurred image, and analyze the spectral energy distribution;

[0177] 403. Determine the spectral compression degree according to the spectral energy distribution;

[0178] 404. Estimate the Gaussian kernel parameters based on the attenuation degree of the edge width and the spectral compression degree;

[0179] 405. Construct a Gaussian point spread function model based on the Gaussian kernel parameters.

[0180] In step 401, for the in-focus blurred image, the image edge is extracted and the average edge width is statistically determined, and the attenuation degree is evaluated by the edge width change (the wider the edge, the more serious the in-focus blur).

[0181] In step 402, the Fourier transform is performed on the image to convert the spatial domain information into frequency domain representation, and the spectral energy distribution characteristics are obtained.

[0182] In step 403, the concentration degree of the spectral energy distribution statistically determined in step 402 is analyzed. If the energy is mainly concentrated in the low frequency region and the high frequency energy proportion is low, the spectral compression degree is high, and the in-focus blur degree is deep.

[0183] In step 404, the standard deviation parameters (kernel parameters) of the Gaussian kernel are estimated by combining the edge attenuation degree (edge width) and the spectral compression degree:

[0184]

[0185] The edge width or spectral energy half The more blurred the focus is, the softer the image edge is, and the larger the corresponding

[0186] In step 405, a Gaussian point spread function model is constructed based on the sigma value estimated in step 404, and the Gaussian point spread function model simulates the physical process of uniform diffusion of pixels in the focus blur to the surrounding.

[0187] In the embodiment, the degree of attenuation is evaluated by statistical edge width, the visual features of focus blur are directly related, and intuitive quantitative basis for blur degree is provided; the essence of focus blur is captured from the frequency domain by analyzing the spectral energy distribution, the spatial domain and frequency domain features are cooperatively judged, and the accuracy of blur degree evaluation is improved; the Gaussian kernel parameters are estimated based on the joint estimation of edge attenuation and spectral compression degree, the error caused by single feature is avoided, and the Gaussian kernel parameters are more consistent with the diffusion law of real focus blur.

[0188] Please refer to Figure 5 An embodiment of a method for obtaining a clear image is provided, which comprises:

[0189] 501, constructing an image degradation model;

[0190] 502, performing Fourier transform on the image degradation model, and converting spatial domain operation into frequency domain operation to obtain a degradation model in frequency domain;

[0191] 503, constructing a Wiener filter based on the parameters of the degradation model in frequency domain;

[0192] 504, applying the Wiener filter to the frequency domain representation of the blurred image to obtain a filtered frequency domain image;

[0193] 505, performing inverse Fourier transform on the filtered frequency domain image to obtain a deblurred clear image.

[0194] Wiener filtering is a classic frequency domain deblurring method, which is suitable for image degradation models with known blur kernel and additive Gaussian white noise. The image degradation model in step 501 is Fourier transformed (i.e. frequency domain representation) as follows:

[0195]

[0196] Wherein, represents the spectrum of the blurred image, represents the spectrum of the blur kernel, represents the spectrum of the original image, represents the spectrum of the noise.

[0197] The Wiener filter is constructed as follows in frequency domain:​

[0198]

[0199] wherein, represents a deconvolution filter;

[0200] represents a blur kernel spectrum;

[0201] represents a conjugate complex of

[0202] represents a noise to image power spectral density (estimable);

[0203] : a blurred image spectrum.

[0204] Applying a filter to the blurred image spectrum: ;

[0205] performing an inverse Fourier transform on the blurred image spectrum to obtain a restored image :

[0206] .

[0207] In addition, the image can also be deblurred and repaired based on a deep learning-based image deblurring neural network DeblurGANv2.

[0208] In the embodiment, by constructing an image degradation model, the degradation process of the blurred image is quantified as a mathematical expression, providing an explicit theoretical basis for deblurring; spatial domain convolution is converted into frequency domain multiplication, simplifying the operation complexity and improving the processing efficiency; based on the frequency domain degradation model parameters, a Wiener filter is constructed, which can effectively compensate the influence of the blur kernel while suppressing noise, balance the deblurring effect and noise amplification problem, and avoid artifacts caused by excessive repair; after filtering, the spatial domain clear image is restored by inverse Fourier transform, the whole process is logically closed loop from modeling to restoration, the scene repair effect is stable for known motion or focus blur point spread function, and the practicability and reliability of the deblurring algorithm are enhanced.

[0209] Referring to Figure 6 , the application provides an embodiment of a delay image acquisition method, comprising:

[0210] 601, if the vibration intensity exceeds the preset threshold, the image acquisition is delayed and the exposure parameter of the image is dynamically adjusted.

[0211] The embodiment is based on real-time three-axis acceleration and angular velocity data collected by an inertial measurement unit, and predicts the vibration intensity in a future preset time through sliding window short-time Fourier transform analysis. If the predicted vibration intensity exceeds the system preset threshold, the pre-collection intervention mechanism is triggered: on the one hand, the image collection time is delayed, and the device vibration is waited to weaken to below the threshold value, so as to avoid motion blur caused by exposure during severe vibration; on the other hand, the exposure parameters are dynamically adjusted, including shortening the exposure time to reduce the accumulation of motion smearing, or appropriately increasing the sensor gain under low exposure conditions, to balance the image brightness and blur risk.

[0212] Through the cooperative control of delay collection and parameter adjustment, the probability of image blur caused by device vibration is reduced from the source, the initial clarity of the collected image is improved, and the pressure of the subsequent repair link is reduced.

[0213] In the embodiment, by delaying the collection when the vibration intensity exceeds the threshold value, the period of severe device vibration is avoided, and the generation of motion blur is reduced from the source; at the same time, the exposure parameters are dynamically adjusted to reduce the blur risk while ensuring the image brightness. The active intervention mechanism reduces the generation of blurred images, reduces the subsequent repair pressure, improves the overall efficiency of the system, enhances the stability and clarity of the image, and improves the collection quality.

[0214] Referring to Figure 7 An embodiment of an image repair device is provided, which includes

[0215] The acquisition unit 701 is configured to acquire multi-dimensional inertial data in real time through an inertial measurement unit, and the multi-dimensional inertial data includes three-axis acceleration and angular velocity data stream.

[0216] The analysis unit 702 is configured to perform frequency domain analysis on the multi-dimensional inertial data by using a sliding window short-time Fourier transform to obtain the vibration intensity.

[0217] The acquisition unit 703 is configured to collect an image if the vibration intensity does not exceed the preset threshold.

[0218] Specifically, the delay unit 704 is further configured to:

[0219] If the vibration intensity exceeds the preset threshold, the collection of the image is delayed and the exposure parameters of the image are dynamically adjusted.

[0220] The calculation unit 705 is configured to calculate the clarity index of the image.

[0221] Specifically, the calculation unit 705 is further configured to:

[0222] The image is grayed and the ROI region is cropped to obtain a sub-region image.

[0223] The sub-region image is subtracted from the mean image pixel by pixel to obtain a residual image.

[0224] calculating mean square error of the residual image as the definition index;

[0225] Specifically, the calculation unit 705 is further configured to:

[0226] performing structural similarity assessment on the sub-region image and a reference image, and taking the inverse value of the assessment value as the definition index, the reference image being generated by mean or Gaussian blur on the current image;

[0227] Specifically, the calculation unit 705 is further configured to:

[0228] calculating the definition index by standard deviation of the pixel gray value in the sub-region.

[0229] The determination unit 706 is configured to determine whether the image is a blurred image according to a preset definition standard and the definition index of the image.

[0230] The determination unit 707 is configured to divide the blurred image into a motion blurred image and a focus blurred image if the image is determined to be a blurred image.

[0231] The construction unit 708 is configured to construct a point spread function model of the motion blurred image and the focus blurred image respectively.

[0232] Specifically, the construction unit 708 is further configured to:

[0233] for the motion blurred image, constructing a directional point spread function model;

[0234] for the focus blurred image, constructing a Gaussian point spread function model.

[0235] Specifically, the construction unit 708 is further configured to:

[0236] obtaining three-axis acceleration and angular velocity data streams of the motion blurred image in the exposure period by time alignment;

[0237] integrating the angular velocity data stream to obtain a rotation trajectory, and integrating the three-axis acceleration twice to obtain a displacement trajectory;

[0238] projecting the rotation trajectory and the displacement trajectory to the plane of the motion blurred image to form a two-dimensional line segment trajectory;

[0239] discretizing the two-dimensional line segment trajectory into a two-dimensional distribution to form a directional convolution kernel as the point spread function model of the motion blurred image;

[0240] performing normalization processing on the directional convolution kernel, so that the total energy of the directional convolution kernel is 1.

[0241] Specifically, the construction unit 708 is further configured to:

[0242] statistically blur the edge width of the image, and evaluate the degree of attenuation of the edge width;

[0243] perform Fourier transform on the in-focus blurred image, and analyze the energy distribution of the spectrum;

[0244] determine the degree of spectrum compression according to the energy distribution of the spectrum;

[0245] estimate the Gaussian kernel parameter based on the degree of attenuation of the edge width and the degree of spectrum compression;

[0246] construct a Gaussian point spread function model based on the Gaussian kernel parameter.

[0247] The processing unit 709 is configured to perform deconvolution processing or deep reconstruction on the blurred image by using the point spread function model to obtain a clear image.

[0248] Specifically, the processing unit 709 is further configured to:

[0249] construct an image degradation model;

[0250] perform Fourier transform on the image degradation model, and convert spatial domain operation into frequency domain operation to obtain a frequency domain form of the degradation model;

[0251] construct a Wiener filter based on the parameters of the frequency domain form of the degradation model;

[0252] apply the Wiener filter to the frequency domain representation of the blurred image to obtain a filtered frequency domain image;

[0253] perform inverse Fourier transform on the filtered frequency domain image to obtain a deblurred clear image.

[0254] The repairing unit 710 is configured to recalculate the sharpness index of the clear image, and if the clear image does not exceed the clear threshold, trigger reacquisition or switch the repair model to perform secondary repair.

[0255] The specific implementation process is described in the Figures 1 to 6 embodiments, which will not be repeated here.

[0256] Please refer to Figure 8 The present application provides an image repairing device, which comprises:

[0257] The processor 801, the memory 802, the input and output unit 804, and the bus 803.

[0258] The processor 801 is connected with the memory 802, the input and output unit 804, and the bus 803.

[0259] The memory 802 stores a program, and the processor 801 calls the program to perform the steps of the method as Figure 1, Figure 2 、 Figure 3 、 Figure 4 、 Figure 5 and Figure 6 .

[0260] The application provides a computer readable storage medium, which stores a program. The program performs the method in Figure 1 、 Figure 2 、 Figure 3 、 Figure 4 、 Figure 5 and Figure 6 when executed on a computer.

[0261] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system, device and unit described above can refer to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0262] In several embodiments provided in the application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. In actual implementation, another division mode can be used. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0263] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units. That is, they can be located in one place, or can be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0264] In addition, each functional unit in each embodiment of the application can be integrated into a processing unit, or each unit can exist physically independently, or two or more units can be integrated into one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0265] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

Claims

1. An image inpainting method characterized by, The method comprises the following steps: real-time acquisition of multi-dimensional inertial data through an inertial measurement unit, the multi-dimensional inertial data comprising three-axis acceleration and angular velocity data streams; frequency domain analysis of the multi-dimensional inertial data by using a sliding window short-time Fourier transform to obtain vibration intensity; if the vibration intensity does not exceed a preset threshold, an image is collected; calculation of a sharpness index of the image; determination of whether the image is a blurred image according to a preset sharpness standard and the sharpness index of the image; if the image is determined to be a blurred image, the blurred image is divided into a motion blurred image and a focus blurred image; point spread function models of the motion blurred image and the focus blurred image are respectively constructed; inverse convolution processing or deep reconstruction of the blurred image is performed through the point spread function models to obtain a clear image; recalculation of the sharpness index of the clear image, if the sharpness index does not exceed a sharpness threshold, re-collection is triggered or a repair model is switched to perform secondary repair; the calculation of the sharpness index of the image comprises the following steps: gray-scale processing and ROI region cropping of the image to obtain a sub-region image; pixel-by-pixel subtraction of the sub-region image from a mean image to obtain a residual image; calculation of a mean square error of the residual image, and taking the mean square error as the sharpness index; the calculation of the sharpness index of the image further comprises the following steps: structure similarity assessment of the sub-region image and a reference image, and taking the inverse value of the assessment value as the sharpness index, the reference image being generated by mean or Gaussian blurring of the current image; the calculation of the sharpness index of the image further comprises the following steps: statistical calculation of a standard deviation of pixel gray-scale values in the sub-region, and calculation of the sharpness index through the standard deviation; the construction of the point spread function models of the motion blurred image and the focus blurred image respectively comprises the following steps: for the motion blurred image, a directional point spread function model is constructed; for the focus blurred image, a Gaussian point spread function model is constructed.

2. The image inpainting method of claim 1, wherein, the construction of the directional point spread function model for the motion blurred image comprises the following steps: time alignment is performed on the three-axis acceleration and angular velocity data streams of the motion blurred image within an exposure period to obtain the three-axis acceleration and angular velocity data streams; integration of the angular velocity data stream to obtain a rotation trajectory, and twice integration of the three-axis acceleration to obtain a displacement trajectory; projection of the rotation trajectory and the displacement trajectory onto the plane of the motion blurred image to form a two-dimensional line segment trajectory; discretization of the two-dimensional line segment trajectory into a two-dimensional distribution to form a directional convolution kernel as the point spread function model of the motion blurred image; normalization processing is performed on the directional convolution kernel to make the total energy of the directional convolution kernel equal to 1.

3. The image inpainting method of claim 1, wherein, the construction of the Gaussian point spread function model for the focus blurred image comprises the following steps: statistical calculation of an edge width of the focus blurred image, and assessment of the attenuation degree of the edge width; Fourier transform of the focus blurred image to analyze the frequency spectrum energy distribution; judgment of the frequency spectrum compression degree according to the frequency spectrum energy distribution; estimation of Gaussian kernel parameters based on the attenuation degree of the edge width and the frequency spectrum compression degree; construction of the Gaussian point spread function model based on the Gaussian kernel parameters.

4. The image inpainting method of claim 1, wherein, The deconvolution processing or deep reconstruction of the blurred image through the point spread function model includes: constructing an image degradation model; performing Fourier transform on the image degradation model and converting spatial domain operation into frequency domain operation to obtain a frequency domain form of the degradation model; constructing a Wiener filter based on parameters of the frequency domain form of the degradation model; applying the Wiener filter to a frequency domain representation of the blurred image to obtain a filtered frequency domain image; performing inverse Fourier transform on the filtered frequency domain image to obtain a deblurred clear image.

5. The image inpainting method of claim 1, wherein, The method further includes: if the vibration intensity exceeds a preset threshold, delaying the acquisition of the image and dynamically adjusting the exposure parameter of the image.

6. An image inpainting apparatus characterized by comprising: The device for performing the method according to any one of claims 1 to 5 includes: an acquisition unit configured to acquire multi-dimensional inertial data in real time through an inertial measurement unit, the multi-dimensional inertial data including three-axis acceleration and angular velocity data streams; an analysis unit configured to perform frequency domain analysis on the multi-dimensional inertial data by using a sliding window short-time Fourier transform to acquire a vibration intensity; an acquisition unit configured to acquire an image if the vibration intensity does not exceed a preset threshold; a calculation unit configured to calculate a sharpness index of the image; a determination unit configured to determine whether the image is a blurred image according to a preset sharpness standard and the sharpness index of the image; a judgment unit configured to determine the blurred image as a motion blurred image or a focus blurred image if the image is determined to be a blurred image; a construction unit configured to construct a point spread function model of the motion blurred image and the focus blurred image, respectively; a processing unit configured to obtain a clear image by performing deconvolution processing or deep reconstruction of the blurred image through the point spread function model; a repair unit configured to recalculate the sharpness index of the clear image, and trigger reacquisition or switch a repair model to perform secondary repair if the sharpness index does not exceed a sharpness threshold.

7. An image inpainting apparatus characterized by comprising: include: a processor, a memory, an input / output unit and a bus, the processor being connected with the memory, the input / output unit and the bus, the memory storing a program, and the processor invoking the program to execute the image repair method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a program, and the program performs the image repair method according to any one of claims 1 to 5 when executed on a computer.

Images