A method and system for removing camera non-uniform blur

Abstract

The present application relates to the technical field of computer vision, and provides a method and system for removing non-uniform camera blur to solve the problem that traditional deblurring methods cannot meet the imaging requirements under the conditions of multi-source disturbance and multi-scale depth of field in road disease detection. The method for removing non-uniform camera blur includes reconstructing the real motion trajectory of the camera within the exposure time into a three-dimensional motion trajectory in the world coordinate system; projecting the reconstructed three-dimensional motion trajectory onto the pixel plane to generate and normalize the spatially variable point spread function; for each channel of the color image, performing a frequency domain filtering restoration operation according to the color image collected by the camera and the normalized spatially variable point spread function to obtain a filtered restoration image and use it as an initial estimate, performing multiplicative iterative updating, and fusing the filtered restoration images after iterative updating of all channels to obtain a final deblurred color image. The method can significantly improve the clarity and structural consistency of road disease detection images.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method and system for removing camera non-uniform shake blur. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] With the rapid development of intelligent transportation and road defect detection technologies, visual inspection cameras have been widely used for the automatic identification of road defects such as cracks, potholes, subsidence, and structural damage. However, in actual road inspection scenarios, detection equipment is often mounted on vehicles, drones, or tripod platforms. Due to the combined effects of vehicle vibration, wind load, steering impact, and uneven ground surfaces, non-uniform motion blur is easily generated during the imaging process. This type of blur exhibits significant non-stationarity and spatial inconsistency, manifesting as varying degrees of blur at different depths and in different areas. Especially under complex lighting and undulating road conditions, the edge texture of defects is severely lost, leading to a significant decrease in the accuracy of detection models in identifying minute cracks and fine potholes.

[0004] Existing deblurring methods are mostly based on fixed depth of field and simplified motion assumptions, assuming that the camera only moves at a constant or uniformly accelerated linear speed along a plane, thus simplifying the blur kernel to a one-dimensional or two-dimensional linear point spread function (PSF). These methods are effective under uniform blur conditions, but they cannot reflect the depth-dependent blur variations caused by multi-degree-of-freedom random jitter in road detection scenarios. Traditional algorithms ignore the differences in projection ratios of different depth regions during the restoration process, resulting in overly sharp distant areas or overly blurred nearby areas, disrupting the overall geometric consistency of the image. Furthermore, while some learning-based deblurring networks can visually restore sharpness, their training samples are mostly drawn from uniformly blurred data under ideal conditions, making it difficult to generalize to non-fixed depth-of-field blurred images containing depth gradients and complex trajectory perturbations.

[0005] Therefore, traditional deblurring methods have significant shortcomings in depth modeling and non-uniform jitter representation, and cannot meet the imaging requirements of multi-source disturbance and multi-scale depth of field conditions in road defect detection. Summary of the Invention

[0006] To address the aforementioned technical problems, this invention provides a method and system for removing camera non-uniform shake blur, which can effectively suppress blur degradation caused by non-uniform shake in complex environments and significantly improve the clarity and structural consistency of road defect detection images.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: The first aspect of the present invention provides a method for removing camera non-uniform shake blur.

[0008] In one or more embodiments, a method for removing camera non-uniform shake blur is provided, comprising: The system acquires color images captured by the camera and their corresponding exposure times, as well as inertial data collected by the inertial measurement unit installed on the camera platform and real-time ranging information collected by the laser ranging module. The inertial data collected by the inertial measurement unit is calibrated, and then attitude solution and displacement integration are performed to recover the true motion trajectory of the camera during the exposure time, so as to realize the spatial representation of non-uniform jitter. The camera's actual motion trajectory during the exposure time is reconstructed into a three-dimensional motion trajectory in the world coordinate system. Combined with the real-time ranging information of the laser ranging module, a dynamic mapping relationship between physical space and pixel plane is established to achieve scale-adaptive modeling under non-fixed depth of field conditions. Based on the dynamic mapping relationship between physical space and pixel plane, the reconstructed 3D motion trajectory is projected onto pixel plane, and a spatially variable point spread function is generated and normalized to characterize the imaging blur characteristics caused by non-uniform jitter. For each channel of the color image, a frequency domain filtering restoration operation is performed based on the color image acquired by the camera and the normalized spatial variable point spread function to obtain the filtered restored image. This image is then used as the initial estimate, and a multiplicative iterative update is performed. Finally, the filtered restored images of all channels are fused together to obtain the final deblurred color image.

[0009] As one implementation method, the process of generating and normalizing the spatially variable point spread function is as follows: Based on the dynamic mapping relationship between physical space and pixel plane, the set of time-series trajectories corresponding to the reconstructed 3D motion trajectory is obtained: The time series trajectory set is normalized to a spatially variable point spread function plane with a fixed resolution to form a two-dimensional matrix of the original spatially variable point spread function. The trajectory points in the time series trajectory set are accumulated and statistically analyzed on it to establish a pixel-level energy distribution model to reflect the diffusion path of pixel energy during the exposure of the camera. The original two-dimensional matrix of the spatially variable point spread function is smoothed by convolution, and then the smoothed two-dimensional matrix of the spatially variable point spread function is normalized.

[0010] As one implementation method, the formula for the normalized spatially variable point diffusion function is: ; in, The grid coordinates are for the point spread function; Point spread function grid coordinates The normalized spatial variable point spread function; Point spread function grid coordinates The energy distribution matrix of the smoothed trajectory; Point spread function grid coordinates The energy distribution matrix of the smoothed trajectory; The grid side length is the spatially variable point diffusion function. To prevent extremely small constants with a denominator of zero, when the variance of the pixel trajectory set is less than a set threshold, the spatially variable point spread function degenerates into a unit pulse kernel: ; It is a two-dimensional discrete pulse function; The coordinates of the center of the spatially variable point diffusion function grid are given.

[0011] As one implementation method, the dynamic mapping relationship between physical space and the pixel plane includes a displacement mapping relationship based on smooth pixel scale: ; ; ; ; ; in, Indicates the first Each sampling time; For the first The original pixel scale corresponding to each sampling time; The pixel scale is after smoothing. Indicates the first The original pixel scale corresponding to each sampling time; Camera focal length, expressed in pixels; For the first The imaging depth measured by the laser ranging module at each sampling time; The radius of the sliding window; This refers to the planar displacement components relative to the start of the exposure in the world coordinate system. These represent pixel displacements in the horizontal and vertical directions, respectively. The reference pixel position corresponding to the start of exposure; For the first The pixel plane trajectory points corresponding to each sampling time.

[0012] As one implementation method, the frequency domain filtering and restoration operation uses Wiener filtering and restoration, and its corresponding formula is: ; ; ; ; in, This is the normalized spatial variable point diffusion function; The frequency domain optical transfer function is the spatially variable point spread function. for The complex conjugate; The input is a blurred image; The spectrum of the input blurred image; This is the initial restored spectrum after Wiener filtering; This is the Wiener noise balance factor; The initial restored image obtained by Wiener filtering; These are the pixel coordinates of the image; The grid coordinates are for the spatially variable point diffusion function. Frequency domain coordinates; For color channels; It is a two-dimensional fast Fourier transform; It is a two-dimensional inverse fast Fourier transform.

[0013] As one implementation method, during the multiplicative iterative update process, the first... The calculation for the next iteration is as follows: ; in, For the first Image restored in one iteration; For the first Image restored in one iteration; Input a blurred image; This is the normalized spatial variable point diffusion function; The inverted adjoint kernel of the spatially variable point diffusion function; This refers to convolution operations in the spatial domain. To prevent extremely small constants with a denominator of zero; These are the pixel coordinates of the image; The grid coordinates are for the spatially variable point diffusion function. For color channels.

[0014] As one implementation method, the final deblurred color image is set as follows: ; in, The final output is a deblurred color image; These are the pixel coordinates of the image; For the first Each color channel at the termination iteration number The restored image obtained below; Assign weights to color channels, and satisfy the following: .

[0015] A second aspect of the present invention provides a system for removing camera non-uniform shake blur.

[0016] In one or more embodiments, a system for removing camera non-uniform shake blur includes: The multi-source data acquisition module is used to acquire color images captured by the camera and their corresponding exposure times, as well as inertial data collected by the inertial measurement unit installed on the camera platform and real-time ranging information collected by the laser ranging module. The real motion trajectory recovery module is used to calibrate the inertial data collected by the inertial measurement unit, and then perform attitude solving and displacement integration to recover the real motion trajectory of the camera during the exposure time, thereby realizing the spatial representation of non-uniform jitter. The dynamic mapping relationship construction module is used to reconstruct the real motion trajectory of the camera during the exposure time into a three-dimensional motion trajectory in the world coordinate system. Combined with the real-time ranging information of the laser ranging module, a dynamic mapping relationship between physical space and pixel plane is established to realize scale adaptive modeling under non-fixed depth of field conditions. The variable point spread function generation module is used to project the reconstructed 3D motion trajectory onto the pixel plane according to the dynamic mapping relationship between physical space and pixel plane, generate and normalize the spatial variable point spread function to characterize the imaging blur characteristics caused by non-uniform jitter. The deblurred color image generation module performs frequency domain filtering and restoration operations on each channel of the color image based on the color image captured by the camera and the normalized spatial variable point spread function. The resulting filtered and restored image is used as the initial estimate, and multiplicative iterative updates are performed. The filtered and restored images of all channels are then fused to obtain the final deblurred color image.

[0017] A third aspect of the present invention provides a computer-readable storage medium.

[0018] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the method for removing camera non-uniform shake blur as described above.

[0019] A fourth aspect of the present invention provides an electronic device.

[0020] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps in the method for removing camera non-uniform shake blur as described above.

[0021] Compared with the prior art, the beneficial effects of the present invention are: The camera non-uniform jitter blur removal method of this invention combines the dynamic attitude information of the inertial measurement unit and the real-time depth measurement of the laser ranging module to establish a dynamic mapping relationship between physical space and the pixel plane. This enables scale-adaptive modeling under non-fixed depth-of-field conditions and high-precision modeling of spatial non-uniform blur caused by complex vibrations. Based on the dynamic mapping relationship between physical space and the pixel plane, the reconstructed 3D motion trajectory is projected onto the pixel plane, generating and normalizing a spatially variable point spread function to characterize the imaging blur characteristics caused by non-uniform jitter. Finally, through frequency domain filtering and frequency domain iterative restoration strategies, noise amplification and artifact generation are effectively suppressed while preserving detailed textures. This achieves joint constraints of the blur kernel and depth of field at the pixel level, solving the problem of uneven restoration caused by the fixed scale of traditional methods. The camera non-uniform jitter blur removal method of this invention can be widely applied in road defect detection, UAV inspection, and intelligent vision measurement, providing reliable physical support and data assurance for image quality restoration and visual inspection algorithms in complex dynamic environments. Attached Figure Description

[0022] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0023] Figure 1 This is a flowchart of a method for removing camera non-uniform shake blur according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the triaxial acceleration zero-bias calibration vector and the triaxial angular velocity zero-bias calibration vector according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the three-dimensional displacement trajectory of the camera platform in the world coordinate system according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the three-dimensional motion trajectory projected onto the pixel plane according to an embodiment of the present invention; Figure 5 This is a schematic diagram of pixel trajectory coordinates normalized to a point spread function grid at a fixed resolution according to an embodiment of the present invention. Figure 6 This is a comparative diagram showing the original road damage image, the simulated motion-blurred damage image, and the images processed by different restoration methods, representing an embodiment of the present invention. Detailed Implementation

[0024] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0025] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0026] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0027] Terminology Explanation IMU (Inertial Measurement Unit): Composed of a three-axis accelerometer and a three-axis gyroscope, it is used to acquire linear acceleration and angular velocity signals of the camera platform to reconstruct attitude and motion trajectory.

[0028] LRM (Laser Ranging Module): A laser ranging module that measures the real-time distance from the camera's optical center to the target surface based on the time-of-flight (ToF) or phase difference principle, providing depth data for pixel-scale calculations.

[0029] PSF (Point Spread Function): The point spread function describes the light intensity diffusion distribution caused by camera shake and is a core parameter for image blurring and restoration calculations.

[0030] Non-fixed depth of field: refers to the fact that different regions in an imaging scene have different distance distributions, causing the blur kernel to exhibit non-uniformity as the depth changes.

[0031] Pixel Scale: The mapping ratio of physical displacement to the image plane, defined as pixels per meter, used to map the IMU reconstructed trajectory to pixel-level motion.

[0032] Wiener filtering (Wiener Deconvolution): A frequency-domain-based linear restoration algorithm that achieves smooth deblurring by using a noise constraint parameter K.

[0033] Richardson–Lucy FFT (RL-FFT) algorithm: A non-blind iterative deblurring method with frequency domain multiplicative updates that can prevent over-enhancement while preserving edge details.

[0034] Figure 1 A schematic diagram illustrating the principle of a camera non-uniform shake blur removal method according to an embodiment of the present invention is provided. Based on... Figure 1 The method for removing camera non-uniform shake blur in this embodiment may include the following steps S101 to S105.

[0035] The specific implementation process of steps S101 to S105 is as follows: Step S101: Acquire the color image captured by the camera and its corresponding exposure time, and simultaneously acquire the inertial data collected by the inertial measurement unit installed on the camera platform and the real-time ranging information collected by the laser ranging module.

[0036] Inertial Measurement Units (IMUs) are widely used for attitude estimation and motion trajectory reconstruction due to their high sampling frequency and miniaturization. Using the three-axis accelerometers and gyroscopes within the IMU, linear acceleration and angular velocity data of the camera mounting platform can be acquired in real time. After static interval zero-bias calibration and gravity separation, the true motion signal under mechanical conditions can be obtained. Further time synchronization and integration operations allow for the reconstruction of the camera's continuous displacement trajectory in three-dimensional space. This trajectory can be used to generate a Position Frame Analyzer (PSF) describing jitter characteristics, enabling fuzzy modeling and restoration based on real motion. However, relying solely on IMUs for trajectory calculation suffers from integral drift and accumulated errors, causing its accuracy to rapidly decline over time. More importantly, IMUs cannot reflect scene depth information and cannot distinguish the differences in blur scale between near and far targets on the image plane, leading to distorted PSF mapping ratios at different depths and a "overcompensated for distant objects, undercompensated for nearby objects" phenomenon in the restored image. PSFs generated by a single IMU trajectory ignore the impact of depth-of-field changes on pixel diffusion, while learning- or statistical network methods lack physical interpretability.

[0037] The introduction of a Laser Ranging Module (LRM) provides a physical basis for blur removal in non-fixed depth-of-field (DOF) scenarios. Based on Time-of-Flight (ToF) or Phase Difference Ranging principles, the LRM can measure the depth information between the camera and the foreground scene in real time. This depth ranging data is used to establish the mapping ratio between physical space and the pixel plane (Pixel Scale), thereby enabling adaptive depth scaling of the IMU trajectory during the blur kernel generation stage. By fusing the 3D displacement trajectory reconstructed by the IMU with the LRM's depth data, a spatially variable point spread function (PSF) with non-fixed depth of field can be formed at the pixel level. Unlike traditional fixed-scale PSFs, this PSF is densely distributed and energy-concentrated in the near-field region, while exhibiting a smoother spread pattern in the far-field region, effectively conforming to the perspective laws of real optical imaging.

[0038] For example, the camera at frame rate Image acquisition is performed, and each frame of the image corresponds to a unique timestamp. The image sequence can be represented as: ; in, Indicates time Images of road defects collected in real time. This represents the total number of frames.

[0039] To address the issue of image blurring caused by uneven shaking of vehicle-mounted cameras under dynamic driving conditions during road defect detection, a novel approach is needed to read... Blurry image in and its corresponding timestamp And record the camera's focal length. Optical center coordinates Imaging geometric parameters are used for subsequent frequency domain deblurring calculations.

[0040] To ensure synchronization of multi-source data, the reference time axis of the camera clock, IMU clock, and LRM clock is unified: ; Simultaneously, the IMU's output triaxial acceleration and triaxial angular velocity signals were acquired, and the raw measurement data are expressed as follows: ; in, Camera acceleration measurement (unit: m / s²); These are the accelerations along the X, Y, and Z axes, respectively. Angular velocity measurement (unit: rad / s); These are the angular velocity measurements along the X, Y, and Z axes, respectively. The IMU data sampling frequency is set to... Time interval ; This represents the transpose of a matrix.

[0041] Acquire LRM depth information. The LRM module measures the distance from the camera's optical center to the road surface in real time, i.e., the depth. The distance measurement relationship can be expressed as: ; in, Speed ​​of light (unit: m / s) The time difference between laser emission and reception (in seconds). The output frequency of the LRM is denoted as... The sampling time interval is .

[0042] Furthermore, to ensure that the asynchronous sampling data from the IMU and LRM are aligned with the image timestamps, linear interpolation or spline interpolation methods are used to resample each time series onto a unified time axis. .

[0043] The interpolation form is: ; in, Can be used as , or .

[0044] In other embodiments, the depth corresponding to each frame of the image can be obtained in the following manner: Mount the LRM and camera on a rigid support, keeping their optical axes as parallel as possible and recording their angular deviation. Read the LRM output file and extract the timestamp sequence. With distance measurement sequence .

[0045] Based on timestamp information Establish a distance interpolation function: ; in, Linear interpolation operator, to obtain the depth D corresponding to each frame of the image. ).

[0046] Step S102: The inertial data collected by the inertial measurement unit is calibrated, and then attitude solution and displacement integration are performed to recover the true motion trajectory of the camera during the exposure time, so as to realize the spatial representation of non-uniform jitter.

[0047] Specifically, the implementation process of step S102 is as follows: Read acceleration from IMU file With angular velocity raw data ; Select the number of samples during the static period at the initial stage of camera startup. Assuming the camera is in a state of no motion during this phase (both velocity and angular velocity are zero), the zero bias vectors of the accelerometer and gyroscope can be calculated as follows: and:

[0048] ; ; in, For timestamp information The corresponding original acceleration measurement value (m / s²). For timestamp information The corresponding original angular velocity measurement value (rad / s). The acceleration and angular velocity signals after zero-bias calibration are respectively... and : ; ; During the stationary phase, the acceleration experienced by the camera platform is primarily due to Earth's gravity. Therefore, the mean static acceleration can be approximated as the component of the gravity vector within the IMU system, yielding the gravity term. : ; in, Its magnitude is close to the gravitational acceleration constant. . Figure 2 (a) in the figure is a comparison diagram of the original acceleration and the calibrated acceleration; Figure 2 (b) in the figure is a comparison diagram of the original angular velocity and the calibrated angular velocity; Figure 2 (c) in the figure is a timestamp comparison distribution map.

[0049] The direction of gravity is used to separate the linear acceleration components. To decouple the linear acceleration during the motion phase, the gravity term needs to be removed from the measured acceleration to obtain pure linear acceleration. : ; Because there are slight differences in the sampling frequency of data from each channel of the IMU, uniform interpolation compensation is performed on the calibrated sequence to maintain time continuity. Let the uniform time step be... Then the acceleration after resampling for: ; Among them, timestamp information express Each sampling time; timestamp information express Each sampling time; For timestamp information The corresponding acceleration; For timestamp information The corresponding acceleration.

[0050] Similarly, we can obtain This ensures that acceleration and angular velocity are synchronized and consistent in the time dimension.

[0051] Step S103: Reconstruct the actual motion trajectory of the camera during the exposure time into a three-dimensional motion trajectory in the world coordinate system, and then combine it with the real-time ranging information of the laser ranging module to establish a dynamic mapping relationship between physical space and pixel plane, so as to realize scale adaptive modeling under non-fixed depth of field conditions.

[0052] Specifically, this step includes 3D motion trajectory reconstruction in the world coordinate system, LRM depth synchronization, dynamic pixel scale calculation, and mapping of 3D trajectory to pixel trajectory.

[0053] First, using the angular velocity calibrated by the IMU Calculate the camera's pose change during the exposure time. To avoid singularities in the Euler angle representation, quaternions are used. Describe the camera pose. The discrete update form of the camera pose quaternion is: ; in, For the first Camera pose quaternions at each sampling time; The angular velocity after zero-bias calibration; To standardize the time step; This is the quaternion update matrix constructed from angular velocities; This represents the quaternion normalization operation, used to suppress numerical drift during integration.

[0054] Furthermore, quaternions Convert to rotation matrix And the linear acceleration under the IMU system Transform to world coordinate system: ; in, The linear acceleration after removing the effects of zero bias and gravity in the machine system; This refers to linear acceleration in the world coordinate system.

[0055] Then, the velocity and displacement of the camera platform in the world coordinate system are solved using discrete integration: ； ; in, The velocity vector in the world coordinate system; This is the camera position vector in the world coordinate system; Indicates the first Each sampling time; Indicates the first Each sampling time.

[0056] This allows us to obtain the continuous three-dimensional motion trajectory within the exposure time: ; in, A set of three-dimensional motion trajectories in the world coordinate system; This represents the number of simultaneous sampling points during the exposure time. For the first The coordinates of the three-dimensional trajectory points corresponding to each sampling time.

[0057] The reconstruction process of the three-dimensional motion trajectory in the world coordinate system is consistent with the above-mentioned attitude integration, coordinate transformation and displacement integration processes. Figure 3 This is used to illustrate the three-dimensional displacement changes of the camera platform during the exposure period and does not constitute an alternative embodiment different from the above formula.

[0058] Furthermore, first-order linear interpolation or spline interpolation is used to synchronize the depth data output by the LRM ranging module to a unified time axis: ; in, This is the original ranging sequence of the LRM; For the first The imaging depth corresponding to each synchronous sampling moment; This represents a linear interpolation or spline interpolation operator.

[0059] Based on the pinhole imaging model, the original pixel scale is defined as: ; in, This is the original pixel scale, in pixels per meter (m). Camera focal length, expressed in pixels; The imaging depth is measured by LRM.

[0060] Considering the potential for instantaneous outliers or jump points in the laser ranging signal, to ensure the stability of scale calculations, [the following is necessary:] Perform smoothing using moving average: ; in, The pixel scale is after smoothing. The radius of the sliding window; The sampling time is within the sliding window.

[0061] To convert the physical motion trajectory reconstructed by the IMU into a pixel-plane trajectory, in order to determine the three-dimensional position at the start of the exposure. For reference, calculate the relative displacement: ; in, For the camera in the The displacement vector of each sampling moment relative to the exposure start moment; These represent the relative displacement components in three directions in the world coordinate system.

[0062] In road inspection scenarios, when the exposure time is short and the local road surface can be approximated as a continuous plane, a smoothed pixel scale is used. Map the horizontal and vertical physical displacements in the world coordinate system to the pixel plane: ; ; ; ; in, The reference pixel position corresponding to the start of exposure; For the first The pixel plane trajectory points corresponding to each sampling time; These represent pixel displacements in the horizontal and vertical directions, respectively.

[0063] This yields a set of time-series trajectories on the pixel plane: ; in, This is the set of pixel trajectories used for subsequent spatially variable point spread function generation.

[0064] In another implementation, a complete pinhole imaging model can be used to establish the projection relationship between world coordinates and the pixel plane: ; in, It is a homogeneous scaling factor; This is the camera intrinsic parameter matrix; The camera pose rotation matrix; Let be the translation vector of the camera in the world coordinate system; These are three-dimensional trajectory points in the world coordinate system.

[0065] The above is based on Both the scale mapping model and the complete pinhole projection model are used to convert 3D motion trajectories into pixel-plane trajectories, such as... Figure 4 As shown. In this embodiment, a smooth pixel scale is preferably used. Depth-adaptive scaling is applied to the IMU reconstructed trajectory to improve the stability of PSF generation under non-fixed depth of field conditions.

[0066] Step S104: Based on the dynamic mapping relationship between physical space and pixel plane, the reconstructed three-dimensional motion trajectory is projected onto pixel plane, and a spatially variable point spread function is generated and normalized to characterize the imaging blur characteristics caused by non-uniform jitter.

[0067] Specifically, based on the set of pixel plane trajectories obtained in step S103: ; Generate a spatially variable point spread function. This trajectory set is jointly determined by the 3D motion trajectory reconstructed by the IMU and the depth information measured by the LRM, where the smoothed pixel scale... Used to convert physical displacement into pixel displacement. Therefore, Directly participate in the pixel trajectory set The generation of this process further influences the shape and scale of the spatially variable point spread function.

[0068] To facilitate matrix-based calculations, the pixel trajectory coordinates are normalized to a fixed-resolution PSF grid, such as... Figure 5 As shown. Let the PSF mesh size be... Then the first The discrete grid coordinates corresponding to each trajectory point are: ; ; in, For the first Discrete coordinates of each trajectory point in the PSF grid; These are the minimum and maximum values ​​of the x-coordinate of the pixel trajectory, respectively; These are the minimum and maximum values ​​of the ordinate of the pixel trajectory, respectively; The side length of the PSF mesh; To prevent extremely small constants with a denominator of zero; This indicates the operation of retrieving the nearest integer.

[0069] Initialize trajectory energy accumulation matrix : ; in, This indicates the camera shake trajectory within the PSF grid coordinates during the exposure time. The accumulated energy value at each trajectory point. Energy accumulation is performed for each trajectory point: ; in, Represents PSF grid coordinates In the current frame, a point is identified as a "potential target point" or a "valid observation point," and its energy value is increased by one unit weight. If a point appears in a position consistent with motion patterns in multiple consecutive frames, its energy value increases linearly; however, random noise, due to its irregular location, is unlikely to accumulate continuously along a specific trajectory, and its energy value remains at a low level. This matrix is ​​used to describe the spatial distribution of pixel energy diffusion along the camera shake trajectory during exposure.

[0070] To avoid point-like energy concentration caused by discrete trajectory sampling and to reduce the ringing effect during frequency domain reconstruction, a normalized smooth convolution kernel is used. Perform convolution smoothing on the energy accumulation matrix: ; in, The smoothed trajectory energy distribution matrix; Normalized smooth convolution kernel; This represents the spatial domain convolution operation.

[0071] To satisfy the energy conservation constraint of the optical system, the smoothed trajectory energy distribution matrix is ​​normalized to obtain the spatially variable point spread function: ; in, The grid coordinates are for the point spread function; Point spread function grid coordinates The normalized spatial variable point spread function; Point spread function grid coordinates The energy distribution matrix of the smoothed trajectory; Point spread function grid coordinates The energy distribution matrix of the smoothed trajectory; The side length of the PSF mesh; To prevent extremely small constants with a denominator of zero.

[0072] When pixel trajectory set The trajectory variance is less than the set threshold When the amplitude of camera motion is small during the exposure, the spatially variable point spread function degenerates into a unit pulse kernel. ; in, It is a two-dimensional discrete pulse function; These represent the coordinates of the center of the spatially variable point diffusion function grid. This case corresponds to an approximately static imaging state.

[0073] Therefore, we obtain Simultaneously, the motion trajectory information including IMU reconstruction and pixel scale information under LRM depth constraints are obtained, which can be used for subsequent frequency domain non-blind restoration.

[0074] Step S105: For each channel of the color image, perform frequency domain filtering restoration operation based on the color image acquired by the camera and the normalized spatial variable point spread function to obtain the filtered restored image and use it as the initial estimate. Perform multiplicative iterative update, fuse the filtered restored images after iterative updates of all channels, and obtain the final deblurred color image.

[0075] In this embodiment, based on the normalized spatial variable point spread function (PSF), a frequency domain convolution operation is performed between the PSF and the blurred image. The calculation is accelerated by Fast Fourier Transform (FFT) to achieve non-blind deblurring restoration. This process is completed in two stages: Wiener preliminary filtering and Richardson–Lucy frequency domain iteration, in order to suppress noise, restore details, and maintain energy conservation.

[0076] Specifically, it includes: Based on the normalized spatial variable point diffusion function obtained in step S104 First, calculate its frequency domain optical transfer function: ; in, This is the frequency domain optical transfer function corresponding to the spatially variable point diffusion function; Represents a two-dimensional Fast Fourier Transform; These are frequency domain coordinates.

[0077] For input blurred image Perform a two-dimensional fast Fourier transform on each color channel: ; in, For the input blurred image, the first The spectrum of each color channel; These are the pixel coordinates of the image; For color channels.

[0078] Furthermore, Wiener filtering is used for preliminary restoration: ； ； in, This is the initial restored spectrum after Wiener filtering; This is the initial restored image; for The complex conjugate; This is the Wiener noise balance coefficient, used to control the trade-off between noise reduction and fidelity. This represents the two-dimensional inverse fast Fourier transform.

[0079] Obtained by Wiener filtering As the initial estimate for the Richardson–Lucy iteration: ; No. The next iteration update is as follows: ; in, For the first Image restored in one iteration; For the first Image restored in one iteration; For spatially variable point spread functions; express The flipping is accompanied by the nucleus; This refers to convolution operations in the spatial domain. To prevent extremely small constants with a denominator of zero.

[0080] In the frequency domain implementation, the above equation can be equivalently expressed as: ; in, The frequency domain optical transfer function of the PSF is given. It is its conjugate.

[0081] To prevent excessive iteration from amplifying noise and causing ringing effects, let the relative rate of change between two adjacent iterations be: ; When the following conditions are met: Or reach the maximum number of iterations The iteration terminates when the iteration number at termination is reached. Then the first The restoration results for each color channel are as follows: .

[0082] The color channels are blended to obtain the final deblurred color image: ; in, Assign weights to color channels, and satisfy the following: 。

[0083] In another implementation, the restored results of the three color channels can be directly stitched together according to the RGB channels to obtain the final deblurred color image: ; To prevent negative values, overflow, or edge artifacts from frequency domain restoration, the output image is intensity-cropped. ; in, This represents the maximum value of the image's grayscale or color intensity. For an 8-bit image, .

[0084] In the blur removal stage, a non-uniform PSF is input to the frequency domain restoration stage. Wiener filtering achieves global smooth restoration by estimating the signal-to-noise ratio parameter (K), suppressing high-frequency noise; while the Richardson–Lucy (RL-FFT) iterative algorithm based on Fast Fourier Transform performs multiplicative updates in the frequency domain, preventing ringing effects caused by over-iteration through an early stopping mechanism. The combination of these two methods enables dynamic adjustment of the deblurring intensity at different depth layers, thereby achieving high-fidelity restoration under non-fixed depth of field. Through the above frequency domain algorithm, each pixel in the image undergoes independent deconvolution based on its corresponding depth and local PSF, achieving gradual restoration of detailed structures and significantly improving the edge sharpness and texture consistency of lesion features.

[0085] Figure 6 This is a comparative diagram showing the original road defect image, the simulated motion-blurred defect image, and the images processed by different restoration methods, according to an embodiment of the present invention. Figure 6 (a) in the image is the original image of the road damage. Figure 6 (b) in the image is a simulated motion-blurred image of the defect; Figure 6 (c) in the figure is the image after processing the simulated motion blur disease image using the Wiener filtering method with an estimated signal-to-noise ratio parameter of 0.01; Figure 6 In the image, (d) is the image after processing the simulated motion blur disease image using the RL-FFT algorithm with 25 iterations.

[0086] The method for removing camera non-uniform shake blur of the present invention utilizes IMU to achieve three-dimensional trajectory reconstruction, establishes an adaptive mapping from physical space to image plane through real-time depth ranging provided by LRM, generates multi-level depth-variable PSF, and completes blur removal by combining frequency domain non-blind restoration algorithm.

[0087] It should be noted that other existing frequency domain filtering and restoration methods can also be used to obtain the filtered and restored image, which will not be described in detail here.

[0088] In one or more embodiments, a camera non-uniform blur removal system is also provided, which can be implemented in software. The camera non-uniform blur removal system includes the following software modules: The multi-source data acquisition module is used to acquire color images captured by the camera and their corresponding exposure times, as well as inertial data collected by the inertial measurement unit installed on the camera platform and real-time ranging information collected by the laser ranging module. The real motion trajectory recovery module is used to calibrate the inertial data collected by the inertial measurement unit, and then perform attitude solving and displacement integration to recover the real motion trajectory of the camera during the exposure time, thereby realizing the spatial representation of non-uniform jitter. The dynamic mapping relationship construction module is used to reconstruct the real motion trajectory of the camera during the exposure time into a three-dimensional motion trajectory in the world coordinate system. Combined with the real-time ranging information of the laser ranging module, a dynamic mapping relationship between physical space and pixel plane is established to realize scale adaptive modeling under non-fixed depth of field conditions. The variable point spread function generation module is used to project the reconstructed 3D motion trajectory onto the pixel plane according to the dynamic mapping relationship between physical space and pixel plane, generate and normalize the spatial variable point spread function to characterize the imaging blur characteristics caused by non-uniform jitter. The deblurred color image generation module performs frequency domain filtering and restoration operations on each channel of the color image based on the color image captured by the camera and the normalized spatial variable point spread function. The resulting filtered and restored image is used as the initial estimate, and multiplicative iterative updates are performed. The filtered and restored images of all channels are then fused to obtain the final deblurred color image.

[0089] It should be noted that each module in the camera non-uniform shake blur removal system of the present invention corresponds one-to-one with each step in the camera non-uniform shake blur removal method in the above embodiments, and their specific implementation processes are the same, so they will not be repeated here.

[0090] The structure of the electronic device according to embodiments of the present invention will be described in detail below. The electronic device provided in the embodiments of the present invention includes: at least one processor, a memory, a user interface, and at least one network interface. The various components in the camera non-uniform blur removal system are coupled together through a bus system. It can be understood that the bus system is used to realize the connection and communication between these components. In addition to a data bus, the bus system also includes a power bus, a control bus, and a status signal bus. The user interface may include a display, keyboard, mouse, trackball, click wheel, buttons, a touchpad, or a touch screen, etc.

[0091] It is understood that the memory can be volatile memory or non-volatile memory, or both. The memory in this embodiment of the invention is capable of storing data to support the operation of the terminal. Examples of this data include any computer programs used to operate on the terminal, such as operating systems and applications. The operating system includes various system programs, such as the framework layer, core library layer, driver layer, etc., used to implement various basic services and handle hardware-based tasks. Applications can include various applications.

[0092] In some embodiments, the camera non-uniform blur removal system provided in this invention can be implemented using a combination of hardware and software. For example, the camera non-uniform blur removal system provided in this invention can be a processor in the form of a hardware decoding processor, programmed to execute the camera non-uniform blur removal method provided in this invention. For instance, the hardware decoding processor can employ one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0093] As an example, a processor can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where a general-purpose processor can be a microprocessor or any conventional processor, etc.

[0094] As an example of the hardware implementation of the camera non-uniform blur removal system provided in this embodiment of the invention, the device provided in this embodiment of the invention can be directly executed by a processor in the form of a hardware decoding processor. For example, it can be executed by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components to implement the camera non-uniform blur removal method provided in this embodiment of the invention.

[0095] The memory in this embodiment of the invention is used to store various types of data to support the operation of the camera non-uniform blur removal system, or to store data for execution. Figure 1 The program code for the method shown. Examples of this data include: any executable instructions for operation on a camera non-uniform blur removal system, such as executable instructions that can be included in the executable instructions to implement the camera non-uniform blur removal method of the embodiments of the present invention.

[0096] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including functions for executing... Figure 1 The program code for the method shown. In such an embodiment, the computer program can be downloaded and installed from a network via a communication component, and / or installed from a removable medium. When the computer program is executed by the central processing unit, it performs the various functions defined in the apparatus of this application.

[0097] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0098] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for removing camera non-uniform shake blur, characterized in that, include: The system acquires color images captured by the camera and their corresponding exposure times, as well as inertial data collected by the inertial measurement unit installed on the camera platform and real-time ranging information collected by the laser ranging module. The inertial data collected by the inertial measurement unit is calibrated, and then attitude solution and displacement integration are performed to recover the true motion trajectory of the camera during the exposure time, so as to realize the spatial representation of non-uniform jitter. The camera's actual motion trajectory during the exposure time is reconstructed into a three-dimensional motion trajectory in the world coordinate system. Combined with the real-time ranging information of the laser ranging module, a dynamic mapping relationship between physical space and pixel plane is established to achieve scale-adaptive modeling under non-fixed depth of field conditions. Based on the dynamic mapping relationship between physical space and pixel plane, the reconstructed 3D motion trajectory is projected onto pixel plane, and a spatially variable point spread function is generated and normalized to characterize the imaging blur characteristics caused by non-uniform jitter. For each channel of the color image, a frequency domain filtering restoration operation is performed based on the color image acquired by the camera and the normalized spatial variable point spread function to obtain the filtered restored image. This image is then used as the initial estimate, and a multiplicative iterative update is performed. Finally, the filtered restored images of all channels are fused together to obtain the final deblurred color image.

2. The method for removing camera non-uniform shake blur as described in claim 1, characterized in that, The process of generating and normalizing the spatially variable point spread function is as follows: Based on the dynamic mapping relationship between physical space and pixel plane, the set of time-series trajectories corresponding to the reconstructed 3D motion trajectory is obtained: The time series trajectory set is normalized to a spatially variable point spread function plane with a fixed resolution to form a two-dimensional matrix of the original spatially variable point spread function. The trajectory points in the time series trajectory set are accumulated and statistically analyzed on it to establish a pixel-level energy distribution model to reflect the diffusion path of pixel energy during the exposure of the camera. The original two-dimensional matrix of the spatially variable point spread function is smoothed by convolution, and then the smoothed two-dimensional matrix of the spatially variable point spread function is normalized.

3. The method for removing camera non-uniform shake blur as described in claim 1, characterized in that, The formula for the normalized spatially variable point diffusion function is: ； in, The grid coordinates are for the point spread function; Point spread function grid coordinates The normalized spatial variable point spread function; Point spread function grid coordinates The energy distribution matrix of the smoothed trajectory; Point spread function grid coordinates The energy distribution matrix of the smoothed trajectory; The grid side length is the spatially variable point diffusion function. To prevent extremely small constants with a denominator of zero, when the variance of the pixel trajectory set is less than a set threshold, the spatially variable point spread function degenerates into a unit pulse kernel: ; It is a two-dimensional discrete pulse function; The coordinates of the center of the spatially variable point diffusion function grid are given.

4. The method for removing camera non-uniform shake blur as described in claim 1, characterized in that, The dynamic mapping relationship between physical space and the pixel plane includes displacement mapping relationships based on smooth pixel scale: ； ； ； ； ； in, Indicates the first Each sampling time; For the first The original pixel scale corresponding to each sampling time; The pixel scale is after smoothing. Indicates the first The original pixel scale corresponding to each sampling time; Camera focal length, expressed in pixels; For the first The imaging depth measured by the laser ranging module at each sampling time; The radius of the sliding window; This refers to the planar displacement components relative to the start of the exposure in the world coordinate system. These represent pixel displacements in the horizontal and vertical directions, respectively. The reference pixel position corresponding to the start of exposure; For the first The pixel plane trajectory points corresponding to each sampling time.

5. The method for removing camera non-uniform shake blur as described in claim 1, characterized in that, The frequency domain filtering and restoration operation uses Wiener filtering and restoration, and its corresponding formula is: ； ； ； ； in, This is the normalized spatial variable point diffusion function; The frequency domain optical transfer function is the spatially variable point spread function. for The complex conjugate; The input is a blurred image; The spectrum of the input blurred image; This is the initial restored spectrum after Wiener filtering; This is the Wiener noise balance factor; The initial restored image obtained by Wiener filtering; These are the pixel coordinates of the image; The grid coordinates are for the spatially variable point diffusion function. Frequency domain coordinates; For color channels; It is a two-dimensional fast Fourier transform; It is a two-dimensional inverse fast Fourier transform.

6. The method for removing camera non-uniform shake blur as described in claim 1, characterized in that, During the multiplicative iterative update process, the first... The calculation for the next iteration is as follows: ； in, For the first Image restored in one iteration; For the first Image restored in one iteration; Input a blurred image; This is the normalized spatial variable point diffusion function; The inverted adjoint kernel of the spatially variable point diffusion function; This refers to convolution operations in the spatial domain. To prevent extremely small constants with a denominator of zero; These are the pixel coordinates of the image; The grid coordinates are for the spatially variable point diffusion function. For color channels.

7. The method for removing camera non-uniform shake blur as described in claim 1, characterized in that, The final deblurred color image is set as follows: ； in, The final output is a deblurred color image; These are the pixel coordinates of the image; For the first Each color channel at the termination iteration number The restored image obtained below; Assign weights to color channels, and satisfy the following: .

8. A system for removing camera non-uniform shake blur, characterized in that, The method for removing camera non-uniform shake blur based on any one of claims 1-7 includes: The multi-source data acquisition module is used to acquire color images captured by the camera and their corresponding exposure times, as well as inertial data collected by the inertial measurement unit installed on the camera platform and real-time ranging information collected by the laser ranging module. The real motion trajectory recovery module is used to calibrate the inertial data collected by the inertial measurement unit, and then perform attitude solving and displacement integration to recover the real motion trajectory of the camera during the exposure time, thereby realizing the spatial representation of non-uniform jitter. The dynamic mapping relationship construction module is used to reconstruct the real motion trajectory of the camera during the exposure time into a three-dimensional motion trajectory in the world coordinate system. Combined with the real-time ranging information of the laser ranging module, a dynamic mapping relationship between physical space and pixel plane is established to realize scale adaptive modeling under non-fixed depth of field conditions. The variable point spread function generation module is used to project the reconstructed 3D motion trajectory onto the pixel plane according to the dynamic mapping relationship between physical space and pixel plane, generate and normalize the spatial variable point spread function to characterize the imaging blur characteristics caused by non-uniform jitter. The deblurred color image generation module performs frequency domain filtering and restoration operations on each channel of the color image based on the color image captured by the camera and the normalized spatial variable point spread function. The resulting filtered and restored image is used as the initial estimate, and multiplicative iterative updates are performed. The filtered and restored images of all channels are then fused to obtain the final deblurred color image.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the method for removing camera non-uniform shake blur as described in any one of claims 1-7.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the method for removing camera non-uniform shake blur as described in any one of claims 1-7.

Images