A raindrop distorted image restoration method based on multi-frame registration

Abstract

The present application relates to the technical field of computer vision, and is a raindrop distorted image recovery method based on multi-frame registration. The method comprises the following steps: acquiring multiple images collected by the same camera at different times or under different viewing angles, and performing pretreatment; determining a reference frame from the pretreated multiple images by using Laplacian response; establishing a reference coordinate system with the reference frame, and performing alignment processing on the pretreated multiple images to obtain aligned auxiliary frames; determining a raindrop distorted area based on pixel residuals between the reference frame and each aligned auxiliary frame; and performing pixel fusion and recovery on the raindrop distorted area indicated by a mask in the reference frame by using area information in the alignment result of the multiple images which is not affected or less affected by raindrops. The method recovers the raindrop distorted area by using complementary information between frames, thereby improving the image imaging quality under raindrop interference conditions.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method for restoring raindrop-distorted images based on multi-frame registration. Background Technology

[0002] With the development of applications such as intelligent transportation, autonomous driving, video surveillance, and mobile visual perception, image / video-based environmental perception technologies have been widely used. However, in complex outdoor weather conditions, camera equipment is susceptible to the effects of rainfall. Raindrops adhering to the lens surface can cause imaging problems such as geometric distortion, blurring, and abnormal brightness in local areas, thereby reducing the usability of images and videos and affecting the stability of visual tasks such as target detection, tracking, and 3D reconstruction.

[0003] To mitigate image degradation caused by raindrop adhesion, existing technologies typically improve both hardware and software. Hardware solutions, such as hydrophobic coatings, mechanical wipers, or heating devices, often require additional hardware support, resulting in high costs, complex maintenance, and limited effectiveness in heavy rain or high-speed motion scenarios. In software solutions, some methods treat the raindrop-affected area as an occlusion region and employ image inpainting; however, because raindrops are usually accompanied by complex geometric distortions and optical aberrations, traditional interpolation or texture replication methods are prone to artifacts or blurring. While deep learning-based methods can improve visual effects, they typically rely on large amounts of paired training data, leading to high data acquisition costs and potentially limited generalization capabilities under different lens conditions, raindrop shapes, and scene variations. Furthermore, single-frame restoration often lacks sufficient information when there is large-area distortion or missing information, making it difficult to stably reconstruct the true structure.

[0004] Compared to single-frame images, video sequences often exhibit camera micro-motion or viewpoint changes between adjacent frames, providing complementary inter-frame information for raindrop distortion recovery. However, existing multi-frame correlation methods primarily focus on denoising or deblurring, neglecting the local nonlinear geometric distortions caused by raindrops. Furthermore, the utilization of multi-frame information typically relies on accurate inter-frame registration; if the registration is not robust, misalignment fusion errors can easily be introduced, resulting in insufficient stability in complex raindrop morphologies or dynamic scenes.

[0005] Therefore, there is an urgent need for a method that can achieve robust registration under multi-frame conditions and utilize inter-frame complementary information to restore the distorted regions of raindrops. Summary of the Invention

[0006] To address the issue of nonlinear image distortion caused by raindrops adhering to the camera lens surface, and the insufficient stability of existing technologies in restoring distorted regions under complex raindrop morphologies and multi-frame perspectives, this invention provides a raindrop-distorted image restoration method based on multi-frame registration. This method utilizes inter-frame complementary information to restore the distorted regions of raindrops, thereby improving image quality under raindrop interference conditions.

[0007] The technical means employed in this invention are as follows:

[0008] A method for raindrop distortion image restoration based on multi-frame registration includes: acquiring multiple frames of images captured by the same camera at different times or under different viewing angles, and preprocessing them; determining a reference frame from the preprocessed multiple frames using the Laplacian response; establishing a reference coordinate system with the reference frame, aligning the preprocessed multiple frames to obtain alignment auxiliary frames; determining raindrop distortion regions based on the pixel residuals between the reference frame and each alignment auxiliary frame; and performing pixel fusion and restoration on the raindrop distortion regions indicated by the mask in the reference frame using information from regions unaffected or minimally affected by raindrops in the multi-frame image alignment results.

[0009] Furthermore, the multi-frame images To describe the same scene content, at least one frame of the image contains a local image distortion region caused by raindrops adhering to the front end of the imaging system; the preprocessing includes: grayscale conversion, noise reduction, contrast normalization, and scale unification; resulting in preprocessed multi-frame images. .

[0010] Furthermore, the reference frame The frames are determined from the preprocessed multi-frame images, and the remaining frames are used as auxiliary frames. The reference frame is used to establish a reference coordinate system for image alignment and serves as the main image to be restored; For any frame Calculate the Laplace response :

[0011] in, The Laplacian operator is used to characterize the second-order spatial transformation properties of an image, and the sharpness evaluation value of each frame of the image is calculated based on the Laplacian response. :

[0012] Among them, the sharpness evaluation value Defined as the variance of the Laplacian response over the entire image range. Represents the set of pixel coordinates of an image. Represents the coordinates of any pixel in the image. Indicates the total number of pixels. Let be the mean of the Laplacian response of the k-th frame; according to Sort selection Larger image as reference frame The selection rule adopts the direct maximum value selection method, letting... Determine the reference frame.

[0013] Furthermore, the step of establishing a reference coordinate system using a reference frame and aligning the preprocessed multi-frame images to obtain an alignment auxiliary frame specifically includes: Extract local feature point sets from the reference frame and any auxiliary frame respectively. With the corresponding feature descriptor subset :

[0014]

[0015] in, This represents the m-th feature point in the reference frame. This represents the total number of feature points detected in the reference frame, and each feature point... A corresponding local image region in the reference frame, whose position can be represented by coordinates. express; Represents feature points in the reference frame The corresponding reference feature descriptor is a vector of length L, i.e. It is used to characterize the local texture and structural information in the neighborhood of the feature point; feature extraction is performed on the auxiliary frame to obtain the feature point set of the auxiliary frame. and corresponding feature descriptor subsets ; This represents the nth feature point in the auxiliary frame. This represents the total number of feature points detected in the auxiliary frame; Indicates feature points in auxiliary frames The corresponding auxiliary feature descriptor has the same dimensions as the reference feature descriptor; an initial matching relationship between the reference frame and the auxiliary frame is established based on the similarity metric between the reference feature descriptor and the auxiliary feature descriptor, for any feature point of the reference frame. In the auxiliary frame description subset The index of the feature point that minimizes the distance to the descriptor is denoted as:

[0016] in, This indicates that in the set of feature points in the auxiliary frame, the m-th feature point is compared with the m-th feature point in the reference frame. Reference feature descriptor, The index of the feature point in the nearest auxiliary frame; Using a preset descriptor distance metric, an initial set of matching point pairs between the reference frame and the auxiliary frame is obtained. :

[0017] in, This represents the m-th feature point in the auxiliary frame compared to the reference frame. Matching feature points.

[0018] Based on the initial set of matching point pairs Estimate the geometric transformation of the auxiliary frame relative to the reference frame. To improve the robustness of the transform estimation, outlier removal is performed on the matching point pairs. A random sample consensus algorithm is used to partition the matching point pairs into inliers and outliers, and the solution is obtained on the set of inliers. Simultaneously, the pixel distance between the matched point and the corresponding point in the reference frame after mapping, i.e., the reprojection error, is used as an evaluation metric to constrain or filter the transformation estimation results. Map the auxiliary frame to the reference frame coordinate system to obtain the aligned auxiliary frame. ;set up x To represent the pixel position in the reference frame coordinate system, the alignment mapping is as follows:

[0019] in, This indicates that the position in the reference coordinate system is reverse-mapped to the position in the auxiliary frame coordinate system. This represents the operation of interpolating and sampling auxiliary frames to obtain pixel values.

[0020] Furthermore, the alignment quality of the alignment auxiliary frame is evaluated using evaluation metrics including the inlier ratio and reprojection error. When the alignment quality is lower than a preset threshold, the corresponding alignment auxiliary frame is removed, or the weight of the auxiliary frame is reduced in the subsequent residual aggregation and restoration fusion process to reduce the impact of misalignment on the determination and restoration results of the distorted region.

[0021] Furthermore, the determination of the raindrop distortion region is based on a reference frame. With each alignment auxiliary frame The pixel residuals between frames are statistically analyzed to obtain a residual response map. When the pixel residuals in the residual response map exceed a preset threshold, it is determined that there is a local distortion caused by raindrops, and the corresponding abnormal area is extracted from the residual response map as the raindrop distortion area. A residual map is constructed based on the pixel-level differences between the reference frame and the alignment auxiliary frame. The residual map It is obtained by weighting the brightness residual and the gradient residual:

[0022] in, For gradient operators, These are preset weighting parameters used to balance the influence of brightness information and structural information in residual calculation; Robust aggregation is performed on the residual maps corresponding to multiple auxiliary frames to obtain a comprehensive residual map. :

[0023] in, For statistical aggregation operators; Based on the comprehensive residual diagram The residual threshold is determined by the statistical distribution of pixel values. And generate an initial raindrop distortion region mask. :

[0024] Morphological processing is performed on the initial mask to remove isolated noise and enhance the spatial coherence of the raindrop region, resulting in an intermediate mask. :

[0025] Where Open is the opening operation and Close is the closing operation. This is a structural element whose size is specified by a preset parameter s; Further Filter the connected components and remove those with areas smaller than a threshold. The connected components are used to obtain the final raindrop distortion region mask M:

[0026] Among them, the output mask , , indicating that pixel x belongs to the raindrop distortion region; Represents the t-th connected component; This represents the area of ​​the connected region. This is a preset area threshold.

[0027] Furthermore, pixel fusion and restoration are performed on the raindrop distortion region. After obtaining the raindrop distortion region mask M, the regions unaffected or minimally affected by raindrops in the multi-frame alignment results are used to perform pixel fusion and restoration on the raindrop distortion region indicated by the mask in the reference frame to obtain the transition output. ; To satisfy The pixel value at position x of the reference frame and each alignment auxiliary frame is used as a candidate information source; the candidate values ​​of the auxiliary frames that are out of bounds, invalid samples or have low alignment quality are eliminated or downweighted to obtain the set of effective candidate frames U. For each valid candidate frame In the local neighborhood of pixel position x Internally calculated quality evaluation value This is used to measure the reliability of the candidate information at that location; the quality evaluation value From candidate frames in The sharpness / texture intensity, consistency residual with the reference frame, and alignment confidence are determined by a weighted combination of one or more factors and normalized for subsequent pixel selection or fusion. Based on the candidate frame set Perform selection or fusion operations on the pixels of the mask region to obtain the initial reconstruction result of the mask region. Smoothing the mask M yields a transition weight map. Reduce seam artifacts at the mask boundaries and fuse the initial restoration result with the reference frame to obtain the transition output. :

[0028] in, Indicates a reference frame; For areas with insufficient candidate information, holes, or texture breaks, patching is performed at the pixel positions indicated by mask M. Perform image inpainting to obtain the final output image. :

[0029] in, For image inpainting operators, These are preset parameters.

[0030] Compared with the prior art, the present invention has the following advantages: This invention addresses the problem of nonlinear image distortion caused by raindrops adhering to the surface of a camera lens. It utilizes complementary information from multiple frames of images to detect and recover the distorted areas caused by raindrops, thereby improving image quality and usability to some extent under raindrop interference conditions.

[0031] This invention aligns and registers multiple frames of images, ensuring that the same scene region in different frames corresponds consistently under a unified reference coordinate system. This provides a basis for the selection and fusion of cross-frame information and reduces fusion errors caused by inter-frame misalignment.

[0032] This invention prioritizes the use of intra / inter-frame information that is less affected by raindrops or has higher imaging quality for compensation and fusion within the raindrop distortion region. This can reduce structural misalignment and boundary artifacts in the distortion region and improve the continuity of the restoration results.

[0033] The method of this invention does not rely on continuously acquired image frames and is adaptable to changes in frame intervals and viewpoints; at the same time, it does not limit specific features, aggregation or repair implementation methods, making it easy to deploy under different lens conditions and application scenarios. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0035] Figure 1 This is a flowchart of the raindrop distortion image restoration method based on multi-frame registration in this invention.

[0036] Figure 2 This is a schematic diagram of the reference frame and multi-view auxiliary frame in this invention.

[0037] Figure 3 This is a schematic diagram of the results of extracting the distortion region of raindrops in this invention.

[0038] Figure 4 This is a schematic diagram of the pixel source distribution based on the clarity preference in this invention.

[0039] Figure 5 This is a schematic diagram comparing the raindrop distortion recovery effect in this invention. Detailed Implementation

[0040] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0041] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the present invention or its application or use. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0042] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0043] Unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps described in these embodiments do not limit the scope of the invention. It should also be understood that, for ease of description, the dimensions of the various parts shown in the drawings are not drawn to actual scale. Techniques, methods, and devices known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and devices should be considered part of the specification. In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values. It should be noted that similar reference numerals and letters in the following figures denote similar items; therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0044] like Figure 1 As shown, this invention provides a method for raindrop distortion image restoration based on multi-frame registration, comprising: acquiring multiple frames of images captured by the same camera at different times or under different viewing angles, and performing preprocessing; in a preferred embodiment of this invention, the multiple frames of images... Used to describe the same scene content, where at least one frame of the image contains local image distortion regions caused by raindrops adhering to the front end of the imaging system; preprocessing includes: grayscale conversion, noise reduction, contrast normalization, and scale unification; resulting in preprocessed multi-frame images. .

[0045] A reference frame is determined from preprocessed multi-frame images using the Laplacian response; in a preferred embodiment of the invention, the reference frame is... The frames are determined from the preprocessed multi-frame images, and the remaining frames are used as auxiliary frames. The reference frame is used to establish a reference coordinate system for image alignment and serves as the main image to be restored; For any frame Calculate the Laplace response :

[0046] in, This represents the Laplacian operator, used to characterize the second-order spatial transformation properties of an image. Sharpness evaluation values ​​for each frame are calculated based on the Laplacian response. :

[0047] Among them, the clarity evaluation value Defined as the variance of the Laplacian response over the entire image range. Represents the set of pixel coordinates of an image. Represents the coordinates of any pixel in the image. Indicates the total number of pixels. Let be the mean of the Laplacian response of the k-th frame; according to Sort selection Larger image as reference frame The selection rule adopts the direct maximum value selection method, letting... Determine the reference frame.

[0048] A reference coordinate system is established using a reference frame, and the preprocessed multi-frame images are aligned to obtain an alignment auxiliary frame. In a preferred embodiment of the invention, local feature point sets are extracted from both the reference frame and any auxiliary frame. With the corresponding feature descriptor subset :

[0049]

[0050] in, This represents the m-th feature point in the reference frame. This represents the total number of feature points detected in the reference frame, and each feature point... A corresponding local image region in the reference frame, whose position can be represented by coordinates. express; Represents feature points in the reference frame The corresponding reference feature descriptor is a vector of length L, i.e. This is used to characterize the local texture and structural information within the neighborhood of a feature point; the reference feature descriptor can be generated using SIFT, SURF, ORB, or other methods. Feature extraction is performed on the auxiliary frame to obtain the auxiliary frame feature point set. and corresponding feature descriptor subsets ; This represents the nth feature point in the auxiliary frame. This represents the total number of feature points detected in the auxiliary frame; Indicates feature points in auxiliary frames The corresponding auxiliary feature descriptor has the same dimensions as the reference feature descriptor; an initial matching relationship between the reference frame and the auxiliary frame is established based on the similarity metric between the reference feature descriptor and the auxiliary feature descriptor, for any feature point of the reference frame. In the auxiliary frame description subset The index of the feature point that minimizes the distance to the descriptor is denoted as:

[0051] in, This indicates that in the set of feature points in the auxiliary frame, the m-th feature point is compared with the m-th feature point in the reference frame. Reference feature descriptor, The index of the feature point in the nearest auxiliary frame; Using a preset descriptor distance metric, an initial set of matching point pairs between the reference frame and the auxiliary frame is obtained. :

[0052] in, This represents the m-th feature point in the auxiliary frame compared to the reference frame. Matching feature points.

[0053] Based on the initial set of matching point pairs Estimate the geometric transformation of the auxiliary frame relative to the reference frame. Geometric transformations can include homography or affine transformations. Outlier culling is performed on matching point pairs to improve the robustness of the transformation estimate. A random sample consensus algorithm is used to partition the matching point pairs into inliers and outliers, and the solution is obtained on the set of inliers. Simultaneously, the pixel distance between the matched point and the corresponding point in the reference frame after mapping, i.e., the reprojection error, is used as an evaluation metric to constrain or filter the transformation estimation results. Map the auxiliary frame to the reference frame coordinate system to obtain the aligned auxiliary frame. ;set up x To represent the pixel position in the reference frame coordinate system, the alignment mapping is as follows:

[0054] in, This indicates that the position in the reference coordinate system is reverse-mapped to the position in the auxiliary frame coordinate system. This represents the operation of interpolating and sampling auxiliary frames to obtain pixel values. The interpolation method can be bilinear interpolation, bicubic interpolation, or other interpolation methods.

[0055] In a specific implementation, as a preferred embodiment of the present invention, the alignment quality of the alignment auxiliary frame is evaluated by evaluation indicators including the inlier ratio and reprojection error. When the alignment quality is lower than a preset threshold, the corresponding alignment auxiliary frame is removed, or the weight of the auxiliary frame is reduced in the subsequent residual aggregation and restoration fusion process to reduce the impact of misalignment on the determination and restoration results of the distorted region.

[0056] The raindrop distortion region is determined based on the pixel residuals between the reference frame and each alignment auxiliary frame; in a preferred embodiment of the present invention, the determination of the raindrop distortion region is based on the reference frame. With each alignment auxiliary frame The pixel residuals between frames are statistically analyzed to obtain a residual response map. When the pixel residuals in the residual response map exceed a preset threshold, it is determined that there is a local distortion caused by raindrops, and the corresponding abnormal area is extracted from the residual response map as the raindrop distortion area. A residual map is constructed based on the pixel-level differences between the reference frame and the alignment auxiliary frame. residual plot It is obtained by weighting the brightness residual and the gradient residual:

[0057] in, For gradient operators, These are preset weighting parameters used to balance the influence of brightness information and structural information in residual calculation; Robust aggregation is performed on the residual maps corresponding to multiple auxiliary frames to obtain a comprehensive residual map. :

[0058] in, For statistical aggregation operators, robust statistical methods such as median, trimmed mean, Huber weighted average, or weighted median can be used. When the alignment quality of some auxiliary frames is poor, their weights can be reduced or the corresponding frames can be removed during aggregation.

[0059] Based on the comprehensive residual diagram The residual threshold is determined by the statistical distribution of pixel values. And generate an initial raindrop distortion region mask. :

[0060] Residual threshold The threshold can be determined using methods such as quantile threshold, mean-variance threshold, or adaptive threshold.

[0061] Morphological processing is performed on the initial mask to remove isolated noise and enhance the spatial coherence of the raindrop region, resulting in an intermediate mask. :

[0062] Where Open is the opening operation and Close is the closing operation. This is a structural element whose size is specified by the preset parameter s; the shape of the structural element can be circular, square, or other shapes.

[0063] Further Filter the connected components and remove those with areas smaller than a threshold. The connected components are used to obtain the final raindrop distortion region mask M:

[0064] Among them, the output mask , , indicating that pixel x belongs to the raindrop distortion region; Represents the t-th connected component; This represents the area of ​​the connected region. This is a preset area threshold.

[0065] By utilizing information from regions unaffected or minimally affected by raindrops in the multi-frame image alignment results, pixel fusion and restoration are performed on the raindrop distortion regions indicated by the mask in the reference frame. Specifically, in a preferred embodiment of this invention, after obtaining the raindrop distortion region mask M, pixel fusion and restoration are performed on the raindrop distortion regions indicated by the mask in the reference frame using information from regions unaffected or minimally affected by raindrops in the multi-frame alignment results, resulting in a transition output. ; To satisfy The pixel value at position x of the reference frame and each alignment auxiliary frame is used as a candidate information source; the candidate values ​​of the auxiliary frames that are out of bounds, invalid samples or have low alignment quality are eliminated or downweighted to obtain the set of effective candidate frames U. For each valid candidate frame In the local neighborhood of pixel position x Internally calculated quality evaluation value This is used to measure the reliability of the candidate information at that location; quality evaluation value. From candidate frames in The sharpness / texture intensity, consistency residual with the reference frame, and alignment confidence are determined by a weighted combination of one or more factors and normalized for subsequent pixel selection or fusion. Based on the candidate frame set Perform selection or fusion operations on the pixels of the mask region to obtain the initial reconstruction result of the mask region. Smoothing the mask M yields a transition weight map. Reduce seam artifacts at the mask boundaries and fuse the initial restoration result with the reference frame to obtain the transition output. :

[0066] in, Indicates a reference frame; It can be obtained by smoothing the mask using Gaussian smoothing, morphological processing, or other methods.

[0067] For areas with insufficient candidate information, holes, or texture breaks, patching is performed at the pixel positions indicated by mask M. Perform image inpainting to obtain the final output image. :

[0068] in, For image inpainting operators, These are preset parameters.

[0069] Example This embodiment provides a raindrop distortion image restoration method based on multi-frame registration. For ease of explanation, this embodiment uses multi-frame image data generated by a computer program as input to simulate the observation results of the same scene under different frame imaging conditions. Specifically, N=6 frames of color checkerboard images are generated, each frame being generated from the same base image through a preset geometric transformation to simulate the inter-frame differences caused by slight camera movement or changes in viewing angle; to simulate the local nonlinear distortion caused by raindrops adhering to the camera lens surface, different degrees of raindrop distortion effects are superimposed on each frame, thereby obtaining multi-frame input data.

[0070] S1. Image Sequence Acquisition Obtain a multi-frame image sequence to describe the same scene. .like Figure 2 As shown, the multi-frame image sequence is the input data generated by the computer program and superimposed with the raindrop distortion effect, which is used for subsequent preprocessing, reference frame determination and registration restoration.

[0071] S2, Image Preprocessing To improve the robustness of subsequent feature extraction and registration alignment, and to reduce the impact of noise and other factors on subsequent residual statistics, the multi-frame image sequence obtained in step S1 is preprocessed. First, each frame of color image... Convert to grayscale image After that, the grayscale image Median filtering is used for denoising to suppress noise interference, resulting in a preprocessed image sequence. .

[0072] S3, Reference Frame Determination According to the sharpness evaluation method in the technical solution, the preprocessed image sequence Calculate the sharpness evaluation value for each frame. The frame with the highest sharpness rating was selected as the reference frame. The remaining frames These serve as auxiliary frames for subsequent registration and alignment.

[0073] S4, Multi-frame Image Alignment Determining the reference frame Then, according to the multi-frame registration and alignment method in the technical solution, each auxiliary frame is... The coordinates are registered to the reference frame coordinate system to establish a cross-frame spatial correspondence. Specifically, in this embodiment, the following processing is performed sequentially on each auxiliary frame: Feature extraction and matching: in the reference frame With any auxiliary frame Local feature points and descriptors are extracted, and initial matching relationships are established based on descriptor distances. This embodiment uses ORB features for feature extraction, with a maximum number of feature points set to 2000. During the matching stage, a Hamming distance-based matcher is used for descriptor matching, and nearest neighbor filtering is enabled to improve matching reliability. This yields a set of matching point pairs for subsequent geometric transformation estimation. .

[0074] Geometric transformation estimation: The random sample consensus algorithm (RANSAC) is used to estimate the geometric transformation from the auxiliary frame to the reference frame. This is done to eliminate mismatched point pairs and improve the robustness of the transform estimation. In this embodiment, the homography matrix is ​​used to describe the inter-frame mapping relationship, and the solution is obtained on the RANSAC in-point set. In this process, the reprojection error threshold of RANSAC is set to 3 pixels, and the maximum number of iterations is set to 2000. When the inlier ratio is less than 0.25, the alignment reliability of the auxiliary frame is deemed insufficient, and the auxiliary frame is removed or its weight is reduced in subsequent processing.

[0075] Coordinate mapping and resampling: using auxiliary frames Mapping to the reference frame coordinate system to obtain the alignment auxiliary frame Specifically, for any pixel position x in the reference frame coordinate system, its corresponding position in the auxiliary frame coordinate system is calculated using a reverse mapping method, and interpolation sampling is performed to obtain the pixel value, resulting in:

[0076] Where x represents the pixel position in the reference frame coordinate system. This indicates the operation of interpolating and sampling the auxiliary frame to obtain pixel values. In this embodiment, bilinear interpolation is used for resampling; when the mapped position is outside the valid range of the auxiliary frame, the corresponding pixel is marked as invalid and discarded or assigned zero weight in subsequent processing.

[0077] Alignment quality evaluation and processing: To reduce the impact of misalignment on subsequent residual statistics and recovery results, this embodiment evaluates the registration results of each auxiliary frame. Specifically, the alignment quality is evaluated using the inlier ratio and reprojection error output by RANSAC; when the inlier ratio is lower than 0.25 or the reprojection error exceeds 3 pixels, the auxiliary frame is determined to be unreliable and is removed in subsequent processing; the alignment results of the remaining reliable auxiliary frames are retained and participate in subsequent steps.

[0078] S5. Raindrop Distortion Region Judgment and Extraction Obtaining alignment auxiliary frames Based on this, a multi-frame residual is constructed between the reference frame and the alignment auxiliary frame, and statistical aggregation is performed to generate a residual response map; then, the raindrop distortion region mask M is extracted through threshold segmentation and morphological processing, such as... Figure 3 As shown, the intermediate results of the raindrop distortion region judgment and extraction process are illustrated. Figure 3 (a) Visualization of the overlay of the distortion region detection results on the reference frame. In the figure, the blue area is the simulated raindrop effect area, and the green area is the detected distortion region. Figure 3 (b) Schematic diagram of the mask for the distortion area where raindrops adhere; Figure 3 (c) is a mask for the detected distorted region; Figure 3 (d) is a schematic diagram of the heatmap obtained by multi-frame aligned residual aggregation, used to reflect the spatial distribution of high residual regions. Specific steps include: Constructing the residual: using the reference frame With alignment auxiliary frame Convert to the same grayscale domain and perform difference analysis to obtain a single-frame residual map. In this embodiment The residual is constructed by weighting the brightness residual and the gradient residual:

[0079] Multi-frame residual statistical aggregation: To suppress the effects of random noise, dynamic interference, or individual frame errors, the residuals of each auxiliary frame are aggregated. Robust aggregation is performed to obtain the comprehensive residual map. This embodiment uses pixel median aggregation:

[0080] Initial raindrop distortion region mask: for the composite residual map Threshold segmentation is performed to obtain the initial raindrop distortion region mask. This embodiment uses a quantile threshold, taking... The threshold is Therefore, we can conclude that:

[0081] Final raindrop distortion region mask: To remove isolated noise and enhance region coherence, the initial mask is modified... Morphological processing is performed. This embodiment employs an "opening operation followed by closing operation" approach, using a circle with a radius of 3 as the structuring element to obtain the intermediate mask. Subsequently, on Perform connected component analysis and remove connected components with an area less than a threshold; in this embodiment, the area threshold is set to... Pixels. The final result is the raindrop distortion region mask M.

[0082] S6, Raindrop Distortion Area Recovery After obtaining the raindrop distortion region mask, information from regions unaffected or minimally affected by raindrops in multiple frames of images is used to modify the reference frame. The raindrop distortion regions indicated by the middle mask are compensated and fused across frames to restore the real scene content of the corresponding regions. This includes the following sub-steps: Candidate frame set construction: After determining the raindrop distortion region mask M, the effective candidate frame set U is used as the source of candidate information for raindrop distortion region recovery. For those satisfying... At pixel position x, perform subsequent recovery processing. At the same time, keep the reference frame pixels unchanged. For each candidate alignment auxiliary frame Its pixel value in the reference frame coordinate system It is considered as one of the candidate compensation information at that location. When a pixel is outside the effective imaging range after alignment and mapping, the pixel is marked as invalid and does not participate in the candidate evaluation and selection of that pixel location.

[0083] Pixel quality evaluation: To select high-quality pixel sources from multiple candidate images within the raindrop distortion region, this embodiment performs pixel-level quality evaluation on the candidate frame set U. Specifically, for any valid candidate frame... The second-order gradient response of the Laplacian operator is calculated to obtain the Laplacian response image. Construct a local window centered at pixel x. (In this embodiment) 5 (5-neighborhood window), and define the variance of the Laplace response within the window as the quality evaluation value of the candidate frame at position x. :

[0084] in, y represents the number of pixels in the neighborhood, and y represents the window size. Any pixel location within the range. This embodiment only applies to masks that satisfy the raindrop distortion region. Calculate the quality evaluation value based on the pixel position; when the candidate frame is at position x or window When there are invalid pixels in the memory, the invalid pixels are ignored in the statistical calculation.

[0085] Generate initial restoration results: Obtain the quality evaluation value at pixel position x for each candidate frame. Subsequently, in this embodiment, the candidate pixel with the best quality is selected for each pixel to be restored within the raindrop distortion region, and an initial restoration result is generated accordingly. Specifically, for any satisfying Given the pixel position x, select the index of the candidate frame that maximizes the quality evaluation value from the candidate frame set U:

[0086] like Figure 4 As shown, a visualization of the source frame distribution of the recovery results during the raindrop distortion region recovery process is presented, where different colors indicate which frame the recovery result at that location ultimately originates from in the reference frame coordinate system: Let Where D1 is the reference frame, and D2-D6 are the aligned auxiliary frames, and the pixel value of D1 in the coordinate system of the reference frame is... This distribution reflects the pixel-by-pixel selection decision made by this invention based on sharpness evaluation among candidate frames. An initial reconstruction result is then generated based on this result.

[0087] To reduce seam artifacts caused by pixel replacement at the boundary of the distorted region, Gaussian smoothing is applied to the mask to obtain a feathered weight map. The initial recovery result is then fused with the reference frame, and this fused result is denoted as... :

[0088] in, Expressed in standard deviation Gaussian smoothing operator with parameters.

[0089] Generate the final restoration result: Obtain the boundary fusion restoration result. Subsequently, to further eliminate any remaining small holes, fracture textures, or local artifacts within the raindrop distortion area, this embodiment performs repair at the pixel positions indicated by mask M to obtain the final repaired image. The repair operation is shown in the following formula:

[0090] in, The image inpainting operator can be implemented using diffusion-based inpainting, fast-moving inpainting, or sample block-based inpainting methods. To preset the repair radius parameter, This represents the preset parameter set for the patching strategy. The patching operator uses effective pixels outside the mask boundary to complete the pixels inside the mask, ensuring that color and texture remain continuous and consistent at the mask boundary. As the output image after restoring the image distortion caused by raindrops. For example... Figure 5 As shown, the left image is the original input with raindrop distortion, and the right image is the final output obtained using the method of this invention. Figure 5 The comparison shows that the structural misalignment in the distorted area of ​​the raindrop is corrected, the texture continuity is improved, and the transition with the surrounding area is natural.

[0091] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for restoring raindrop-distorted images based on multi-frame registration, characterized in that, include: Acquire multiple frames of images captured by the same camera at different times or from different angles, and perform preprocessing; The reference frame is determined from the preprocessed multi-frame image using the Laplacian response; A reference coordinate system is established using the reference frame, and the preprocessed multi-frame images are aligned to obtain an alignment auxiliary frame. Based on the pixel residuals between the reference frame and each alignment auxiliary frame, the raindrop distortion region is determined; By utilizing information from regions unaffected or minimally affected by raindrops in the multi-frame image alignment results, pixel fusion and restoration are performed on the raindrop distortion regions indicated by the mask in the reference frame.

2. The raindrop distortion image restoration method based on multi-frame registration according to claim 1, characterized in that, The multi-frame images To describe the same scene content, at least one frame of the image contains a local image distortion region caused by raindrops adhering to the front end of the imaging system; the preprocessing includes: grayscale conversion, noise reduction, contrast normalization, and scale unification; resulting in preprocessed multi-frame images. .

3. The raindrop distortion image restoration method based on multi-frame registration according to claim 1, characterized in that, The reference frame The frames are determined from the preprocessed multi-frame images, and the remaining frames are used as auxiliary frames. The reference frame is used to establish a reference coordinate system for image alignment and serves as the main image to be restored; For any frame Calculate the Laplace response : in, The Laplacian operator is used to characterize the second-order spatial transformation properties of an image, and the sharpness evaluation value of each frame of the image is calculated based on the Laplacian response. : Among them, the sharpness evaluation value Defined as the variance of the Laplacian response over the entire image range. Represents the set of pixel coordinates of an image. Represents the coordinates of any pixel in the image. Indicates the total number of pixels. Let be the mean of the Laplacian response of the k-th frame; according to Sort selection Larger image as reference frame The selection rule adopts the direct maximum value selection method, letting... Determine the reference frame.

4. The raindrop distortion image restoration method based on multi-frame registration according to claim 1, characterized in that, The step of establishing a reference coordinate system using a reference frame and aligning the preprocessed multi-frame images to obtain an alignment auxiliary frame specifically includes: Extract local feature point sets from the reference frame and any auxiliary frame respectively. With the corresponding feature descriptor subset : in, This represents the m-th feature point in the reference frame. This represents the total number of feature points detected in the reference frame, and each feature point... A corresponding local image region in the reference frame, whose position can be represented by coordinates. express; Represents feature points in the reference frame The corresponding reference feature descriptor is a vector of length L, i.e. It is used to characterize the local texture and structural information in the neighborhood of the feature point; feature extraction is performed on the auxiliary frame to obtain the feature point set of the auxiliary frame. and corresponding feature descriptor subsets ; This represents the nth feature point in the auxiliary frame. This represents the total number of feature points detected in the auxiliary frame; Indicates feature points in auxiliary frames The corresponding auxiliary feature descriptor has the same dimensions as the reference feature descriptor; an initial matching relationship between the reference frame and the auxiliary frame is established based on the similarity metric between the reference feature descriptor and the auxiliary feature descriptor, for any feature point of the reference frame. In the auxiliary frame description subset The index of the feature point that minimizes the distance to the descriptor is denoted as: in, This indicates that in the set of feature points in the auxiliary frame, the m-th feature point is compared with the m-th feature point in the reference frame. Reference feature descriptor The index of the feature point in the nearest auxiliary frame; Using a preset descriptor distance metric, an initial set of matching point pairs between the reference frame and the auxiliary frame is obtained. : in, This represents the m-th feature point in the auxiliary frame compared to the reference frame. Matching feature points; Based on the initial set of matching point pairs Estimate the geometric transformation of the auxiliary frame relative to the reference frame. To improve the robustness of the transform estimation, outlier removal is performed on the matching point pairs. A random sample consensus algorithm is used to partition the matching point pairs into inliers and outliers, and the solution is obtained on the set of inliers. Simultaneously, the pixel distance between the matched point and the corresponding point in the reference frame after mapping, i.e., the reprojection error, is used as an evaluation metric to constrain or filter the transformation estimation results. Map the auxiliary frame to the reference frame coordinate system to obtain the aligned auxiliary frame. ;set up x To represent the pixel position in the reference frame coordinate system, the alignment mapping is as follows: in, This indicates that the position in the reference coordinate system is reverse-mapped to the position in the auxiliary frame coordinate system. This represents the operation of interpolating and sampling auxiliary frames to obtain pixel values.

5. The raindrop distortion image restoration method based on multi-frame registration according to claim 4, characterized in that, The alignment quality of the alignment auxiliary frame is evaluated using evaluation metrics including inlier ratio and reprojection error. When the alignment quality is below a preset threshold, the corresponding alignment auxiliary frame is removed, or the weight of the auxiliary frame is reduced in the subsequent residual aggregation and restoration fusion process to reduce the impact of misalignment on the determination and restoration results of distorted regions.

6. The raindrop distortion image restoration method based on multi-frame registration according to claim 1, characterized in that, The determination of the raindrop distortion region is based on a reference frame. With each alignment auxiliary frame The pixel residuals between frames are statistically analyzed to obtain a residual response map. When the pixel residuals in the residual response map exceed a preset threshold, it is determined that there is a local distortion caused by raindrops, and the corresponding abnormal area is extracted from the residual response map as the raindrop distortion area. A residual map is constructed based on the pixel-level differences between the reference frame and the alignment auxiliary frame. The residual map It is obtained by weighting the brightness residual and the gradient residual: in, For gradient operators, These are preset weighting parameters used to balance the influence of brightness information and structural information in residual calculation; Robust aggregation is performed on the residual maps corresponding to multiple auxiliary frames to obtain a comprehensive residual map. : in, For statistical aggregation operators; Based on the comprehensive residual diagram The residual threshold is determined by the statistical distribution of pixel values. And generate an initial raindrop distortion region mask. : Morphological processing is performed on the initial mask to remove isolated noise and enhance the spatial coherence of the raindrop region, resulting in an intermediate mask. : Where Open is the opening operation and Close is the closing operation. This is a structural element whose size is specified by a preset parameter s; Further Filter the connected components and remove those with areas smaller than a threshold. The connected components are used to obtain the final raindrop distortion region mask M: Among them, the output mask , , indicating that pixel x belongs to the raindrop distortion region; Represents the t-th connected component; This represents the area of ​​the connected region. This is a preset area threshold.

7. The raindrop distortion image restoration method based on multi-frame registration according to claim 1, characterized in that, Pixel fusion and restoration are performed on the raindrop distortion region. After obtaining the raindrop distortion region mask M, the regions unaffected or minimally affected by raindrops in the multi-frame alignment results are used to perform pixel fusion and restoration on the raindrop distortion region indicated by the mask in the reference frame to obtain the transition output. ; To satisfy The pixel value at position x of the reference frame and each alignment auxiliary frame is used as a candidate information source; the candidate values ​​of the auxiliary frames that are out of bounds, invalid samples or have low alignment quality are eliminated or downweighted to obtain the set of effective candidate frames U. For each valid candidate frame In the local neighborhood of pixel position x Internally calculated quality evaluation value This is used to measure the reliability of the candidate information at that location; the quality evaluation value From candidate frames in The sharpness / texture intensity, consistency residual with the reference frame, and alignment confidence are determined by a weighted combination of one or more factors and normalized for subsequent pixel selection or fusion. Based on the candidate frame set Perform selection or fusion operations on the pixels of the mask region to obtain the initial reconstruction result of the mask region. ; Smoothing the mask M yields a transition weight map. Reduce seam artifacts at the mask boundaries and fuse the initial restoration result with the reference frame to obtain the transition output. : in, Indicates a reference frame; For areas with insufficient candidate information, holes, or texture breaks, patching is performed at the pixel positions indicated by mask M. Perform image inpainting to obtain the final output image. : in, For image inpainting operators, These are preset parameters.

Images