A post-processing method for multi-level iterative collaborative representation of rain-removed images

Abstract

This invention discloses a post-processing method for multi-level iterative collaborative representation of derained images, comprising the following steps: obtaining an initial image set, which includes several sets of derained images and corresponding rainless images, wherein the derained images are images obtained after a deraining preprocessing algorithm on rainy images, and the rainless images are images collected for the corresponding scene that do not contain rain streaks; constructing an algorithm matrix to remove mismatches; obtaining the images after removing mismatches, and using them to construct deraining network data, and obtaining the final derained image based on it. This invention combines the Hessian-Affine and SIFT algorithms, focusing on the strong correlation between feature point neighborhood information and feature points, resulting in higher feature matching accuracy during feature matching, and providing more refined assistance for image post-processing.

Description

Technical Field

[0001] This invention relates to the field of image processing, specifically to a post-processing method for multi-level iterative collaborative representation of derained images. Background Technology

[0002] Images captured in complex rainy conditions are often affected by raindrops, reducing visibility. Especially in heavy rain, rain from all directions can blur the background, degrading the visual quality of the image and severely interfering with the work of many computer vision applications, such as affecting the accuracy of tasks like object detection and image segmentation. Therefore, effectively removing the interference of rainy weather on image imaging and improving image quality has very important practical application value.

[0003] Existing technologies provide a variety of image deraining methods, such as removing rain streaks based on traditional mathematical modeling, learning rain streak features using deep learning networks (CN202210278399.4), and image deraining methods based on generative adversarial networks (CN202210763079.8).

[0004] However, the above methods are prone to blurring some image details during processing, which leads to the inability to effectively recover the blurred details in the image, such as the texture and edges of the target. Moreover, their processing efficiency and accuracy are low. Therefore, it is necessary to design a post-processing method for derained images with advantages in accuracy, robustness and efficiency. Summary of the Invention

[0005] The purpose of this invention is to provide a post-processing method for multi-level iterative collaborative representation of rain-removed images, so as to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a post-processing method for multi-level iterative collaborative representation of rain-removed images, comprising the following steps:

[0007] S1: Obtain the initial image set, which includes several sets of rain-removed images and corresponding rain-free images. The rain-removed images are the images obtained after the rain-removed images have undergone a rain-removed preprocessing algorithm, and the rain-free images are the images collected for the corresponding scene that do not contain rain streak information.

[0008] S2: Construct an algorithm matrix to remove mismatches;

[0009] S3: Obtain the image after removing mismatches, and use it to construct the deraining network data, and obtain the final derained image based on it.

[0010] Preferably, S1 specifically includes:

[0011] S101: Given a set of images based on the initial image set, each set of images includes at least two images with rain removed and one without rain. Extract independent key point sets from the two images. The key point sets are the position coordinates of key points in the images. Extract the key points and form feature descriptors.

[0012] S102: Compare the feature descriptors of the two keypoint sets above to obtain the initial image set.

[0013] Preferably, S2 specifically includes:

[0014] S201: The Hessian-Affine feature point detection algorithm and SIFT local feature descriptor are used to detect and describe the features of the image, and extract scale and invariant feature points;

[0015] S202: Construct a matrix using the local invariant information and the positional information of the key points. This matrix describes the positional coordinates and local structure of the matching key points.

[0016] S203: Perform a hypothetical match, that is, define a compatibility error representing local structural changes to measure the difference;

[0017] S204: Construct a neighborhood based on the compatibility ranking of the assumed match, and aggregate the features within the neighborhood to construct a graph feature, which is the feature of the neighborhood elements of the assumed match.

[0018] S205: Extract the topological information of the assumed match and its neighborhood;

[0019] S206: Use the topological information obtained above as the classification feature of the assumed match to classify and remove mismatches.

[0020] Preferably, the process of image feature detection and description based on the Hessian-Affine feature point detection algorithm in S201 includes: using the Hessian-Affine feature point detection algorithm to detect image features. Perform detection to obtain the corresponding local feature point set. ,in The total number of images. This represents the number of local feature points in each image.

[0021] The process of feature detection and description using the SIFT local feature descriptor in S201 includes:

[0022] S201a: For the constructed Gaussian pyramid image, candidate extreme points are detected at all scales of the image using the difference function;

[0023] The image space is described as follows:

[0024]

[0025] in and This represents the dimension of the Gaussian template. It is a scale space factor. The smaller the value, the less the image is smoothed, and the smaller the scale. Large scales correspond to the overall features of the image, while small scales correspond to the detailed features of the image.

[0026] The Gaussian difference scale space is represented as:

[0027]

[0028] S201b: Locates the precise position and scale of extreme points through function fitting, while filtering out edge points in the candidate extreme point set;

[0029] S201c: Combining the gradient information of the neighboring pixels of the key point, assign a main direction to each key point;

[0030] S201d: Feature Descriptor Generation: The gradient magnitude and direction of the pixels in the neighborhood of the keypoint are statistically analyzed to obtain the feature description of the keypoint, i.e. .

[0031] Preferably, S3 specifically includes: merging the rain-removed image (after removing mismatches) and the rain-free image through dimensionality compression, residual dense unit feature transformation, and feature stepwise fusion unit fusion to obtain a rain-removing network model; and obtaining the final clear image after rain removal after processing by the rain-removing network model.

[0032] Preferably, after the features in S3 are gradually fused, the rain-removed image features are extracted from the mapping matrix through multi-level iterative collaborative representation, thereby optimizing the rain-removed image.

[0033] Preferably, in the training process after residual feature transformation in S3, the Adam optimization algorithm is used for training, and the network parameters (including the connection weights between neurons) are saved after training to obtain the trained feature fusion deraining network.

[0034] Compared with the prior art, the beneficial effects of the present invention are:

[0035] 1. This invention, given a set of images, extracts independent sets of keypoints from the images. An initial image matching set is obtained by measuring the similarity of the feature descriptors of these two independent keypoint sets. The goal is to remove mismatches from this initial matching set, which can be reduced to a binary classification problem. Specifically, each assumed match is categorized as either a correct or incorrect match, and then mismatches are removed. In the post-processing of the de-rained image, this mismatch removal logic enables faster matching and processing.

[0036] 2. This invention combines the Hessian-Affine and SIFT algorithms, focusing on the strong correlation between the neighborhood information of feature points and feature points, thereby achieving higher feature matching accuracy and providing more refined assistance for image post-processing.

[0037] 3. This invention utilizes a multi-level iterative network to continuously optimize the derained image in the corresponding iterations, effectively improving the visual quality of the derained image. Attached Figure Description

[0038] Figure 1 This is a flowchart of the method of the present invention;

[0039] Figure 2 This is a flowchart illustrating the specific method for acquiring the initial image set and removing mismatches in an embodiment of the present invention.

[0040] Figure 3 This is a flowchart of the rain removal network construction process in an embodiment of the present invention. Detailed Implementation

[0041] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0042] Please see Figure 1-3 This invention provides a technical solution: a post-processing method for multi-level iterative collaborative representation of rain-removed images, comprising the following steps:

[0043] S1: Obtain the initial image set, which includes several sets of rain-removed images and corresponding rain-free images. The rain-removed images are the images obtained after the rain-removed images have undergone a rain-removed preprocessing algorithm, and the rain-free images are the images collected for the corresponding scene that do not contain rain streak information.

[0044] S2: Construct an algorithm matrix to remove mismatches;

[0045] S3: Obtain the image after removing mismatches, and use it to construct the deraining network data, and obtain the final derained image based on it.

[0046] In this embodiment, S1 specifically includes:

[0047] S101: Given a set of images based on the initial image set, each set of images includes at least two images with rain removed and one without rain. Extract independent key point sets from the two images. The key point sets are the position coordinates of key points in the images. Extract the key points and form feature descriptors.

[0048] S102: Compare the feature descriptors of the two keypoint sets above to obtain the initial image set.

[0049] In this embodiment, S2 specifically includes:

[0050] S201: The Hessian-Affine feature point detection algorithm and SIFT local feature descriptor are used to detect and describe the features of the image, and extract scale and invariant feature points;

[0051] S202: Construct a matrix using the local invariant information and the positional information of the key points. This matrix describes the positional coordinates and local structure of the matching key points.

[0052] S203: Perform a hypothetical match, that is, define a compatibility error representing local structural changes to measure the difference;

[0053] S204: Construct a neighborhood based on the compatibility ranking of the assumed match, and aggregate the features within the neighborhood to construct a graph feature, which is the feature of the neighborhood elements of the assumed match.

[0054] S205: Extract the topological information of the assumed match and its neighborhood;

[0055] S206: Use the topological information obtained above as the classification feature of the assumed match to classify and remove mismatches.

[0056] In this embodiment, the process of image feature detection and description based on the Hessian-Affine feature point detection algorithm in S201 includes: using the Hessian-Affine feature point detection algorithm to detect image features. Perform detection to obtain the corresponding local feature point set. ,in The total number of images. This represents the number of local feature points in each image.

[0057] The process of feature detection and description using the SIFT local feature descriptor in S201 includes:

[0058] S201a: For the constructed Gaussian pyramid image, candidate extreme points are detected at all scales of the image using the difference function;

[0059] The image space is described as follows: ,

[0060]

[0061] in and This represents the dimension of the Gaussian template. It is a scale space factor. The smaller the value, the less the image is smoothed, and the smaller the scale. Large scales correspond to the overall features of the image, while small scales correspond to the detailed features of the image.

[0062] The Gaussian difference scale space is represented as:

[0063]

[0064] S201b: Locates the precise position and scale of extreme points through function fitting, while filtering out edge points in the candidate extreme point set;

[0065] S201c: Combining the gradient information of the neighboring pixels of the key point, assign a main direction to each key point;

[0066] S201d: Feature Descriptor Generation: The gradient magnitude and direction of the pixels in the neighborhood of the keypoint are statistically analyzed to obtain the feature description of the keypoint, i.e. .

[0067] In this embodiment, S3 specifically includes: merging the rain-removed image (after removing mismatches) and the rain-free image through dimensionality compression, residual dense unit feature transformation, and feature stepwise fusion unit to obtain a rain-removing network model; and obtaining the final clear image after rain removal after processing by the rain-removing network model.

[0068] In this embodiment, after the features are gradually fused in S3, the rain-removed image features are extracted from the mapping matrix through multi-level iterative collaborative representation, thereby optimizing the rain-removed image.

[0069] In this embodiment, during the training process after residual feature transformation in S3, the Adam optimization algorithm is used for training, and the network parameters (including the connection weights between neurons) are saved after training to obtain the trained feature fusion deraining network.

[0070] In this embodiment, the core of this application lies in: given a set of images, extracting independent sets of key points from the images, and obtaining an initial image matching set by measuring the similarity of the feature descriptors of these two independent key point sets. That is, removing mismatches based on the initial matching set, which can be reduced to a binary classification problem. Specifically, each assumed match is classified as either a correct match or an incorrect match, and then mismatches are removed. In the post-processing of the de-rained image, based on the mismatch removal logic, the matching and processing process can be achieved more quickly.

[0071] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A post-processing method for multi-stage iterative collaborative representation of rain-removed images, characterized in that, The method comprises the steps of: S1: obtaining an initial image set, the initial image set comprising a plurality of rain-removed images and corresponding rain-free images, wherein the rain-removed image is an image obtained by performing a rain-removal preprocessing algorithm on a rainy image, and the rain-free image is an image collected from a corresponding scene without rain strip information; S2: constructing an algorithm matrix to remove false matches; Specifically comprising: S201: performing feature detection and description on the images by using a Hessian-Affine feature point detection algorithm and a SIFT local feature descriptor, and extracting scale and invariant feature points; S202: constructing a matrix using the local invariant information of the key points and the position information of the key points, the matrix describing the position coordinates and local structure of the matched key points; S203: performing a hypothetical match, that is, defining a compatibility error representing the change of the local structure to measure the difference; S204: constructing a neighborhood according to the compatibility order of the hypothetical match, and aggregating and constructing graph features of the features in the neighborhood of the hypothetical match, the graph features being the neighborhood element features of the hypothetical match; S205: extracting the topological information of the hypothetical match and its neighborhood; S206: classifying the topological information obtained above as classification features of the hypothetical match to remove false matches; S3: obtaining the images after removing false matches, and constructing a rain-removal network data based thereon to obtain a final rain-removed image.

2. The post-processing method of multi-stage iterative collaborative representation for rain removal image according to claim 1, characterized in that: The S1 specifically comprises: S101: based on the initial image set, a group of images is given, each group of images comprising at least two rain-removed images and rain-free images, independent key point sets are extracted from the two images, the key point set being the position coordinates of the key points in the image, and the key points are extracted to form feature descriptors; S102: comparing the feature descriptor similarity of the two key point sets to obtain the initial image set.

3. The method of claim 1, wherein: The process of image feature detection and description based on the Hessian-Affine feature point detection algorithm in S201 includes: detecting pictures by using the Hessian-Affine feature point detection algorithm to obtain a corresponding local feature point set is the total number of pictures, is the number of local feature points in each picture;​​ The process of feature detection and description of the SIFT local feature descriptor in the S201 comprises: S201a: for the constructed Gaussian pyramid image, a difference function is used to detect candidate extreme points at all scales of the image; where the image space is described as , wherein and denotes the dimension of the Gaussian template, is a scale space factor, the smaller the value the less the image is smoothed and the smaller the scale. Large scales correspond to the general features of the image and small scales to the detailed features of the image. The Gaussian difference scale space is represented as: S201b: the accurate position and scale of the extreme points are located by function fitting, and the edge points in the candidate extreme point set are filtered out; S201c: a main direction is given to each key point in combination with the gradient information of the key point neighborhood pixels. S201d: Feature descriptor generation: the gradient amplitude and direction of the key point neighborhood pixels are counted to obtain the feature description of the key point, i.e. .

4. The method of claim 1, wherein: In the S3, the rain-removed image and the rain-free image after removing false matches are subjected to dimension compression, residual dense unit feature conversion, and feature step-by-step fusion unit fusion to obtain a rain-removal network model, and a final clear image after rain removal is obtained after processing by the rain-removal network model.

5. The method of claim 4, wherein: In the S3, the rain-removed image features are extracted from the mapping matrix through multi-level iterative collaborative representation after the feature step-by-step fusion, and the rain-removed image is further optimized.

6. The method of claim 4, wherein: In the training process after the residual feature conversion in the S3, the Adam optimization algorithm is used for training, the network parameters after training are saved, and a feature fusion rain-removal network after training is obtained.

Images