Single image rain removal method based on meta-learning relation and representation learning network
Through the relationship and representation learning network based on meta-learning, learning the relationship and embedding of rainy images and clean images, the problems of insufficient generalization ability and lack of real values in the existing technology are solved, and better image rain removal effect and background are achieved. Fidelity.
Patent Information
- Application Number
- CN202411335950.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-24
- Publication Date
- 2025-01-07
AI Technical Summary
When processing rain-day images, the existing image de-raining algorithms lack generalization capabilities, cannot effectively maintain background fidelity, and the lack of benchmark real values limits the performance of deep learning methods.
Using a meta-learning-based relationship and representation learning network, the relationship and embedding between rainy images and clean images is learned by constructing a meta-learning discriminator encoder, relationship network and rainy network generator, and using similarity function and ternary loss Function training relation vectors, combining mean square error loss and structural similarity objective function training model to achieve image rain removal.
The generalization ability of image rain removal is improved, the fidelity of background is maintained, the problems of diversity and limited samples are solved, the performance of rain removal is improved, and the performance limitations of existing methods are overcome.
Smart Images
Figure CN119273585A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of image processing, and mainly relates to a single image deraining method based on a meta-learning relationship and representation learning network. Background Art
[0002] With the development of computer vision technology, image deraining has become an important algorithm in its development process in recent years. This algorithm is mainly used to improve the quality of images taken under rainy days, and has been widely used in many outdoor computer vision tasks, such as video surveillance, character recognition, and target detection. However, the current mainstream algorithms directly "translate" the entire rainy image including rain streaks and clean background into a clean image, or indirectly characterize the rain residual and remove it from the embedding of the rainy image, which has insufficient generalization ability. Therefore, how to obtain better image deraining results is one of the hot research topics in the field of computer vision.
[0003] In response to the above problems, researchers have proposed some methods. Recently, an article titled "ERL-Net: Entangled Representation Learning for Single Image De-Raining" was published at the IEEE International Conference on Computer Vision Proceeding, the world's top academic conference in the field of computer science in 2019. This article decomposes the image conversion model of "rainy image to clean image" into an embedding learning network (encoder) and a conditional generator (decoder), and assumes that there is a smooth connection from the basic embedding to the optimal embedding in the potential embedding space, and proposes to use residual learning to learn this connection. However, this method does not deeply consider the endless diversity of the background of rainy images. When one type of background is far from the data distribution of the training set, this situation may destroy the balance between removing rain marks and maintaining background fidelity, resulting in blurred images. In addition, rainy images collected from the real world do not have benchmark truth values, which limits the performance of traditional deep learning-based deraining methods trained on synthetic datasets. Summary of the invention
[0004] In order to solve the shortcomings of the above-mentioned prior art, the present invention proposes a single image deraining method based on meta-learning relationship and representation learning network, in order to ensure the fidelity of the background in rainy images, better realize image deraining of an infinite number of complex scenes, and improve the generation quality of clean images and the generalization ability of deraining tasks.
[0005] In order to solve the technical problem, the present invention adopts the following technical solution:
[0006] The single image deraining method based on meta-learning relationship and representation learning network of the present invention is characterized in that it is performed according to the following steps:
[0007] Step 1: Get the image training set ,in, Indicates images, represents the number of images in the training set, and , Represents a set of rainy day images, represents the set of clean images; Indicates Rainy day image, express Matching A clean image;
[0008] Step 2: Construct a deraining network MLDN, including: Encoder based on meta-learning discriminator , Meta-learning-based relational network, deraining network generator :
[0009] Step 2.1: Encoder based on meta-learned discriminator right Process it and get Image features of an image ;
[0010] Step 2.2: Randomly initialize the generated relationship vector ;
[0011] Step 2.3: Use formula (1) to calculate the Image With Image Similarity between ;
[0012] (1)
[0013] In formula (1), T represents transposition, Represents the activation function of the neural network; Indicates Image The image features, Indicates Image With Image The correlation characteristics between
[0014] Step 2.4: Calculate the ternary loss function using formula (2) , and by updating the relationship vector To minimize the ternary loss , thereby training the relationship vector , get the trained relationship vector :
[0015] (2)
[0016] In formula (2), Indicates Rainy day images With Clean image The similarity between Indicates Rainy day images With Rainy day images The similarity between Indicates Rainy day images With Clean image The correlation characteristics between Indicates Rainy day images With Rainy day images The correlation characteristics between , and are all random numbers from 1 to n, and ;
[0017] Step 2.5: Use formula (3) to construct the objective function , and used to train the encoder Parameters , thus obtaining the trained encoder :
[0018] (3)
[0019] In formula (3), Indicates Rainy day images With Image The similarity between Indicates Rainy day images With Image The correlation characteristics between
[0020] Step 2.6: Use equation (4) to construct the objective function , and used to train the deraining network generator Parameters , thus obtaining the trained rain removal network generator :
[0021] (4)
[0022] In formula (4), is the structural similarity;
[0023] Step 3: Use formula (5) to input any rainy image Perform rain removal to obtain the corresponding clean image after rain removal :
[0024] (5).
[0025] An electronic device of the present invention includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the single image deraining method according to claim 1, and the processor is configured to execute the program stored in the memory.
[0026] The present invention provides a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and the computer program executes the steps of the single image deraining method according to claim 1 when executed by a processor.
[0027] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0028] 1. The present invention takes into account the contradiction between the infinite diversity of rainy images in the real world and the limited diversity of training samples. To this end, a task-customized meta-learning relational network is proposed to describe the general relationship between rainy / clean images in the relation vector, so as to better distinguish rain streaks under different backgrounds.
[0029] 2. To address the problem of lack of ground truth values for rainy images collected in the real world, this paper proposes a meta-learning-based representation learning network to learn transferable embeddings of rainy / clean images. Compared with existing methods, the meta-learning network can explore more inherent patterns of rainy images, thereby improving the deraining performance, maintaining the fidelity of the image background, and improving the generalization ability.
[0030] 3. The image deraining method of the present invention learns transferable embeddings of rainy / clean images by combining meta-learning technology, and describes the potential differences between rainy / clean images by learning relationship vectors, thereby improving the generalization ability of the deraining task. It solves the problem of limited generalization ability caused by the contradiction between the endless diversity of the background of rainy images and the limited diversity of training samples in most current image deraining methods for deraining images, as well as the problem of limiting the performance of traditional deraining methods based on deep learning due to the fact that rainy images collected from the real world do not have benchmark true values. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] Figure 1 is a flow chart of the method of the present invention;
[0032] Figure 2 It is a schematic diagram of the experimental results of the present invention. DETAILED DESCRIPTION
[0033] In this embodiment, a single image deraining method based on a meta-learning relation and representation learning network is to obtain a training set using a rainy day image and a corresponding clean image data set; then the similarity is calculated by a constructed similarity function; secondly, the similarity is introduced into the constructed ternary loss function to train and update the relationship vector; the mean square error loss is used to train the model again, and the encoding layer of the meta-learning-based discriminator is updated according to the designed objective function; then the objective function is designed to train the deraining network generator; finally, the trained parameters are substituted into the constructed image deraining function to obtain the general process of the deraining method; according to the image deraining function, the rainy day image data is input to obtain the corresponding clean image. Figure 1 Specifically, the method is performed in the following steps:
[0034] Step 1: Get the image training set ,in, Indicates images, represents the number of images in the training set, and , Represents a set of rainy day images, represents the set of clean images; Indicates Rainy day image, express Matching A clean image.
[0035] Step 2: Construct a deraining network MLDN, including: Encoder based on meta-learning discriminator , Meta-learning-based relational network, deraining network generator :
[0036] Step 2.1: Encoder based on meta-learned discriminator right Process and extract image features to obtain the Image features of an image ;
[0037] Step 2.2: Randomly initialize the generated relationship vector ,This vector characterizes the difference between the rainy image and the clean image, retains the characteristics of rain streaks, and provides a target that needs to be removed from the rainy image.
[0038] Step 2.3: Use formula (1) to calculate the Image With Image Similarity between :The equation first quantifies the correlation between each given image and the relationship vector through the inner product. Then the comprehensive similarity is calculated based on the product of the two similarities .
[0039] (1)
[0040] In formula (1), T represents transposition, Represents the activation function of the neural network; Indicates Image The image features, Indicates Image With Image The correlation characteristics between them.
[0041] Step 2.4: Calculate the ternary loss function using formula (2) , and by updating the relationship vector To minimize the ternary loss , thereby training the relationship vector , get the trained relationship vector : The difference between rainy / clean images is described by reducing the distance between rainy images and expanding the distance between rainy images and clean images, and the characteristics of rain streaks are trained.
[0042] (2)
[0043] In formula (2), Indicates Rainy day images With Clean image The similarity between Indicates Rainy day images With Rainy day images The similarity between Indicates Rainy day images With Clean image The correlation characteristics between Indicates Rainy day images With Rainy day images The correlation characteristics between , and are all random numbers from 1 to n, and .
[0044] Step 2.5: Use formula (3) to construct the objective function , and used to train the encoder Parameters , thus obtaining the trained encoder : Use mean square error loss to train the model and score the similarity Regression is performed with the ground truth. For pairs where the images are truly related, the similarity score is set to 1. For pairs where the images are unrelated, the similarity score is set to 0. By updating To minimize equation (3).
[0045] (3)
[0046] In formula (3), Indicates Rainy day images With Image The similarity between Indicates Rainy day images With Image The correlation characteristics between them.
[0047] Step 2.6: Use equation (4) to construct the objective function , and used to train the deraining network generator Parameters , thus obtaining the trained rain removal network generator : The deraining network generator is optimized by reducing the similarity between the derained image generated by the deraining network and the real clean image.
[0048] (4)
[0049] In formula (4), is the structural similarity, which is used to measure the perceptual loss between rainy images and the corresponding clean images; Used to accurately remove rain streaks from rainy images without causing further degradation.
[0050] Step 3: Use formula (5) to input any rainy image Perform rain removal to obtain the corresponding clean image after rain removal :
[0051] (5).
[0052] In this embodiment, an electronic device includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the above method, and the processor is configured to execute the program stored in the memory.
[0053] In this embodiment, a computer-readable storage medium stores a computer program on the computer-readable storage medium, and the computer program executes the steps of the above method when executed by a processor.
[0054] like Figure 2 As shown in Figure 2, the left side is a real rain scene image collected from the real world (without a benchmark truth value), and the right side is the deraining result of MLDN. Figure 2 It can be seen that although the backgrounds of the input rainy images vary greatly, this method can still achieve satisfactory results.
Claims
1. A single image deraining method based on meta-learning relation and representation learning network, characterized in that: The steps are as follows: Step 1: Get the image training set ,in, Indicates images, represents the number of images in the training set, and , Represents a set of rainy day images, represents the set of clean images; Indicates Rainy day image, express Matching A clean image; Step 2: Construct a deraining network MLDN, including: Encoder based on meta-learning discriminator , Meta-learning-based relational network, deraining network generator : Step 2.1: Encoder based on meta-learned discriminator right Process it and get Image features of an image ; Step 2.2: Randomly initialize the generated relationship vector ; Step 2.3: Use formula (1) to calculate the Image With Image Similarity between ; (1) In formula (1), T represents transposition, Represents the activation function of the neural network; Indicates Image The image features, Indicates Image With Image The correlation characteristics between Step 2.4: Calculate the ternary loss function using formula (2) , and by updating the relationship vector To minimize the ternary loss , thereby training the relationship vector , get the trained relationship vector : (2) In formula (2), Indicates Rainy day images With Clean image The similarity between Indicates Rainy day images With Rainy day images The similarity between Indicates Rainy day images With Clean image The correlation characteristics between Indicates Rainy day images With Rainy day images The correlation characteristics between , and are all random numbers from 1 to n, and ; Step 2.5: Use formula (3) to construct the objective function , and used to train the encoder Parameters , thus obtaining the trained encoder : (3) In formula (3), Indicates Rainy day images With Image The similarity between Indicates Rainy day images With Image The correlation characteristics between Step 2.6: Use equation (4) to construct the objective function , and used to train the deraining network generator Parameters , thus obtaining the trained rain removal network generator : (4) In formula (4), is the structural similarity; Step 3: Use formula (5) to input any rainy image Perform rain removal to obtain the corresponding clean image after rain removal : (5)。 2. An electronic device, comprising a memory and a processor, characterized in that: The memory is used to store a program that supports the processor to execute the single image deraining method according to claim 1, and the processor is configured to execute the program stored in the memory.
3. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the single image deraining method according to claim 1 are executed.