Image rain removal reconstruction method based on expectation maximization algorithm and local modeling

Abstract

According to the image rain removal reconstruction method based on the expectation maximization algorithm and the local modeling, rain lines in the image can be effectively removed under the condition of low parameter quantity and calculation quantity, and a clear background image can be recovered. Comprising the following steps of 1, performing shallow feature extraction on a rain image, and capturing shallow basic information of the image; 2, inputting shallow basic information of the captured picture into the constructed image rain removal network based on an expectation maximization algorithm and local modeling; step 3, using SSIM as a training loss function to train the image rain removal network to obtain a final trained rain removal reconstruction network model; and step 4, loading the rain removal reconstruction network model, and inputting a picture needing rain into the rain removal reconstruction network model to obtain a rain-free image corresponding to the rain image.

Description

Technical Field

[0001] The invention belongs to the field of computer vision, relates to image processing technology, and specifically relates to an image rain removal and reconstruction method based on a maximum expectation algorithm and local modeling. Background Art

[0002] Image deraining technology is an image processing method that aims to remove noise and interference caused by rain or raindrop occlusion in images and improve image quality. In images taken on rainy days, raindrops or raindrops may have a significant impact on visual quality, resulting in blurred object edges, loss of background details, and increased overall visual noise. Therefore, image deraining technology plays an important role in ensuring image clarity and detail performance, especially in the fields of monitoring, autonomous driving, and visual navigation, which can help the system maintain accurate perception of the environment under severe weather conditions. The challenges of image deraining technology mainly come from the complex relationship between raindrops and the background. Raindrops or raindrops often appear randomly distributed in the image, and factors such as their morphology, transparency, and lighting changes make it more difficult to separate raindrops from the background. Therefore, how to effectively remove raindrops while retaining the background information and details of the image is a research focus in the field of image deraining.

[0003] Current image deraining methods can be mainly divided into three categories: methods based on traditional image processing, methods based on physical models, and methods based on deep learning.

[0004] Methods based on traditional image processing: These methods mainly rely on image filtering techniques, and try to remove raindrop noise by processing the image in the spatial domain or frequency domain. For example, using median filtering, guided filtering, or bilateral filtering can remove some raindrops. However, such methods often perform poorly in complex scenes and may result in loss of details or over-smoothing of the image.

[0005] Methods based on physical models: These methods describe the movement trajectory, shape, and impact of raindrops on image brightness by modeling the physical properties of raindrops. The appearance of raindrops in an image can be regarded as a foreground occlusion. Physical model methods attempt to restore the original image by separating the foreground (raindrops) from the background. However, these methods have strong assumptions about the physical properties of raindrops and are limited in practical applications by the complexity of environmental conditions.

[0006] Deep learning-based methods: With the rapid development of deep learning, image deraining technology based on methods such as convolutional neural networks (CNNs) and generative adversarial networks (GANs) has made significant progress. Deep learning methods learn the characteristics of raindrops through large-scale training data and combine adaptive network structures to remove raindrops while retaining background details. For example, the deraining network (DerainNet) learns the task of removing raindrops in an end-to-end manner and achieves better results than traditional methods. Recently, Transformer-based models have also begun to be applied to image deraining tasks. They capture long-range dependencies in images through self-attention mechanisms, effectively handle complex raindrop scenes and restore more details. However, despite the success of these methods in performance, their high computational complexity and high resource consumption during training and inference limit their promotion in practical applications. Summary of the invention

[0007] In order to solve the problems of high computational complexity and large resource consumption of existing deep learning-based methods, the present invention proposes an image deraining and reconstruction method based on maximum expectation algorithm and local modeling. This method can effectively remove rain lines in the image and restore a clear background image with lower parameter and computational complexity.

[0008] To achieve the above object, the technical solution of the present invention is:

[0009] An image rain removal and reconstruction method based on maximum expectation algorithm and local modeling comprises the following steps:

[0010] Step 1: Extract shallow features of rainy images to capture shallow basic information of the image;

[0011] Step 2: Input the shallow basic information of the captured image into the constructed image deraining network based on the maximum expectation algorithm and local modeling;

[0012] Step 3: Use SSIM as the training loss function to train the image deraining network to obtain the final trained deraining and reconstruction network model;

[0013] Step 4: Load the above-mentioned rain removal and reconstruction network model, input the required rainy picture into the rain removal and reconstruction network model, and obtain the rain-free image corresponding to the rainy image.

[0014] Furthermore, step 1 specifically includes:

[0015] Given an image disturbed by rain, we first use a 3×3 convolutional layer to perform shallow feature extraction to capture the basic information of the image. The specific formula is as follows:

[0016] F b =f 3×3(I) (1)

[0017] where F b represents the shallow features extracted by the 3×3 convolutional layer; f 3×3 (·) represents a 3×3 convolution; I represents the input low-resolution image corrupted by rain.

[0018] Furthermore, step 2 specifically includes:

[0019] Step 2.1: Input the result obtained by shallow feature extraction in step 1 into a four-level symmetrical encoder-decoder for deep feature extraction. The structure of the encoder-decoder includes an optimal iterative attention mechanism and an optimal iterative feedforward neural network. The specific formula is as follows:

[0020] F′ b =F b +f IOAB (LN(F b )) (2)

[0021] F m =F′ b +f IOFN (LN(F′ b )) (3)

[0022] Where: F' b represents the features extracted by the optimal iterative attention mechanism, LN represents the LayerNorm normalization method, and f IOAB represents the optimal iterative attention process, F m represents the features extracted by the optimal iterative feedforward neural network, f IOFN It represents the optimal iterative feedforward neural network process.

[0023] Furthermore, the specific process in step 2.1 includes:

[0024] Step 2.1.1: According to formula (2), given an input tensor F z ,,f IOAB The specific formula of the process is as follows:

[0025] F z =LN(F b ) (4)

[0026]

[0027] Where D t is the attention weight distribution of the optimal iterative attention, t represents the number of iterations of the maximum expected algorithm operation, λ represents the hyperparameter, and V represents the value in the attention mechanism;

[0028] Step 2.1.2: According to formula (3), given an input tensor F k , f IOFN The specific formula of the process is as follows:

[0029] F k =LN(F′ b ) (6)

[0030]

[0031] where ⊙ represents element-wise multiplication, G (t) Represents the optimal iterative process value; W d represents 1×1 convolution; W p Shows a 3×3 depthwise convolution.

[0032] Step 2.2: To further enhance the local information modeling capability of the model, a local modeling residual block is introduced. By integrating two local model blocks and a series of convolution and activation functions, local feature extraction is promoted synergistically and detail reproduction is improved. Given an input tensor F l , the specific formula is as follows:

[0033] F′ l =F l +f LMB (Relu(Conv(F l ))) (8)

[0034] F u =F′ l +f LMB (Relu(Conv(F′ l ))) (9)

[0035] where F' l represents the intermediate features of the local modeling residual block; F u represents the output of the local modeling residual block; Conv(·) represents the 1×1 convolution function; ReLu(·) represents the Relu activation function; f LMB Represents a local model nugget.

[0036] The local model block is a key component of the local modeling residual block. According to Formula 9, given an input tensor F i , the specific formula of the local model block is as follows:

[0037] F i =Relu(Conv(F l )) (10)

[0038] F′ i =F i+Sigmoid((AvePool(LN(AvePool(F i ))))) (11)

[0039] Where Sigmoid(·) represents the Sigmoid operation; F' i represents the output features of the local model block; AvePool(·) represents the average pooling operation.

[0040] Step 2.3: Image reconstruction: The features extracted by the deep feature extraction and local modeling residual block are sent to the image reconstruction module, upsampled using a 3×3 convolutional layer, and added to the original input image to generate the final clear background image.

[0041] Furthermore, the step 3 is specifically as follows:

[0042] In the whole network training process, the present invention adopts SSIM as the training loss function. Specifically, for given paired training data x and y, the SSIM training loss function formula can be expressed as follows:

[0043]

[0044] Among them, μ x and μ y represents the mean of x and y, σ x and σ y represents the variance of x and y, σ xy represents the covariance of x and y; c1 represents the constant to prevent division by zero, which is 6.5; c2 represents the constant to prevent division by zero, which is 58.5.

[0045] Further, the step 4 specifically includes:

[0046] Load the model model_best trained in step 3, input the rainy image into the model, and output the reconstructed rain-free image and the corresponding evaluation indicators.

[0047] The beneficial effects of the present invention include:

[0048] This method can effectively remove rain lines from images and restore clear background images through shallow feature extraction, deep feature extraction (including optimal iterative attention mechanism and optimal iterative feedforward neural network), local modeling residual block and image reconstruction. The local modeling residual block is introduced to promote local feature extraction by integrating two local model blocks and a series of convolution and activation functions, improve the ability of detail reproduction, and make the reconstructed image clearer and more accurate in details. In the whole network training process, structural similarity (SSIM) is used as the training loss function, which helps to improve the structural similarity between the reconstructed image and the original rain-free image, thereby obtaining better visual effects.

[0049] In summary, the present invention integrates the maximum expectation algorithm and local modeling technology, which can effectively extract local and global information of the image while maintaining low complexity. Through the maximum expectation algorithm, the network parameters are gradually optimized, more pixels are activated, and more features are extracted while maintaining a low amount of calculation. Compared with the traditional rain removal method, the present invention significantly improves the image rain removal effect while reducing the number of parameters and the amount of calculation, making the rain-removed image clearer and more detailed. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 This is the complete network structure diagram of the image deraining and reconstruction method based on the maximum expectation algorithm and local modeling;

[0051] Figure 2 This is the complete structure diagram of the optimal iterative attention mechanism and the optimal iterative feedforward neural network;

[0052] Figure 3 This is the complete structure diagram of the local model block. DETAILED DESCRIPTION

[0053] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0055] The present invention proposes a single image deraining and reconstruction method based on maximum expectation algorithm and local modeling. First, a 3×3 convolution layer is used for shallow feature extraction to capture the basic information of the image. Secondly, a four-level symmetrical codec is used as a deep feature extraction module, and the expectation maximization algorithm is used to iteratively optimize the attention distribution, reduce redundant information, and generate a clear feature map. In addition, a local model residual block (LMRB) is introduced to promote feature extraction and improve detail reproduction by integrating two local model blocks and a series of convolution and activation functions. Finally, the features extracted by the deep feature extraction module and the local model residual block are sent to the image reconstruction module, upsampled by a 3×3 convolution layer, and the original input image is added to generate the final clear background image. In addition, SSIM is used as the training loss function, and a large amount of data is used for training to obtain the final deraining and reconstruction network model. The present invention integrates the advantages of the maximum expectation algorithm and local modeling, achieves a good balance between the size and performance of the model, and has a significant improvement in the objective evaluation index of the image.

[0056] Example 1

[0057] This embodiment uses Windows system as the development environment, PyCharm as the development platform, Python as the development language, and adopts the image deraining and reconstruction method based on maximum expectation algorithm and local modeling of the present invention to complete the reconstruction of rainy images into rain-free images.

[0058] In this embodiment, the image rain removal and reconstruction method based on the maximum expectation algorithm and local modeling includes the following steps:

[0059] Step 1: Extract shallow features of rainy images to capture shallow basic information of the image;

[0060] Step 2: Build an image deraining network based on the maximum expectation algorithm and local modeling, and input the shallow basic information of the captured image into the network;

[0061] Step 3: Use SSIM as the training loss function for training to obtain the final trained deraining and reconstruction network model;

[0062] Step 4: Load the above model and input the required rainy pictures into the image deraining model to obtain the corresponding rain-free images and corresponding evaluation indicators. Peak Signal to Noise Ratio (PSNR) and SSIM (the Structural Similarity, SSIM) are used as objective evaluation indicators. The calculation formula of PSNR is as shown below:

[0063]

[0064] Here is the MAX I 2 It represents the maximum value of the color of the image point, and MSE represents the mean square error.

[0065] According to the above steps, the model proposed in the present invention is compared with the advanced methods on five different test sets (Rain200H, Rain200L, Rain1200, Rain1400 and Rain12). As can be seen from Table 1, the method proposed in the present invention can achieve better PSNR and SSIM values ​​on the five common test sets.

[0066] Table 1 Quantitative comparison on five datasets

[0067]

[0068] Note: The best results are highlighted.

[0069] Obviously, the above embodiments are merely examples for the purpose of clear explanation, and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived therefrom are still within the scope of protection of the invention.

Claims

1. An image deraining and reconstruction method based on maximum expectation algorithm and local modeling, characterized in that: The method comprises the following steps: Step 1: Perform shallow feature extraction on rainy images to capture shallow basic information of the image; Step 2: Input the shallow basic information of the captured image into the constructed image deraining network based on the maximum expectation algorithm and local modeling; Step 3: Use SSIM as the training loss function to train the image deraining network to obtain the final trained deraining and reconstruction network model; Step 4: Load the above-mentioned rain removal and reconstruction network model, input the required rainy picture into the rain removal and reconstruction network model, and obtain the rain-free image corresponding to the rainy image.

2. The image deraining and reconstruction method based on maximum expectation algorithm and local modeling according to claim 1, characterized in that: The step 1 specifically includes: Given an image disturbed by rain, we first use a 3×3 convolutional layer to perform shallow feature extraction to capture the basic information of the image. The specific formula is as follows: Fb=f3 ×3 (I) (1) Where Fb represents the shallow features extracted by the 3×3 convolution layer; f3 ×3 (·) represents a 3×3 convolution; I represents the input image disturbed by rain.

3. The image deraining and reconstruction method based on maximum expectation algorithm and local modeling according to claim 2, characterized in that: Step 2 specifically includes: Step 2.1: Input the result obtained by shallow feature extraction in step 1 into a four-level symmetrical encoder-decoder for deep feature extraction. The structure of the encoder-decoder includes an optimal iterative attention mechanism and an optimal iterative feedforward neural network. The specific formula is as follows: F′ b =F b +f IOAB (LN(F b )) (2) F m =F′ b +f IOFN (LN(F′ b )) (3) Where: F′ b represents the features extracted by the optimal iterative attention mechanism, LN represents the LayerNorm normalization method, and f IOAB represents the optimal iterative attention process, F m represents the features extracted by the optimal iterative feedforward neural network, f IOFN It represents the optimal iterative feedforward neural network process; Step 2.2: Integrate the two local model blocks and 1×1 convolution and activation function to form a local modeling residual block for local feature extraction. Given an input tensor F l , the specific formula of the local modeling residual block is as follows: F′ l =F l +f LMB (Relu(Conv(F l ))) (4) F u =F′ l +f LMB (Relu(Conv(F′ l ))) (5) where F′ l represents the intermediate features of the local modeling residual block; F u represents the output of the local modeling residual block; Conv(·) represents the 1×1 convolution function; Relu(·) represents the Relu activation function; f LMB Represents a local model block; The local model block, given an input tensor F i , the specific formula of the local model block is as follows: F i =Relu(Conv(F l )) (6) F′ i =F i +Sigmoid((AvePool(LN(AvePool(F i ))))) (7) Where Sigmoid(·) represents the Sigmoid operation; F′ i Represents the output features of the local model block; AvePool(·) represents the average pooling operation; Step 2.3: Image reconstruction: The extracted deep features and local features are sent to the image reconstruction module, upsampled using a 3×3 convolutional layer, and added to the original input image to generate the final clear background image.

4. The image deraining and reconstruction method based on maximum expectation algorithm and local modeling according to claim 3, characterized in that: The step 2.1 specifically includes: Step 2.1.1: According to formula (2), given an input tensor F z , f IOAB The specific formula of the process is as follows: F z =LN(F b ) (8) Where D t is the attention weight distribution of the optimal iterative attention mechanism, t represents the maximum expected number of iterations of the algorithm operation, λ represents the hyperparameter, and V represents the value in the attention mechanism; Step 2.1.2: According to formula (3), given an input tensor F k , f IOFN The specific formula of the process is as follows: F k =LN(F′ b ) (10) where ⊙ represents element-wise multiplication, G (t) Represents the optimal iterative process value; W d represents 1×1 convolution; W p Shows a 3×3 depthwise convolution.

5. The image deraining and reconstruction method based on maximum expectation algorithm and local modeling according to any one of claims 1 to 4, characterized in that: The SSIM training loss function described in step 3, for a given pair of training data x and y, is expressed as follows: Among them, μ x and μ y represents the mean of x and y, σ x and σ y represents the variance of x and y, σ xy represents the covariance of x and y; c1 represents the constant to prevent division by zero, which is 6.5; c2 represents the constant to prevent division by zero, which is 58.5.

Images