Night image rain removal method with enhanced position perception
By constructing a night rain removal network model, using rain precedent and position enhancement perception modules, the problem of rain pattern processing in night scenes is solved, more accurate rain pattern removal is achieved, and the effect of rain removal at night image is improved.
Patent Information
- Application Number
- CN202411430743.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-14
- Publication Date
- 2024-11-12
AI Technical Summary
Existing image rain removal methods are difficult to effectively deal with rain patterns in night scenes, especially due to the difference in the light source lighting, the rain patterns have different appearance and difficulty in distinguishing, resulting in poorer rain removal effect at night than during the day.
The night image rain removal method with position enhancement perception is adopted. By constructing a night rain removal network model, including a rainfall prior module and a position enhancement perception module, the cyclic residual learning and in-depth information exploration are used to accurately capture the location of the rain pattern and remove the rain pattern.
Rain patterns are captured and removed more accurately in night scenes, improving the image's rain removal effect and significantly improving the quality of rain-containing images at night.
Smart Images

Figure CN118941801A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to a method for removing rain from nighttime images with enhanced position perception. Background Art
[0002] In night scenes, photos taken on rainy days usually suffer from reduced image quality and poor visibility due to the interference of rain, which affects the performance of visual tasks such as target detection, video surveillance and unmanned driving. Therefore, image deraining can serve visual tasks such as target detection, video surveillance and unmanned driving.
[0003] Traditional image deraining methods mainly use hand-crafted priors (such as Gaussian mixture models, sparse representation learning, and directional gradient priors) to provide additional constraints to separate rain components. However, such hand-crafted priors are derived based on existing empirical knowledge, cannot model the inherent properties of clear images, often lack good generalization capabilities, and are usually difficult to adapt to a variety of scenarios.
[0004] At present, deep learning-based rain removal methods are increasingly being used in the field of rain removal. However, most of the existing deep learning-based rain removal methods are designed for daytime rain scenes, without considering the characteristics of nighttime scenes (the biggest difference between rain scenes at night and during the day is that there is no uniform light source illumination, which will cause rain streaks at different spatial positions on the same rain-containing image to have different appearances and identification difficulties due to differences in light source illumination. Rain streaks near the light source are clearly visible and highly recognizable, while rain streaks far from the light source are the opposite. Existing technologies are difficult to effectively process such complex and changeable rain streaks). Therefore, the effect of directly removing rain from nighttime scenes is worse than that of removing rain from daytime scenes. Therefore, it is necessary to seek a method that can accurately remove rain streaks in nighttime scenes. Summary of the invention
[0005] The purpose of this application is to provide a position-enhanced perception night image rain removal method, which can accurately capture the position of rain streaks in the image in night scenes, thereby accurately removing rain streaks.
[0006] In a first aspect, the present application provides a location-enhanced perception nighttime image deraining method, which is used to derain a nighttime rainy image, comprising the steps of: A1. Constructing a nighttime rain removal network model; the nighttime rain removal network model includes a rainfall prior module and a position enhancement perception module, wherein the rainfall prior module is used to obtain preliminary position information of rain streaks in nighttime rainy images based on rainfall prior, and the position enhancement perception module is used to further extract position information of rain streaks in nighttime rainy images; A2. Training the nighttime rain removal network model; A3. Use the trained nighttime rain removal network model to perform rain removal processing on the nighttime rainy image to be processed.
[0007] This method uses a nighttime rain removal network model to remove rain from nighttime rainy images. In the nighttime rain removal network model, the preliminary position information of rain streaks in nighttime rainy images is first obtained based on rainfall priors, and then the position information of rain streaks in nighttime rainy images is further extracted using a position enhancement perception module, thereby being able to more accurately capture the position of rain streaks in the image, and then accurately remove rain streaks from nighttime rainy images.
[0008] Preferably, the nighttime rain removal network model further includes an input layer, a U-Net network and an output layer; The input layer, the rainfall prior module, the location enhancement perception module, the U-Net network and the output layer are connected in sequence from front to back, and the input layer is also connected to the location enhancement perception module.
[0009] Preferably, the rainfall prior module performs the following operations: ; in, is the output of the rainfall prior module at the kth cycle stage, is the output of the rainfall prior module at the k-1th cycle stage, Indicates channel cascade, is the main function of the rainfall prior module, N is the total number of preset cycle stages, is the initial prior mapping, It is the input of the rainfall prior module (1).
[0010] Preferably, the rainfall priori module comprises a trunk part, the output end of the trunk part is connected to the input end of the rainfall priori module after channel cascading, and the output end of the trunk part is also connected to the location enhancement perception module as the output end of the rainfall priori module; The backbone part includes a first convolutional layer, an LSTM module, a plurality of residual modules and a second convolutional layer which are sequentially connected from front to back.
[0011] Therefore, the rainfall prior module adopts the architecture of cyclic residual learning, in which the residual module can effectively model rain streaks as residual components. The rainfall prior module learns to gradually capture rain streaks by recursively designing the high-frequency features of rain. When a nighttime rainy image is input and the initial prior mapping is set, the rainfall prior module extracts the deep features of the nighttime rainy image and cyclically updates the prior mapping. Thus, the preliminary position information of rain streaks in the nighttime rainy image is obtained through the rainfall prior.
[0012] Preferably, the position enhancement perception module comprises a spatial position encoding module and an efficient channel attention module connected front and back, and the spatial position encoding module is connected to the input layer and the output end of the rainfall prior module; The spatial position encoding module is used to perform position encoding processing on the input feature map; The efficient channel attention module is used to adjust the attention weight of each pixel of the feature map output by the spatial position encoding module.
[0013] Compared with the traditional Transformer module, this position-enhanced perception module can better utilize depth information exploration to distinguish foreground rain streaks and background objects, thereby better utilizing depth information to restore the three-dimensional details obscured by rain streaks.
[0014] Preferably, the spatial position encoding module includes an encoding module and a nonlinear embedding module, the input end of the encoding module is connected to the output end of the rainfall prior module, the input end of the nonlinear embedding module is connected to the input layer, and the output end of the encoding module and the output end of the nonlinear embedding module are channel-concatenated to serve as the output end of the spatial position encoding module.
[0015] Preferably, the encoding module is used to perform the following operations: The encoding information of each pixel in the three coordinate directions is calculated according to the following formula: ; ; in, is the position of the pixel in the image, x, y, and z represent the three coordinate directions respectively. for The coordinate value corresponding to the j coordinate direction, is the feature dimension in the j-coordinate direction, i is a positive integer, for The dimension number of the corresponding pixel in the j coordinate direction, for The encoding information of the corresponding pixel in the j coordinate direction; The coding information of each pixel in three coordinate directions is concatenated to form the position code of each pixel.
[0016] Preferably, the efficient channel attention module comprises a main body part, an input end of the main body part is connected to an input end of the efficient channel attention module, and an output of the main body part is multiplied element-by-element with the input of the efficient channel attention module as the output of the efficient channel attention module; The main part includes an average pooling layer, a third convolutional layer, a ReLU activation function layer, a fourth convolutional layer and a Sigmoid activation function layer which are connected sequentially from front to back.
[0017] Preferably, the U-Net network comprises a plurality of downsampling layers and a plurality of upsampling layers, and both the downsampling layers and the upsampling layers are Restormer blocks.
[0018] Preferably, the downsampling operation of the downsampling layer adopts a pixel deshuffle operation, and the upsampling operation of the upsampling layer adopts a pixel shuffle operation.
[0019] Beneficial effect: The position-enhanced perception night image deraining method provided by the present application utilizes a night deraining network model to perform deraining processing on night rainy images. In the night deraining network model, preliminary position information of rain streaks in night rainy images is first obtained based on rainfall priors, and then the position information of rain streaks in night rainy images is further extracted using a position-enhanced perception module, thereby being able to more accurately capture the position of rain streaks in the image, and further accurately remove rain streaks in night rainy images. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] Figure 1 A flowchart of a location-enhanced perceived nighttime image deraining method provided in an embodiment of the present application.
[0021] Figure 2 This is a schematic diagram of the structure of the nighttime rain removal network model.
[0022] Figure 3 This is a structural diagram of the rainfall prior module.
[0023] Figure 4 Schematic diagram of the structure of the spatial position encoding module.
[0024] Figure 5 Schematic diagram of the structure of the efficient channel attention module.
[0025] Figure 6 This is the first set of comparison images.
[0026] Figure 7 This is the second set of comparison images.
[0027] Figure 8 This is the third set of comparison images.
[0028] Explanation of numbers: 1. Rainfall prior module; 101. Main part; 102. First convolutional layer; 103. LSTM module; 104. Residual module; 105. Second convolutional layer; 2. Position enhancement perception module; 3. Input layer; 4. U-Net network; 5. Output layer; 6. Spatial position encoding module; 601. Encoding module; 602. Non-linear embedding module; 7. Efficient channel attention module; 701. Main part; 702. Average pooling layer; 703. Third convolutional layer; 704. ReLU activation function layer; 705. Fourth convolutional layer; 706. Sigmoid activation function layer. DETAILED DESCRIPTION
[0029] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.
[0030] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.
[0031] Please refer to Figure 1 , Figure 1 A method for removing rain from nighttime images with enhanced position perception in some embodiments of the present application is used to remove rain from nighttime rainy images (rainy images taken in an outdoor nighttime environment), comprising the steps of: A1. Construct a nighttime rain removal network model; the nighttime rain removal network model includes a rainfall prior module 1 and a position enhancement perception module 2. The rainfall prior module 1 is used to obtain preliminary position information of rain streaks in nighttime rainy images based on rainfall priors, and the position enhancement perception module 2 is used to further extract position information of rain streaks in nighttime rainy images; A2. Train the nighttime rain removal network model; A3. Use the trained nighttime rain removal network model to remove rain from the nighttime rainy images to be processed.
[0032] The method uses a nighttime rain removal network model to remove rain from nighttime rainy images. In the nighttime rain removal network model, the preliminary position information of rain streaks in nighttime rainy images is first obtained based on rainfall priors, and then the position information of rain streaks in nighttime rainy images is further extracted using the position enhancement perception module 2, so as to more accurately capture the position of rain streaks in the image, and then accurately remove rain streaks in nighttime rainy images.
[0033] Specifically, see Figure 2 ,The nighttime rain removal network model also includes an input layer 3, a U-Net network 4 and an output layer 5; The input layer 3, the rainfall prior module 1, the location enhancement perception module 2, the U-Net network 4 and the output layer 5 are connected in sequence from front to back, and the input layer 3 is also connected to the location enhancement perception module 2.
[0034] Among them, the rainfall prior module 1 performs the following operations: ; in, is the output of rainfall prior module 1 at the kth cycle stage, is the output of rainfall prior module 1 at the k-1th cycle stage (for the kth cycle stage, is a priori mapping), Indicates channel cascade, is the main function of the rainfall prior module 1 (i.e., the function of the trunk part 101 hereinafter), N is the total number of preset cycle stages, is the initial prior mapping, It is the input of rainfall prior module 1.
[0035] Among them, N and It can be set according to actual needs. For example, in one possible implementation, N is 6. All values of are 0.5; however, they are not limited to this.
[0036] Further, see Figure 3 The rainfall prior module 1 includes a main part 101, the output end of the main part 101 is connected to the input end of the rainfall prior module 1 after channel cascading, and the output end of the main part 101 is also connected to the location enhancement perception module 2 as the output end of the rainfall prior module 1; The backbone part 101 includes a first convolutional layer 102, an LSTM module 103 (i.e., Long Short-Term Memory module, long short-term memory network module), a plurality of residual modules 104 and a second convolutional layer 105 which are sequentially connected from front to back.
[0037] Therefore, the rainfall prior module 1 actually adopts the architecture of cyclic residual learning, wherein the residual module 104 can effectively model rain streaks as residual components. The rainfall prior module 1 learns to gradually capture rain streaks by recursively designing the high-frequency features of rain. When a nighttime rainy image is input and the initial prior mapping is set, the rainfall prior module 1 extracts the depth features of the nighttime rainy image and cyclically updates the prior mapping, thereby obtaining the preliminary position information of rain streaks in the nighttime rainy image through the rainfall prior.
[0038] Preferably, see Figure 2 , the position enhancement perception module 2 includes a spatial position encoding module 6 and an efficient channel attention module 7 connected front and back, and the spatial position encoding module 6 is connected to the input layer 3 and the output end of the rainfall prior module 1; The spatial position encoding module 6 is used to perform position encoding processing on the input feature map (referring to the feature map input to the spatial position encoding module 6); The efficient channel attention module 7 is used to adjust the attention weight of each pixel of the feature map output by the spatial position encoding module 6.
[0039] The traditional Transformer module cannot reflect complex two-dimensional or three-dimensional spatial information and may ignore important information such as the actual distance between pixels. Compared with the traditional Transformer module, the position enhanced perception module 2 can better utilize depth information exploration to distinguish foreground rain streaks and background objects, thereby better utilizing depth information to restore the three-dimensional details obscured by rain streaks.
[0040] Specifically, see Figure 4 The spatial position encoding module 6 includes an encoding module 601 and a nonlinear embedding module 602. The input end of the encoding module 601 is connected to the output end of the rainfall prior module 1, and the input end of the nonlinear embedding module 602 is connected to the input layer 3. The output end of the encoding module 601 and the output end of the nonlinear embedding module 602 are channel-concatenated as the output end of the spatial position encoding module 6.
[0041] The encoding module 601 is used to perform the following operations: The encoding information of each pixel in three coordinate directions (x, y, and z) is calculated according to the following formula: ; ; in, is the position of the pixel in the image, x, y, and z represent the three coordinate directions respectively. for The coordinate value corresponding to the j coordinate direction, is the feature dimension in the j coordinate direction (the feature dimensions in each coordinate direction can be set according to actual needs, for example, the feature dimensions in the x, y, and z coordinate directions are 32, 32, and 16 respectively, but not limited thereto), i is a positive integer, for The dimension number of the corresponding pixel in the j coordinate direction ( ), for The corresponding pixel encoding information in the j coordinate direction; that is, when When it is an even number, calculate according to the above formula ,when When it is an odd number, calculate according to the following formula in the above formula ; The coding information of each pixel in three coordinate directions is concatenated to form the position code of each pixel.
[0042] Thus, the spatial position encoding module 6 can perform fine encoding on the position of each pixel, not only considering the x, y two-dimensional spatial coordinates, but also adding depth information according to the prior of the rainfall prior module 1. The depth information is essential for distinguishing foreground rain streaks from background objects, understanding the three-dimensional structure of the scene, and restoring the details blocked by rain streaks. The encoding method of the spatial position encoding module 6 enables the nighttime rain removal network model to accurately capture the spatial relationship between pixels, thereby more accurately removing the influence of rain streaks.
[0043] Further, see Figure 5 The efficient channel attention module 7 includes a main body 701, an input end of the main body 701 is connected to an input end of the efficient channel attention module 7, and an output of the main body 701 is multiplied element by element with the input of the efficient channel attention module 7 as the output of the efficient channel attention module 7; The main part 701 includes an average pooling layer 702, a third convolutional layer 703, a ReLU activation function layer 704, a fourth convolutional layer 705 and a Sigmoid activation function layer 706 which are connected sequentially from front to back.
[0044] Thus, the operation performed by the efficient channel attention module 7 can be expressed as: ; ; in, is the output of the efficient channel attention module 7, is the input of the efficient channel attention module 7, is the average pooling function, is the convolution function, is the ReLU activation function, is the Sigmoid activation function, A is the channel attention weight matrix, Represents element-wise multiplication.
[0045] The efficient channel attention module 7 can adjust the corresponding channel weight according to the position encoding of each pixel, so that even in the same channel, pixels at different positions may obtain different attention weights (the attention weight reflects the relative importance of the features at the corresponding positions), thereby more comprehensively considering the interaction between each channel feature and the rain streak position information, which helps the nighttime rain removal network model to pay attention to the relationship between channels while fully considering the influence of position factors on feature importance, strengthen the cognitive representation of rain streak information, and improve the ability to process and remember this information.
[0046] Specifically, the U-Net network 4 includes multiple downsampling layers and multiple upsampling layers, and both the downsampling layers and the upsampling layers are Restormer blocks (i.e., Restoration Transformer blocks, which are efficient Transformer blocks). The number of downsampling layers and upsampling layers can be set according to actual needs, for example, four layers, but not limited thereto.
[0047] Preferably, the downsampling operation of the downsampling layer adopts a pixel unshuffle operation (i.e., a pixel-unshuffle operation), and the upsampling operation of the upsampling layer adopts a pixel shuffle operation (i.e., a pixel-shuffle operation), so that a high-quality clear output image can be finally reconstructed.
[0048] Among them, in step A2, the synthetic dataset GTAV-NightRain provided by python can be used to train the night rain removal network model. The specific training process is existing technology and will not be described in detail here.
[0049] Among them, step A3 includes: obtaining a nighttime rainy image to be processed; inputting the nighttime rainy image to be processed into a trained nighttime rain removal network model to obtain a rain-free image output by the trained nighttime rain removal network model.
[0050] refer to Figure 6-Figure 8, the figure shows the comparison images of the processing results of the location-enhanced perception night image deraining method of the present application and several existing image deraining methods, wherein, figure a is the input image, figure b is the image deraining result using the DiG-CoM model, figure c is the image deraining result using the PreNet model, figure d is the image deraining result using the Unet model, figure e is the image deraining result using the Uformer model, figure f is the image deraining result using the RLP model, figure g is the image deraining result using the location-enhanced perception night image deraining method of the present application, and figure h is a real rain-free image. It can be seen from the figure that the location-enhanced perception night image deraining method of the present application has better deraining performance for night rainy images.
[0051] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.
[0052] In addition, the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, and may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0053] Furthermore, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.
[0054] In this document, relational terms such as first and second, etc. are used merely to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any such actual relationship or order between these entities or operations.
[0055] The above description is only an embodiment of the present application and is not intended to limit the protection scope of the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims
1. A location-enhanced perception night image deraining method, used for deraining night rainy images, characterized in that: Includes steps: A1. Constructing a nighttime rain removal network model; the nighttime rain removal network model comprises a rainfall prior module (1) and a position enhancement perception module (2), wherein the rainfall prior module (1) is used to obtain preliminary position information of rain streaks in a nighttime rainy image based on rainfall prior information, and the position enhancement perception module (2) is used to further extract position information of rain streaks in a nighttime rainy image; A2. Training the nighttime rain removal network model; A3. Use the trained nighttime rain removal network model to perform rain removal processing on the nighttime rainy image to be processed.
2. The method for removing rain from nighttime images with enhanced position perception according to claim 1, characterized in that: The nighttime rain removal network model also includes an input layer (3), a U-Net network (4) and an output layer (5); The input layer (3), the rainfall prior module (1), the location enhancement perception module (2), the U-Net network (4) and the output layer (5) are connected in sequence from front to back, and the input layer (3) is also connected to the location enhancement perception module (2).
3. The method for removing rain from nighttime images with enhanced position perception according to claim 2, characterized in that: The rainfall prior module (1) performs the following operations: ; in, is the output of the rainfall prior module (1) at the kth cycle stage, is the output of the rainfall prior module (1) at the k-1th cycle stage, Indicates channel cascade, is the main function of the rainfall prior module (1), N is the total number of preset cycle stages, is the initial prior mapping, It is the input of the rainfall prior module (1).
4. The location-enhanced perception nighttime image deraining method according to claim 3, characterized in that: The rainfall priori module (1) comprises a main part (101), the output end of the main part (101) is connected to the input end of the rainfall priori module (1) after channel cascading, and the output end of the main part (101) is also connected to the location enhancement perception module (2) as the output end of the rainfall priori module (1); The backbone part (101) comprises a first convolutional layer (102), an LSTM module (103), a plurality of residual modules (104) and a second convolutional layer (105) which are sequentially connected from front to back.
5. The location-enhanced nighttime image deraining method according to claim 2, characterized in that: The position enhancement perception module (2) comprises a spatial position encoding module (6) and an efficient channel attention module (7) connected front and back, and the spatial position encoding module (6) is connected to the input layer (3) and the output end of the rainfall prior module (1); The spatial position encoding module (6) is used to perform position encoding processing on the input feature map; The efficient channel attention module (7) is used to adjust the attention weight of each pixel of the feature map output by the spatial position encoding module (6).
6. The location-enhanced perception nighttime image deraining method according to claim 5, characterized in that: The spatial position encoding module (6) comprises an encoding module (601) and a nonlinear embedding module (602); the input end of the encoding module (601) is connected to the output end of the rainfall priori module (1); the input end of the nonlinear embedding module (602) is connected to the input layer (3); the output end of the encoding module (601) and the output end of the nonlinear embedding module (602) are cascaded through channels to serve as the output end of the spatial position encoding module (6).
7. The location-enhanced perception nighttime image deraining method according to claim 6, characterized in that: The encoding module (601) is used to perform the following operations: The encoding information of each pixel in the three coordinate directions is calculated according to the following formula: ; ; in, is the position of the pixel in the image, x, y, and z represent the three coordinate directions respectively. for The coordinate value corresponding to the j coordinate direction, is the feature dimension in the j-coordinate direction, i is a positive integer, for The dimension number of the corresponding pixel in the j coordinate direction, for The encoding information of the corresponding pixel in the j coordinate direction; The coding information of each pixel in three coordinate directions is concatenated to form the position code of each pixel.
8. The location-enhanced perception nighttime image deraining method according to claim 5, characterized in that: The efficient channel attention module (7) comprises a main body (701), an input end of the main body (701) is connected to an input end of the efficient channel attention module (7), and an output of the main body (701) is multiplied element-by-element with the input of the efficient channel attention module (7) to serve as the output of the efficient channel attention module (7); The main part (701) includes an average pooling layer (702), a third convolutional layer (703), a ReLU activation function layer (704), a fourth convolutional layer (705) and a Sigmoid activation function layer (706) which are connected in sequence from front to back.
9. The location-enhanced perception nighttime image deraining method according to claim 2, characterized in that: The U-Net network (4) includes a plurality of downsampling layers and a plurality of upsampling layers, and both the downsampling layers and the upsampling layers are Restormer blocks.
10. The location-enhanced perception nighttime image deraining method according to claim 9, characterized in that: The downsampling operation of the downsampling layer adopts a pixel deshuffle operation, and the upsampling operation of the upsampling layer adopts a pixel shuffle operation.
Citation Information
Patent Citations
Automobile 360-degree look-around image rain removal method based on Transform network
CN114913389A
Night image rain removing method and system based on rain line position prior
CN116152089A
Video rain removal method, system and device and storage medium
CN117808706A
