A night image rain removal method based on refined illumination modeling

Abstract

The application provides a night image rain removal method based on refined lighting modeling, and relates to the technical field of computer vision. The method comprises the following steps: acquiring an RGB three-channel night image, and respectively converting the RGB three-channel night image into YCbCr color space to separate Y channels and HSV color space to separate V channels; constructing a refined lighting modeling module based on the Y channels, generating a light source type weight map distinguishing point light sources and area light sources; calculating a dynamic lighting threshold pair based on V channel statistical features and generating a lighting coefficient matrix; calculating local lighting information by using the lighting coefficient matrix, fusing the local lighting information with the light source type weight map and Y channel features, and obtaining restored Y channel features; fusing the restored Y channel features with Cb and Cr channels to obtain a preliminary rain removal image; and finally inputting the preliminary rain removal image into a pre-trained encoder-decoder network to output a final restored image. The application effectively removes rain mark interference under complex night lighting by dual color space decoupling and refined lighting perception.

Description

Technical Field

[0001] This invention relates to the field of computer vision and image processing technology, and in particular to a method for removing rain from nighttime images based on refined lighting modeling. Background Technology

[0002] In applications such as urban security monitoring, autonomous driving visual perception, and all-weather outdoor imaging, rain at night is a common factor leading to image quality degradation. Unlike daytime scenes with uniform lighting, nighttime environments typically include various artificial light sources such as streetlights, vehicle lights, and advertising light boxes. These light sources vary greatly in brightness and shape, and raindrops produce complex scattering and refraction effects under different light sources, forming high-intensity rain streaks and diffuse halos in the image. This results in reduced image contrast, obscured object textures, and loss of key information, directly affecting the accuracy of subsequent video analysis tasks.

[0003] Most existing image deraining techniques focus on processing daytime images with relatively good lighting conditions, or use end-to-end deep learning networks to directly learn rain streak distribution in the RGB color space. This approach often ignores the independence of luminance and chrominance components in nighttime images, easily confusing rain streak features with background colors during the deraining process, thus destroying the original background tone. Furthermore, existing methods typically lack the ability to differentiate between point and area light sources, and using globally uniform convolution kernels or threshold parameters struggles to adapt to the extremely wide dynamic range at night. This results in rain streaks or artifacts remaining in strong light-emitting areas, while excessive smoothing in dark areas can lead to loss of background details. Summary of the Invention

[0004] The purpose of this invention is to provide a method for deraining nighttime images based on refined lighting modeling, in order to solve the problems mentioned in the background art.

[0005] This invention provides a nighttime image deraining method based on refined illumination modeling, comprising the following steps: Step S1: Obtain the input RGB three-channel night image, convert it to YCbCr color space to separate the Y channel, and at the same time convert the RGB three-channel night image to HSV color space to separate the V channel; Step S2: Based on the Y channel, construct a refined lighting modeling module to generate a light source type weight map that distinguishes between point light sources and area light sources; Step S3: Calculate dynamic lighting threshold pairs based on the statistical characteristics of the V channel, and use the dynamic lighting threshold pairs to calculate the lighting coefficient matrix; Step S4: Extract features from the Y channel to obtain Y channel features, calculate local lighting information using the lighting coefficient matrix, and fuse the local lighting information, the light source type weight map, and the Y channel features to obtain the restored Y channel features; Step S5: Fuse the recovered Y channel features with the Cb and Cr channels from step S1, and convert them to the RGB color space to obtain a preliminary derained image; Step S6: Input the preliminary derained image into the pre-trained encoder-decoder network and output the final restored image.

[0006] Optionally, the specific process of generating the light source type weight map in step S2 includes: The Y channel is input into the first convolutional layer, and the first output feature is obtained after normalization and nonlinear activation function processing. The first output feature is input into the second convolutional layer, and the second output feature is obtained after normalization and nonlinear activation function processing. The second output feature is input into the third convolutional layer, and the output is constrained by the Sigmoid activation function. The interval is used to obtain a pixel-level light source type weight map. ; Among them, the pixel-level light source type weight map It includes a channel 0 component and a channel 1 component, where the channel 0 component represents the characteristics of a point light source and the channel 1 component represents the characteristics of a surface light source.

[0007] Optionally, step S2 may further include weighting the pixel-level light source type map. After normalization and filtering, the final light source type weight map is obtained. The specific formula is as follows: Calculate the weights and graph. The formula is: ; in This indicates that the channel 0 component is... This represents the channel 1 component; Calculate the normalized light source type weight map The formula is: ; During the calculation process, if the corresponding coordinates are If the value is less than a preset threshold of 0.5, then the corresponding coordinate at that location... Set to 0 to filter out low-confidence light source areas.

[0008] Optionally, the specific process for calculating the dynamic lighting threshold pair in step S3 is as follows: Calculate the histogram of the V channel. ; The histogram The input consists of a network with two fully connected layers, and the output is a dynamic lighting threshold pair. ,in The lower threshold is... This is the upper limit threshold.

[0009] Optionally, the specific formula for calculating the lighting coefficient matrix in step S3 is as follows: ; in, Represents the illumination coefficient matrix in coordinates The value at that location, This indicates that the V channel is in coordinates Pixel value at that location, and For the dynamic lighting threshold pair.

[0010] Optionally, the specific formula for calculating local lighting information in step S4 is as follows: ; in, Indicates the currently calculated pixel Local lighting information at the location; Represented in pixels Centered Neighborhood; Indicates the The coordinates of any pixel in the neighborhood; This represents the illumination coefficient at the pixel coordinates; The Gaussian weight coefficient matrix has the following numerical distribution: .

[0011] Optionally, the specific process of obtaining the recovered Y-channel features in step S4 includes: The Y channel separated in step S1 is processed by a multi-layer Transformer or CNN encoder to extract the Y channel features. ; The local lighting information Weighted graph of final light source type By splicing the images together, the lighting characteristics can be obtained. ; The lighting features With the Y channel features The data is stitched together, input into a multilayer perceptron (MLP), and outputs the recovered Y-channel features. .

[0012] Optionally, the encoder-decoder network described in step S6 can be constructed using a multi-layer Transformer or a Convolutional Neural Network (CNN).

[0013] Optionally, the refined lighting modeling module, the fully connected network, the encoder, and the encoder-decoder network are jointly optimized during the training phase by minimizing a loss function, which includes the Y-channel MSE loss function. Charbonnier loss for complete image recovery SSIM loss for fully recovering the image .

[0014] Optionally, the loss function may also include hyperparameters for balancing the weights. The final total loss function The definitions of each sub-loss are as follows: ; ; ; ; in, and These are the height and width of the image, respectively; This represents the Y channel brightness value of the true image. This represents the recovered Y-channel characteristics; This refers to the final restored image. Represents the true value of a rainless image; For smoothing coefficients; This represents the structural similarity function.

[0015] The present invention has achieved the following beneficial effects: This invention utilizes the characteristic of separating luminance and chromaticity in the YCbCr color space, focusing on the structural recovery of the Y channel. This avoids interference with background colors caused by direct processing in the RGB space, thus ensuring the color fidelity of the image. The refined lighting modeling module constructed in this invention can identify and distinguish between point light sources and area light sources at the pixel level. Combined with a dynamic lighting threshold generated based on V channel statistical features, it achieves adaptive perception and differentiated processing for regions with different light intensities and light source shapes. This mechanism allows the algorithm to effectively suppress bright halos and remove rain streaks while preserving texture details in dark areas. It solves the problem of incomplete rain removal or loss of background details caused by the single lighting model in existing technologies, effectively improving the clarity and visibility of images in complex nighttime lighting environments.

[0016] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description and the accompanying drawings.

[0017] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0018] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart illustrating a nighttime image deraining method based on refined lighting modeling, as described in an embodiment of the present invention. Figure 2 This is a schematic diagram of the composition structure of a nighttime image deraining system based on refined lighting modeling in an embodiment of the present invention. Detailed Implementation

[0019] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0020] This application discloses a nighttime image deraining method based on refined lighting modeling. This method is primarily applied in the fields of computer vision and image processing, and is particularly suitable for scenarios such as visual perception systems for autonomous vehicles, urban intelligent traffic monitoring systems, and outdoor security monitoring terminals. In these scenarios, nighttime environments are often accompanied by complex lighting conditions, such as point light sources from streetlights and vehicle headlights, and area light sources from billboards and shop windows. Simultaneously, the presence of rainwater scatters and refracts these lights, forming high-contrast rain streaks and halos, severely affecting image clarity and the accuracy of downstream tasks (such as object detection and semantic segmentation). Most existing technologies perform deraining directly in the RGB space, easily ignoring the non-uniformity of light distribution, leading to artifacts or background color distortion near the light source in the derained image. To address these issues, this application proposes a deraining scheme based on the fusion of YCbCr and HSV dual color space features. By constructing a refined lighting modeling module and a dynamic threshold generation mechanism, it achieves adaptive perception and processing of different light source types.

[0021] In one specific embodiment of this application, the nighttime image deraining method based on refined lighting modeling can run on various types of electronic devices. These electronic devices can be servers or workstations equipped with high-performance graphics processors, or embedded computing devices such as vehicle domain controllers or edge computing boxes. The electronic device typically includes at least one processor, memory, and a communication interface. The processor, as the computing core, can be a central processing unit (CPU), graphics processing unit (GPU), neural network processor (NPU), field-programmable gate array (FPGA), or application-specific integrated circuit (ASIC) with data processing capabilities. Considering the convolutional neural network (CNN), Transformer module, and numerous matrix operations involved in the embodiments of this application, preferably, the processor uses a GPU or NPU that supports parallel computing architecture to meet the requirements of real-time processing. The memory is used to store computer program code and data during processing, such as the input nighttime image, intermediate feature maps, lighting weight maps, and trained network model parameters. The memory can include high-speed random access memory (RAM) and non-volatile memory (such as hard disks, flash memory, EEPROM, etc.). The communication interface is used to interact with other devices, such as receiving video stream data from an external camera, or sending processed de-rained images to a display terminal or downstream decision module.

[0022] In practical hardware deployment scenarios, such as on autonomous vehicles, this electronic device is connected to an onboard optical sensor. This optical sensor is responsible for acquiring images of the environment in front of the vehicle. Due to the low ambient light intensity at night, the sensor typically operates in high-gain or long-exposure modes. This results in the acquired raw images containing not only rain streaks but also glare and halos caused by streetlights or vehicle headlights, as well as thermal noise from the sensor itself. The method described in this application aims to solve this complex hybrid degradation problem and improve the environmental adaptability of nighttime perception from the source.

[0023] Please see Figure 1 The specific implementation steps and underlying working principles of the method provided in the embodiments of this application will be described in detail below.

[0024] Step 1: Data acquisition and multi-color space conversion.

[0025] First, the embodiments of this application perform data acquisition and multi-color space conversion steps.

[0026] In this stage, the system first acquires the input RGB three-channel nighttime image, denoted as the input image. The input image It has three color channels, namely red (R), green (G), and blue (B), and its data dimensions are usually represented as follows: ,in Represents the height of the image (number of pixel rows). This represents the width of the image (number of pixel columns). In the computer's internal storage, each channel value of each pixel is typically represented as an 8-bit unsigned integer (range 0-255) or a normalized floating-point number (range 0.0-1.0).

[0027] In order to better separate rain streaks from background color features, this application does not perform rain removal processing directly in the RGB space, but adopts a multi-color space decoupling strategy.

[0028] On one hand, the system converts the RGB three-channel nighttime image into the YCbCr color space. The YCbCr color space is a standard color model widely used in digital image and video processing, where the Y component represents luminance, the Cb component represents chromaticity shift (blue), and the Cr component represents chromaticity shift (red). In physical optics, raindrops, as transparent media, scatter and refract light, primarily altering the propagation path and intensity of light, thus forming high-brightness stripes or spots on the image. This drastic change in brightness is mainly concentrated in the Y channel, while the impact on the Cb and Cr chromaticity channels is relatively small and gradual. Therefore, by converting to the YCbCr space and separating the Y channel, the processing focus can be on the image's structure and brightness information, avoiding damage to the inherent colors of background objects during the deraining operation.

[0029] The specific conversion process can be performed according to the BT.601 standard published by the International Telecommunication Union (ITU). For each pixel coordinate in the image... Its RGB value Convert to YCbCr value The mathematical formula is as follows: ; ; ; Through the above calculations, the system separates independent Y-channel data, Cb-channel data, and Cr-channel data. In subsequent steps of this embodiment, the Y-channel data will be used as the core input and fed into the refined lighting modeling module and rain removal network for deep processing; while the Cb-channel data and Cr-channel data will be temporarily retained and will participate in the final image reconstruction after the luminance components are restored.

[0030] On the other hand, in order to perceive the illumination intensity distribution of the image from a global statistical perspective, this application embodiment converts the RGB three-channel night image into the HSV color space in parallel and separates the V channel from it.

[0031] The HSV color space consists of three components: Hue, Saturation, and Value. The Value (V) channel is typically defined as the maximum value among the three RGB components. ; Although both the Y and V channels describe the brightness attributes of an image, they have different physical meanings and application values ​​in nighttime rain removal tasks. The Y channel, calculated using a weighted average, reflects the human eye's comprehensive perception of different color light intensities, making it suitable for extracting texture details and geometric structures from an image. The V channel, on the other hand, directly takes the maximum value from the RGB values, enabling it to accurately capture extreme regions in an image. In nighttime scenes, artificial light sources (such as direct headlights or the center of streetlights) often cause pixel values ​​in local image areas to reach saturation (e.g., 255). In such cases, the V channel accurately reflects the distribution range and extreme intensity of these highlight areas, which is statistically significant for determining whether an image is overexposed or underexposed. Therefore, in this embodiment, the separated V channel data is specifically used for subsequent global illumination threshold calculation and light intensity adjustment, complementing the Y channel used for structure restoration.

[0032] Step 2: Refine the lighting model.

[0033] Secondly, the embodiments of this application perform a fine lighting modeling step based on the Y channel.

[0034] Artificial light sources in nighttime images vary in form, and different types of light sources have drastically different effects on the imaging of rain streaks. For example, point light sources, such as distant streetlights or oncoming vehicle headlights, typically appear as bright spots with high-frequency characteristics in images. Their radiation range is small, but their intensity gradient is large. Raindrops illuminated by point light sources will form sharp, thin, and high-contrast bright lines. Area light sources, such as large advertising light boxes or walls illuminated by floodlights, typically appear as smooth bright areas with low-frequency characteristics. Their radiation range is large and their intensity is relatively uniform. Raindrops against the background of area light sources often appear as low-contrast, foggy obscurations or blurred halos.

[0035] Existing rain removal algorithms often ignore the differences in light source types, using uniform parameters or convolution kernels for processing. This can lead to insufficient rain removal in point light source areas, leaving behind bright noise; or excessive rain removal in area light source areas, causing background textures (such as text on billboards) to be erased.

[0036] To address this issue, this application embodiment constructs a refined lighting modeling module on the Y channel. This module is a deep learning-based feature extraction and classification network designed to identify and distinguish light source types in images at the pixel level. The specific workflow of this module is as follows: The first step is shallow feature extraction and normalization.

[0037] The separated Y-channel data (denoted as the input tensor) is input into the first convolutional layer of the module. The first convolutional layer contains several convolutional kernels (e.g., ...). (number of convolutional kernels), the size of which is preferably set to [number of kernels]. With a stride of 1 and padding of 1, this configuration ensures that the spatial size of the output feature map remains consistent with the input image, while effectively extracting local brightness gradients, edge and corner features. The convolution operation maps the single-channel Y image to... Feature map of the channel.

[0038] Subsequently, the output features of the first convolutional layer are normalized. Given the significant differences in the distribution of nighttime image data across different scenes (e.g., urban centers versus rural roads) and the extremely strong contrast within a single image, this embodiment preferably employs instance normalization (IN) or layer normalization (LN) rather than batch normalization. Instance normalization standardizes the mean and variance within the feature channels of a single image, better preserving the unique contrast information of each image and is not limited by batch size.

[0039] After normalization, the data is activated by the GELU (Gaussian Error Linear Unit) non-linear activation function. Compared to the traditional ReLU function, the GELU function has a smooth, non-zero gradient in the negative region, which can introduce a stochastic regularization effect. When processing nighttime images, many background pixel values ​​are close to zero. Using GELU can effectively prevent neuron failure and preserve weak texture features in dark areas, thus obtaining the first output feature. .

[0040] The second step is deep semantic feature extraction.

[0041] The first output feature The input is fed into the second convolutional layer. The purpose of the second convolutional layer is to further expand the receptive field, integrating the gradient features from the shallow layers into morphological features with semantic information. The number of output channels for this layer is set to... (e.g., 64 or 128), the kernel size remains constant. By stacking multiple convolutional layers, the network can perceive pixel brightness distribution patterns over a larger area, thereby distinguishing between isolated bright spots (point light source features) and continuous smooth bright areas (area light source features).

[0042] Similarly, the output of the second convolutional layer is also processed by data normalization (such as instance normalization) and the GELU non-linear activation function to obtain the second output features. At this point, each node in the feature map no longer simply represents a brightness value, but rather encodes the lighting pattern information of that location and its neighborhood.

[0043] The third step is probability mapping of light source types.

[0044] The second output feature The input is fed into the third convolutional layer. This layer is the classification decision layer, and its output channel number is strictly set to 2. These two channels correspond to the confidence maps of the point light source and the area light source, respectively.

[0045] To transform the unbounded values ​​of the convolution output into physically meaningful probability weights, this layer is followed by a sigmoid activation function. The sigmoid function strictly maps all output values ​​to... Within the interval. The output obtained at this time is called the pixel-level light source type weight map, denoted as . .

[0046] It is a dimension The tensor of channel 0. This represents the probability that each pixel in the image belongs to a point light source feature; the components of channel 1. This represents the probability that each pixel belongs to the features of an area light source. Through end-to-end supervised learning, the network can automatically learn how to determine which type of light source the current pixel best matches based on the input brightness texture.

[0047] The fourth step is weight normalization and background filtering.

[0048] Directly generated The image may contain some noise, especially in dark background areas, and the network may output some meaningless random fluctuations. To obtain a clean and reliable light source weight map... In this application, the embodiments introduce a normalization and filtering mechanism.

[0049] First, the system calculates the weights and graph. For any pixel coordinate on the image plane Its weights and sums It equals the sum of the weights of the point light source and the area light source at that location, that is: ; This value intuitively reflects the network's overall confidence level that an artificial light source exists at this location.

[0050] Next, normalization calculations are performed to obtain a preliminary weighted map of light source types. The calculation formula is: ; This step ensures that the sum of the relative proportions of point light source components and area light source components within the light source area is 1, thus clarifying the dominant attribute of the light source.

[0051] Most importantly, this application embodiment sets a preset confidence threshold, such as 0.5. The system scans pixel by pixel. If a certain location If the confidence level is less than 0.5, the region is considered a low-confidence region (i.e., a dark background without a light source). In this case, the system forces the corresponding location to be weighted in the final light source type weight map. All channel values ​​are set to 0.

[0052] This operation is equivalent to providing a precise light source mask for the rain removal network. It ensures that subsequent illumination adaptation processing only applies to the actual light source areas, while background areas that are already dark and do not reflect rain streaks are skipped or given zero weight, thus avoiding the introduction of artificial artifacts in dark areas and greatly improving processing efficiency and image purity.

[0053] Step 3: Dynamic threshold calculation and illumination coefficient matrix generation.

[0054] Furthermore, the embodiments of this application perform the steps of dynamic threshold calculation and illumination coefficient matrix generation based on the V channel.

[0055] Step S2 addresses the issue of light source type, while this step aims to address the issue of light intensity adaptation. Nighttime images have an extremely wide dynamic range, and the brightness distribution varies drastically across different scenes (such as bustling commercial streets versus quiet residential areas). A fixed brightness threshold cannot adapt to all scenes. Therefore, this embodiment utilizes the statistical characteristics of the V channel to construct a dynamic threshold generation mechanism.

[0056] The first step is histogram statistical analysis.

[0057] The V-channel image separated by the aforementioned steps The system calculates its grayscale histogram. Assuming the image bit depth is 8 bits, the histogram... It is a vector with dimension 256, where the first... Each element records the brightness value in the image. The number of pixels. Histogram It is a statistical summary of the global illumination environment of an image, which implicitly contains key information such as the image's average brightness, contrast, and whether there are overexposed or underexposed areas.

[0058] The second step is dynamic threshold prediction.

[0059] In order to intelligently infer the optimal brightness segmentation threshold from histogram features, embodiments of this application utilize a pre-trained fully connected network (Multi-Layer Perceptron, MLP).

[0060] The system will calculate the histogram vector. As input, it is fed into the fully connected network. This network contains at least two fully connected layers. The first fully connected layer maps the high-dimensional histogram features to the hidden layer space, extracting deep patterns of the illumination distribution; after processing by a non-linear activation function (such as ReLU), it enters the second fully connected layer. The second fully connected layer directly regresses and outputs two scalar values, namely the dynamic illumination threshold pair. .

[0061] in, This is the lower threshold value used to define the boundary between extremely dark backgrounds and areas of effective information. These two thresholds are upper limits used to define the boundary between the effective information area and the extremely bright saturated area. They are dynamically generated by the network based on the histogram features of the current input image, thus possessing strong scene adaptability.

[0062] The third step is to calculate the lighting coefficient matrix.

[0063] Based on the generated dynamic threshold pair The system calculates the illumination coefficient matrix for the entire map. For each pixel in the V channel Different lighting coefficients are assigned based on the range in which the value falls. The specific calculation logic is as follows: if If the pixel is in an extremely dark region, it is determined that the region is dark. These regions typically have extremely low signal-to-noise ratios and mainly consist of sensor noise. To prevent the amplification of this noise during the rain removal process, embodiments of this application set its illumination coefficient to half of the original brightness, i.e. .

[0064] if If the pixel is in an extremely bright region, it is determined that the pixel is in an extremely bright region. These regions are usually the center of the light source, where the pixel value is close to saturation and texture information is severely lost. To prevent highlight clipping or halo artifacts caused by rain removal operations, this embodiment also sets its illumination coefficient to half of the original brightness, i.e. .

[0065] if If the pixel is found to be in the mid-tone region, then it is determined that the region is the richest in texture, has the most typical rain streaks, and contains the most effective background information in nighttime images. In this embodiment, the original brightness value is retained as the illumination coefficient. .

[0066] Through this piecewise mapping, the illumination coefficient matrix In essence, it becomes an illumination attention map. It indicates that subsequent networks should focus on restoring details in the intermediate brightness regions, while applying soft suppression to the extreme brightness regions at both ends.

[0067] Step 4: Y-channel feature extraction and fusion.

[0068] Next, the embodiment of this application performs the Y-channel feature extraction and fusion steps.

[0069] This step is the core of the entire algorithm, aiming to integrate the previously acquired structural features (from the Y channel) and the prior knowledge of the light source type (from...). ) and prior knowledge of light intensity (from Deep fusion is performed to generate the Y channel features after rain removal.

[0070] The first step is to extract the depth features of the Y channel.

[0071] The system inputs the raw Y-channel image separated in step S1 into a deep feature extraction network. To capture long-range dependencies in the image (such as long rain streaks running across the screen), the network preferably employs an encoder structure containing multiple Transformer modules, or a deep convolutional neural network (CNN). The Y-channel image is transformed into a high-dimensional feature tensor through layers of the encoder. This feature It not only contains the original semantic and structural information of the image, but also implicitly contains the rain streak pattern to be removed.

[0072] The second step is to calculate the local lighting information.

[0073] Light does not abruptly change between pixels when propagating in a medium, but rather exhibits a smooth transition and diffusion. To simulate this physical process, the embodiments of this application do not directly use the illumination coefficient of a single point. Instead, it calculates local lighting information. .

[0074] The system defines a value based on the current pixel. Centered Neighborhood Within this neighborhood, for The matrix is ​​then subjected to a Gaussian weighted summation. The specific weight distribution is determined by the Gaussian weight coefficient matrix. Decision. In the embodiments of this application, The values ​​are precisely designed to approximate a two-dimensional Gaussian distribution: the center point has a weight of 0.368, the four orthogonal neighborhoods (top, bottom, left, and right) have a weight of 0.135, and the four diagonal neighborhoods have a weight of 0.018.

[0075] The calculation formula is: ; in belong .

[0076] The local lighting information obtained through this convolutional weighted calculation It not only reflects the light intensity at the current point, but also incorporates the contribution of ambient light, resulting in better spatial continuity and effectively simulating the halo effect in rainy and foggy environments at night.

[0077] The third step is the splicing and fusion of multi-source features.

[0078] The system will calculate the single-channel local lighting information. Dual-channel light source type weighting diagram and multi-channel Y-channel characteristics Concatenation is performed along the channel dimension to form a hybrid feature tensor containing rich prior information.

[0079] To enable the network to learn to utilize this prior information to guide rain removal, the concatenated features are input into a multilayer perceptron (MLP) or convolutional fusion layer. This fusion layer acts as a non-linear mapper, dynamically adjusting the features according to illumination conditions. The processing strategy. For example, in Indicates a point light source and In areas designated as high brightness, the network automatically enhances its suppression of high-frequency rain streaks; while... For areas designated as background, the network tends to preserve the original texture.

[0080] The output of the fusion layer is the recovered Y-channel feature. At this time Most of the rain streaks have been removed, and a natural transition has been maintained in areas of abrupt changes in illumination (such as halo edges), laying a solid foundation for subsequent color image reconstruction.

[0081] Step 5: Multi-channel fusion and inverse color space transformation.

[0082] Next, the embodiments of this application perform multi-channel fusion and color space inverse transformation steps.

[0083] This step corresponds to step S5 in the technical solution of this application. In the aforementioned steps S1 to S4, all depth feature extraction, lighting refinement modeling, and rain removal texture restoration operations are concentrated within the Y-channel feature space of the luminance. This is based on a profound understanding of the physical characteristics of nighttime imaging: the scattering effect of raindrops under complex nighttime lighting mainly manifests as drastic fluctuations in luminance (such as bright stripes or halos), while having an extremely weak impact on the chromaticity attributes (Cb and Cr channels) of the light reflected from the background object. Therefore, after completing the core luminance channel rain removal, the restored Y-channel features (denoted as...) are obtained. After that, it must be realigned and fused with color information to reconstruct a color image that conforms to human visual perception.

[0084] Specifically, the system first processes the recovered Y-channel features output in step S4. Perform dimensional checks and standardization. Because... These are high-dimensional features processed by deep neural networks (such as MLPs or convolutional layers), and their physical values ​​may fall within the normalized floating-point range (e.g., 0.0 to 1.0). The system retrieves the original Cb channel data and the original Cr channel data separated and temporarily stored in the cache memory in step S1. Although these two chroma channels have not undergone dedicated deep deraining network processing, it is precisely this operation-free strategy that maximizes the preservation of the original color fidelity of the image and avoids the color cast or saturation degradation problems commonly found in traditional end-to-end deraining networks.

[0085] The system performs a channel splicing operation. This splices the single-channel... Single-channel Cb and single-channel Cr data are stacked in the depth direction to form a three-channel YCbCr data tensor. Then, an inverse color space transformation is performed. This transformation process strictly follows the linear algebraic transformation relationships defined by digital video broadcasting standards (such as ITU-R BT.601). For each pixel coordinate on the image plane... The formula for calculating its inverse transform is as follows: ; ; ; When performing the above matrix operations, considering the precision characteristics of computer floating-point operations, this embodiment uses double-precision floating-point numbers (float64) for intermediate calculations to reduce accumulated errors. The calculated R, G, and B values ​​should theoretically fall within the range of... Within the specified range, however, due to the non-linear mapping of deep networks, some pixel values ​​may overflow. Therefore, the system further performs numerical truncation and quantization operations: forcibly correcting all values ​​less than 0 to 0, forcibly correcting all values ​​greater than 255 to 255, and converting the data type to an 8-bit unsigned integer (uint8).

[0086] At this point, the system outputs a preliminary derained image, denoted as... At this time Most rain streaks have been removed from the macroscopic structure, and thanks to the lighting modeling in steps S2 and S3, the brightness distribution in the point and area light source regions has become more natural. However, as an intermediate product, it may still have some minor imperfections, such as slight jaggedness at the edge junction of the Y channel and the CbCr channel, or a small amount of residual noise in extremely dark backgrounds.

[0087] Step Six: Global Fine-grained Recovery.

[0088] Subsequently, embodiments of this application perform a global fine-grained recovery step based on the encoder-decoder network.

[0089] This step corresponds to step S6 in the technical solution of this application. In order to obtain the preliminary rain-removed image... To enhance the final restored image to meet top-tier image quality requirements, this application introduces an end-to-end deep neural network model. This model employs a classic encoder-decoder architecture and incorporates the latest Transformer or Convolutional Neural Network (CNN) techniques to achieve powerful feature extraction and reconstruction capabilities.

[0090] The network's workflow mainly consists of three stages: encoding, bottleneck transformation, and decoding. The first stage is the encoding stage. Preliminary rain-removed image. The input is fed into the encoder. The encoder consists of multiple cascaded downsampling modules. Each downsampling module extracts local features of the image through convolution operations and reduces the spatial resolution of the feature map through convolution or pooling operations with a stride of 2, while increasing the number of feature channels. As the network depth increases, the feature map gradually transforms from low-level texture features (such as edges and colors) to high-level semantic features (such as the shape of objects and the structure of scenes). In a preferred embodiment of this application, a Transformer module is embedded in the encoder, utilizing its multi-head self-attention mechanism to calculate the correlation between any two pixels in the image. This allows the network to use areas not obscured by rain streaks as references to perform non-local information completion of obscured areas, thereby effectively solving the problem of information loss caused by dense rain streaks in nighttime images.

[0091] The second stage is the bottleneck stage. At the end of the encoder, image features are compressed to the smallest spatial size but contain the richest contextual semantic information. Here, the network performs a deep nonlinear transformation on the features by stacking multiple residual dense blocks or Swin Transformer Blocks. This stage is crucial for the network to understand image content and distinguish noise (rain streaks, halos) from signals (background objects). The network utilizes prior knowledge learned from large datasets to suppress anomalous responses in feature maps and connect broken texture features.

[0092] The third stage is the decoding stage. The deep features, after being processed by the bottleneck layer, enter the decoder. The decoder's structure is symmetrical to the encoder, consisting of multiple cascaded upsampling modules. The upsampling operation gradually restores the spatial resolution of the feature map through transposed convolution or bilinear interpolation combined with convolution, ultimately bringing it back to a level similar to the original. Figure 1 The exact dimensions.

[0093] To address the issue of deep networks easily losing high-frequency details (such as the sharpness of text and the veins of leaves), this application establishes skip connections between corresponding layers of the encoder and decoder. Skip connections directly stitch the high-resolution shallow features extracted by the encoder onto the corresponding layer features of the decoder. This design allows the decoder to utilize both deep semantic information to remove rain streaks and shallow detail information to restore clear edges when reconstructing images, effectively avoiding the smoothing or blurring effects commonly found in de-raining images.

[0094] The final output layer of the network uses a convolutional kernel of size [size missing]. The convolutional layer maps the high-dimensional features back to the 3-channel RGB space, obtaining the final reconstructed image, denoted as [image name missing] in the technical solution. (Corresponding to Y, for the sake of precision, it is denoted here as Y) This image is the final result output by the method of this application. It not only completely removes rain streaks but also corrects the halo effect unique to nighttime, and has natural colors and rich details.

[0095] To ensure that the entire system, including the refined lighting modeling module, the fully connected network, the feature extraction network, and the encoder-decoder network, can work together and achieve optimal performance, this application embodiment constructs a scientifically rigorous loss function system and optimizes the network parameters by minimizing this loss function during the model training phase.

[0096] The total loss function proposed in this application embodiment It is a composite loss function, consisting of the mean square error loss of the Y channel ( Charbonnier loss for complete image recovery ) and structural similarity loss for complete image recovery ( It consists of three parts. The mathematical definition of each sub-loss function and its physical significance in this scheme are explained in detail below.

[0097] First, define the Y-channel MSE loss function. .

[0098] Since this application focuses on independent and refined modeling and rain removal of the Y channel (luminance), it is necessary to analyze the intermediate product, i.e., the recovered Y channel features. Explicit supervisory constraints are imposed. The mean squared error (MSE) loss function is chosen as the supervisory metric for brightness restoration due to its characteristics of being differentiable everywhere, having good convexity, and converging quickly. Its calculation formula is as follows: ; in, Indicates the height of the image. Indicates the width of the image. Represents pixel coordinates. The Y channel brightness value represents the ground truth image (i.e., a clear image of the same scene without rain); This represents the recovered Y-channel luminance value output from step S4. This formula calculates the average of the squares of the differences between the predicted and actual luminance. Its physical meaning is to force the luminance map generated by the network to numerically approximate the luminance distribution of the real scene as closely as possible. Since MSE is highly sensitive to large errors (square amplification effect), it effectively drives the network to prioritize the elimination of high-brightness rain streaks (i.e., areas with huge pixel value differences), thereby ensuring the accuracy of the lighting coefficient matrix generated by the refined lighting modeling module. It can accurately reflect the intensity of light.

[0099] Secondly, define the Charbonnier loss for fully recovering the image. .

[0100] For the final output color image This application embodiment does not continue to use MSE loss, but instead selects Charbonnier loss. Charbonnier loss is essentially a smooth L1 norm loss (Differentiable L1 Loss). In the field of image restoration, traditional L2 loss (MSE) tends to produce a blurred average solution, while L1 loss, although better at preserving edges, is not differentiable at zeros, leading to oscillations in the later stages of training. Charbonnier loss combines the advantages of both. Its calculation formula is as follows: ; in, This represents the pixel values ​​of the finally recovered image. This represents the pixel value of the true rainless image. It is a very small smoothing coefficient (Constant), which is preferably set to in this embodiment. or .

[0101] The physical meaning of this formula is that when the prediction error is large, The loss function is negligible, approximating as L1 loss (linear growth), which is robust to outliers and effectively preserves image edges and textures. When the prediction error approaches 0, the function approximates as L2 loss (quadratic growth), exhibiting a smooth gradient, which facilitates minor parameter adjustments during later training phases and converges to a higher-precision solution. In nighttime rain removal tasks, images contain significant noise and halos; using Charbonnier loss can significantly improve image clarity and visual appeal.

[0102] Next, define the SSIM loss for fully recovering the image. .

[0103] The aforementioned MSE and Charbonnier losses are both based on point-to-point pixel errors, which easily ignore the spatial correlation between image pixels. However, the human eye's judgment of image quality is highly dependent on the structural information of objects. In order to make the reconstructed image visually closer to the real scene, this application introduces structural similarity (SSIM) loss.

[0104] The calculation formula is as follows: ; in, The function measures the similarity between two images from three dimensions: luminance, contrast, and structure. For a local window in the image... and The calculation formula is as follows: ; In this formula, the parameters are defined as follows: Represents the predicted image window The mean value reflects the local brightness; Truth image window The mean; Represents the predicted image window The variance reflects local contrast; Truth image window The variance; express and The covariance reflects the degree of structural similarity; and It is a constant used to maintain computational stability and prevent the denominator from being zero. It is usually taken as... ,in For pixel dynamic range (e.g., 255). .

[0105] The physical significance lies in maximizing the structural similarity between the generated image and the ground truth image. It forces the network to focus on the high-frequency edge information of objects (such as the straight lines of lampposts and the outlines of vehicles), preventing structural distortion or blurring in the image after rain removal.

[0106] Finally, define the total loss function. .

[0107] This application's embodiment combines the above three losses through a weighted summation to form the final optimization objective: ; in, It is a weighted hyperparameter. This parameter is used to balance the importance of Y-channel luminance recovery and global color structure recovery. After extensive experimental verification, this application will... Setting the loss function within the range of 0.01 to 0.2 yields the best training results. This combined loss function not only ensures the correctness of the image lighting physical model (by...) This ensures both the protection and the perceptual quality of the final output image (provided by...). and Assure).

[0108] To fully verify the effectiveness and advancement of the methods described in the embodiments of this application, a standardized experimental environment was constructed for detailed testing and comparative analysis.

[0109] In terms of experimental hardware configuration, a server workstation equipped with a high-performance deep learning accelerator card (such as an NVIDIA GeForce RTX 3090 with 24GB of video memory) was used. The central processing unit (CPU) was an Intel Xeon series, and the memory was 128GB.

[0110] In terms of experimental software configuration, the operating system was Ubuntu 20.04 LTS, and the deep learning framework used was PyTorch version 1.10.

[0111] In terms of training strategy, the AdamW optimizer (Adam with Weight Decay) is used, with its parameters set to... The weight decay factor is 0.02. The initial learning rate is set to... The learning rate was gradually reduced to a certain value during 300,000 iterations using a cosine annealing strategy. To increase data diversity and prevent overfitting, data augmentation operations such as random cropping, random horizontal flipping, and 90-degree rotation were performed on the input images during training.

[0112] The experiment used the industry-recognized HQ-NightRain dataset as the benchmark. This dataset contains a large number of paired nighttime images with rain / no rain, covering a variety of complex lighting scenarios such as urban roads, highways, and commercial districts.

[0113] The comparative experiment selected four of the most representative advanced rain removal algorithms in the current field of computer vision: Restormer, DRSformer, IDT, and NeRD.

[0114] The evaluation metrics used are Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM). A higher PSNR value indicates less image distortion and higher quality; a SSIM value closer to 1 indicates better image structure restoration.

[0115] Experimental results show that: Restormer method: PSNR is 41.8844dB, SSIM is 0.9907; DRSformer method: PSNR is 42.8107dB, SSIM is 0.9922; IDT method: PSNR is 42.4204dB, SSIM is 0.9918; NeRD method: PSNR is 42.7139 dB, SSIM is 0.9923; The method in this embodiment achieved a PSNR of 42.8856 dB and an SSIM of 0.9925.

[0116] As can be seen from the above objective data, the method in this application surpasses all comparable methods in both key metrics, achieving state-of-the-art performance. In particular, the leading performance on the SSIM metric demonstrates that this application effectively preserves the structural information of the image through refined lighting modeling and a dynamic thresholding strategy, avoiding the light source halo and background blur problems that are prone to occur in traditional algorithms.

[0117] In a comparison of subjective visual effects, the advantages of the method proposed in this application are more significant. For point light source areas (such as streetlights), the image generated by the method of this application shows clear light source outlines with no obvious halo diffusion; for area light source areas (such as billboards), the method of this application completely preserves the text texture on the billboard surface without any accidental deletion; for dark background areas, the image purity is high and noise is effectively suppressed. This fully demonstrates the effectiveness and practicality of the YCbCr / HSV dual-spatial feature fusion scheme proposed in this application.

[0118] In addition, please see Figure 2 This application also relates to a nighttime image deraining device based on refined lighting modeling for performing the above-described method. The device logically includes functional modules that correspond one-to-one with the steps of the above-described method.

[0119] Specifically, the device includes: The color space conversion module is used to acquire the input RGB three-channel night image, convert it to the YCbCr color space to separate the Y channel, and at the same time convert the RGB three-channel night image to the HSV color space to separate the V channel. The refined lighting modeling module is used to generate a light source type weight map that distinguishes between point light sources and area light sources based on the Y channel. This module contains a first convolution unit, a second convolution unit, and a probability mapping unit. The dynamic threshold calculation module is used to calculate dynamic lighting threshold pairs based on the statistical characteristics of the V channel, and to calculate the lighting coefficient matrix using the dynamic lighting threshold pairs. This module integrates a histogram statistical unit and a fully connected prediction network. The feature extraction and fusion module is used to extract features from the Y channel to obtain Y channel features, calculate local lighting information using the lighting coefficient matrix, and fuse the local lighting information, the light source type weight map, and the Y channel features to obtain the restored Y channel features. The color reconstruction module is used to fuse the recovered Y channel features with the Cb and Cr channels and convert them into the RGB color space to obtain a preliminary derained image. The global refinement module is used to input the preliminary derained image into a pre-trained encoder-decoder network and output the final restored image.

[0120] These modules can be stored in memory as software code and executed by a processor, or they can be embedded in a chip as hardware logic circuits (such as FPGAs and ASICs) to meet the processing speed and power consumption requirements of different application scenarios.

[0121] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the nighttime image deraining method based on refined lighting modeling described in any of the above embodiments.

[0122] The electronic device may be, but is not limited to, a desktop computer, a laptop computer, a smartphone, a tablet computer, an in-vehicle computer (ECU), a network video recorder (NVR), or a cloud server.

[0123] The memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, like read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0124] The processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0125] This application also provides a computer-readable storage medium storing a computer program thereon. When the program is executed by an electronic device, the electronic device implements the nighttime image deraining method based on refined lighting modeling described in any of the above embodiments.

[0126] The computer-readable storage medium can be any medium capable of storing program code, including but not limited to: floppy disks, hard disks, optical discs (CD-ROM, DVD-ROM), magneto-optical discs, magnetic tapes, flash memory cards (USB flash drives, SD cards), as well as ROM, RAM, etc.

[0127] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for deraining rain from nighttime images based on refined illumination modeling, characterized in that, Includes the following steps: Step S1: Obtain the input RGB three-channel night image, convert it to YCbCr color space to separate the Y channel, and at the same time convert the RGB three-channel night image to HSV color space to separate the V channel; Step S2: Based on the Y channel, construct a refined lighting modeling module to generate a light source type weight map that distinguishes between point light sources and area light sources; Step S3: Based on the statistical characteristics of the V channel, calculate the dynamic illumination threshold pair using a fully connected network containing two fully connected layers, and calculate the two-dimensional illumination coefficient matrix corresponding to the pixel coordinates of the V channel according to the pixel value of the V channel at each coordinate and the dynamic illumination threshold pair. Step S4: Extract features from the Y channel to obtain Y channel features, calculate local lighting information using the lighting coefficient matrix, and fuse the local lighting information, the light source type weight map, and the Y channel features to obtain the restored Y channel features; Step S5: Fuse the recovered Y channel features with the Cb and Cr channels from step S1, and convert them to the RGB color space to obtain a preliminary derained image; Step S6: Input the preliminary derained image into the pre-trained encoder-decoder network and output the final restored image.

2. The nighttime image deraining method based on refined lighting modeling according to claim 1, characterized in that, The specific process of generating the light source type weight map in step S2 includes: The Y channel is input into the first convolutional layer, and the first output feature is obtained after normalization and nonlinear activation function processing. The first output feature is input into the second convolutional layer, and the second output feature is obtained after normalization and nonlinear activation function processing. The second output feature is input into the third convolutional layer, and the output is constrained by the Sigmoid activation function. The interval is used to obtain a pixel-level light source type weight map. ; Among them, the pixel-level light source type weight map It includes a channel 0 component and a channel 1 component, where the channel 0 component represents the characteristics of a point light source and the channel 1 component represents the characteristics of a surface light source.

3. The nighttime image deraining method based on refined lighting modeling according to claim 2, characterized in that, Step S2 also includes weighting the pixel-level light source type map. After normalization and filtering, the final light source type weight map is obtained. The specific formula is as follows: Calculate the weights and graph. The formula is: ； in This indicates that the channel 0 component is... This represents the channel 1 component; Calculate the normalized light source type weight map The formula is: ； During the calculation process, if the corresponding coordinates are If the value is less than a preset threshold of 0.5, then the corresponding coordinate at that location... Set to 0 to filter out low-confidence light source areas.

4. The nighttime image deraining method based on refined lighting modeling according to claim 1, characterized in that, The specific process for calculating the dynamic lighting threshold pair in step S3 is as follows: Calculate the histogram of the V channel. ; The histogram Input the fully connected network and output dynamic lighting threshold pairs ,in The lower threshold is... This is the upper limit threshold.

5. A nighttime image deraining method based on refined lighting modeling according to claim 4, characterized in that, The specific formula for calculating the lighting coefficient matrix in step S3 is as follows: ； in, Represents the illumination coefficient matrix in coordinates The value at that location, This indicates that the V channel is in coordinates Pixel value at that location, and For the dynamic lighting threshold pair.

6. A nighttime image deraining method based on refined lighting modeling according to claim 5, characterized in that, The specific formula for calculating local lighting information in step S4 is as follows: ； in, Indicates the currently calculated pixel Local lighting information at the location; Represented in pixels Centered Neighborhood; Indicates the The coordinates of any pixel in the neighborhood; This represents the illumination coefficient at the pixel coordinates; The Gaussian weight coefficient matrix has the following numerical distribution: .

7. A nighttime image deraining method based on refined lighting modeling according to claim 6, characterized in that, The specific process of obtaining the recovered Y channel features in step S4 includes: The Y channel separated in step S1 is processed by a multi-layer Transformer or CNN encoder to extract the Y channel features. ; The local lighting information Weighted graph of final light source type By splicing the images together, the lighting characteristics can be obtained. ; The lighting features With the Y channel features The data is stitched together, input into a multilayer perceptron (MLP), and outputs the recovered Y-channel features. .

8. A nighttime image deraining method based on refined lighting modeling according to claim 1, characterized in that, The encoder-decoder network described in step S6 is constructed using a multi-layer Transformer or a Convolutional Neural Network (CNN).

9. A nighttime image deraining method based on refined lighting modeling according to claim 7, characterized in that, The refined lighting modeling module, fully connected network, encoder, and encoder-decoder network are jointly optimized during the training phase by minimizing a loss function, which includes the Y-channel MSE loss function. Charbonnier loss for complete image recovery SSIM loss for fully recovering the image .

10. A nighttime image deraining method based on refined lighting modeling according to claim 9, characterized in that, The loss function also includes hyperparameters for balancing the weights. The final total loss function The definitions of each sub-loss are as follows: ； ； ； ； in, and These are the height and width of the image, respectively; This represents the Y channel brightness value of the true image. This represents the recovered Y-channel characteristics; This refers to the final restored image. Represents the true value of a rainless image; For smoothing coefficients; This represents the structural similarity function.

Images