A rain picture rain intensity calculation method based on raindrop extraction-rain intensity calculation

Abstract

The application belongs to the field of municipal engineering rainwater real-time measurement, and provides a rainfall picture rainfall intensity calculation method based on raindrop extraction-rainfall intensity calculation, comprising: obtaining rainfall video and decomposing it into rainfall images according to frames; constructing a target function of image raindrop extraction, and extracting raindrop images by using an ADMM algorithm; modifying a CNN model and training the same; and predicting rainfall intensity. The application can prevent random noise in the image from being extracted together, so that the extracted raindrop image is more accurate. The image raindrop extraction algorithm proposed in the application has higher calculation efficiency, can efficiently extract raindrops in the image, and can realize real-time calculation of the algorithm. The application can effectively improve the generalization ability of the CNN model, especially in complex scenes such as night with insufficient light. Rainfall events with large rainfall usually occur at night, and the prediction ability in the night scene has very important significance for the prevention and control of urban waterlogging.

Description

Technical Field

[0001] This invention belongs to the field of real-time rainwater measurement in municipal engineering, and specifically relates to a method for calculating rainfall intensity from rainfall images based on raindrop extraction and rainfall intensity calculation. Background Technology

[0002] Heavy rain exhibits significant spatial heterogeneity, resulting in substantial differences in the severity of damage across different urban areas. Accurately obtaining real-time rainfall levels in various regions is fundamental for monitoring, controlling, and responding to urban flooding. Currently, convolutional neural networks (CNNs) are used to calculate rainfall intensity from rain images, which performs well in both synthetic rain images and real rainfall during the day. However, directly using CNN models to calculate rainfall intensity from rain images still has some limitations, namely (1) requiring a large amount of data for training, making it difficult to collect a sufficient amount of training data; and (2) the model's generalization ability is relatively low, especially for nighttime scenarios.

[0003] Furthermore, existing image raindrop extraction algorithms also have limitations. In the field of computer image processing, raindrops in images are generally considered noise that affects image quality. Therefore, research on rainfall images focuses on removing raindrops to obtain a clearer background, i.e., image deraining algorithms. These algorithms can remove raindrops from videos and still images. Deraining algorithms utilize the physical properties of raindrops in images (such as color and texture) and the motion differences between adjacent frames in a rainfall video to remove raindrops from the image. However, because deraining algorithms are designed to remove raindrops to obtain a clearer background image, they often remove random noise in the image (such as the swaying of branches and leaves, changes in brightness, etc.). Therefore, the rain layer extracted directly using deraining algorithms usually contains a lot of noise. As a result, existing technical solutions cannot accurately quantify rainy images, especially in scenes such as nighttime. Summary of the Invention

[0004] To address the above problems, the present invention provides the following technical solution:

[0005] This invention provides a method for calculating rainfall intensity from rainfall images based on raindrop extraction and rainfall intensity calculation, wherein the quantization method includes:

[0006] (1) Acquire rainfall video and decompose it into rainfall images frame by frame;

[0007] (2) Construct an objective function for raindrop extraction from images and use an optimization algorithm to extract raindrop images;

[0008] (3) Modify the CNN model and train it;

[0009] (4) Predicting rainfall intensity

[0010] Preferably, the optimization algorithm can be ADMM, FISTA, etc. The specific optimization algorithm is not improved in this solution. The calculation is performed using existing technology, and this solution will not provide further detailed explanation.

[0011] Furthermore, the specific steps for constructing the objective function for image raindrop extraction in step (2) are as follows:

[0012] (21) Perform grayscale processing on the rainfall image;

[0013] (22) Use a low-rank matrix to decompose the background of the rainfall image;

[0014] (23) Use Markov random fields (MRF) to represent the position of raindrops;

[0015] (24) Calculate the noise layer in the rainfall image;

[0016] (25) Combine steps (22) to (23) to obtain the objective function of the raindrop extraction algorithm.

[0017] Furthermore, the specific process of step (22) is as follows:

[0018] For a continuous rainfall video, the background is considered constant over time, and each frame of the rainfall image X is viewed as a two-dimensional matrix. Each frame of the rainfall image X (X∈R) is then processed. h×w Expand into a one-dimensional vector x (x∈R) hw ), and arrange them in time into a new two-dimensional matrix X′ (X′∈R hw×t From the low-rank matrix decomposition, we can obtain:

[0019] X′≈UΣV (1)

[0020] Where ∑ is a diagonal matrix, with diagonal elements being the first r singular values ​​of X, the column vectors of U being the left singular vectors of X, and the column vectors of V being the right singular vectors of X; U∈R hw×r , Σ∈R r×r V∈R n×r r <rank(X′);

[0021] The background layer B of the rainfall image is represented as:

[0022] B≈Fold(X′)≈Fold(UΣV) (2)

[0023] Expressed using the F-norm:

[0024] min||B-Fold(UΣV)|| F (3);

[0025] Where B∈R h×w×t ;

[0026] Fold() means to restore each column of the matrix to an image of size h×w;

[0027] The Frobenius norm (F-norm) is used to represent the distance of a matrix to the origin.

[0028] Furthermore, the specific process of step (23) is as follows:

[0029] Raindrops exhibit sparsity in the image, while the background and noise are continuous; this is a significant characteristic of raindrops in the image. Markov random fields (MRF) are used to represent the positions H of the raindrops. R , is represented as:

[0030]

[0031] Among them, H R ∈R h×w×t ;

[0032] The position H of raindrops is described using l1 regularization. R The sparsity of is represented as

[0033] min||H R ||1 (5).

[0034] Furthermore, the specific process of step (24) is as follows:

[0035] The grayscale processed rainfall image is viewed as a two-dimensional matrix X∈R h×w , represented as

[0036] X = B + R + N (6)

[0037] Changes in light and the effects of wind can cause random disturbances in the image, which are considered as noise layers. The noise layer N of the rainfall image is obtained from equation (6):

[0038]

[0039] Random disturbances are usually small, resulting in relatively low noise. Therefore, it is necessary to control the noise level to a minimum, which can be represented using the F-norm:

[0040]

[0041] Where B represents the background layer of the rainfall image, B∈R h×w ;

[0042] R represents the rain layer in the rainfall image, R∈R h×w ;

[0043] N represents the noise layer of the rainfall image, N∈R h×w ;

[0044] h and w are the height and width of the rainfall image, respectively;

[0045] This represents the Hadamard product of matrices, which is the element-wise product of two matrices. Further, the objective function of the raindrop extraction algorithm in step (25) is:

[0046]

[0047] Among them, λ1, λ2, and λ3 are adjustable parameters; the values ​​of λ1, λ2, and λ3 affect the raindrop extraction effect, but different parameter values ​​can be regarded as systematic errors. Since deep learning algorithms have strong fitting capabilities, they can automatically learn the systematic error and are robust to different parameter values; the recommended parameter values ​​for λ1, λ2, and λ3 are 1, 0.1, and 1, respectively.

[0048] Furthermore, the specific process of step (3) is as follows:

[0049] Add a linear regression layer to the original CNN model;

[0050] Model training was performed using the ImageNet open-source dataset.

[0051] The CNN model was further trained using raindrop image data to determine the parameter values ​​of the final CNN model.

[0052] Furthermore, predict rainfall intensity The expression is as follows:

[0053]

[0054] Where W represents the parameters of the linear regression layer, which are automatically determined during the training of the CNN model;

[0055] A represents the output of the penultimate layer of the CNN model;

[0056] T is the matrix transpose operator.

[0057] b is the bias term.

[0058] The present invention has the following beneficial effects:

[0059] (1) The present invention extracts raindrop images from rainfall images and then inputs the extracted raindrop images into a CNN model for calculation, which can effectively avoid the influence of background changes on the model and improve the generalization ability of the model.

[0060] (2) This invention divides the rainfall image into three parts: background layer, rain layer, and noise layer. A new raindrop extraction algorithm is designed for the image characteristics of each part, which can effectively extract raindrops. The raindrop image extracted by the raindrop extraction algorithm is then further processed by a convolutional neural network to calculate the rainfall intensity, i.e., the two-stage algorithm of raindrop extraction and rainfall intensity recognition. The two-stage algorithm can effectively improve the generalization ability of the model and greatly improve the prediction effect of the model in nighttime and other scenarios.

[0061] (3) Compared with traditional computer image rain removal algorithms, the image raindrop extraction algorithm proposed in this invention integrates the features of the background layer, raindrop layer and noise layer, which can prevent random noise in the image from being extracted together, making the extracted raindrop image more accurate.

[0062] (4) Compared with traditional computer image deraining algorithms, the image raindrop extraction algorithm proposed in this invention has higher computational efficiency, can efficiently extract raindrops in images, and can realize real-time calculation of the algorithm.

[0063] (5) The two-stage algorithm proposed in this invention can effectively improve the generalization ability of CNN models, especially in complex scenarios with insufficient light, such as at night. Since heavy rainfall events usually occur at night, improving the prediction ability of CNN models in night scenarios is of great significance for the prevention and control of urban flooding. Attached Figure Description

[0064] Figure 1 This is a flowchart of the quantization method of the present invention.

[0065] Figure 2 This is an overall framework diagram of the present invention.

[0066] Figure 3 This is the objective function for constructing image raindrop extraction in this invention.

[0067] Figure 4 This is a flowchart of the construction process of the CNN model of the present invention.

[0068] Figure 5 This is a schematic diagram of the calculation of instantaneous rainfall intensity using the linear interpolation method in the experimental example.

[0069] Figure 6 This is a rainfall map of the daytime scene in Comparison 1.

[0070] Figure 7This is a raindrop image extracted using the present invention in a daytime scene in Comparative Example 1.

[0071] Figure 8 This is a rainfall map of a nighttime scene in Comparison 1.

[0072] Figure 9 This is a raindrop image extracted using the present invention in a nighttime scene in Comparative Example 1.

[0073] Figure 10 This is a verification graph for using a CNN model with random sampling in Comparison 1.

[0074] Figure 11 This is a verification diagram of the model in this invention used in Comparative Example 1, which is verified using random sampling.

[0075] Figure 12 This is a verification of the independent rainfall event used in Example 2, which directly uses a CNN model.

[0076] Figure 13 This is a verification graph of Comparative Example 2, which uses the model in this invention to verify independent rainfall events. Detailed Implementation

[0077] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be noted that the embodiments are only specific illustrations of the invention and should not be regarded as limitations on the invention. The purpose of the embodiments is to enable those skilled in the art to better understand and reproduce the technical solution of the present invention. The scope of protection of the present invention should still be determined by the scope defined in the claims.

[0078] like Figure 1-2 As shown, this invention provides a method for calculating rainfall intensity from rainfall images based on raindrop extraction and rainfall intensity calculation. The quantization method includes:

[0079] S1, acquire the rainfall video and decompose it into rainfall images frame by frame;

[0080] S2, construct the objective function for raindrop extraction from the image, and use the optimization algorithm to extract the raindrop image;

[0081] like Figure 3 As shown, the specific steps for constructing the objective function for raindrop extraction from an image are as follows:

[0082] S21, perform grayscale processing on the rainfall image;

[0083] S22, using a low-rank matrix to decompose the background of a rainfall image;

[0084] For a continuous rainfall video, the background is considered constant over time, and each frame of the rainfall image X is viewed as a two-dimensional matrix. Each frame of the rainfall image X (X∈R) is then processed.h×w Expand into a one-dimensional vector x (x∈R) hw ), and arrange them in time into a new two-dimensional matrix X′ (X′∈R hw×t From the low-rank matrix decomposition, we can obtain:

[0085] X′≈UΣV (1)

[0086] Where ∑ is a diagonal matrix, with diagonal elements being the first r singular values ​​of X, the column vectors of U being the left singular vectors of X, and the column vectors of V being the right singular vectors of X; U∈R hw×r , Σ∈R r×r V∈R n×r ,

[0087] r <rank(X′);

[0088] The background layer B of the rainfall image is represented as:

[0089] B≈Fold(X′)≈Fold(UΣV) (2)

[0090] Expressed using the F-norm:

[0091] min||B-Fold(UΣV)|| F (3);

[0092] Where B∈R h×w×t ;

[0093] Fold() means to restore each column of the matrix to an image of size h×w;

[0094] The Frobenius norm (F-norm) is used to represent the distance of a matrix from the origin.

[0095] S23, using Markov random fields (MRF) to represent the position of raindrops;

[0096] Raindrops exhibit sparsity in the image, while the background and noise are continuous; this is a significant characteristic of raindrops in the image. Markov random fields (MRF) are used to represent the positions H of the raindrops. R , is represented as:

[0097]

[0098] Among them, H R ∈R h×w×t ;

[0099] The position H of raindrops is described using l1 regularization. RThe sparsity of is expressed as:

[0100] min||H R ||1 (5).

[0101] S24, Calculate the noise layer in the rainfall image;

[0102] The grayscale processed rainfall image is viewed as a two-dimensional matrix X∈R h×w , is represented as:

[0103] X = B + R + N (6)

[0104] Changes in light and the effects of wind can cause random disturbances in the image, which are considered as noise layers. The noise layer N of the rainfall image is obtained from equation (6):

[0105]

[0106] Where B represents the background layer of the rainfall image, B∈R h×w ;

[0107] R represents the rain layer in the rainfall image, R∈R h×w ;

[0108] N represents the noise layer of the rainfall image, N∈R h×w ;

[0109] h and w are the height and width of the rainfall image, respectively;

[0110] The Hadamard product represents the element-wise product of two matrices. Random perturbations are usually small, resulting in relatively small noise. Therefore, it is necessary to control the amount of noise to be small. Equation (7) is expressed using the F-norm as follows:

[0111]

[0112] S25, combining steps S22 to S24, we obtain the objective function of the raindrop extraction algorithm;

[0113] The objective function is:

[0114]

[0115] Among them, λ1, λ2, and λ3 are adjustable parameters; the values ​​of λ1, λ2, and λ3 affect the raindrop extraction effect, but different parameter values ​​can be regarded as systematic errors. Since deep learning algorithms have strong fitting capabilities, they can automatically learn the systematic error and are robust to different parameter values; the recommended parameter values ​​for λ1, λ2, and λ3 are 1, 0.1, and 1, respectively.

[0116] S3, Modify the CNN model and train it;

[0117] like Figure 4 As shown in S31, a linear regression layer (output layer) is added to the last layer of the original CNN model to directly output the specific rainfall value.

[0118] S32 uses the ImageNet open-source dataset for model training;

[0119] Because the CNN model to be trained has a large number of parameters, training the model from scratch requires a large number of rainfall images. To address this issue, the ImageNet open-source dataset (consisting of 1.28 million images in 1000 classes) was used to pre-train the CNN model.

[0120] S33, use raindrop image data to further train the CNN model to determine the parameter values ​​of the final CNN model. The parameter values ​​of the final CNN model include the specific parameter values ​​of all parameters in the original CNN model. The number of layers, convolution kernel size and other parameters of the model are determined in the CNN model building stage. These are the specific values ​​of each parameter obtained by training the model.

[0121] S4, Predicted Rainfall Intensity The expression is as follows:

[0122]

[0123] Where W represents the parameters of the linear regression layer, which are automatically determined during the training of the CNN model;

[0124] A represents the output of the penultimate layer of the CNN model;

[0125] T is the matrix transpose operator.

[0126] b is the bias term.

[0127] Example

[0128] Six nighttime rainfall scenes were captured using a fixed camera over a specific time period. A tipping bucket rain gauge was used near the fixed camera to record the rainfall intensity, with a time accuracy of 1 minute and a rainfall intensity accuracy of 0.1 mm / min (6 mm / h). The relevant rainfall data recorded by the rain gauge are shown in Table 1.

[0129] Table 1

[0130]

[0131] Rain gauges record rainfall at one-minute intervals, recording the cumulative rainfall over the previous minute. However, the exposure time of a single frame in a photograph or video is extremely short (approximately 1 / 200 of a second), recording the instantaneous image characteristics of rainfall. Therefore, the cumulative rainfall recorded by a rain gauge does not match the instantaneous rainfall intensity in time. Since the change in rainfall intensity within one minute is relatively small, it can be assumed that the rainfall intensity changes linearly within one minute, and a linear interpolation method can be used to resolve this. For example... Figure 5 As shown, since the change in rainfall intensity is relatively small within a short time interval Δt, it is assumed that the rainfall intensity changes linearly within this short time interval Δt. It is also assumed that the midpoint of each rainfall interval is precisely the cumulative rainfall intensity value for that period. At time t... L t R The rainfall intensity values ​​are respectively I L I R Then at time t L t R Instantaneous rainfall intensity I at any time t within the time interval t for:

[0132]

[0133] Where △t represents 1 minute.

[0134] To further verify the effectiveness of the present invention, the following methods were used for verification.

[0135] I. Comparison of the effects of using the method of this invention and directly using a CNN model to calculate rainfall intensity.

[0136] The performance evaluation metric for the model is the mean absolute percentage error (MAPE), expressed as follows:

[0137]

[0138] Where n represents the total number of data points, Y i Let i be the target value for the i-th sample. This represents the predicted value for the i-th sample. A lower MAPE indicates a better model performance.

[0139] Using a random sampling validation method, 4820 images (80%) were randomly selected for training, and 1205 images (20%) were used for validation. The two-stage algorithm proposed in this invention significantly outperforms the CNN model (MAPE value 19.73%). The proposed two-stage algorithm first extracts raindrop images from rainfall images, and then inputs the extracted raindrop images into the CNN model for calculation. Compared to directly using the CNN model to calculate rainfall intensity, this effectively avoids the influence of background changes on the model and improves the model's generalization ability.

[0140] Figure 6-9 This is an example of the effect of the image raindrop extraction algorithm of the present invention in different scenarios. It can be seen that the raindrop extraction algorithm can extract raindrops well from rainfall images in both daytime and nighttime scenarios.

[0141] Figure 10-11 The results are from verification using random sampling. Figure 10 Using the results directly from the CNN model, the average MAPE of the validation results was 28.35%, while Figure 11 The average MAPE of the validation results using the method of this invention is 19.73%. The validation results of the method of this invention are significantly better than those of directly using a CNN model. This is mainly because the imaging of raindrops at night relies primarily on the reflection of other light sources. At the same time, the light at night is relatively cluttered, making it difficult to capture raindrops clearly. Therefore, directly using the original rain image would result in poor performance. However, the extracted raindrop image can better avoid the influence of light, thus achieving better results.

[0142] II. Verification Methods for Independent Rainfall Events

[0143] Rainfall event 5 was used for validation, and the rest were used for training. The two-stage algorithm proposed in this invention also significantly outperforms the direct use of the CNN model (MAPE value 20.88%).

[0144] Figure 12-13 The validation results were obtained using independent rainfall events. Figure 12 Using the results directly from the CNN model, the average MAPE of the validation results was 26.24%, while Figure 13 The average MAPE of the validation results using the method of this invention is 20.88%. The results of validation using independent rainfall events are similar to those of random validation, and the validation results of the method of this invention are also significantly better than those of directly using a CNN model.

[0145] In summary, the two-stage rain image quantization method proposed in this invention, which involves raindrop extraction and rain intensity calculation, has significant advantages over directly using a CNN model. It can effectively improve the generalization and prediction performance of the CNN model, especially in nighttime scenes.

[0146] It should be noted that any technical features not described in detail in this invention can be implemented using any existing technology.

[0147] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

Claims

1. A method for calculating rainfall intensity from rainfall images based on raindrop extraction and rainfall intensity calculation, characterized in that, The calculation method includes: (1) Acquire rainfall video and decompose it frame by frame into rainfall images; (2) Construct the objective function for raindrop extraction from the image and solve it using an optimization algorithm to extract the raindrop image; (3) Modify the CNN model and train it; (4) Predicting rainfall intensity ; The specific steps for constructing the objective function for image raindrop extraction in step (2) are as follows: (21) Perform grayscale processing on the rainfall image; (22) Use a low-rank matrix to decompose the background of the rainfall image; (23) Use Markov random fields to represent the position of raindrops; (24) Calculate the noise layer in the rainfall image; (25) Combining steps (22) to (23), we obtain the objective function of the raindrop extraction algorithm; The specific process of step (22) is as follows: For a continuous rainfall video, the background is considered constant over time, and each frame of the rainfall image X is viewed as a two-dimensional matrix. X Expand into a one-dimensional vector x Arranged into a new two-dimensional matrix according to time. ,in, , , From low-rank matrix decomposition, we can obtain: in, , , , ∑ It is a diagonal matrix with diagonal elements. X The former r A singular value, U The column vector is X left singular vector, V The column vector is X The right singular vector; , h , w These represent the height and width of the rainfall image, respectively. The background layer of the rainfall image B Represented as: Expressed using the F-norm: in, ; Fold ( ) means restoring each column of the matrix to its original state. Image of size; The objective function of the raindrop extraction algorithm in step (25) is: in, , , These are adjustable parameters. Indicates the location of the raindrops.

2. The method for calculating rainfall intensity from rainfall images based on raindrop extraction and rainfall intensity calculation according to claim 1, characterized in that, The specific process of step (23) is as follows: Raindrops are sparse in the image, while the background and noise are continuous; this is a significant characteristic of raindrops in the image. Markov random fields are used to represent the positions of the raindrops. , is represented as: in, ; use l 1 Regular expressions describe the position of raindrops The sparsity of is expressed as: 。 3. The method for calculating rainfall intensity from rainfall images based on raindrop extraction and rainfall intensity calculation according to claim 2, characterized in that, The specific process of step (24) is as follows: The grayscale processed rainfall image X is represented as: Changes in light and the effects of wind can cause random disturbances in the image, which can be considered as a noise layer. The noise layer of the rainfall image can be obtained from equation (6). N： Using the F-norm, it can be represented as: in, B This represents the background layer of a rainfall image. R’ The rain layer in the rainfall image. N Represents the noise layer in a rainfall image. This represents the matrix dot product.

4. The method for calculating rainfall intensity from rainfall images based on raindrop extraction and rainfall intensity calculation according to claim 1, characterized in that, The specific process of step (3) is as follows: Add a linear regression layer to the original CNN model; Model training was performed using the ImageNet open-source dataset. The CNN model was further trained using raindrop image data to determine the parameter values ​​of the final CNN model.

5. The method for calculating rainfall intensity from rainfall images based on raindrop extraction and rainfall intensity calculation according to claim 4, characterized in that, Predicted rainfall intensity The expression is: in, W This represents the parameters of the linear regression layer; This represents the output of the penultimate layer of the CNN model; T is the matrix transpose operator. b It is a bias term.

Images