A Video Rainfall Inversion Method Based on Transfer Learning

Abstract

A rainfall inversion algorithm based on transfer learning includes the following steps: dataset preparation, dataset classification, dataset preprocessing, and construction of a multivariate Gaussian mixture model featuring grayscale changes and saturation, further filtering candidate raindrops within the ROI region. The raindrops obtained after the above filtering steps are used to invert rainfall amounts using a Gamma distribution model based on the definition of raindrop spectra. This invention, due to the adoption of the above technical solution, has the following advantages: it can effectively invert rainfall amounts using videos captured by traffic cameras without relying on other professional rainfall monitoring equipment; it has stronger anti-interference performance in complex environments; it reduces the probability of misjudging rainfall in sunny environments; and it has greater practical significance and application value.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and computer graphics, specifically to image imaging principles, image feature extraction, and fusion, and particularly to a video rainfall inversion method based on transfer learning. Background Technology

[0002] Meteorological observation information has a significant impact on social production and life. Meteorological monitoring is an important component of smart city construction, the foundation for weather early warning and forecasting, climate prediction and forecasting, meteorological services and scientific research, and a driving force for the development of meteorological science.

[0003] Currently, the main operational instruments for precipitation measurement in China include rain gauges, tipping bucket rain gauges, and weighing rain gauges. These instruments can only measure rainfall amount and intensity, and the type and nature of precipitation still require manual judgment, making automatic identification impossible. They also lack the ability to measure the microscopic characteristics of precipitation. Precipitation measurement technologies based on vibration, optical, and photographic methods suffer from problems such as underestimating the number of small raindrops, environmental noise affecting particle count accuracy, inability to measure raindrop shape and velocity, and poor accuracy in measuring solid precipitation particles. On the other hand, existing raindrop detection technologies based on CCD imaging mainly use high-speed imaging to obtain the shape of precipitation particles, but cannot measure velocity, and suffer from depth-of-field effects, resulting in a low probability of collecting larger particles.

[0004] Currently, most meteorological information products based on video image recognition on the market require additional camera units and markers to assist them, resulting in poor applicability and difficulty in applying them to existing social surveillance video networks.

[0005] Therefore, a deep learning method is needed to reduce false detections in clear weather and obtain high spatiotemporal resolution rainfall data to improve scene utilization. Summary of the Invention

[0006] To overcome the problems of high cost, manpower requirements, terrain influence, and false detections in clear weather associated with the aforementioned monitoring instruments, this invention designs a video rainfall inversion method based on transfer learning. This method enables rainfall monitoring without requiring specialized rainfall detection equipment. Specifically, this invention uses a transfer learning model to filter and differentiate between clear and rainy video images, employs semantic segmentation and temporal filtering-based background subtraction to select suitable Regions of Interest (ROIs), then uses a Gaussian mixture model to filter raindrops; finally, it defines and inverts rainfall based on raindrop spectra.

[0007] The technical solution adopted by this invention to solve its technical problem is:

[0008] The video rainfall inversion method based on transfer learning mainly consists of five parts: dataset preparation, dataset classification, data preprocessing, raindrop selection, and rainfall inversion. Dataset preparation is the first step before subsequent algorithm design and testing, and is crucial to the entire algorithm process. Dataset classification uses a transfer learning rainfall classification model to distinguish between sunny and rainy conditions. Data preprocessing uses semantic segmentation and temporal filtering-based background subtraction to select appropriate Regions of Interest (ROIs) and remove interfering environmental elements to improve algorithm accuracy. Then, a multivariate Gaussian mixture model is used to select raindrops. Finally, the inverted rainfall is defined based on the raindrop spectrum.

[0009] A video rainfall inversion method based on transfer learning includes the following steps:

[0010] Step 1: Dataset Preparation: A large amount of video data from different time periods was collected from video surveillance stations along highways and in urban areas. The data was matched with rainfall information from meteorological monitoring based on the time and location of the video capture. Damaged and problematic video images were removed to construct a video image dataset.

[0011] Step 2: Dataset Classification: Existing raindrop detection algorithms suffer from over-separation of rain stripes and are not suitable for video images in clear weather, significantly interfering with subsequent rainfall estimation. Therefore, we first classify the dataset into two categories, rainy and clear, through transfer learning of a rainfall classification model, thereby effectively improving the accuracy of rainfall estimation.

[0012] Step 3, Data Preprocessing: For the rainfall image dataset obtained in Step 2, semantic segmentation and time-based background subtraction methods are used to select appropriate ROI regions and remove the interfering environment to improve the accuracy of the algorithm.

[0013] Step 4, Raindrop Screening: Research found that the saturation value of real raindrops in the HSV color model is relatively low. Based on this feature, a multivariate Gaussian mixture model with grayscale changes and saturation as features was constructed to further screen candidate raindrops in the ROI region.

[0014] Step 5, Rainfall Inversion: Based on the raindrops obtained after screening in Step 4, the rainfall is inverted using the Gamma distribution model according to the definition of raindrop spectrum. Attached Figure Description

[0015] Figure 1 This is the fully connected layer structure based on Extreme Learning Machine (ELM) in this scheme.

[0016] Figure 2 This solution includes the overall transfer learning network architecture comprising Inception v3 and a fully connected layer. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further explained below in conjunction with specific embodiments and accompanying drawings.

[0018] Step 1: Dataset Preparation. Image data from video surveillance stations along highways and in urban areas, as well as meteorological data from weather stations, are collected at regular intervals. Based on the temporal and spatial information from the video surveillance and weather stations, data cleaning is performed using a computer to obtain video image data from 8:00 AM to 5:00 PM. The filtered data is then manually reviewed, and data from failed acquisitions or those heavily obscured by foreign objects are deleted. Finally, video images of the same scene appearing repeatedly are randomly deleted to ensure a balanced distribution of data across different scenes and rainfall amounts. This completes the dataset preparation.

[0019] Step 2: Dataset Classification: Since the raindrop detection algorithm is not suitable for video images of sunny weather, the dataset is classified into two categories—rainy and sunny—through transfer learning of the rainfall classification model, effectively improving the accuracy of rainfall estimation. First, the parameters of the Inception v3 model trained on ImageNet are transferred to the rain / sunshine classification task to extract video image features, obtaining 2048-dimensional feature vectors. Then, the feature maps are input into a fully connected network to train the network parameters to perform binary classification (sunny images and rainy images) on these feature vectors. The fully connected network is implemented using a super-limited learning machine, and the network structure is as follows: Figure 1 As shown. The overall network architecture for transfer learning, including Inception v3 and fully connected layers, is as follows. Figure 2 As shown, the left side displays the convolutional and pooling parts of the Inception v3 deep convolutional model trained on the ImageNet dataset, which has the ability to extract image features. Its structure and parameters are kept unchanged to extract the image features obtained in step 1. The fully connected network is then fine-tuned using the image features extracted in step 1 from the Inception v3 model for rain / sun classification.

[0020] Step 3: Data Preprocessing: For the rainfall image dataset obtained in Step 2, the HRFormer network is used for semantic segmentation to extract regions such as trees and buildings as candidate regions for rainfall calculation, reducing interference from moving objects like vehicles. This further reduces the influence of non-raindrop noise. Non-raindrop moving regions are then detected through shape and other features. The difference between these two regions is used to obtain candidate regions of interest (ROIs) for raindrops, avoiding interference from excessively large gray skies and irrelevant areas in raindrop identification.

[0021] Step 4, Raindrop Filtering: Research revealed that the changes in the three primary color pixels of raindrops are approximately equal, while the changes in moving pixels of non-raindrops show significant differences. Utilizing these pixel difference characteristics, a clustering method was employed to detect raindrops. The mathematical description of the raindrop pixel change characteristics is as follows:

[0022] Δr≈Δg≈Δb≈c

[0023] In the formula, Δr, Δg, and Δb represent the pixel variations of the three primary colors of raindrops in two adjacent frames, and c is a constant. The formula for calculating saturation S in the HSV color model is as follows:

[0024]

[0025] This leads to the conclusion that the actual raindrop saturation value is relatively low. Based on this characteristic, a multivariate Gaussian mixture model is constructed with grayscale variation and saturation as features to further filter candidate raindrops within the ROI region.

[0026] Step 5, Rainfall Inversion: Calculate the raindrop diameter d using the raindrop length l on the image plane, the camera focal plane position z, the focal length f, and the exposure time T.

[10] :

[0027]

[0028] Based on the meteorological definition of raindrop spectrum, the number of raindrops with diameter d falling within the interval [d-0.5, d+0.5] is counted to obtain the raindrop spectrum N(d).

[0029] Using the Gamma distribution model, its mathematical description is as follows:

[0030] N(d)=N0d μ e (-ωd)

[0031] In the formula, N0 is the raindrop spectral intercept parameter, and N(d) is the number of raindrops per unit volume whose particle size is within the range of d as the center and the interval is the unit diameter scale.

[0032]

[0033] In the formula, μ is the shape parameter of the raindrop spectrum, ω is the parameter affecting the slope of the raindrop spectrum, and Γ(·) is the gamma function. Finally, the rainfall is obtained by inversion using this formula.

[0034] The present invention has the following advantages due to the adoption of the above technical solution: it can effectively use the video captured by traffic cameras to obtain rainfall without relying on other professional rainfall monitoring equipment, has stronger anti-interference performance in complex environments, reduces the probability of misjudging rain in sunny environments, and has more practical significance and application value.

Claims

1. A video rainfall inversion method based on transfer learning, the method comprising the following steps: Step 1: Dataset Preparation: A large amount of video data at different times was collected from video surveillance stations on highways and in urban areas. The data was matched with rainfall information from meteorological monitoring based on the time and location of the video collection. Damaged and problematic video images were deleted to construct a video image dataset. Step 2, Dataset Classification: By transferring learning to the rainfall classification model, the dataset is classified into two categories: rainfall and sunny, thereby effectively improving the accuracy of rainfall estimation; Step 3, Data Preprocessing: For the rainfall image dataset obtained in Step 2, semantic segmentation and time-based background subtraction methods are used to select appropriate ROI regions and remove the environment that causes interference to improve the accuracy of the algorithm. Step 4, Raindrop Filtering: The saturation value of real raindrops in the HSV color model is relatively low. Based on this feature, a multivariate Gaussian mixture model with grayscale changes and saturation as features is constructed to further filter candidate raindrops in the ROI area. Step 5, Rainfall Inversion: Based on the raindrops obtained after screening in Step 4, the rainfall is inverted using the Gamma distribution model according to the definition of raindrop spectrum.

2. The video rainfall inversion method based on transfer learning as described in claim 1, characterized in that, In step 1, the time and location of video acquisition are matched with rainfall information from meteorological monitoring. The filtered data are manually reviewed, and data that failed to be acquired or was obscured by large areas of foreign objects are deleted. Finally, video images of the same scene that appear repeatedly are randomly deleted to ensure a balanced distribution of data with different rainfall amounts in different scenes.

3. The video rainfall inversion method based on transfer learning as described in claim 1 or 2, characterized in that, In step 2, the dataset is classified into two categories, rainy and sunny, by transferring the learning of the rainfall classification model, which effectively improves the accuracy of rainfall estimation. First, the parameters of the Inception v3 model trained on ImageNet are transferred to the rainy / sunny classification task to extract video image features and obtain 2048-dimensional feature vectors. Then, the feature maps are input into the fully connected network of the overlimit learning machine to train the network parameters to perform binary classification on these feature vectors, resulting in sunny and rainy images.

4. The video rainfall inversion method based on transfer learning as described in claim 1, characterized in that, In step 3, the HRFormer network is used to perform semantic segmentation on the rainy image obtained in step 1, and then non-raindrop movement regions are obtained through shape feature detection. The difference between the two regions is used to obtain the candidate region ROI of the raindrop, thus avoiding interference from excessive gray sky and irrelevant regions on raindrop recognition.

5. The video rainfall inversion method based on transfer learning as described in claim 1, characterized in that, In step 4, the study found that the changes in the three primary color pixels of raindrops were approximately equal, while the changes in moving pixels of non-raindrops showed significant differences. Simultaneously, utilizing the pixel difference characteristics, a clustering method was used to detect raindrops. The mathematical description of the raindrop pixel change characteristics is as follows: Δr≈Δg≈Δb≈c In the formula, Δr, Δg, and Δb represent the pixel variations of the three primary colors of raindrops in two adjacent frames, c is a constant, and the formula for calculating saturation S in the HSV color model is as follows: It can be concluded that the actual raindrop saturation value is relatively low. Based on this characteristic, a multivariate Gaussian mixture model with grayscale change and saturation as features is constructed to further screen candidate raindrops in the ROI region.

6. The video rainfall inversion method based on transfer learning as described in claim 1, characterized in that, In step 5, the raindrop diameter d is calculated using the raindrop length l on the image plane, the camera focal plane position z, the focal length f, and the exposure time T. Based on the meteorological definition of raindrop spectrum, the number of raindrops with diameter d falling within the interval [d-0.5, d+0.5] is counted to obtain the raindrop spectrum N(d); Using the Gamma distribution model, its mathematical description is as follows: N(d)=N0d μ e (-ωd) In the formula, N0 is the raindrop spectral intercept parameter, and N(d) is the number of raindrops per unit volume whose particle size is within the range of d as the center and the interval is the unit diameter scale. In the formula, μ is the shape parameter of the raindrop spectrum, ω is the parameter that affects the slope of the raindrop spectrum, Γ(·) is the gamma function, and finally the rainfall is obtained by inversion according to this formula.

Images