Multi-feature rainfall intensity estimation method based on audio of monitoring camera

Abstract

The invention relates to a multi-feature rainfall intensity estimation method based on monitoring camera audio, which adopts monitoring audio multi-domain feature signals to describe rainfall information, and comprises four parts of audio data processing, multi-feature combination, model training and rainfall intensity estimation. In the multi-feature combination part, rainfall audio features are obtained from a time domain, a frequency domain and a cepstrum domain according to attributes of different audio features and fused, and an audio multi-feature combination vector is obtained; in the rainfall intensity estimation part, training is carried out in combination with a mainstream deep network model, a multi-feature combination vector is input into the network model to obtain a monitoring audio multi-feature rainfall intensity estimation model, then the model is used in practical application, the method can be deployed at different positions such as a cloud, an edge and an end, rainfall monitoring is realized by using an existing monitoring camera, and the accuracy of rainfall intensity estimation is improved. The method and technical support are provided for regional low-cost high-temporal-spatial-resolution rainfall, and the method has wide market application prospects.

Description

Technical Field

[0001] The invention belongs to the field of meteorology, relates to computer auditory recognition technology, and specifically relates to a multi-feature rainfall intensity estimation method based on surveillance camera audio. Background Art

[0002] High-temporal and spatial resolution rainfall information is an indispensable and important part of meteorological data, and is the direct supporting data for disaster warning, emergency response, waterlogging assessment, precision agriculture, etc. With the combined influence of global warming and the El Niño phenomenon, extreme rainfall events have become one of the most important meteorological factors threatening human social security, which has put forward more urgent requirements for the acquisition of high-temporal and spatial resolution rainfall data products.

[0003] The high spatiotemporal heterogeneity of rainfall events makes it challenging for traditional observation methods such as ground observation stations (rain gauges) and aerial observations (remote sensing, weather radar) to produce high-resolution rainfall data products. Among them, ground rain gauges have insufficient spatial resolution, with a global distribution of less than 1%, and aerial observations are limited by insufficient temporal resolution and complex precision verification. In recent years, new, efficient, economical, and high spatiotemporal resolution rainfall observation methods based on the idea of ​​opportunity perception (some studies also call it "crowdsourcing" or "citizen science") have gradually become the research focus of rainfall observation. Sensing methods such as microwave link rain gauges, vehicle rain gauges, smartphone rain gauges, and surveillance camera rain gauges have been proposed. Among them, surveillance cameras can not only continuously record audio-visual information of rainfall, but also provide a hardware foundation for high spatiotemporal resolution rainfall observations due to their dense distribution. They have become a very promising rainfall observation method. Compared with surveillance videos, audio data perceives rainfall intensity by capturing the sound signals generated by rainfall. It has the advantages of all-weather, unaffected by lighting conditions, and small data volume. It can provide rainfall information in the auditory dimension when visual signals are blocked, thus becoming a good supplementary data source.

[0004] However, when using surveillance audio or other acoustic sensors to monitor rainfall, it will be disturbed by other noises in the mixed audio besides the rain sound. The existing method model relies on a single acoustic feature to directly establish a rainfall intensity estimation model. However, the noise mixed in the rain sound is diverse, and it is often difficult to accurately characterize the rainfall intensity using a single audio feature, which greatly limits the accuracy of the model in estimating rainfall intensity. Summary of the invention

[0005] Purpose of the invention: In order to solve the problem of insufficient accuracy of the existing rainfall intensity estimation model based on a single audio feature, a multi-feature rainfall intensity estimation method based on surveillance camera audio is provided.

[0006] Technical solution: To achieve the above purpose, the present invention provides a multi-feature rainfall intensity estimation method based on surveillance camera audio, obtains highly robust monitoring rain sound description features, and realizes high-precision estimation of rainfall intensity based on surveillance audio, comprising the following steps:

[0007] S1: Extract audio data from rainfall event monitoring videos collected in different scenarios, annotate the rainfall true value according to the rainfall intensity value obtained by the rain gauge at the weather station, and the duration of the audio data is consistent with the time resolution of the rainfall data collected by the rain gauge at the weather station, which is 1 minute. All annotated 1-minute audio data are cropped into 4-second audio to construct a rainfall monitoring audio dataset containing rainfall intensity values;

[0008] S2: Starting from the analysis of audio time domain, frequency domain, and spectrum domain, multiple audio features closely related to rain sound are selected and combined to obtain a multi-feature combination vector of audio, forming a multi-dimensional, stable, and comprehensive fusion feature expression for rainfall monitoring audio, namely, multi-domain fusion feature;

[0009] S3: Based on the training set in the dataset, the deep network model is trained and fed back to the model to update the network weights and optimize the model parameters to obtain a multi-feature rainfall intensity estimation model;

[0010] S4: input the multi-feature combination vector of the audio to be tested into the multi-feature rainfall intensity estimation model, perform feature extraction and rainfall intensity regression, and output the estimated rainfall intensity;

[0011] S5: Compare the rainfall intensity estimated by the model with the true data from the weather station to evaluate the accuracy of the model estimation.

[0012] Furthermore, the audio data collected in step S1 involves multiple rainfall intensity levels such as light, medium, heavy, and torrential rain, and the collection time is from 2022.09.15 to 2023.12.15, which has rich diversity in time scale.

[0013] Furthermore, the process of splicing and combining the multi-feature combination vectors of the audio in step S2 is as follows: calculating the power spectral density (PSD), spectral centroid (SC), spectral roll-off (SRO) and spectral bandwidth (SB) of the frequency bands with high correlation with rainfall intensity in the frequency domain; the root mean square energy (RMS) and signal to noise ratio (SNR) in the time domain; and the Mel-scale Frequency Cepstral Coefficients (MFCC) in the cepstrum domain. The above seven features are combined along the frequency domain dimension, and the obtained multi-feature combination vector makes up for the limitation of a single feature in not fully describing the rain sound information from different domains.

[0014] Furthermore, step S3 is specifically as follows: the rainfall intensity label is used as the response variable to train the model, and the model weight is randomly generated in the initial stage, and the performance is consistent with the pre-trained model; a warm-up strategy is adopted to slow down the overfitting phenomenon in the initial stage of training, thereby maintaining the stability of parameter distribution; and Smooth L1 loss is used as the loss function to alleviate the impact of extreme rainfall intensity values.

[0015] Furthermore, the specific process of model feature extraction and regression in step S4 is as follows: taking the Transformer baseline model as an example, feature extraction is performed through the encoder, and the feature dimension after extraction is the same as the original feature dimension. After obtaining the two-dimensional feature vector in the global average pooling layer, rainfall intensity estimation is performed through the fully connected layer. Compared with other baseline models, Transformer fully perceives the spatial distribution of features while combining the temporal characteristics of audio information.

[0016] Furthermore, the prediction accuracy of the evaluation model used in step S5 is: mean absolute error (MAE), root mean square error (RMSE) and determination coefficient (R-Squared / R2).

[0017] The mean absolute error (MAE) calculates the absolute value of the difference between the rainfall intensity estimate and the true value of the rain gauge, and then sums them up to get the average value. i is the true value, f(x i )and is the model prediction value, and n is the number of samples.

[0018]

[0019] The root mean square error (RMSE) is the square root of the mean square error:

[0020]

[0021] The coefficient of determination (R-Squared / R2) determines the explanatory power of the model used, R 2 It is generally a value between 0 and 1. The closer it is to 1, the better the model effect is, and the closer it is to 0, the worse the model effect is. represents the mean value of y.

[0022]

[0023] Beneficial effects: The present invention focuses on studying the fusion of multiple features in different audio domains, analyzes different audio feature attributes, and selects audio features that are closely related to rain sounds for feature fusion. In the same data set, the combination of multiple features can effectively improve the accuracy of rainfall intensity estimation compared to a single feature. The present invention improves the temporal resolution and spatial resolution of rainfall detection, and has high computational efficiency and low equipment cost for rainfall intensity prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is the overall workflow diagram of the method of the present invention. DETAILED DESCRIPTION

[0025] The present invention is further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, various equivalent forms of modifications to the present invention by those skilled in the art all fall within the scope defined by the claims attached to this application.

[0026] like Figure 1 As shown, the present invention provides a multi-feature rainfall intensity estimation method based on surveillance camera audio, comprising the following steps:

[0027] S1: Extract audio data from rainfall event monitoring videos collected in different scenarios, annotate the rainfall true value according to the rainfall intensity value obtained by the rain gauge at the weather station, and the duration of the audio data is consistent with the time resolution of the rainfall data collected by the rain gauge at the weather station, which is 1 minute. All annotated 1-minute audio data are cropped into 4-second audio to construct a rainfall monitoring audio dataset containing rainfall intensity values;

[0028] S2: Starting from the analysis of audio time domain, frequency domain, and spectrum domain, multiple audio features closely related to rain sound are selected and combined to obtain a multi-feature combination vector of audio, forming a multi-dimensional, stable, and comprehensive fusion feature expression for rainfall monitoring audio, namely, multi-domain fusion feature;

[0029] S3: Based on the training set in the dataset, the deep network model is trained and fed back to the model to update the network weights and optimize the model parameters to obtain a multi-feature rainfall intensity estimation model;

[0030] S4: input the multi-feature combination vector of the audio to be tested into the multi-feature rainfall intensity estimation model, perform feature extraction and rainfall intensity regression, and output the estimated rainfall intensity;

[0031] S5: Compare the rainfall intensity estimated by the model with the true data from the weather station to evaluate the accuracy of the model estimation.

[0032] In step S1, the collected audio data involves multiple rainfall intensity levels such as light, medium, heavy, and torrential rain. The collection time is from September 15, 2022 to December 15, 2023, and has rich diversity in time scale.

[0033] In step S2, the power spectral density (PSD), spectral centroid (SC), spectral roll-off (SRO) and spectral bandwidth (SB) of the frequency band with high correlation with rainfall intensity in the frequency domain are calculated; the root mean square energy (RMS) and signal to noise ratio (SNR) in the time domain; and the Mel-scale Frequency Cepstral Coefficients (MFCC) in the cepstrum domain are calculated. The above seven features are combined along the frequency domain dimension, and the obtained multi-feature combination vector makes up for the limitation of a single feature in not fully describing the rain sound information from different domains.

[0034] In step S3, the rainfall intensity label is used as the response variable to train the model. The model weights are randomly generated in the initial stage, and the performance is consistent with the pre-trained model. A warm-up strategy is adopted to reduce overfitting in the initial stage of training, thereby maintaining the stability of parameter distribution. Smooth L1 loss is used as the loss function to alleviate the impact of extreme rainfall intensity values.

[0035] In step S4, the specific process of model feature extraction and regression is as follows: taking the Transformer baseline model as an example, feature extraction is performed through the encoder, and the feature dimension after extraction is the same as the original feature dimension. After obtaining the two-dimensional feature vector in the global average pooling layer, rainfall intensity estimation is performed through the fully connected layer. Compared with other baseline models, Transformer fully perceives the spatial distribution of features while combining the temporal characteristics of audio information.

[0036] The model accuracy evaluation in step S5 is specifically: using the mean absolute error (MAE), root mean square error (RMSE) and determination coefficient (R-Squared / R2) indicators to evaluate the model prediction accuracy.

[0037] Based on the above content, in order to verify the effectiveness of the method of the present invention, this embodiment applies the above solution to an example, as follows:

[0038] (1) Data preprocessing. Taking the monitored rainfall audio collected on campus as an example, after data format conversion, annotation integration, and label matching, considering the temporal continuity of the rainfall audio, the length of each 1-minute audio sample is set to 4 seconds for cropping; the acquisition time, rainfall intensity, monitoring camera name, audio data duration, and audio segmentation are marked in the file name, for example:

[0039] "2022-09-15_17-57-00_0.64_hiv00013_60_segment[4,8]"

[0040] The 17,093 audio data sets after preprocessing are divided into training and validation sets according to a ratio of 7:3, resulting in 11,965 training sets and 5,128 validation sets. The statistical information of rainfall events in different periods is shown in Table 1.

[0041] Table 1 Statistical information of rainfall events

[0042]

[0043]

[0044] (2) Calculate audio features. The audio features of the 4s audio files in the training set and the validation set were calculated respectively. In order to maintain the continuity of the audio information, the overlapping frame method was adopted, and the overlap was set to 50%. After framing, librosa was used to extract MFCC, RMS, SC, SB, and SRO features from the audio; the power spectrum density of the audio was divided from the frequency perspective, and the correlation between the PSD value and the rainfall intensity was calculated. The power spectrum density of the frequency band with high correlation was selected to represent the PSD of the entire audio segment, and the part with insufficient feature vector length was filled with 0 vectors; SNR was calculated based on the difference between the maximum decibel value in the decibel envelope and the decibel value of the background noise. The background noise was obtained by adjusting the decibel waveform value. The dimensions of the above 7 features in the training set and the validation set are shown in Table 2 below. After feature dimension conversion and splicing, the dimensions of the multi-feature combination obtained in the training set and the test set were 11965×87×345 and 5128×87×345, respectively.

[0045] Table 2 Feature dimensions of training set and validation set

[0046]

[0047] (3) Transformer model parameter setting. The supervised training method is adopted, and the data set features and true values ​​are input into the model in multiple batches. A small learning rate is set at the beginning of training. After the error between the true value and the predicted value tends to be stable, the preset learning rate is selected to accelerate the convergence speed. The number of samples selected for one training is set to 64, and the number of rounds is set to 100.

[0048] (4) Model training and testing. Based on the constructed monitoring audio rainfall intensity dataset, the multi-feature combination vectors calculated from the divided training set and validation set were input into the model in turn, and the predicted rainfall intensity value and the evaluation index value of each audio file in the validation set were obtained. Compared with the current model performance with a single feature MFCC input and higher prediction accuracy, the results are shown in Table 3.

[0049] Table 3 Performance comparison of single-feature and multi-feature models

[0050]

[0051] As can be seen from Table 3, the model performance of this paper with multiple features as input has higher prediction accuracy than the single feature input of the same model, and has a greater advantage. Specifically, the method of the present invention has a 6.02% reduction in the result error value (MAE) and a 1.02% reduction in the model fitting performance (R 2 ), which has great practical application value for rainfall intensity prediction based on acoustic characteristics.

Claims

1. A multi-feature rainfall intensity estimation method based on surveillance camera audio, characterized in that: Obtain highly robust monitoring rain sound description features and achieve high-precision estimation of rainfall intensity based on monitoring audio; the specific steps are as follows: S1: Extract audio data from rainfall event monitoring videos collected in different scenarios, annotate the rainfall true value according to the rainfall intensity value obtained by the rain gauge at the weather station, and the duration of the audio data is consistent with the time resolution of the rainfall data collected by the rain gauge at the weather station, which is 1 minute. All annotated 1-minute audio data are cropped into 4-second audio to construct a rainfall monitoring audio dataset containing rainfall intensity values; S2: Starting from the analysis of audio time domain, frequency domain, and spectrum domain, multiple audio features closely related to rain sound are selected and combined to obtain a multi-feature combination vector of audio, forming a multi-dimensional, stable, and comprehensive fusion feature expression for rainfall monitoring audio, namely, multi-domain fusion feature; S3: Based on the training set in the dataset, the deep network model is trained and fed back to the model to update the network weights and optimize the model parameters to obtain a multi-feature rainfall intensity estimation model; S4: input the multi-feature combination vector of the audio to be tested into the multi-feature rainfall intensity estimation model, perform feature extraction and rainfall intensity regression, and output the estimated rainfall intensity; S5: Compare the rainfall intensity estimated by the model with the true data from the weather station to evaluate the accuracy of the model estimation.

2. The multi-feature rainfall intensity estimation method based on surveillance camera audio according to claim 1 is characterized in that: The audio data collected in step S1 involves multiple rainfall intensity levels including light, medium, heavy and torrential rain.

3. The multi-feature rainfall intensity estimation method based on surveillance camera audio according to claim 1, characterized in that: The process of splicing and combining the multi-feature combination vectors of the audio in step S2 is as follows: calculating the power spectral density PSD, spectral centroid SC, spectral roll-off SRO and spectral bandwidth SB of the frequency band with high correlation with rainfall intensity in the frequency domain; the root mean square energy RMS and signal-to-noise ratio SNR in the time domain; and the Mel-frequency cepstral coefficients MFCC in the cepstral domain; combining the above 7 features along the frequency domain dimension, and obtaining a multi-feature combination vector to make up for the limitation of a single feature in not fully describing the rain sound information from different domains.

4. The multi-feature rainfall intensity estimation method based on surveillance camera audio according to claim 1, characterized in that: The step S3 is specifically as follows: the rainfall intensity label is used as the response variable to train the model, and the model weight is randomly generated in the initial stage, and the performance is consistent with the pre-trained model; a warm-up strategy is adopted to reduce the overfitting phenomenon in the initial stage of training, thereby maintaining the stability of parameter distribution; and Smooth L1 loss is used as the loss function to alleviate the impact of extreme rainfall intensity values.

5. The multi-feature rainfall intensity estimation method based on surveillance camera audio according to claim 1, characterized in that: The specific process of model feature extraction and regression in step S4 is as follows: using the Transformer baseline model, feature extraction is performed through the encoder, the feature dimension after extraction is the same as the original feature dimension, and rainfall intensity estimation is performed through the fully connected layer after obtaining the two-dimensional feature vector in the global average pooling layer. The Transformer fully senses the spatial distribution of features while combining the temporal characteristics of audio information.

6. The multi-feature rainfall intensity estimation method based on surveillance camera audio according to claim 5, characterized in that: In step S5, the mean absolute error MAE, the root mean square error RMSE and the determination coefficient R-Squared / R 2 Indicators evaluate the prediction accuracy of the model, including: Mean absolute error MAE, calculate the absolute value of the difference between the rainfall intensity estimate and the true value of the rain gauge, and then sum and average them; where y i is the true value, f(x i )and is the model prediction value, n is the number of samples; Root mean square error RMSE, based on the square root of the mean square error: Coefficient of determination R-Squared / R 2 , to determine the explanatory power of the model used, R 2 It is a value between 0 and 1. The closer it is to 1, the better the model effect is, and the closer it is to 0, the worse the model effect is. represents the mean value of y;

Images