A rainfall intensity real-time monitoring method based on audio-video multi-modal fusion

Abstract

This invention discloses a real-time rainfall intensity monitoring method based on audio-video multimodal fusion, belonging to the interdisciplinary field of hydrological monitoring and artificial intelligence. The method includes: acquiring raw rainfall video data and rain gauge data of the target monitoring area; constructing a multimodal synchronous dataset containing visual image frame sequences, audio clips, and real rainfall intensity labels; training a visual single-modal prediction network and an audio single-modal prediction network respectively based on the multimodal synchronous dataset; constructing a multimodal gated fusion monitoring network using the trained visual and audio single-modal prediction networks as two encoders, and training it based on the multimodal synchronous dataset; acquiring real-time rainfall video of the target monitoring area; and predicting the real-time rainfall intensity using the trained multimodal gated fusion monitoring network. Combining the ability of vision to track rainfall dynamics with the ability of audio to calibrate environmental noise, the method achieves better accuracy and trend fitting than single-modal methods.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of hydrological and meteorological monitoring and artificial intelligence, and in particular relates to a method for real-time monitoring of rainfall intensity based on audio and video multimodal fusion. Background Technology

[0002] Due to the combined effects of global climate change and rapid urbanization, extreme short-duration heavy rainfall is characterized by strong localization, suddenness, and high intensity, easily triggering secondary disasters such as urban flooding, road waterlogging, and drainage system overload. Precise urban stormwater control relies heavily on the ability to "refinedly perceive and rapidly assess" rainfall processes, and rainfall intensity, as the core boundary condition driving urban hydrological and hydrodynamic models (runoff generation and runoff evolution), is an indispensable basic data for achieving short-term early warning.

[0003] Current methods for acquiring rainfall intensity mainly include rain gauge observation, weather radar inversion, and satellite remote sensing estimation. While rain gauges can provide high-precision point observations, their spatial distribution is usually sparse due to the high cost of network construction and maintenance, making it difficult to capture the significant spatial heterogeneity of localized urban rainstorms. Although weather radar and satellite remote sensing have wide-area coverage capabilities, they are often limited by tall buildings, clutter interference, and vertical resolution when estimating near-surface rainfall in urban areas. These traditional observation methods often cannot fully meet the requirements for "high spatiotemporal resolution and low latency" rainfall perception at the urban block scale.

[0004] With the advancement of smart city construction, "opportunistic observation" using widely distributed urban surveillance equipment has become a low-cost supplementary approach with significant engineering potential. However, existing purely visual rainfall measurement methods face severe robustness challenges in practical applications: the visual representation of rainfall is highly susceptible to interference from changes in lighting conditions (such as dusk and night), complex background textures, and lens obstruction by water droplets. Especially at night or under low visibility conditions, purely visual models often suffer from severe information loss and estimation bias, leading to a significant reduction in all-weather monitoring capabilities.

[0005] To compensate for the blind spots of visual perception, some studies have begun to introduce environmental audio as an auxiliary modality, using the sound of raindrops hitting the ground to invert rainfall intensity. Audio features are insensitive to illumination and can provide robust fallback discrimination in dark environments. However, single audio models also have limitations; they are easily disturbed by complex urban background sound fields such as strong wind noise and traffic rumble. Currently, most deep learning-based rainfall measurement methods remain at the stage of single-modal independent observation, failing to fully explore the dynamic complementary advantages of audiovisual information in different environments. Summary of the Invention

[0006] To address the technical problems existing in the prior art, this invention provides a real-time rainfall intensity monitoring method based on audio-visual multimodal fusion. By constructing a multimodal dynamic fusion mechanism, it relies on auditory compensation when vision is limited and uses visual calibration when hearing is disturbed, thereby stably outputting high-precision rainfall intensity estimation results in complex and ever-changing urban environments, providing reliable data support for refined hydrological simulation and urban flood control.

[0007] According to a first aspect of the embodiments of this application, a method for real-time monitoring of rainfall intensity based on audio and video multimodal fusion is provided, comprising:

[0008] S1: Obtain raw rainfall video data and rain gauge data of the target monitoring area, and construct a multimodal synchronous dataset containing visual image frame sequences, audio clips and real rainfall intensity labels;

[0009] S2: Based on the visual image frame sequence and real rain intensity labels in the multimodal synchronous dataset, train a visual unimodal prediction network;

[0010] S3: Based on the audio segments and real rain intensity labels in the multimodal synchronization dataset, train an audio unimodal prediction network;

[0011] S4: Using the trained visual monomodal prediction network and audio monomodal prediction network as two encoders, construct a multimodal gated fusion monitoring network based on the two encoders, and train the multimodal gated fusion monitoring network based on the multimodal synchronous dataset;

[0012] S5: Acquire real-time rainfall videos of the target monitoring area and use the trained multimodal gating fusion monitoring network to predict the real-time rainfall intensity.

[0013] Further, step S1 includes:

[0014] S11: Acquire raw rainfall video data and rain gauge data of the target monitoring area and align the data;

[0015] S12: The aligned raw rainfall video data is segmented according to a predetermined window length. Visual image frames and audio signal data are extracted from the raw rainfall video data of each segmented time window. Based on the aligned rain gauge data, the real rainfall intensity data of each time window is determined, thereby constructing a multimodal synchronous dataset.

[0016] Furthermore, the visual monomodal prediction network in step S2 adopts a SlowFast network architecture, which includes two parallel paths that process inputs at different frame rates, an average pooling layer, and a single-neuron linear regression layer.

[0017] The two parallel paths are the fast channel and the slow channel. The fast channel processes the input visual image frames with a relatively higher temporal resolution and a relatively smaller channel capacity, while the slow channel processes the input visual image frames with a relatively lower temporal resolution and a relatively larger channel capacity. The fast channel and the slow channel interact with each other through lateral connections in the residual stage of the network, thereby obtaining fast features and slow features respectively.

[0018] The average pooling layer performs global spatiotemporal average pooling on the fast and slow features to obtain the visual feature vector;

[0019] The single-neuron linear regression layer generates predicted rainfall intensity labels based on the visual feature vector.

[0020] Further, in step S3, the waveform of the audio segment is converted into a log-Mel spectrum, which is used as the input of the audio single-modal prediction network. The audio single-modal prediction network includes six sequentially connected convolutional blocks, two global average pooling layers, one mean pooling layer, and a single-layer linear regression head. The network is used to generate minute-level predicted rainfall intensity based on the log-Mel spectrum. Each convolutional block includes two convolutional layers, a batch normalization layer, an activation function, and an average pooling layer.

[0021] Furthermore, in the multimodal gating fusion monitoring network described in step S4:

[0022] The trained visual unimodal prediction network and audio unimodal prediction network are used as visual encoder and audio encoder, respectively, to extract visual feature vectors and audio feature vectors from visual image frame sequences and audio segments, respectively.

[0023] Visual feature vectors and audio feature vectors are mapped to a unified dimension by independent linear projection layers and layer normalization, resulting in visual projection features and audio projection features.

[0024] The visual feature vector and audio feature vector are concatenated along the feature dimension, and a gated weight vector is generated by linear transformation and activation function. The visual projection feature and audio projection feature are then adaptively weighted and fused based on the gated weight vector to obtain the fused feature.

[0025] The predicted rainfall intensity is generated based on the fused features using a multilayer perceptron regression prediction head.

[0026] Furthermore, during the training of the multimodal gating fusion monitoring network in step S4, the parameters of the visual encoder and audio encoder are frozen.

[0027] According to a second aspect of the embodiments of this application, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the method as described in the first aspect.

[0028] According to a third aspect of the embodiments of this application, an electronic device is provided, comprising:

[0029] One or more processors;

[0030] Memory, used to store one or more programs;

[0031] When the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method as described in the first aspect.

[0032] According to a fourth aspect of the embodiments of this application, a computer-readable storage medium is provided that stores computer instructions thereon, which, when executed by a processor, implement the steps of the method as described in the first aspect.

[0033] The technical solutions provided by the embodiments of this application may include the following beneficial effects:

[0034] (1) Adaptive and complementary fusion of audiovisual information is achieved through a gating fusion mechanism. The gating network dynamically adjusts the contribution ratio of visual and audio features on a sample-by-sample and dimension-by-dimensional basis based on the Sigmoid gating weight. Compared with the simple splicing fusion strategy, the prediction accuracy is significantly improved, the noise interference of low-quality modalities is effectively suppressed, and the accuracy and robustness of rainfall intensity monitoring under all-weather and cross-scene conditions are improved.

[0035] (2) Audiovisual multimodal fusion is significantly better than any single modality. The prediction error of the multimodal gated fusion model is significantly reduced compared to both the optimal visual single modality and the optimal audio single modality, and the coefficient of determination is much lower. This is the highest among all models, validating the complementary gain effect of visual and audio information in the rainfall intensity prediction task.

[0036] (3) Introducing the SlowFast dual-path spatiotemporal network into the visual branch, the low frame rate slow channel captures spatial semantics such as the change of dryness and wetness of the ground, and the high frame rate fast channel captures temporal dynamics such as the movement of raindrops, providing a high-quality visual feature foundation for multimodal fusion.

[0037] In summary, the proposed real-time urban rainfall intensity monitoring method based on audiovisual multimodal deep learning utilizes existing urban surveillance cameras. It extracts visual image frame sequences and audio signals from rainfall videos, employing a SlowFast dual-stream spatiotemporal network and pre-trained PANNs for feature encoding, respectively. Furthermore, a gating fusion mechanism is combined to achieve adaptive weighted fusion of audiovisual information and rainfall intensity regression prediction. This method is characterized by simple deployment, rapid response, and low cost, effectively solving the problems of high equipment cost, low spatiotemporal resolution, and high maintenance difficulty commonly found in traditional rainfall monitoring methods. It significantly improves the accuracy and real-time performance of urban rainfall intensity monitoring. This method provides technical support for scientific early warning and emergency decision-making in urban rainstorm and flood disasters, and is of great significance for improving urban disaster prevention and mitigation systems, possessing high promotional value and practical engineering application potential.

[0038] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0039] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0040] Figure 1 This is a flowchart illustrating a real-time rainfall intensity monitoring method based on audio-video multimodal fusion, according to an exemplary embodiment.

[0041] Figure 2 This is a diagram of the audio and video multimodal fusion architecture based on the SlowFast and PANNs models in the embodiment.

[0042] Figure 3 This is a schematic diagram illustrating the performance of the SlowFast single video module on a rainfall video test set in the embodiment.

[0043] Figure 4 This is a schematic diagram illustrating the performance of PANNs-based single-audio modal on a rainfall video test set in the embodiment.

[0044] Figure 5 This is a schematic diagram illustrating the performance of the visual-audio multimodal method fusion in this embodiment on a rainfall video test set.

[0045] Figure 6 This is a block diagram illustrating a real-time rainfall intensity monitoring device based on audio-video multimodal fusion, according to an exemplary embodiment.

[0046] Figure 7 This is a schematic diagram of an electronic device according to an exemplary embodiment. Detailed Implementation

[0047] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application.

[0048] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0049] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0050] The present invention will now be described in detail with reference to the accompanying drawings, so that the advantages and features of the present invention will be more readily understood by those skilled in the art. In general, the features of the embodiments and implementations described below may overlap.

[0051] like Figure 1 This invention provides a method for real-time monitoring of rainfall intensity based on audiovisual multimodal fusion, the monitoring method comprising the following steps:

[0052] S1: Obtain raw rainfall video data and rain gauge data of the target monitoring area, and construct a multimodal synchronous dataset containing visual image frame sequences, audio clips and real rainfall intensity labels;

[0053] Specifically, this step may include the following sub-steps:

[0054] S11: Acquire raw rainfall video data and rain gauge data of the target monitoring area and align the data;

[0055] Specifically, since there may be a clock discrepancy between the video capture device and the rain gauge, it is necessary to calculate the time offset. Align the two timelines. Let the video start time be... The rain gauge recording start time is The offset is:

[0056]

[0057] The effective start time after alignment is:

[0058]

[0059] exist In this case, skip the asynchronous period between the start of the video and the start of the rain gauge; In this case, skip the asynchronous period between the start of the rain gauge and the start of the video to ensure that the visual image frame sequence, audio data and rainfall labels are on the same time axis in each subsequent time window segment.

[0060] S12: The aligned raw rainfall video data is divided into segments according to a predetermined window length. Visual image frames and audio signal data are extracted from the raw rainfall video data of each segmented time window. Based on the aligned rain gauge data, the real rainfall intensity data of each time window is determined, thereby constructing a multimodal synchronous dataset.

[0061] In this embodiment, the time window can be set to one minute according to actual needs and the input scale that the subsequent network can accept. However, this method is only one feasible implementation method. In specific implementation, the size of the time window for data segmentation can be selected according to the actual situation and needs.

[0062] Specifically, the raw rainfall video data is divided into several minute-level video segments. For each minute-level video segment, a sequence of visual image frames is extracted and stored as a frame-by-frame image file. At the same time, the audio track is extracted to obtain the raw audio signal. Because the original sampling rates of the built-in microphones in video capture devices (such as surveillance cameras) may differ, the original audio needs to be uniformly resampled to the target sampling rate to ensure consistency in subsequent acoustic feature extraction. This ensures versatility with different data acquisition devices. In this embodiment... The resampling process can be represented as:

[0063]

[0064] in The original audio sampling rate is used; Resample indicates a resampling operation. After resampling, the audio is exported as a minute-long mono WAV file. For multi-channel audio, the average value of each channel is taken to convert it to mono.

[0065]

[0066] in For the number of vocal tracts, For the first The sound channel at the sampling point The amplitude at that point.

[0067] The rain gauge reports the cumulative rainfall (in mm) once per minute. The difference between the cumulative rainfall at two adjacent times can be used to obtain the rainfall in that minute, which is used as the true rainfall intensity label y (in mm / min) for the corresponding sample in that minute.

[0068] After data partitioning, the visual branch organizes all image frames corresponding to each minute into a spatiotemporal segment directory; based on the input scale of subsequent networks, the audio branch segments the aligned audio into 60-second segments per minute, according to a fixed 1-minute time window. Each audio clip is 10 seconds long. Sampling points are filled with zeros if necessary. Using minute-level rainfall intensity as the supervisory label, a multimodal synchronous dataset is derived, consisting of visual image frame sequences, audio clips, and real rainfall intensity labels. This multimodal synchronous dataset includes multiple sets of data, each containing a one-minute visual image frame sequence, an audio clip, and a minute-level real rainfall intensity label.

[0069] S2: Based on the visual image frame sequence and real rain intensity labels in the multimodal synchronous dataset, train a visual unimodal prediction network;

[0070] In one embodiment, the visual unimodal prediction network employs a SlowFast network architecture, which is built on 3D ResNet-50 and includes two parallel paths that process inputs at different frame rates.

[0071] The frame sampling process is as follows: Let the total number of frames in the video frame sequence be... Fast lane frame rate is (This embodiment) ), the inter-frame sampling step size is (This embodiment) Then, the original frame span covered by a video clip is:

[0072]

[0073] During the training phase, the starting frame index of a segment is randomly selected:

[0074]

[0075] The set of frame indices for the Fast pathway is:

[0076]

[0077] The slow pathway takes frames from the fast pathway at intervals. (This embodiment) Perform a second extraction:

[0078]

[0079] in Frames. Therefore, the frames in the slow channel are a strict subset of the frame sequence in the fast channel. The two channels always observe the rainfall process within the same time window, only differing in temporal resolution.

[0080] The slow channel processes the input with a lower temporal resolution (8 frames) but a larger channel capacity (64 channels), focusing on extracting spatial semantic information from the rainfall scene, including slowly changing features such as ground wetness, water accumulation area distribution, overall rain density, and ambient lighting conditions. The fast channel processes the input with a higher temporal resolution (32 frames) but a smaller channel capacity (channel ratio...). The input is processed with a focus on capturing rapidly changing temporal features such as the trajectory of falling raindrops, the transient state of raindrop splashes, and the motion of water ripples. Both channels are built on a ResNet-50 residual network, each containing four residual stages (res2 ~ res5), progressively increasing the number of channels while decreasing the spatial resolution. The slow channel starts with a base of 64 channels, expanding to 256, 512, 1024, and 2048 channels after four residual stages; the fast channel increases the number of channels by a certain ratio. Starting with a base number of 8, the numbers expand sequentially to 32, 64, 128, and 256.

[0081] The fast and slow channels interact in terms of features across the four residual stages of the network via lateral connections. These lateral connections concatenate the features of the fast channel (downsampled over time) with those of the slow channel, achieving cross-timescale information fusion. Let the features of the slow channel in layer l be... Fast lane features are

[0082] in Number the residual stage. and The slow lane and the fast lane are respectively in the... The number of channels in the layer and These represent the corresponding time dimension sizes. and This represents the spatial height and width of the feature map for this layer.

[0083] Horizontal connection can be represented as:

[0084]

[0085] in Downsampling operations in the time dimension (e.g., 3D convolution kernel size is...) (convolutional layers) to align the time dimension of the fast channel with that of the slow channel.

[0086] After all the residual stages, the features of the two channels are concatenated after global spatiotemporal average pooling to obtain... 3D visual feature vector (in this embodiment) ). The original SlowFast Classification Head ( Replace the number of action recognition categories with a single-neuron linear regression layer:

[0087]

[0088] in , .in For the weights of the regression layer, For bias terms, The result obtained after global spatiotemporal average pooling A 3D visual feature vector. The regression layer is initialized using the Kaiming normal distribution.

[0089] The training loss of the visual unimodal prediction network uses mean squared error:

[0090]

[0091] in For batch size, For the first Visual modality prediction values ​​for each sample. This corresponds to the actual rainfall intensity label.

[0092] S3: Based on the audio segments and real rain intensity labels in the multimodal synchronization dataset, train an audio unimodal prediction network;

[0093] Specifically, firstly, each minute-level sample corresponds to... A 10-second clip Loaded as a mono waveform at a sampling rate of 32kHz Length of each segment Sampling point.

[0094] This embodiment uses the CNN14 architecture from PANNs (Pretrained Audio Neural Networks) as the audio unimodal prediction network. CNN14 is a deep convolutional neural network containing 14 layers of convolutions. It is pre-trained on the large-scale audio event dataset AudioSet (containing more than 2 million 10-second audio clips and 527 audio event categories) and has the ability to distinguish various environmental sounds (including rain sounds, wind sounds, traffic sounds, etc. of different intensities).

[0095] First, the original mono waveform with a sampling rate of 32kHz is transformed using a short-time Fourier transform (window length 1024, frame shift 320). The data is converted to a spectrogram, then mapped to a log-Mel spectrogram using a Mel filter bank (64 Mel filters, frequency range 50~14000Hz), which serves as the input to CNN14. During training, SpecAugmentation data augmentation (temporal masking width 64, frequency masking width 8) is applied to the Mel spectrogram to improve the model's generalization ability.

[0096] CNN14 comprises six sequentially connected convolutional blocks, two global average pooling layers, one mean pooling layer, and a single-layer linear regression head. The network generates minute-level predicted rainfall intensity based on the log-Melogram spectrogram. Specifically, the spectral features are then used to extract time-frequency features layer by layer through six convolutional blocks. Each convolutional block contains two 3×3 convolutions, batch normalization, and ReLU activation. A 2×2 average pooling layer is applied after each convolutional block for downsampling (the last convolutional block uses 1×1 pooling to maintain resolution). Dropout is applied after the output of each convolutional block. Regularization is then applied. The number of channels in the six convolutional blocks are 64, 128, 256, 512, 1024, and 2048, respectively. The last convolutional block outputs a feature map with 2048 channels. Global average pooling is then performed on the frequency and time dimensions respectively to obtain a 2048-dimensional segment-level audio feature vector:

[0097]

[0098] Within the same minute The segment-level feature vectors of each segment are aggregated using an inter-segment feature mean pooling strategy, that is, mean pooling is performed at the segment level to form a minute-level audio feature representation:

[0099]

[0100] The aggregated features are then processed by Dropout. After regularization, the input to the single-layer linear regression head Output minute-level rainfall intensity forecast values (Unit: mm / min). During the training phase, a full-parameter fine-tuning strategy is adopted, with all parameters of the CNN14 convolutional backbone and linear regression head participating in gradient updates; a smooth L1 loss function is used. As a training objective:

[0101]

[0102] This loss function combines the sensitivity of mean squared error to small biases with the robustness of absolute error to large biases, making it suitable for robust regression of samples containing abnormally heavy rainfall.

[0103] In one alternative implementation, the regression prediction of the audio modality also supports a segment-by-segment prediction strategy: for each 10-second segment... Independently using CNN14 feature extraction and linear regression head to output fragment-level rainfall intensity prediction values And then The arithmetic mean of the predicted values ​​is used as the minute-level prediction result:

[0104]

[0105] The difference between this strategy and the aforementioned inter-segment feature mean pooling strategy is that segment-by-segment prediction aggregates at the predicted value level, while inter-segment mean pooling aggregates at the feature level. This embodiment prefers the latter.

[0106] S4: Using the trained visual monomodal prediction network and audio monomodal prediction network as two encoders, construct a multimodal gated fusion monitoring network based on the two encoders, and train the multimodal gated fusion monitoring network based on the multimodal synchronous dataset;

[0107] Specifically, the multimodal gating fusion monitoring network includes:

[0108] (1) Parallel visual encoder and audio encoder

[0109] The visual encoder uses the SlowFast R50 network trained in step S2, taking a full-minute video frame sequence as input as a spatiotemporal segment, and generating the encoder through one forward propagation. 3D visual feature vector (This embodiment) ).

[0110] Audio encoder: Using the PANNs-CNN14 network trained in step S3, it processes audio within the same minute. Generate minute-level audio feature vectors from 10-second audio clips. .

[0111] After loading the weights obtained from training on the rainfall data in steps S2 and S3, all parameters of the two encoders are frozen, and only the gated fusion module and the regression prediction head are trained. Specifically, the two encoders have been fully trained for the rainfall scenario in steps S2 and S3 and have good feature extraction capabilities. Even after freezing, they can still output high-quality visual and audio features. The gated fusion module adaptively weights and fuses the two features through learnable gate weights, which has sufficient learning capacity to model cross-modal complementarity relationships. Freezing the encoders can effectively prevent the converged encoder parameters from being destroyed due to gradient backpropagation during the fusion training phase.

[0112] (2) Linear projection and layer normalization

[0113] Due to the dimensions of visual features With audio feature dimensions Since there are differences, the two feature paths are mapped to a unified dimension through independent linear projection layers and layer normalization. Hidden space (in this embodiment) ), to obtain visual projection features and audio projection features :

[0114]

[0115]

[0116] in , For visual projection layer parameters, , These are the parameters for the audio projection layer.

[0117] (3) Gating fusion module

[0118] Original visual features With audio features After concatenating the feature dimensions, a linear transformation and a sigmoid activation function are applied to generate a gated weight vector with the same dimensions as the hidden layers.

[0119]

[0120] in , For gating network parameters, For the Sigmoid function, [v; a] indicates concatenation along the feature dimension.

[0121] Based on gating weights Adaptive weighted fusion of the two projection features yields the fused features:

[0122]

[0123] in This represents element-wise multiplication. Gating weight. Each component takes values ​​in the range (0, 1). When a certain dimension... When the value approaches 1, the fused features are mainly contributed by visual projection features; when... When the value approaches 0, the audio projection feature becomes dominant. This mechanism enables the network to adaptively adjust the contribution ratio of the two modalities on a dimension-by-dimensional and sample-by-sample basis.

[0124] (4) Regression prediction head

[0125] Fusion features The data is fed into a multilayer perceptron (MLP) regression prediction head, which includes a GELU activation function and a Dropout regularization layer. The system performs two linear transformations to ultimately generate the predicted rainfall intensity. :

[0126]

[0127]

[0128]

[0129]

[0130] in , These are the parameters of the first fully connected layer. , These are the parameters for the second fully connected layer.

[0131] In practical implementation, the multimodal gating fusion monitoring network is trained using a smooth L1 loss function. ):

[0132]

[0133] The AdamW optimizer was used (learning rate 1×10⁻). 4 Weight decay 1×10⁻ 4The learning rate is scheduled using cosine annealing. The model selection strategy is as follows: after each training epoch, the validation set is used for evaluation, and the validation set MAE (mean absolute error) is used as the model selection criterion. The checkpoint with the lowest MAE is saved as the optimal model for final testing.

[0134] S5: Acquire real-time rainfall videos of the target monitoring area and use the trained multimodal gated fusion monitoring network to predict the real-time rainfall intensity;

[0135] Specifically, after undergoing the same preprocessing operation as in step S1, the real-time rainfall video is used to output the predicted rainfall intensity value through the trained multimodal gating fusion monitoring network.

[0136] Example

[0137] To verify the effectiveness of the method of this invention, data acquisition and model training experiments were conducted in a real outdoor rainfall scenario. A fixed observation point was selected for the experiment, and continuous data was collected from several natural rainfall events between June and September 2025. Video images, environmental audio, and the true rainfall values ​​at corresponding times were recorded simultaneously. The video acquisition equipment was a Hikvision AOV 4G solar-powered PTZ camera (model HK-T1S4M-GL), with a resolution of 2560×1440, IP66 protection rating, and support for infrared / white light dual supplementary lighting to ensure all-weather recording. The rainfall observation equipment was an RS-YLS-N01-6 double-tip bucket rain gauge from Shandong Renke Measurement and Control Technology Co., Ltd., with a resolution of 0.1 mm and uploading cumulative rainfall data once per minute. The rain gauge was installed approximately 10 m away from the camera, and both were in the same observation area, meeting the spatial consistency requirement.

[0138] Approximately 1007 minutes of valid labeled data were obtained. The rainfall intensity distribution in the dataset exhibits a typical long-tail characteristic: light rain (0–10 mm / h) accounts for 61%, moderate rain (10–25 mm / h) accounts for 30.5%, heavy rain (25–50 mm / h) accounts for 5.3%, and torrential rain (≥50 mm / h) accounts for 3.3%. Daytime samples account for 85.6%, and nighttime samples account for 14.4%, covering different lighting conditions from early morning to late at night. Following a rainfall event-based partitioning strategy, complete rainfall events were distributed across the training, validation, and test sets to ensure that different time segments of the same rainfall event are not distributed across sets.

[0139] The visual unimodal prediction network, audio unimodal network, and multimodal fusion network are trained separately, with the following specific configurations:

[0140] (1) Visual monomodal (SlowFast R50): 32 frames for the fast channel, 8 frames for the slow channel, frame sampling step size of 2, batch size of 4, learning rate of 10⁻ 4 AdamW optimizer, MSE loss function, and full parameter fine-tuning.

[0141] (2) Audio Single-Mode (PANNs CNN14): Input is a 10-second audio clip with a sampling rate of 32 kHz, batch size of 16, and learning rate of 3×10⁻ 5 Adam optimizer, SmoothL1 loss function ( Full parameter fine-tuning was performed; two strategies, segment-by-segment prediction and inter-segment feature mean pooling prediction, were tested respectively.

[0142] (3) Multimodal fusion (SlowFast R50 + PANNs CNN14): Both the visual encoder and the audio encoder are loaded with their respective single-modal pre-trained weights and the encoder parameters are frozen. The fusion module is trained with a fusion hidden layer dimension of 1024, a dropout rate of 0.2, and a SmoothL1 loss function. The AdamW optimizer was tested, along with two strategies: feature concat fusion and gate fusion.

[0143] All experiments used the validation set MAE as the model selection criterion, and the optimal checkpoint was saved to evaluate the final performance on the test set. Evaluation metrics included MAE (mm / min) and the coefficient of determination. .

[0144] The performance of each model on the test set is as follows: Figure 3-5 As shown, the performance comparison is shown in Table 1.

[0145] Table 1 Performance Comparison of Test Sets for Each Modal Model

[0146]

[0147] The following conclusions can be drawn from the experimental results:

[0148] (1) Multimodal fusion is significantly better than any single modality. The multimodal gated fusion model (SlowFast + PANNs, Gate Fusion) achieved the lowest MAE = 0.065 mm / min and the highest R² = 0.89, which is 16.7% lower than the best visual single modality (SlowFast R50, MAE = 0.078) and 34.3% lower than the best audio single modality (PANNs CNN14 segmented prediction, MAE = 0.099), verifying the significant gain effect of audiovisual information fusion on rainfall intensity prediction.

[0149] (2) Gated fusion mechanism is superior to simple concatenation fusion. The multimodal model using gated fusion has an MAE of 0.065 and an R² of 0.89, compared to the MAE of simple concatenation fusion (Concat) of 0.089 and an R² of 0.86. The MAE is reduced by 27.0%, and the R² is increased by 0.03. This indicates that the gated mechanism, through the adaptive adjustment of the contribution ratio of visual and audio features on a dimension-by-dimensional and sample-by-sample basis using sigmoid gate weights, can effectively suppress noise interference from low-quality modalities and achieve better cross-modal information integration than fixed equal-weight concatenation.

[0150] (3) The visual modality contributes more to rainfall intensity prediction than the audio modality. The MAE of the SlowFast visual monomodality is 0.078 and the R² is 0.87, both of which are better than the best results of the PANNs audio monomodality (MAE = 0.099 and R² = 0.76). This indicates that the dual-path spatiotemporal modeling of SlowFast can effectively capture visual cues such as ground wetness and dryness changes and rain streak movement in rainfall scenarios, and has a strong explanatory power for the regression prediction of rainfall intensity. Although the audio modality performs weaker than the visual modality on its own, it provides complementary information in multimodal fusion, which further improves the R² of the fusion model from 0.87 for the visual monomodality to 0.89.

[0151] (4) The training strategy of the audio encoder has a significant impact on the prediction performance. PANNs CNN14 achieved MAE = 0.099 and R² = 0.76 under the segment-by-segment prediction strategy, while MAE = 0.102 and R² = 0.80 under the inter-segment fusion strategy. The two strategies each have their advantages: segment-by-segment prediction is slightly better in terms of MAE, while inter-segment fusion performs better in terms of R², indicating that different aggregation methods of audio segment features affect the model's ability to fit the trend of rainfall intensity changes.

[0152] The above experimental results fully verify the effectiveness of the proposed real-time urban rainfall intensity monitoring method based on audiovisual multimodal deep learning, particularly the superiority of the gated fusion mechanism in cross-modal information integration. Experiments show that gated fusion not only significantly outperforms single-modal and simple splicing fusion schemes in prediction accuracy (MAE), but also achieves the best performance among all models in terms of the coefficient of determination (R²), demonstrating that adaptive gate weights can effectively coordinate the complementary advantages of the audiovisual and visual modalities, achieving more accurate rainfall intensity prediction.

[0153] Corresponding to the aforementioned embodiments of the real-time rainfall intensity monitoring method based on audio-video multimodal fusion, this application also provides embodiments of a real-time rainfall intensity monitoring device based on audio-video multimodal fusion.

[0154] Figure 6 This is a block diagram of a real-time rainfall intensity monitoring device based on audio-video multimodal fusion, according to an exemplary embodiment. (Refer to...) Figure 6 The device may include:

[0155] The dataset construction module M1 is used to acquire raw rainfall video data and rain gauge data of the target monitoring area and construct a multimodal synchronous dataset containing visual image frame sequences, audio clips and real rainfall intensity labels.

[0156] The visual unimodal training module M2 is used to train the visual unimodal prediction network based on the visual image frame sequence and real rain intensity labels in the multimodal synchronous dataset.

[0157] The audio unimodal training module M3 is used to train the audio unimodal prediction network based on the audio segments and real rain intensity labels in the multimodal synchronous dataset.

[0158] The multimodal gating fusion training module M4 is used to construct a multimodal gating fusion monitoring network based on the trained visual monomodal prediction network and audio monomodal prediction network as two encoders, and to train the multimodal gating fusion monitoring network based on the multimodal synchronous dataset.

[0159] The real-time prediction module M5 is used to acquire real-time rainfall videos of the target monitoring area and use the trained multimodal gating fusion monitoring network to predict the real-time rainfall intensity.

[0160] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0161] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0162] Accordingly, this application also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the above-described method for real-time monitoring of rainfall intensity based on audio-video multimodal fusion.

[0163] Accordingly, this application also provides an electronic device, including: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the above-described method for real-time monitoring of rainfall intensity based on audio-video multimodal fusion. Figure 7 The diagram shown is a hardware structure diagram of any device with data processing capabilities, including a real-time rainfall intensity monitoring device based on audio and video multimodal fusion provided in an embodiment of the present invention. (Except for...) Figure 7 In addition to the processor, memory, and network interface shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0164] Accordingly, this application also provides a computer-readable storage medium storing computer instructions, which, when executed by a processor, implement the above-described method for real-time monitoring of rainfall intensity based on audio-video multimodal fusion. The computer-readable storage medium can be an internal storage unit of any data-processing device as described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units of any data-processing device and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the data-processing device, and can also be used to temporarily store data that has been output or will be output.

[0165] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

Claims

1. A method for real-time monitoring of rainfall intensity based on audio-video multimodal fusion, characterized in that, include: S1: Obtain raw rainfall video data and rain gauge data of the target monitoring area, and construct a multimodal synchronous dataset containing visual image frame sequences, audio clips and real rainfall intensity labels; S2: Based on the visual image frame sequence and real rain intensity labels in the multimodal synchronous dataset, train a visual unimodal prediction network; S3: Based on the audio segments and real rain intensity labels in the multimodal synchronization dataset, train an audio unimodal prediction network; S4: Using the trained visual monomodal prediction network and audio monomodal prediction network as two encoders, construct a multimodal gated fusion monitoring network based on the two encoders, and train the multimodal gated fusion monitoring network based on the multimodal synchronous dataset; S5: Acquire real-time rainfall videos of the target monitoring area and use the trained multimodal gating fusion monitoring network to predict the real-time rainfall intensity.

2. The method according to claim 1, characterized in that, Step S1 includes: S11: Acquire raw rainfall video data and rain gauge data of the target monitoring area and align the data; S12: The aligned raw rainfall video data is segmented according to a predetermined window length. Visual image frames and audio signal data are extracted from the raw rainfall video data of each segmented time window. Based on the aligned rain gauge data, the real rainfall intensity data of each time window is determined, thereby constructing a multimodal synchronous dataset.

3. The method according to claim 1, characterized in that, The visual monomodal prediction network in step S2 adopts a SlowFast network architecture, which includes two parallel paths that process inputs at different frame rates, an average pooling layer, and a single-neuron linear regression layer. The two parallel paths are the fast channel and the slow channel. The fast channel processes the input visual image frames with a relatively higher temporal resolution and a relatively smaller channel capacity, while the slow channel processes the input visual image frames with a relatively lower temporal resolution and a relatively larger channel capacity. The fast channel and the slow channel interact with each other through lateral connections in the residual stage of the network, thereby obtaining fast features and slow features respectively. The average pooling layer performs global spatiotemporal average pooling on the fast and slow features to obtain the visual feature vector; The single-neuron linear regression layer generates predicted rainfall intensity labels based on the visual feature vector.

4. The method according to claim 1, characterized in that, In step S3, the waveform of the audio segment is converted into a log-Mel spectrum, which is used as the input of the audio single-modal prediction network. The audio single-modal prediction network includes six sequentially connected convolutional blocks, two global average pooling layers, one mean pooling layer, and a single-layer linear regression head. The network is used to generate minute-level predicted rainfall intensity based on the log-Mel spectrum. Each convolutional block includes two convolutional layers, a batch normalization layer, an activation function, and an average pooling layer.

5. The method according to claim 1, characterized in that, In the multimodal gating fusion monitoring network described in step S4: The trained visual unimodal prediction network and audio unimodal prediction network are used as visual encoder and audio encoder, respectively, to extract visual feature vectors and audio feature vectors from visual image frame sequences and audio segments, respectively. Visual feature vectors and audio feature vectors are mapped to a unified dimension by independent linear projection layers and layer normalization, resulting in visual projection features and audio projection features. The visual feature vector and audio feature vector are concatenated along the feature dimension, and a gated weight vector is generated by linear transformation and activation function. The visual projection feature and audio projection feature are then adaptively weighted and fused based on the gated weight vector to obtain the fused feature. The predicted rainfall intensity is generated based on the fused features using a multilayer perceptron regression prediction head.

6. The method according to claim 5, characterized in that, During the training of the multimodal gating fusion monitoring network in step S4, the parameters of the visual encoder and audio encoder are frozen.

7. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method as described in any one of claims 1-6.

8. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method as described in any one of claims 1-6.

9. A computer-readable storage medium storing computer instructions thereon, characterized in that, When executed by the processor, this instruction implements the steps of the method as described in any one of claims 1-6.

Images