Rainfall monitoring methods, devices, equipment, storage media and program products

Abstract

This application provides a rainfall monitoring method, apparatus, device, storage medium, and program product. Raindrop optical flow maps, water wave spectrograms, and interferometric energy heatmaps are extracted from video data of rainfall scenes. Features are extracted from the raindrop optical flow maps, water wave spectrograms, and interferometric energy heatmaps using a rainfall monitoring model, and these features are fused to obtain multimodal fused features. Rainfall is monitored using the multimodal fused features through the rainfall monitoring model to obtain rainfall monitoring results. The rainfall monitoring model is trained based on the energy transfer and wave transfer laws during raindrop impact on the water surface, which can effectively improve the accuracy of rainfall monitoring while reducing system cost and complexity.

Description

Technical Field

[0001] This application belongs to the field of meteorological early warning technology, and in particular relates to a rainfall monitoring method, device, equipment, storage medium and program product. Background Technology

[0002] Existing rainfall monitoring technologies mainly include satellite remote sensing, weather radar, and traditional rain gauges. However, these methods generally suffer from technical complexity, high construction and maintenance costs, and difficulty in guaranteeing monitoring accuracy under extreme weather conditions. For example, satellite remote sensing technology has insufficient spatial resolution, making it impossible to accurately monitor rainfall over small areas. Furthermore, satellite remote sensing technology relies on satellite communication for long-distance data transmission over large areas, making it susceptible to weather conditions that can block satellite signals, ultimately leading to distorted monitoring data. Weather radar equipment is expensive and has high maintenance costs, and its propagation is easily affected by weather conditions, resulting in distorted monitoring data. Traditional rain gauges come in two types: siphon-type and tipping-bucket-type. Siphon-type rain gauges use the siphon principle to prevent overflow and ensure high-precision measurements, but their design is complex, and maintenance and installation are difficult. Tipping-bucket-type rain gauges have a simple structure, but errors may occur during heavy rain due to incomplete tipping.

[0003] With the continuous advancement of video surveillance technology, using video data for rainfall monitoring has become an emerging research direction. Existing video monitoring methods mostly rely on large models to estimate the features of different types of rainfall images. Although this reduces equipment complexity and cost, relying solely on rainfall image features for inference leads to low accuracy of monitoring results.

[0004] Therefore, how to effectively improve the accuracy of rainfall monitoring while reducing system costs and complexity has become an urgent technical problem to be solved. Summary of the Invention

[0005] This application provides a rainfall monitoring method, apparatus, device, storage medium, and program product, which can effectively improve the accuracy of rainfall monitoring while reducing system cost and complexity.

[0006] In a first aspect, embodiments of this application provide a rainfall monitoring method, the method comprising: Extract raindrop optical flow maps, water wave spectrum maps, and interferometric energy thermograms from video data of rainfall scenes; By extracting features from raindrop optical flow maps, water wave spectrum maps, and interferometric energy thermograms using a rainfall monitoring model and fusing these features, multimodal fusion features are obtained. Rainfall monitoring results are obtained by using a rainfall monitoring model to analyze multimodal fusion features. The rainfall monitoring model is trained based on the energy transfer and wave transfer laws during the process of raindrops hitting the water surface.

[0007] In one feasible implementation, the method further includes: Acquire rainfall monitoring training set data, which includes multiple rainfall monitoring training samples and corresponding rainfall labels. Each rainfall monitoring training sample includes: raindrop optical flow map sample, water wave spectrum map sample, and interferometric energy heat map sample. For each rainfall monitoring training sample, perform the following steps: Input the rainfall monitoring training sample into the rainfall monitoring model to be trained, and obtain the rainfall prediction result of the training sample; Based on the rainfall prediction results of the training sample, the loss function value is calculated. The loss function value is calculated based on the raindrop kinetic energy, ripple energy, wave equation and rainfall difference corresponding to the rainfall prediction results. The rainfall difference is calculated based on the rainfall prediction results and rainfall labels. Determine whether the loss function value meets the preset training stopping condition, which is that the loss function value is minimized. If the conditions are not met, adjust the model parameters of the rainfall monitoring model to be trained, and continue training the rainfall monitoring model using the rainfall monitoring training set until the preset training stop condition is met, and obtain the trained rainfall monitoring model.

[0008] In one feasible implementation, the method further includes: For the image frame sequence corresponding to the video data, extract the raindrop features corresponding to each frame to obtain the raindrop feature map sequence; The motion velocity of raindrops is calculated between adjacent raindrop feature maps in the raindrop feature map sequence using an optical flow algorithm; Based on the movement speed of raindrops, an optical flow map of raindrops is generated.

[0009] In one feasible implementation, the method further includes: For the image frame sequence corresponding to the video data, the water wave region corresponding to each frame image is extracted to obtain the water wave image sequence; Perform a Fourier transform on the water wave image sequence to obtain the frequency and amplitude characteristics of the water waves. A water wave spectrum is generated based on frequency and amplitude characteristics.

[0010] In one feasible implementation, the method further includes: For each frame of water wave image, the spatial modulation degree of the interference fringes is calculated. The spatial modulation degree is calculated based on the maximum and minimum values ​​of pixels within a local window of each frame of water wave image. The spatial modulation is converted into an image representing the spatial distribution of energy, generating an interferometric energy heatmap.

[0011] In one feasible implementation, the method further includes: The system acquires video data captured by a camera device equipped with a polarizing filter to suppress reflections from the water surface. By using motion estimation and image registration techniques, the video data is adjusted to obtain the adjusted video data.

[0012] Secondly, embodiments of this application provide a rainfall monitoring device, the device comprising: The extraction module is used to extract raindrop optical flow maps, water wave spectrum maps, and interferometric energy heat maps from video data of rainfall scenes; The feature fusion module is used to extract features from raindrop optical flow maps, water wave spectrum maps, and interferometric energy thermograms through a rainfall monitoring model and perform feature fusion to obtain multimodal fused features. The prediction module is used to monitor rainfall by using a rainfall monitoring model to analyze multimodal fusion features and obtain rainfall monitoring results. The rainfall monitoring model is trained based on the energy transfer and wave transfer laws during the process of raindrops hitting the water surface.

[0013] Thirdly, embodiments of this application provide an electronic device, the device including: a processor, and a memory storing computer program instructions; The processor reads and executes computer program instructions to implement a rainfall monitoring method that is any one of the first aspects.

[0014] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer program instructions, which, when executed by a processor, implement a rainfall monitoring method as described in any of the first aspects.

[0015] Fifthly, embodiments of this application provide a computer program product in which instructions, when executed by a processor of an electronic device, cause the electronic device to perform a rainfall monitoring method as described in the first aspect.

[0016] The rainfall monitoring method, apparatus, equipment, storage medium, and program products provided in this application extract raindrop optical flow maps, water wave spectrum maps, and interferometric energy heat maps from video data of rainfall scenes. Then, a rainfall monitoring model extracts features from the raindrop optical flow maps, water wave spectrum maps, and interferometric energy heat maps and performs feature fusion to obtain multimodal fused features. Finally, the rainfall monitoring model monitors the rainfall using the multimodal fused features to obtain rainfall monitoring results. Compared to existing technologies where models rely solely on rainfall image features for estimation when assessing different types of rainfall image features, leading to low accuracy in rainfall monitoring results, this application extracts raindrop optical flow maps, water wave spectrum maps, and interferometric energy thermograms. This allows for the comprehensive capture of multi-dimensional rainfall dynamic information, including raindrop motion direction and velocity, water wave frequency and amplitude, and raindrop impact interference energy distribution. Furthermore, feature fusion is performed on the raindrop optical flow maps, water wave spectrum maps, and interferometric energy thermograms to obtain multi-modal fusion features. This fully utilizes the ripple features generated when raindrops impact the water surface, avoiding distortion from single features that could negatively affect rainfall monitoring results. Moreover, because the rainfall monitoring model is trained based on the energy transfer and wave transfer laws during raindrop impacts, it can force the model to output rainfall monitoring results that conform to physical laws, further improving the accuracy and reliability of rainfall monitoring results. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart illustrating a rainfall monitoring method according to an embodiment of this application is shown; Figure 2 This paper presents a schematic diagram of the overall process of a rainfall monitoring method provided in this application; Figure 3 This application provides an architecture diagram of a rainfall monitoring system. Figure 4 A schematic diagram of the structure of a rainfall monitoring device provided in this application is shown; Figure 5 A schematic diagram of the hardware structure of the electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0019] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0020] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0021] It should be noted that the acquisition, storage, use, and processing of data in this application embodiment all comply with the relevant provisions of national laws and regulations.

[0022] Currently, rainfall monitoring mainly utilizes satellite remote sensing, weather radar, and traditional rain gauges. However, these methods generally suffer from technical complexity, high construction and maintenance costs, and difficulty in guaranteeing monitoring accuracy under extreme weather conditions. With the continuous advancement of video surveillance technology, using video data for rainfall monitoring has become an emerging research direction. Existing video monitoring methods mostly rely on large models to estimate the characteristics of different types of rainfall images. Although this reduces equipment complexity and cost, relying solely on rainfall image characteristics for inference leads to low accuracy of monitoring results.

[0023] Based on this, embodiments of this application provide a rainfall monitoring method, device, equipment, storage medium, and program product. By extracting raindrop optical flow maps, water wave spectrum maps, and interference energy thermograms from video data of rainfall scenes, it can comprehensively capture multi-dimensional dynamic information of rainfall, such as the direction and speed of raindrop movement, water wave frequency and amplitude, and the distribution of interference energy from raindrop impacts. Then, feature fusion is performed on the raindrop optical flow maps, water wave spectrum maps, and interference energy thermograms to obtain multi-modal fusion features. This fully utilizes the ripple features generated when raindrops hit the water surface, avoiding the distortion of single features from affecting the rainfall monitoring results. Furthermore, since the rainfall monitoring model is trained based on the energy transfer and wave transfer laws during the raindrop impact process, it can force the model to output rainfall monitoring results that conform to physical laws, further improving the accuracy and reliability of the rainfall monitoring results.

[0024] The following is a description of a rainfall monitoring method provided in the embodiments of this application.

[0025] Figure 1 A schematic flowchart of a rainfall monitoring method according to an embodiment of this application is shown. Figure 1 As shown, the method may include the following steps: S101: Extract raindrop optical flow maps, water wave spectrum maps, and interferometric energy thermograms from video data of rainfall scenes; S102: Features are extracted from raindrop optical flow maps, water wave spectrum maps, and interferometric energy thermograms using a rainfall monitoring model, and then fused to obtain multimodal fused features; S103: Rainfall monitoring is performed by using a rainfall monitoring model to analyze multimodal fusion features and obtain rainfall monitoring results. The rainfall monitoring model is trained based on the energy transfer and wave transfer laws during the process of raindrops hitting the water surface.

[0026] In this embodiment, raindrop optical flow maps, water wave spectrograms, and interferometric energy heatmaps are extracted from video data of rainfall scenes. Then, features are extracted from the raindrop optical flow maps, water wave spectrograms, and interferometric energy heatmaps using a rainfall monitoring model, and feature fusion is performed to obtain multimodal fusion features. Finally, rainfall monitoring is performed on the multimodal fusion features using a rainfall monitoring model to obtain rainfall monitoring results. Compared to existing technologies where models rely solely on rainfall image features for estimation when assessing different types of rainfall image features, leading to low accuracy in rainfall monitoring results, this application extracts raindrop optical flow maps, water wave spectrum maps, and interferometric energy thermograms. This allows for the comprehensive capture of multi-dimensional rainfall dynamic information, including raindrop motion direction and velocity, water wave frequency and amplitude, and raindrop impact interference energy distribution. Furthermore, feature fusion is performed on the raindrop optical flow maps, water wave spectrum maps, and interferometric energy thermograms to obtain multi-modal fusion features. This fully utilizes the ripple features generated when raindrops impact the water surface, avoiding distortion from single features that could negatively affect rainfall monitoring results. Moreover, because the rainfall monitoring model is trained based on the energy transfer and wave transfer laws during raindrop impacts, it can force the model to output rainfall monitoring results that conform to physical laws, further improving the accuracy and reliability of rainfall monitoring results.

[0027] The specific implementation methods for each of the above steps are described below.

[0028] S101: Extract raindrop optical flow maps, water wave spectrum maps, and interference energy thermograms from video data of rainfall scenes.

[0029] In this embodiment, single-modal image features cannot capture the dynamic information of a rainfall scene. This application monitors rainfall by considering the ripple features generated when raindrops hit the water surface. Specifically, this application extracts raindrop optical flow maps, water wave spectrum maps, and interferometric energy thermograms from video data of rainfall scenes to monitor rainfall. Among them, the raindrop optical flow map reflects the motion state of raindrops; the water wave spectrum map reflects the frequency and amplitude of water waves; and the interferometric energy thermogram reflects the intensity of raindrop impact.

[0030] In one example, a specific implementation of extracting raindrop optical flow maps from video data of a rainfall scene is as follows: For the image frame sequence corresponding to the video data, raindrop features corresponding to each frame are extracted to obtain a raindrop feature map sequence.

[0031] In this embodiment, raindrop features are extracted for each image frame sequence corresponding to the video data. In one example, a Convolutional Neural Network (CNN) can be used to extract the corresponding raindrop features from each image frame. In one example, the raindrop features include the shape, density, and trajectory of the raindrops. Based on the extracted raindrop features, a raindrop feature map can be generated to ensure that subsequent optical flow calculations are based on the pixels corresponding to the raindrops, avoiding misjudging the movement of water waves and shoreline objects as raindrop movement. Specifically, the shape of the raindrops in the obtained raindrop feature map can be determined based on the trajectory to determine whether the raindrops are formed by raindrops, removing raindrop shapes formed by water waves.

[0032] In one example, the CNN extracts the shape of raindrops from each frame of the image, then calculates the raindrop density in each frame based on a density estimation algorithm, and extracts the trajectory of the raindrops from adjacent frames based on a trajectory tracking algorithm. In another example, an interference fringe analysis model can be constructed based on the Huygens-Fresnel principle to calculate the raindrop density, where the Huygens-Fresnel principle formula is: (1) Where E(P) represents the complex amplitude of the observation point P; S represents the wavefront surface, i.e., the wavefront surface of the initial wavefront formed by the raindrop impacting the water surface; E(Q) represents the complex amplitude of the source point Q; r represents the distance from the source point Q to the observation point P; and k represents the wave number. The tilt factor represents the change in amplitude of the secondary wave with respect to direction. This represents the complex exponential phase factor, which describes the phase change of the wavelet as it propagates from Q to P.

[0033] The motion velocity of raindrops is calculated by using an optical flow algorithm between adjacent raindrop feature maps in a sequence of raindrop feature maps.

[0034] In this embodiment, an optical flow algorithm is used to calculate the velocity of raindrops between adjacent raindrop feature maps in the raindrop feature map sequence. In one example, the Lucas-Kanade optical flow algorithm can be used for calculation. (2) in, This indicates the speed of a pixel's movement in the x-direction; This indicates the speed of the pixel's movement in the y-direction; This represents the gradient of a pixel in the x-direction; This represents the gradient of a pixel in the y-direction; This represents the gradient of a pixel in the time t direction.

[0035] In one example, based on the pixels occupied by the raindrop in the raindrop feature map, the velocity of each pixel in the pixels occupied by the raindrop can be calculated using the Lucas-Kanade optical flow algorithm. Finally, the average velocity of all pixels in the pixels occupied by the raindrop is taken as the speed of the raindrop.

[0036] Based on the movement speed of raindrops, an optical flow map of raindrops is generated.

[0037] In this embodiment of the application, a raindrop optical flow map is generated based on the movement speed of the raindrops. In one example, based on the raindrop feature map, arrows are used to mark the movement direction and speed of each raindrop. The direction of the arrows is consistent with the displacement direction, and the length of the arrows is proportional to the speed, thereby generating a raindrop optical flow map that can intuitively reflect the overall movement state of the raindrops.

[0038] In this embodiment, raindrop features are extracted from each image frame sequence corresponding to the video data to obtain a raindrop feature map sequence. Background interference is eliminated, allowing subsequent focus on the calculation of raindrop motion velocity. The motion velocity of the raindrops is calculated between adjacent raindrop feature maps in the raindrop feature map sequence using an optical flow algorithm. Finally, based on the motion velocity of the raindrops, a raindrop optical flow map is generated, thereby generating a raindrop optical flow map that reflects the motion state of the raindrops and providing raindrop kinematic features for subsequent rainfall monitoring.

[0039] In one example, a specific implementation of extracting water wave spectrograms from video data of a rainfall scene is as follows: For the image frame sequence corresponding to the video data, the water wave region corresponding to each frame image is extracted to obtain the water wave image sequence.

[0040] In this embodiment, for the image frame sequence corresponding to the video data, the water wave region corresponding to each frame is extracted to avoid pixel interference in the extraction of frequency and amplitude from non-water wave regions such as the shore and sky. In one example, the water wave region can be filtered out based on image segmentation technology to obtain a water wave image sequence.

[0041] Perform a Fourier transform on the water wave image sequence to obtain the frequency and amplitude characteristics of the water waves.

[0042] In this embodiment of the application, a Fourier transform is performed on the water wave image sequence to convert the pixel brightness change signal in the time domain into frequency and amplitude signals in the frequency domain, thereby obtaining the frequency and amplitude features corresponding to the water waves. In one example, the Fourier transform can be a fast Fourier transform.

[0043] In one example, the formula for calculating the Fourier transform is: (3) in, Represents a discrete-time signal in the frequency domain, each It is a complex number, including the first... k The amplitude and phase of each frequency component correspond to the frequency and amplitude characteristics of water waves; This represents a discrete-time signal, showing the change in pixel brightness over time in a sequence of water wave images; N represents the number of sampling points, such as the number of pixels in a column of a water wave image; j represents the imaginary unit. This represents the complex exponential basis function.

[0044] A water wave spectrum is generated based on frequency and amplitude characteristics.

[0045] In this embodiment of the application, a water wave spectrum is generated based on frequency and amplitude features. In one example, with frequency as the horizontal axis and amplitude as the vertical axis, the frequency and amplitude features of each frame of water wave image are marked as points on the spectrum, and the water wave spectrum is generated in chronological order.

[0046] In this embodiment, for the image frame sequence corresponding to the video data, the water wave region corresponding to each frame image is extracted to obtain the water wave image sequence. The interference of non-water waves on the construction of the water wave spectrum map is eliminated. Then, the water wave image sequence is subjected to Fourier transform to obtain the frequency features and amplitude features corresponding to the water waves. Finally, based on the frequency features and amplitude features, a water wave spectrum map reflecting the frequency and amplitude of the water waves is generated to provide water wave features for rainfall monitoring.

[0047] In one example, a specific implementation of extracting an interferometric energy heatmap from video data of a rainfall scene is as follows: For each frame of the water wave image, the spatial modulation degree of the interference fringes is calculated. The spatial modulation degree is calculated based on the maximum and minimum values ​​of pixels within a local window of each frame of the water wave image.

[0048] In this embodiment, the sharpness of the interference fringes directly reflects the energy of the raindrop impact. The spatial modulation is calculated by using the maximum and minimum values ​​of a local window to quantify the fringes sharpness.

[0049] In one example, the formula for calculating spatial modulation is: (4) in, This represents the maximum brightness value within a local window centered at (x, y), such as a 3×3 pixel area. This represents the minimum brightness value within a local window centered at (x, y); This indicates the spatial modulation within the local area.

[0050] The spatial modulation is converted into an image representing the spatial distribution of energy, generating an interferometric energy heatmap.

[0051] In this embodiment of the application, the spatial modulation is converted into an image representing the spatial distribution of energy to generate an interferometric energy heat map. In one example, the spatial modulation can be mapped to different colors to generate an interferometric energy heat map that can intuitively show the spatial distribution of raindrop impact energy.

[0052] In this embodiment, the spatial modulation of the interference fringes is calculated for each frame of water wave image, thereby calculating the sharpness of the interference fringes. Since the sharpness can directly reflect the energy of raindrops hitting the water surface, the higher the sharpness, the greater the impact energy. Based on the calculated spatial modulation of the interference fringes, it is converted into an image characterizing the spatial distribution of energy, generating an interference energy heat map that reflects the impact intensity of raindrops, thereby providing water wave energy characteristics for rainfall monitoring.

[0053] To improve the accuracy of extracting raindrop optical flow maps, water wave spectrograms, and interferometric energy thermograms, the following steps are included before step S101: The system acquires video data captured by a camera equipped with a polarizing filter to suppress reflections from the water surface.

[0054] In this embodiment, camera equipment is deployed in the target area to fully cover the rainfall scene and ensure that complete video data of the rainfall scene can be captured. To improve the imaging quality of water ripples, a polarizing filter is configured on the camera equipment. The polarizing filter filters the polarization direction of light, removes random polarization components in reflected and scattered light, reduces glare and fog interference, and enhances image contrast and clarity.

[0055] By using motion estimation and image registration techniques, the video data is adjusted to obtain the adjusted video data.

[0056] In this embodiment, motion estimation and image registration techniques are used to correct image jitter and displacement deviation caused by wind and waves in real time, thereby adjusting the video data to obtain adjusted video data, ensuring the stability of the acquired video data, and thus ensuring the accuracy of subsequent analysis.

[0057] In one example, to improve the efficiency of subsequent image processing, the acquired RGB (Red, Green, Blue) video data is converted into grayscale video data, thereby reducing computational complexity and improving processing efficiency.

[0058] In another example, to ensure the quality of the acquired video data, Gaussian blur noise is used for denoising to reduce the interference of sensor noise and environmental clutter on the quality of the video data. Then, the video image is segmented into multiple sub-regions using adaptive thresholding, and a local threshold is calculated for each sub-region to achieve local binarization. The binarized image is then normalized to map pixel values ​​to a uniform range to enhance image contrast. Finally, a visibility index is calculated based on image features, such as brightness and contrast. The brightness and contrast of the image are directly adjusted according to the visibility index to ensure that image features can still be identified under low visibility conditions.

[0059] In this embodiment, by configuring a polarizing filter on the camera device, random polarization components in reflected and scattered light are removed, improving the image contrast and clarity of the acquired video data. Then, through motion estimation and image registration techniques, image jitter and displacement deviation caused by wind and waves are corrected in real time, ensuring the stability of the acquired video data. This provides accurate and clear video data for subsequent extraction of raindrop optical flow maps, water wave spectrum maps, and interferometric energy thermograms.

[0060] S102: Features are extracted from raindrop optical flow maps, water wave spectrum maps, and interferometric energy thermograms using a rainfall monitoring model, and then fused to obtain multimodal fusion features.

[0061] In this embodiment, the three feature maps have different information dimensions. Using any one feature alone will lead to errors due to incomplete information. By using a rainfall detection model, features are extracted from raindrop optical flow maps, water wave spectrum maps, and interferometric energy thermograms and then fused to improve the sensitivity of the rainfall monitoring model to changes in rainfall intensity, thereby ensuring the accuracy of the monitoring results.

[0062] In one example, raindrop optical flow features, water wave spectral features, and interference energy thermal features can be fused using a cross-attention mechanism: (5) Where Q represents the optical flow characteristics of raindrops; K represents the spectral characteristics of water waves; and V represents the thermodynamic characteristics of interference energy. Feature dimensions representing the spectral characteristics of water waves; Similarity matrix between raindrop optical flow characteristics and water wave spectral characteristics.

[0063] The contributions of raindrop optical flow features, water wave spectral features, and interference energy thermal features are dynamically balanced through a cross-attention mechanism. When the similarity between raindrop optical flow features and water wave spectral features is high, the weight of the corresponding interference energy thermal features will be increased. When the similarity between raindrop optical flow features and water wave spectral features is low, the weight of the corresponding interference energy thermal features will be reduced, thus avoiding interference information from affecting the fusion result.

[0064] S103: Rainfall monitoring is performed by using a rainfall monitoring model to analyze multimodal fusion features and obtain rainfall monitoring results. The rainfall monitoring model is trained based on the energy transfer and wave transfer laws during the process of raindrops hitting the water surface.

[0065] In this embodiment, rainfall is monitored using a rainfall monitoring model based on multimodal fusion features to obtain rainfall monitoring results. The rainfall monitoring model is trained based on the energy transfer and wave transfer laws during raindrop impact on the water surface. In one example, these laws include raindrop kinetic energy, ripple energy, and wave equations. In another example, meteorological sensors and water level sensors can be combined to infer the rainfall results obtained in this application, thereby providing more comprehensive meteorological and hydrological information.

[0066] In one example, before extracting and fusing features from raindrop optical flow maps, water wave spectrograms, and interferometric energy thermograms using a rainfall monitoring model to obtain multimodal fused features, it is necessary to train the rainfall monitoring model to obtain a trained rainfall monitoring model. A specific training method for the rainfall monitoring model includes: Acquire rainfall monitoring training set data, which includes multiple rainfall monitoring training samples and corresponding rainfall labels. Each rainfall monitoring training sample includes: raindrop optical flow map sample, water wave spectrum map sample, and interferometric energy heat map sample.

[0067] In this embodiment, rainfall monitoring training data is acquired. This training data includes multiple rainfall monitoring training samples and corresponding rainfall labels. Each training sample includes: a raindrop optical flow map sample, a water wave spectrum map sample, and an interferometric energy heat map sample. In one example, rainfall monitoring training data covering various rainfall scenarios, such as light rain, moderate rain, and heavy rain, can be collected to avoid poor generalization ability caused by the rainfall monitoring model learning only a single scenario. This ensures that the rainfall monitoring model can accurately predict rainfall under different rainfall types.

[0068] For each rainfall monitoring training sample, perform the following steps: Input the rainfall monitoring training sample into the rainfall monitoring model to be trained, and obtain the rainfall prediction result of the training sample; In this embodiment of the application, for each rainfall monitoring training sample, a rainfall monitoring model to be trained is input, and the rainfall monitoring model to be trained performs a forward propagation calculation to obtain the rainfall prediction result for that training sample. The rainfall prediction calculation formula is as follows: (6) in, This represents the predicted rainfall value at time t; The scaling factor is used to adjust the magnitude of the output to match the actual rainfall range; the ReLU (Rectified Linear Unit) function represents the modified linear unit activation function; W represents the weight matrix. represents the hidden layer feature vector after multimodal fusion; b represents the bias term.

[0069] Based on the rainfall prediction results of the training sample, the loss function value is calculated. The loss function value is calculated based on the raindrop kinetic energy, ripple energy, wave equation and rainfall difference corresponding to the rainfall prediction results. The rainfall difference is calculated based on the rainfall prediction results and rainfall labels.

[0070] In this embodiment of the application, a loss function value is calculated based on the rainfall prediction results of the training sample. The loss function value is calculated based on the raindrop kinetic energy, ripple energy, wave equation, and rainfall difference corresponding to the rainfall prediction results.

[0071] In one example, the formula for calculating the kinetic energy of a raindrop is: (7) in, This represents the kinetic energy of the raindrops; The mass of the raindrop is represented by 'v', which can be estimated using the raindrop shape extracted by a CNN convolutional neural network; the velocity of the raindrop is represented by 'v', which can be calculated using an optical flow algorithm. In one example, the formula for calculating ripple energy is: (8) in, The energy of the ripples on the water surface after a raindrop impacts the surface; A represents the amplitude of the water wave, which can be calculated using Fourier transform; f represents the dominant frequency of the water wave, which can also be calculated using Fourier transform. In one example, during model training, the computation... The difference is used as a loss term and is used together with the rainfall difference to optimize the parameters of the rainfall monitoring model, so as to ensure that the rainfall results conform to both the rainfall label and the physical laws of energy transfer.

[0072] In one example, the propagation of interference energy is constrained by a two-dimensional wave equation, which is: (9) Where u represents water surface displacement; c represents wave velocity; This represents the Laplace operator, which describes the second-order partial derivative of the water surface displacement with respect to spatial coordinates, reflecting the curvature and diffusion trend of the ripples in space.

[0073] In one example, when training the model, the residual of the two-dimensional wave equation (9) is calculated as the loss term. The smaller the residual, the more the water wave propagation conforms to the physical laws. Together with the rainfall difference, the parameters of the rainfall monitoring model are optimized to ensure that the rainfall results conform to both the rainfall label and the wave law of water wave propagation.

[0074] Determine whether the loss function value meets the preset training stopping condition, which is the minimum loss function value.

[0075] In this embodiment of the application, it is determined whether the loss function value is the minimum value. When the loss function value is the minimum value, training is stopped and the currently trained rainfall monitoring model is the trained rainfall monitoring model.

[0076] If the conditions are not met, adjust the model parameters of the rainfall monitoring model to be trained, and continue training the rainfall monitoring model using the rainfall monitoring training set until the preset training stop condition is met, and obtain the trained rainfall monitoring model.

[0077] In this embodiment, if the current loss function value is not the minimum, the model parameters of the rainfall monitoring model to be trained are adjusted, and the rainfall monitoring model is continued to be trained using the rainfall monitoring training set until a preset training stopping condition is met, thus obtaining the trained rainfall monitoring model. In one example, the model parameters of the rainfall monitoring model to be trained can be adjusted based on a genetic optimization algorithm.

[0078] In another example, the loss function value can be the difference between the rainfall prediction result and the rainfall label. That is, by using a genetic optimization algorithm, the optimal rainfall estimate is derived by minimizing the difference between the rainfall prediction result and the rainfall label. The inversion process is as follows: (10) in, This indicates the final rainfall forecast result; This represents the i-th observation data; This represents the i-th feature value predicted by the model when the predicted rainfall is R.

[0079] When the model yields the optimal rainfall estimate, it further determines whether the raindrop kinetic energy and ripple energy corresponding to this rainfall estimate conform to physical laws, i.e., whether the difference between the two is less than or equal to a preset threshold. If the preset threshold is not met, the model parameters of the rainfall monitoring model are retrained.

[0080] In one example, the propagation of interference energy is constrained by the two-dimensional wave equation, and the temporal causal relationship between raindrop characteristics and water wave characteristics is verified by Granger causality analysis to eliminate environmental noise interference. That is, the vertical displacement of each point on the water surface is extracted from the interference energy heat map and substituted into both sides of the above formula (7). If the deviation of the calculation results on both sides of the equation is greater than or equal to the preset threshold, it indicates that the water wave is a water wave caused by non-rainfall interference. Therefore, the characteristics corresponding to the water wave are eliminated to ensure that the water wave characteristics of the input model are completely caused by raindrops.

[0081] In another example, gravitational acceleration can be inferred from the observed water wave wavelength. If the inferred gravitational acceleration deviates from the standard value of gravitational acceleration by a preset threshold, it indicates that the water wave is not generated by raindrop impacts, but by non-rainfall disturbances such as wind disturbances and leaf impacts. Therefore, the feature corresponding to this water wave can be removed to ensure that the water wave features in the input model are entirely caused by raindrops. The water wave calculation formula is as follows: (11) in, Indicates the wavelength of water waves; This represents the gravitational acceleration to be calculated; This represents the surface tension coefficient of water; Indicates the density of water; This indicates the frequency of the water waves.

[0082] In this embodiment of the application, by acquiring rainfall monitoring training set data, the rainfall monitoring model to be trained is trained based on the rainfall monitoring training set data. During the training process, physical constraints such as raindrop kinetic energy, ripple energy, and wave equation are embedded into the model training process, so that the output result of the model is not only closest to the rainfall label, but also needs to conform to the physical constraints, thereby enabling the trained rainfall monitoring model to accurately predict rainfall.

[0083] In one example, the trained rainfall monitoring model can independently infer rainfall based on both raindrop and water wave features. If the difference between the inferred rainfall exceeds a preset threshold, it indicates that at least one feature is being interfered with by non-rainfall factors, triggering an environmental anomaly alarm to ensure the accuracy of rainfall monitoring.

[0084] In another example, to improve the accuracy of rainfall monitoring, an interferometric fingerprint database for different rainfall types is first constructed. Water wave interferometric images are acquired under varying rainfall intensities, raindrop sizes, and wind speeds. Key features (such as fringe dominant frequency, energy distribution, and spatial modulation) are extracted from each interferometric image, and the corresponding real rainfall type is labeled. The mapping relationship of these feature rainfall types is stored in a database, forming the interferometric fingerprint database. Therefore, features of the interference fringes can be extracted from real-time acquired water wave images. The extracted features are then compared with standard features in the interferometric fingerprint database. The standard fingerprint with the highest similarity is selected, and its corresponding rainfall type is the determination result. Combining this determined rainfall type provides prior knowledge for the rainfall monitoring model, improving the inversion accuracy.

[0085] Figure 2 A schematic diagram of the overall process of a rainfall monitoring method provided in this application is shown. Figure 2 As shown, it includes: S201: video acquisition; S202: data preprocessing; S203: polarization filtering and wind and wave compensation; S204: raindrop feature extraction; S205: water wave feature extraction; S206: multimodal feature fusion; S207: physical constraints; S208: inversion calculation; S209: data output.

[0086] S201: Video capture, which uses a high-definition camera to capture video data, ensuring data integrity and accuracy, and providing high-quality material for subsequent processing.

[0087] S202: Data preprocessing, which performs operations such as color space conversion, noise reduction, and normalization on the acquired video data to improve image quality.

[0088] S203: Polarization filtering and wave compensation. It uses a polarizer to filter the polarization direction of light, removes random polarization components in reflected and scattered light, reduces glare and fog interference, and enhances image contrast and clarity. Through motion estimation and image registration technology, it accurately captures dynamic changes and eliminates jitter and displacement deviations caused by wind and waves, achieving stable image output and improving the accuracy of subsequent analysis.

[0089] S204: Raindrop feature extraction, using a Convolutional Neural Network - Long Short-Term Memory (CNN-LSTM) to extract parameters such as the shape, velocity, and density of raindrops.

[0090] S205: Water wave feature extraction, extracting dynamic features such as wavelength and modulation of water waves through Fast Fourier Transform (FFT) and Huygens-Fresnel principle.

[0091] S206: Multimodal feature fusion, integrating raindrop optical flow maps, water wave spectrograms, and interferometric energy thermodynamics. Figure 3 The model utilizes modal features and enhances its sensitivity to key features through a cross-attention mechanism.

[0092] S207: Physical constraints, physical constraints for constructing energy conservation and wave equations.

[0093] S208: Inversion calculation, combining the physical constraint model of energy conservation and wave equation to perform rainfall inversion calculation, ensuring high accuracy and reliability of monitoring results.

[0094] S209: Data Output. The calculated rainfall data is formatted and output as a real-time rainfall monitoring report or visualization interface. This supports further analysis and decision-making by users, providing strong support for meteorological early warning, hydrological monitoring, and urban drainage system management. It can also utilize recurrent neural networks (RNNs) and deep learning models for time-series modeling and real-time prediction, outputting rainfall monitoring reports and visualization interfaces.

[0095] Figure 3 This application illustrates a rainfall monitoring system architecture diagram. This system can be integrated with a video IoT platform to achieve real-time data sharing and remote monitoring. Users can view rainfall conditions anytime via mobile phone or computer. The rainfall monitoring system architecture diagram provided in this application includes: a video acquisition module 301; a video preprocessing module 302; a rainfall feature extraction module 303; a rainfall inversion calculation module 304; and a result output module 305.

[0096] The video acquisition module 301 includes: deploying a high-definition camera network in the river and lake area to fully cover the rainfall scene and ensure that the entire rainfall process can be captured. A dedicated water surface monitoring camera is added, equipped with a polarizing filter to optimize the imaging quality of water surface ripples and effectively suppress water surface reflection interference. The water surface monitoring camera is positioned at a 45° angle to the lake surface.

[0097] The video preprocessing module 302 includes: color space conversion, converting RGB images to grayscale images to reduce computational complexity and improve processing efficiency; denoising and enhancement, employing Gaussian blur denoising combined with adaptive threshold normalization to calculate visibility index correction parameters, ensuring image quality; polarization filtering, using polarizers to filter the polarization direction of light, removing random polarization components from reflected and scattered light, reducing glare and fog interference, and enhancing image contrast and clarity; wind and wave compensation processing, performing dynamic analysis on the acquired video data, accurately capturing dynamic changes and eliminating jitter and displacement deviations caused by wind and waves through motion estimation and image registration techniques, achieving stable image output and improving the accuracy of subsequent analysis; and data caching, setting up a dedicated data processing server to store the preprocessed image sequences, providing high-quality input for subsequent analysis.

[0098] The rainfall feature extraction module 303 includes: raindrop feature extraction, which extracts features such as shape (size, distribution), velocity (optical flow method), density (density estimation algorithm) and trajectory (trajectory tracking algorithm) of raindrops through CNN; water wave feature extraction, which performs fast Fourier transform on the water surface wave region to extract the dominant frequency and energy distribution of the ripples; and interference fringes based on CNN.

[0099] The rainfall inversion calculation module 304 includes: dual-modal fusion, which fuses raindrop optical flow maps and water wave spectrum maps through a cross-attention mechanism to improve the model's sensitivity to changes in rainfall intensity; tri-modal enhancement, which adds an interferometric energy heatmap (resolution 0.1 mm / pixel) and uses a three-branch cross-attention network to fuse features, further improving monitoring accuracy; a physical constraint model, which combines energy conservation constraints and wave equation constraints to ensure that the model output conforms to physical laws and improves the reliability of monitoring results; and inversion calculation, which uses an inversion algorithm to invert the optimal rainfall estimate based on the fused features.

[0100] The results output module 305 includes support for generating formatted reports and displaying visual interfaces, facilitating further analysis and decision-making by users. It can also be integrated with a geographic information system to monitor rainfall in real time and predict potential flooding areas, automatically sending early warning information to relevant departments and residents. Furthermore, it utilizes big data analytics to analyze historical rainfall data, predicting future rainfall trends and providing a scientific basis for urban planning and flood control measures.

[0101] Figure 4 A schematic diagram of a rainfall monitoring device provided in this application is shown. Figure 4 As shown, the rainfall monitoring device 400 provided in this application includes: Extraction module 401 is used to extract raindrop optical flow map, water wave spectrum map and interference energy heat map from video data of rainfall scene; The feature fusion module 402 is used to extract features from raindrop optical flow map, water wave spectrum map and interferometric energy thermogram through rainfall monitoring model and perform feature fusion to obtain multimodal fused features; The prediction module 403 is used to monitor rainfall by using a rainfall monitoring model to analyze the multimodal fusion features and obtain rainfall monitoring results. The rainfall monitoring model is trained based on the energy transfer and wave transfer laws during the process of raindrops hitting the water surface.

[0102] In one example, the rainfall monitoring device 400 also includes: The first acquisition module is used to acquire rainfall monitoring training set data. The rainfall monitoring training set data includes multiple rainfall monitoring training samples and corresponding rainfall labels. Each rainfall monitoring training sample includes: raindrop optical flow map sample, water wave spectrum map sample and interferometric energy heat map sample. For each rainfall monitoring training sample, perform the following steps: The input module is used to input rainfall monitoring training samples into the rainfall monitoring model to be trained, and obtain the rainfall prediction results of the training samples. The calculation module is used to calculate the loss function value based on the rainfall prediction results of the training sample. The loss function value is calculated based on the raindrop kinetic energy, ripple energy, wave equation and rainfall difference corresponding to the rainfall prediction results. The rainfall difference is calculated based on the rainfall prediction results and rainfall labels. The judgment module is used to determine whether the loss function value meets the preset training stopping condition, which is that the loss function value is minimized. The training module is used to adjust the model parameters of the rainfall monitoring model to be trained when the conditions are not met, and to continue training the rainfall monitoring model using the rainfall monitoring training set until the preset training stop condition is met, thus obtaining the trained rainfall monitoring model.

[0103] In one example, extraction module 401 includes: The first extraction submodule is used to extract raindrop features corresponding to each frame of the image frame sequence corresponding to the video data, and obtain a raindrop feature map sequence. The first calculation submodule is used to calculate the movement speed of raindrops between adjacent raindrop feature maps in the raindrop feature map sequence using an optical flow algorithm; The first generation submodule is used to generate raindrop optical flow maps based on the movement speed of raindrops.

[0104] In one example, extraction module 401 includes: The second extraction submodule is used to extract the water wave region corresponding to each frame of the video data to obtain the water wave image sequence. The processing submodule is used to perform Fourier transform on the water wave image sequence to obtain the frequency and amplitude features of the water waves. The second generation submodule is used to generate water wave spectrum diagrams based on frequency and amplitude characteristics.

[0105] In one example, extraction module 401 includes: The second calculation submodule is used to calculate the spatial modulation of the interference fringes for each frame of water wave image. The spatial modulation is calculated based on the maximum and minimum values ​​of pixels within a local window of each frame of water wave image. The third generation submodule is used to convert spatial modulation into an image representing the spatial distribution of energy, generating an interferometric energy heatmap.

[0106] In one example, the rainfall monitoring device 400 also includes: The second acquisition module is used to acquire video data collected by the camera device, which is equipped with a polarizing filter to suppress water surface reflection; The adjustment module is used to adjust video data using motion estimation and image registration techniques to obtain adjusted video data.

[0107] Figure 5 A schematic diagram of the hardware structure of the electronic device provided in an embodiment of this application is shown.

[0108] An electronic device may include a processor 501 and a memory 52 storing computer program instructions.

[0109] Specifically, the processor 501 may include a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0110] Memory 502 may include mass storage for data or instructions. For example, and not limitingly, memory 502 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. In one instance, memory 702 may include removable or non-removable (or fixed) media, or memory 502 may be a non-volatile solid-state memory.

[0111] In one instance, memory 502 may be read-only memory (ROM). In one instance, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or flash memory, or a combination of two or more of these.

[0112] Memory 502 may include read-only memory (ROM), random access memory (RAM), disk storage media device, optical storage media device, flash memory device, electrical, optical, or other physical / tangible memory storage device. Therefore, generally, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to one aspect of this disclosure.

[0113] The processor 501 implements a rainfall monitoring method in the above-described embodiment by reading and executing computer program instructions stored in the memory 502.

[0114] In one example, the electronic device may also include a communication interface 503 and a bus 504. For example, Figure 5 As shown, the processor 501, memory 502, and communication interface 503 are connected through bus 504 and complete communication with each other.

[0115] The communication interface 503 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0116] Bus 504 includes hardware, software, or both, that couples components of an online data traffic metering device together. For example, and not as a limitation, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or a combination of two or more of these. Where appropriate, bus 504 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, this application contemplates any suitable bus or interconnect.

[0117] In addition, in conjunction with the rainfall monitoring method in the above embodiments, this application embodiment can provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the rainfall monitoring methods in the above embodiments.

[0118] This application also provides a computer program product, including a computer program that, when executed, implements any of the rainfall monitoring methods described in the above embodiments.

[0119] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0120] The functional blocks shown in the above block diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, read-only memory (ROM), flash memory, erasable read-only memory (EROM), floppy disks, compact disc read-only memory (CD-ROM), optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0121] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0122] The aspects of this disclosure have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by special-purpose hardware performing the specified functions or actions, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0123] The above are merely specific embodiments of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A method for monitoring rainfall, characterized in that, include: Extract raindrop optical flow maps, water wave spectrum maps, and interferometric energy thermograms from video data of rainfall scenes; The extraction of interferometric energy heatmaps from video data of rainfall scenes includes: For the image frame sequence corresponding to the video data, extract the water wave region corresponding to each frame to obtain the water wave image sequence; For each frame of water wave image, the spatial modulation degree of the interference fringes is calculated. The spatial modulation degree is calculated based on the maximum and minimum values ​​of pixels within a local window of each frame of water wave image. The spatial modulation is converted into an image characterizing the spatial distribution of energy to generate the interferometric energy heatmap; Acquire rainfall monitoring training set data, which includes multiple rainfall monitoring training samples and corresponding rainfall labels. Each rainfall monitoring training sample includes: raindrop optical flow map sample, water wave spectrum map sample, and interferometric energy thermogram sample. For each of the rainfall monitoring training samples, perform the following steps: The rainfall monitoring training sample is input into the rainfall monitoring model to be trained to obtain the rainfall prediction result of the training sample; Based on the rainfall prediction results of the training sample, a loss function value is calculated. The loss function value is calculated based on the raindrop kinetic energy, ripple energy, wave equation, and rainfall difference corresponding to the rainfall prediction results. The rainfall difference is calculated based on the rainfall prediction results and rainfall labels. Determine whether the loss function value meets the preset training stopping condition; If the conditions are not met, the model parameters of the rainfall monitoring model to be trained are adjusted, and the rainfall monitoring model is trained again using the rainfall monitoring training set until the preset training stop condition is met, and the trained rainfall monitoring model is obtained. Features are extracted from the raindrop optical flow map, the water wave spectrum map, and the interferometric energy thermogram using a rainfall monitoring model, and then fused to obtain multimodal fused features. Rainfall monitoring results are obtained by using the multimodal fusion features through the rainfall monitoring model. The rainfall monitoring model is trained based on the energy transfer and wave transfer laws during the process of raindrops hitting the water surface.

2. The method according to claim 1, characterized in that, The extraction of raindrop optical flow maps from video data of rainfall scenes includes: For the image frame sequence corresponding to the video data, extract the raindrop features corresponding to each frame to obtain a raindrop feature map sequence; The motion velocity of raindrops is calculated using an optical flow algorithm between adjacent raindrop feature maps in the raindrop feature map sequence. The optical flow map of the raindrops is generated based on their movement speed.

3. The method according to claim 1, characterized in that, The extraction of water wave spectrograms from video data of rainfall scenes includes: Perform a Fourier transform on the water wave image sequence to obtain the frequency and amplitude characteristics of the water waves; A water wave spectrum is generated based on the frequency and amplitude characteristics.

4. The method according to claim 1, characterized in that, Before extracting raindrop optical flow maps, water wave spectrograms, and interferometric energy thermograms from video data of rainfall scenes, the following steps are also included: The system acquires video data collected by a camera device equipped with a polarizing filter to suppress reflections from the water surface. The video data is adjusted using motion estimation and image registration techniques to obtain adjusted video data.

5. A rainfall monitoring device, characterized in that, The device includes: The extraction module is used to extract raindrop optical flow maps, water wave spectrum maps, and interferometric energy heat maps from video data of rainfall scenes; The extraction module is specifically used for: For the image frame sequence corresponding to the video data, the water wave region corresponding to each frame image is extracted to obtain the water wave image sequence; for each frame of water wave image, the spatial modulation degree of the interference fringes is calculated, which is calculated based on the maximum and minimum values ​​of pixels within a local window of each frame of water wave image; the spatial modulation degree is converted into an image representing the spatial distribution of energy to generate the interference energy heatmap; The first acquisition module is used to acquire rainfall monitoring training set data, which includes multiple rainfall monitoring training samples and corresponding rainfall labels. Each rainfall monitoring training sample includes: raindrop optical flow map sample, water wave spectrum map sample and interferometric energy heat map sample. For each of the rainfall monitoring training samples, perform the following steps: The input module is used to input the rainfall monitoring training sample into the rainfall monitoring model to be trained, and obtain the rainfall prediction result of the training sample. The calculation module is used to calculate the loss function value based on the rainfall prediction result of the training sample. The loss function value is calculated based on the raindrop kinetic energy, ripple energy, wave equation and rainfall difference corresponding to the rainfall prediction result. The rainfall difference is calculated based on the rainfall prediction result and the rainfall label. The judgment module is used to determine whether the loss function value meets the preset training stopping condition; The training module is used to adjust the model parameters of the rainfall monitoring model to be trained when the conditions are not met, and to continue training the rainfall monitoring model using the rainfall monitoring training set until the preset training stop condition is met, so as to obtain the trained rainfall monitoring model. The feature fusion module is used to extract features from the raindrop optical flow map, the water wave spectrum map and the interferometric energy thermogram through the rainfall monitoring model and perform feature fusion to obtain multimodal fused features; The prediction module is used to monitor rainfall using the multimodal fusion features through the rainfall monitoring model to obtain rainfall monitoring results. The rainfall monitoring model is trained based on the energy transfer and wave transfer laws during the process of raindrops hitting the water surface.

6. An electronic device, characterized in that, The device includes: a processor and a memory storing computer program instructions; the processor reads and executes the computer program instructions to implement a rainfall monitoring method as described in any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement a rainfall monitoring method as described in any one of claims 1-4.

8. A computer program product, characterized in that, When the instructions in the computer program product are executed by the processor of the electronic device, the electronic device performs a rainfall monitoring method as described in any one of claims 1-4.

Images