Video real-time rainfall inversion method and system

Abstract

The present application relates to the technical field of meteorological measurement and computer vision, and particularly relates to a video real-time rainfall inversion method and system. Through a fast target detection backbone network improved based on YOLOv11, a dynamic target area is positioned and segmented in real time; a multi-scale raindrop feature extraction subnetwork adopts a transpose multi-head attention mechanism MDTA to capture the spatiotemporal features of rain stripes at different distances; based on a cross attention module, the detected target mask is used for raindrop feature enhancement; and a time series rainfall estimator based on a Transformer fuses multi-frame information and a Gamma raindrop spectrum distribution model to output rainfall. The present application can realize effective suppression of dynamic interference and stable inversion of real-time rainfall by using existing monitoring videos, and is suitable for fields such as urban meteorological monitoring network construction, rainfall detection and urban waterlogging early warning.

Description

Technical Field

[0001] This invention relates to the field of meteorological measurement and computer vision interdisciplinary technology, and in particular to a method and system for real-time video rainfall inversion. Background Technology

[0002] Rainfall measurement is a fundamental data source for meteorological forecasting, hydrological simulation, disaster prevention and mitigation, and refined urban management. Currently, the mainstream methods for observing rainfall mainly rely on traditional point instruments and remote sensing. Traditional point instruments, such as tipping bucket rain gauges and weighing rain gauges, directly measure rainfall through mechanical or weighing methods, offering advantages such as high standardization and reliable accuracy. However, they are limited by sparse equipment distribution, high maintenance costs, and susceptibility to interference from environmental factors such as terrain and wind fields, failing to effectively capture the spatial heterogeneity of rainfall, especially exhibiting significant coverage blind spots in complex urban terrain and localized rainstorms. Remote sensing methods include weather radar, satellite remote sensing, and laser raindrop spectrometers. Among these, radar can achieve regional scanning, satellites provide wide-area coverage, and laser raindrop spectrometers can accurately acquire raindrop spectral distribution. However, these methods generally face problems such as high equipment costs, complex maintenance, and limited spatiotemporal resolution. Furthermore, radar is susceptible to obstruction by ground objects and echo interference, and laser equipment is difficult to widely adopt.

[0003] In recent years, with the widespread deployment of surveillance cameras and the rapid development of computer vision technology, using visual technology to retrieve rainfall has become a highly promising low-cost supplementary observation method. The main advantages of this method are: it fully utilizes existing massive amounts of surveillance data, eliminating the need for additional site construction and achieving high-density coverage of key areas such as urban roads and intersections; it features high video frame rates and fine spatial resolution, enabling the construction of high spatiotemporal resolution rainfall field monitoring through multi-camera networking; and its processing flow supports real-time video stream access with update cycles down to the second level, making it suitable for dynamic early warning scenarios.

[0004] Therefore, it is necessary to provide a new method for real-time rainfall inversion from video, which can effectively suppress dynamic interference, accurately capture multi-scale rain stripes, and achieve real-time rainfall inversion under physical constraints. Summary of the Invention

[0005] The main objective of this invention is to provide a method and system for real-time video rainfall inversion. Based on a YOLOv11-Transformer deep learning hybrid architecture, this method is particularly suitable for achieving high spatiotemporal resolution and real-time quantitative rainfall inversion using surveillance video. It fully utilizes existing surveillance video resources and, through deep fusion of deep learning and physical models, achieves accurate extraction of rain streaks in dynamic scenes, capture of multi-scale spatiotemporal features, temporal continuity modeling, and high-precision quantitative inversion of rainfall intensity, thereby solving at least one of the technical problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides a method for real-time video rainfall inversion, comprising the following steps: S1, Fast Target Detection Based on YOLOv11 Improvement: Input the sequence of dynamic scene monitoring video frames to be analyzed. Utilize the fast target detection backbone network based on YOLOv11 improvement to perform real-time target detection and instance segmentation on each frame. Locate and segment the dynamic interference target region corresponding to the moving object in the video frame, as well as the non-dynamic background and rain region. Generate the corresponding dynamic target binary mask M, which is used to suppress non-rain interference in subsequent steps. Among them, the dynamic target region is marked as 1, and the non-dynamic background and rain region is marked as 0. S2, Multi-scale raindrop feature extraction: Construct a multi-scale raindrop feature extraction sub-network, input the video frames processed in step S1 into this sub-network, use the transposed multi-head attention mechanism MDTA to calculate and capture the spatiotemporal features of rain stripes at different distances and scales, and output multi-scale raindrop feature maps; S3, Raindrop Feature Enhancement Based on Cross-Attention: A cross-attention module is introduced, using the raindrop feature map output in step S2 as the query, and the mask condition features of the dynamic target mask M generated in step S1 after feature extraction as the key K and value V, to perform cross-attention calculation, thereby achieving targeted enhancement of rain stripe features and further suppression of dynamic interference noise, resulting in an enhanced rain feature map. S4, Transformer-based temporal rainfall estimation: Establish a Transformer-based temporal rainfall estimator, perform sequential modeling of the enhanced rain feature map obtained in step S3 in the time dimension, fuse multi-frame continuous information, utilize a fully connected regression head and a Gamma raindrop spectrum distribution model, and output the real-time rainfall intensity R, where R is in mm / h. Among them, the raindrop diameter is calculated based on the length l of the rain streaks on the image plane, the camera focal length f, the position z of the camera focal plane, and the exposure time T, according to the perspective imaging relationship. : ; The Gamma raindrop spectral distribution model is as follows: In the formula, This represents the number of raindrops within a unit volume with a diameter of d. Here, μ is the raindrop spectral intercept parameter, and μ is the shape parameter. The slope parameter of the raindrop spectrum; The rainfall intensity R was calculated by fitting the Gamma raindrop spectral distribution model. In the formula, Γ(·) is the gamma function, for , , The parameters are fitted with moment quantities to calculate the real-time rainfall intensity R.

[0007] Preferably, the improved fast target detection backbone network of YOLOv11 includes: using the C3k2 module to optimize feature extraction efficiency, supporting anchor-free detection head and instance segmentation branch, and outputting dynamic target bounding boxes and pixel-level segmented dynamic interference target regions corresponding to moving objects in the video frame, as well as non-dynamic background and rain regions.

[0008] Preferably, the calculation process of the transposed multi-head attention mechanism (MDTA) is as follows: after applying layer normalization to the feature map, 1×1 convolution and 3×3 depth convolution are used to generate query Q, key K, and value V, and transposed multi-head attention operation is performed to capture the long-distance dependence and multi-scale spatiotemporal features of rain stripes.

[0009] Preferably, step S3 includes: Raindrop feature map As a query; The mask features obtained by convolutional projection of the dynamic target mask M are used as the key K and value V; Calculate cross attention weights ;in, The dimension of the key; Output enhanced rain feature map This enables the suppression of dynamic interference and semantic enhancement of rain features.

[0010] Preferably, step S4 includes: processing the enhanced rain feature maps from multiple frames. The model inputs a TransformerEncoder to perform self-attention modeling to capture the temporal continuity of rain streaks. Then, the model parameters are optimized through backpropagation and a fully connected regression head to directly output the rainfall intensity R.

[0011] Preferably, the method also includes a preprocessing step, in which the dynamic scene monitoring video to be analyzed is classified and predicted for rainfall before input. Subsequent steps are only performed on video sequences that are judged to have rain, so as to avoid invalid calculations and erroneous inversions in sunny scenes.

[0012] Preferably, it also includes the steps of network training and optimization: The training set was composed of a synthetic rain dataset and a dataset consisting of real surveillance videos. During training, the loss function is based on the mean squared error of the main regression task. To optimize the gap between predicted rainfall intensity and actual values; The optimizer chosen is AdamW, and the initial learning rate is set to... The batch size is 16, and the training lasts for 100 epochs. Training will automatically stop when the validation set loss stops decreasing for a number of consecutive iterations.

[0013] The present invention also provides a video real-time rainfall inversion system, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the video real-time rainfall inversion method described in any of the above claims.

[0014] This application proposes a method and system for real-time rainfall inversion from video. It utilizes a fast target detection backbone network based on an improved YOLOv11 to locate and segment dynamic target regions in real time; a multi-scale raindrop feature extraction sub-network employs a transposed multi-head attention mechanism (MDTA) to capture the spatiotemporal features of rain streaks at different distances; a cross-attention module is used to enhance raindrop features using detected target masks; and a Transformer-based temporal rainfall estimator fuses multi-frame information with a Gamma raindrop spectrum distribution model to output rainfall. This invention can effectively suppress dynamic interference and stably invert real-time rainfall using existing surveillance videos, making it applicable to fields such as urban meteorological monitoring network construction, rainfall detection, and urban flooding early warning. This invention has at least the following advantages: (1) Effective suppression of dynamic interference is achieved through YOLOv11 dynamic masking; (2) MDTA multi-scale extraction combined with Transformer temporal modeling significantly improves the robustness of rain stripe features; (3) The inclusion of the Gamma model ensures the interpretability and accuracy of the inversion; (4) The method has strong real-time performance, high computational efficiency, and is suitable for all-weather video monitoring scenarios. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of a video real-time rainfall inversion method provided in this embodiment.

[0016] Figure 2 This is a schematic diagram of the hardware structure of a video real-time rainfall inversion system provided in this embodiment. Detailed Implementation

[0017] To better explain and facilitate understanding of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0018] like Figure 1 As shown in the figure, this embodiment provides a method for real-time rainfall inversion from video, which includes steps S1 to S4.

[0019] Preferably, a preprocessing step is performed on the dynamic scene monitoring video frame sequence. Before inputting the dynamic scene monitoring video to be analyzed, a rainfall classification prediction is performed on the dynamic scene monitoring video to be analyzed. Subsequent steps are only performed on video sequences that are judged to have rain, so as to avoid invalid calculations and erroneous inversions in sunny scenes.

[0020] Specifically, to avoid invalid calculations due to sunny or rainless videos, rainfall classification and prediction are performed before inputting the video frame sequence. This involves a combination of image processing and statistical features. First, background subtraction is performed on multiple consecutive frames (e.g., 5-10 frames) to extract foreground variation regions. Then, histograms of grayscale variation amplitude and direction are statistically analyzed within these foreground variation regions. Finally, an empirical threshold is used to determine whether rain streaks appear. If the proportion of regions meeting the criteria in consecutive frames is below the threshold, the sequence is classified as rainless, and subsequent steps are only performed on video sequences classified as rainy.

[0021] S1, Fast Target Detection Based on YOLOv11 Improvement: Input the sequence of dynamic scene monitoring video frames to be analyzed, and use the fast target detection backbone network based on YOLOv11 improvement to perform real-time target detection and instance segmentation on each frame. Locate and segment the dynamic interference target region corresponding to the moving object in the video picture as well as the non-dynamic background and rain region, and generate the corresponding dynamic target binary mask M. The dynamic target binary mask M is used to suppress non-rain interference in the subsequent process. Among them, the dynamic target region is marked as 1, and the non-dynamic background and rain region is marked as 0.

[0022] Preferably, the YOLOv11 improved fast target detection backbone network includes: using the C3k2 module to optimize feature extraction efficiency, supporting anchor-free detection head and instance segmentation branch, outputting dynamic target bounding boxes and pixel-level segmented dynamic interference target regions corresponding to moving objects in the video frame as well as non-dynamic background and rain regions in real time, and generating dynamic target binary mask M.

[0023] Specifically, for dynamic scene monitoring video frame sequences, a modified YOLOv11 backbone network is used for real-time object detection and instance segmentation. This network uses the C3k2 module to optimize feature extraction efficiency, while combining SPPF spatial pooling and C2PSA attention mechanisms to enhance channel-space interaction. In the Neck part, multi-scale features are fused through a Concat operation. The Head uses an anchor-free detection head and an instance segmentation branch, directly outputting dynamic target bounding boxes and pixel-level segmentation of dynamic interference target regions corresponding to moving objects in the video frame, as well as non-dynamic background and rain regions. Then, a corresponding dynamic target binary mask M is generated. The mask M is a binary image, with dynamic target regions (vehicles, pedestrians, umbrellas, etc.) marked as 1, and non-dynamic background and rain regions marked as 0. The improvements of YOLOv11 include the use of the C3k2 module to optimize feature extraction efficiency, support for an anchor-free detection head and instance segmentation branch, and the ability to output pixel-level masks in real-time in high frame rate video streams. The purpose of this step is to "exclude interference" for subsequent rain feature extraction, preventing the edges or trajectories of dynamic objects from being misidentified as rain stripes, thereby improving the overall robustness of the inversion. This step has a short processing time per frame, meeting the requirements of real-time video processing. The output mask M serves as the conditional input for the subsequent cross-attention module, used to suppress dynamic interference.

[0024] S2, Multi-scale raindrop feature extraction: Construct a multi-scale raindrop feature extraction sub-network, input the video frames processed in step S1 into this sub-network, use the transposed multi-head attention mechanism MDTA to calculate and capture the spatiotemporal features of rain stripes at different distances and scales, and output multi-scale raindrop feature maps.

[0025] Preferably, the MDTA module originates from an efficient image restoration network, and its core is to achieve local and non-local pixel interaction through multi-head transposed attention. The calculation process of the transposed multi-head attention mechanism MDTA is as follows: after applying layer normalization to the feature map, 1×1 convolution and 3×3 depth convolution are used to generate query Q, key K, and value V, and transposed multi-head attention operation is performed to capture the long-distance dependence and multi-scale spatiotemporal features of rain stripes. This step solves the problem of uneven feature extraction of raindrops at different distances using traditional methods.

[0026] S3, Raindrop Feature Enhancement Based on Cross-Attention: A cross-attention module is introduced. The raindrop feature map output in step S2 is used as the query, and the mask condition features of the dynamic target mask M generated in step S1 after feature extraction are used as the key K and value V. Cross-attention calculation is performed to achieve targeted enhancement of rain stripe features and further suppression of dynamic interference noise, resulting in an enhanced rain feature map. .

[0027] Preferably, step S3 includes: Raindrop feature map As a query; The mask features obtained by convolutional projection of the dynamic target mask M are used as the key K and value V; Calculate cross attention weights ;in, The dimension of the key; Output enhanced rain feature map This enables the suppression of dynamic interference and semantic enhancement of rain features.

[0028] This module utilizes the dynamic mask provided by YOLOv11 as a "domain knowledge" condition to suppress the attention weights of motion-interference regions, while enhancing the semantic expression of real rain stripes and improving rain feature extraction in complex dynamic scenes.

[0029] S4, Transformer-based temporal rainfall estimation: A Transformer-based temporal rainfall estimator is established. The enhanced rainfall feature map obtained in step S3 is used for sequential modeling in the time dimension. Multi-frame continuous information is fused, and a fully connected regression head and a Gamma raindrop spectral distribution model are used to output the real-time rainfall intensity R, where R is in mm / h. Specifically, the enhanced rainfall feature map obtained in step S3... A sequence is constructed along the time dimension and input into a Transformer Encoder (e.g., 6 layers, 8 attention heads, 512 hidden dimensions) for self-attention modeling to capture the temporal continuity of rain streaks. After the Encoder outputs aggregated features, the real-time rainfall intensity is directly output through a fully connected regression head (e.g., two-layer MLP, 256 hidden dimensions, ReLU activation, and finally linear output).

[0030] Among them, based on the length of the rain stripes on the image plane l Camera focal length f Camera focal plane position z and exposure time T Calculate the diameter of raindrops based on perspective imaging relationships. : .

[0031] Among them, formula The process of pushing down and tracing the origin is as follows: pass , You can get , The unit is ; And the diameter of the raindrops This can be expressed as, , Pick (Reference: Chen Zan et al., Chen Zan, Xu Jianhua, Liang Zhuoran et al. Rainfall inversion based on social surveillance video [J]. Optoelectronics / Laser, 2023, 34(11): 1142–1149. DOI: 10.16136 / j.joel.2023.11.0476.)

[0032] in, The unit is mm. The unit is m / s. The unit is s; and Use the same unit of length (e.g., pixels or mm for all). It is the distance from the lens to the focal area where the raindrop is located, measured in meters (m); therefore The unit is m / s, divide by the unit. Later obtained Take again The result of raising it to the power of 2 is mm, which is consistent with the dimensions.

[0033] The Gamma raindrop spectral distribution model is as follows: In the formula, The inner diameter per unit volume is d The number of raindrops within the range, For raindrop spectral intercept parameters, μ For shape parameters, The slope parameter of the raindrop spectrum; The rainfall intensity R was calculated by fitting the Gamma raindrop spectral distribution model. In the formula, Γ(·) is the gamma function, for , , The parameters are fitted with moment quantities to calculate the real-time rainfall intensity R.

[0034] Preferably, step S4 includes: processing the enhanced rain feature maps from multiple frames. The model inputs a TransformerEncoder to perform self-attention modeling to capture the temporal continuity of rain streaks. Then, the model parameters are optimized through backpropagation and a fully connected regression head to directly output the rainfall intensity R.

[0035] The calculation principle is as follows: Let the raindrop spectrum be: ; in The unit is , for , The unit follows Change, for .

[0036] Rainfall intensity is calculated by integrating the formula "number of raindrops per unit volume × volume of a single droplet × falling velocity" and converting it to mm / h. ; Lee et al. (LEE J, BYUN J, BAIK J, et al. Estimation of raindrop size distribution and rain rate with infrared surveillance camera in dark conditions[J]. Atmospheric Measurement Techniques, 2023, 16: 707–725. DOI:10.5194 / amt-16-707-2023.) gave the equivalent volumetric integral form of raindrop intensity in equation (17). Substituting... By integrating Gamma, we can obtain: ; Therefore, 0.0071 is an approximation of the above-mentioned geometric, time, and length unit conversion factors. Dong et al. (DONG R, LIAO J, LI B, et al. Measurements of rainfall rates from videos[C] / / 2017 10th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI). IEEE, 2017: 1–9. DOI: 10.1109 / CISP-BMEI.2017.8302066.) directly listed the corresponding results in equations (27)-(31).

[0037] for , , These data points need to be obtained by combining the raindrop diameter and number concentration within a specific time window during the observation process. The calculation method is as follows: ; This is the actual number of raindrops identified. It is the effective sampling volume after processing by field of view, depth of field, time window, and repetition count. It refers to the particle size range.

[0038] Then let From the above formula, we can deduce that , , .

[0039] Preferably, it also includes the steps of network training and optimization: The training set was composed of a synthetic rain dataset and a dataset consisting of real surveillance videos. During training, the loss function is based primarily on the mean squared error of the main regression task. To optimize the gap between predicted rainfall intensity and actual values; The optimizer chosen is AdamW, and the initial learning rate is set to... The batch size is 16, and the training lasts for 100 epochs. To prevent overfitting, an early stopping strategy is introduced, which automatically stops training when the validation set loss stops decreasing for a certain number of consecutive iterations.

[0040] In addition, the rainfall classification model in the preprocessing stage can be trained independently to ensure its classification performance is stable and reliable.

[0041] In surveillance video scenarios, this method can effectively suppress dynamic interference from vehicles, pedestrians, etc., achieve accurate extraction of rain stripe features and temporal continuity modeling, and finally output stable and real-time rainfall intensity.

[0042] like Figure 2 As shown, this application also provides a video real-time rainfall inversion system, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of a video real-time rainfall inversion method as described in any of the above claims.

[0043] Figure 2 This is a schematic diagram of the hardware structure for running a real-time video rainfall inversion method according to an embodiment of the present invention. Figure 2 As shown, this embodiment / computer 6 includes: a processor 60, a memory 61, and a computer program 62 stored in the memory 61 and executable on the processor 60, such as a program for running a video real-time rainfall inversion method. When the processor 60 executes the computer program 62, it implements the steps in the various embodiments described above for running a video real-time rainfall inversion method. Alternatively, when the processor 60 executes the computer program 62, it implements the functions of each module / unit in the various device embodiments described above.

[0044] For example, the computer program 62 may be divided into one or more modules / units, which are stored in the memory 61 and executed by the processor 60 to complete the present invention. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program 62 in the computer 6.

[0045] The computer 6 can be a desktop computer, laptop, handheld computer, cloud server, or other computing device. The computer 6 may include, but is not limited to, a processor 60 and a memory 61. Those skilled in the art will understand that... Figure 2 This is merely an example of computer 6 and does not constitute a limitation on computer 6. It may include more or fewer components than shown, or combine certain components, or different components. For example, computer 6 may also include input / output devices, network access devices, buses, etc.

[0046] The processor 60 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0047] The memory 61 can be an internal storage unit of the computer 6, such as a hard drive or memory. The memory 61 can also be an external storage device of the computer 6, such as a plug-in hard drive, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the terminal device. Furthermore, the memory 61 can include both internal storage units and external storage devices of the computer 6. The memory 61 is used to store the computer program and other programs and data required by the terminal device. The memory 61 can also be used to temporarily store data that has been output or will be output.

[0048] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0049] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0050] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0051] In the embodiments provided by this invention, it should be understood that the disclosed apparatus / terminal devices and methods can be implemented in other ways. For example, the apparatus / terminal device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0052] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0053] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0054] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0055] The above are merely specific application examples of the present invention and do not constitute any limitation on the scope of protection of the present invention. In addition to the above embodiments, the present invention may have other implementations. All technical solutions formed by equivalent substitution or equivalent transformation fall within the scope of protection claimed by the present invention.

Claims

1. A method for real-time rainfall inversion from video, characterized in that, Includes the following steps: S1, Fast Target Detection Based on YOLOv11 Improvement: Input the sequence of dynamic scene monitoring video frames to be analyzed. Utilize the fast target detection backbone network based on YOLOv11 improvement to perform real-time target detection and instance segmentation on each frame. Locate and segment the dynamic interference target region corresponding to the moving object in the video frame, as well as the non-dynamic background and rain region. Generate the corresponding dynamic target binary mask M, which is used to suppress non-rain interference in subsequent steps. Among them, the dynamic target region is marked as 1, and the non-dynamic background and rain region is marked as 0. S2, Multi-scale raindrop feature extraction: Construct a multi-scale raindrop feature extraction sub-network, input the video frames processed in step S1 into this sub-network, use the transposed multi-head attention mechanism MDTA to calculate and capture the spatiotemporal features of rain stripes at different distances and scales, and output multi-scale raindrop feature maps; S3, Raindrop Feature Enhancement Based on Cross-Attention: A cross-attention module is introduced, using the raindrop feature map output in step S2 as the query, and the mask condition features of the dynamic target mask M generated in step S1 after feature extraction as the key K and value V, to perform cross-attention calculation, thereby achieving targeted enhancement of rain stripe features and further suppression of dynamic interference noise, resulting in an enhanced rain feature map. S4, Transformer-based temporal rainfall estimation: Establish a Transformer-based temporal rainfall estimator, perform sequential modeling of the enhanced rain feature map obtained in step S3 in the time dimension, fuse multi-frame continuous information, utilize a fully connected regression head and a Gamma raindrop spectrum distribution model, and output the real-time rainfall intensity R, where R is in mm / h. Among them, based on the length of the rain stripes on the image plane l Camera focal length f Camera focal plane position z and exposure time T Calculate the diameter of raindrops based on perspective imaging relationships. : ; The Gamma raindrop spectral distribution model is as follows: In the formula, The inner diameter per unit volume is d The number of raindrops within the range, For raindrop spectral intercept parameters, μ For shape parameters, The slope parameter of the raindrop spectrum; The rainfall intensity R was calculated by fitting the Gamma raindrop spectral distribution model. In the formula, Γ(·) is the gamma function, for , , The parameters are fitted with moment quantities to calculate the real-time rainfall intensity R.

2. The method for real-time rainfall inversion based on video according to claim 1, characterized in that, The improved fast target detection backbone network of YOLOv11 includes: using the C3k2 module to optimize feature extraction efficiency, supporting anchor-free detection head and instance segmentation branch, and outputting dynamic target bounding boxes and pixel-level segmented dynamic interference target regions corresponding to moving objects in video frames, as well as non-dynamic background and rain regions in real time.

3. The method for real-time rainfall inversion based on video according to claim 2, characterized in that, The calculation process of the transposed multi-head attention mechanism (MDTA) is as follows: after applying layer normalization to the feature map, 1×1 convolution and 3×3 depth convolution are used to generate query Q, key K, and value V, and transposed multi-head attention operation is performed to capture the long-distance dependence and multi-scale spatiotemporal features of rain stripes.

4. The method for real-time rainfall inversion based on video according to claim 3, characterized in that, Step S3 includes: Raindrop feature map As a query; The mask features obtained by convolutional projection of the dynamic target mask M are used as the key K and value V; Calculate cross attention weights ;in, The dimension of the key; Output enhanced rain feature map This enables the suppression of dynamic interference and semantic enhancement of rain features.

5. The method for real-time rainfall inversion based on video according to claim 4, characterized in that, Step S4 includes: processing the enhanced rain feature maps from multiple frames. The model input is a Transformer Encoder that performs self-attention modeling to capture the temporal continuity of rain streaks. Then, the model parameters are optimized through backpropagation and a fully connected regression head, and the rainfall intensity R is directly output.

6. A method for real-time rainfall inversion based on video according to any one of claims 1-5, characterized in that, It also includes a preprocessing step, which performs rainfall classification and prediction on the dynamic scene monitoring video to be analyzed before inputting the video. Subsequent steps are only performed on video sequences that are identified as having rain, in order to avoid invalid calculations and erroneous inversions in sunny scenes.

7. A method for real-time rainfall inversion based on video according to any one of claims 1-5, characterized in that, It also includes steps, network training and optimization: The training set was composed of a synthetic rain dataset and a dataset consisting of real surveillance videos. During training, the loss function is based on the mean squared error of the main regression task. To optimize the gap between predicted rainfall intensity and actual values; The optimizer chosen is AdamW, and the initial learning rate is set to... The batch size is 16, and the training lasts for 100 epochs. Training will automatically stop when the validation set loss stops decreasing for a number of consecutive iterations.

8. A video real-time rainfall inversion system, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of a video real-time rainfall inversion method as described in any one of claims 1 to 7.

Images