Video waterlogging identification simulation method, device and system and storage medium

Abstract

The invention relates to the technical field of image processing, and discloses a video waterlogging identification simulation method, device and system and a storage medium, and the method comprises the steps: obtaining a plurality of pictures containing waterlogging, carrying out the waterlogging contour marking of each picture, making a marking data set, and dividing the marking data set into a training set and a test set; improving the RTMDet to construct a waterlogging detection model; inputting the training set into a waterlogging detection model for training to obtain a weight file of the model; deploying the trained waterlogging detection model into a reasoning framework; detecting frames of the urban camera monitoring video by using the obtained weight file, and correcting a detection result of a single frame in combination with real-time rainfall information; and judging whether waterlogging occurs in the corresponding video point based on the number of the frames of which waterlogging is detected. According to the method, the RTMDet model is improved, automatic station rainfall data correction and model reasoning acceleration are combined, and a waterlogging detection model integrating accuracy, real-time performance and practicability is constructed.

Description

Technical Field

[0001] The present invention relates to the technical field of smart cities, computer vision and instance segmentation, and in particular to a video waterlogging recognition simulation method, device, system and storage medium. Background Art

[0002] As global climate change continues to intensify and extreme weather events occur frequently, waterlogging issues have become more urgent and complex in urban management. Traditional methods use water level sensors to detect the depth of water accumulation, but the installation cost of sensors is high, construction and maintenance are difficult, and in actual use, they are easily damaged by the environment. Therefore, seeking more economical and efficient solutions has become the top priority of current urban waterlogging prevention and control work.

[0003] In recent years, the application of deep learning in waterlogging detection has gradually increased. It is faster and more convenient to use deep learning models to identify waterlogging in urban surveillance videos. However, existing models still have certain limitations in waterlogging detection tasks. For example, semantic segmentation models are prone to confuse ground wetness and waterlogging, resulting in a high false alarm rate. The image classification model detects too little information and cannot distinguish between small puddles and waterlogging based on the area of ​​the waterlogged area. Some instance segmentation models are still insufficient in detection accuracy and real-time performance. In addition, the detection results of a single image may be wrong. Making judgments based on the detection results of multiple consecutive images can reduce errors. Video-based waterlogging detection can provide higher accuracy than single-image-based waterlogging detection. Summary of the invention

[0004] In order to solve the above problems, the present invention provides a video waterlogging recognition simulation method, device, system and storage medium to effectively improve the accuracy of waterlogging detection, while ensuring a low false alarm rate of the model, reduce the confusion between ground wetness and waterlogging, thereby reducing the false alarm rate. In addition, the waterlogging detection efficiency is greatly improved.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0006] According to a first embodiment of the present invention, a video waterlogging recognition simulation method is provided, the method comprising:

[0007] Acquire multiple pictures containing waterlogging, annotate the waterlogging contours of each picture, create an annotated data set, and divide the annotated data set into a training set and a test set;

[0008] Improve RTMDet to build a waterlogging detection model; the improvements to RTMDet include: fusing the C2 layer features of the backbone network to the C3 layer features after using the spatial attention mechanism; introducing MGD distillation learning, and using RTMDet-x as a teacher model to guide the training of the RTMDet-m student model;

[0009] Input the training set into the waterlogging detection model training to obtain the model weight file;

[0010] Deploy the trained waterlogging detection model to the inference framework;

[0011] The obtained weight file is used to detect the frames of the urban camera surveillance video, and the detection results of the single frame are corrected in combination with the real-time rainfall information;

[0012] Based on the number of frames in which waterlogging is detected, it is determined whether waterlogging occurs at the corresponding video point.

[0013] Furthermore, a plurality of pictures containing waterlogging are obtained, and after waterlogging contours are annotated for each picture, annotated data sets are prepared, and the annotated data sets are divided into a training set and a test set, including:

[0014] Based on multiple pictures containing waterlogging, the waterlogged areas in the pictures are annotated to obtain an annotation file, which contains the category, position and contour information of the target in the image; the annotation file and the corresponding original picture file are divided into a training set and a test set according to a set ratio, and then the data set is converted into a unified format.

[0015] Furthermore, RTMDet is improved to build a waterlogging detection model, including:

[0016] Extract the C2 layer features that were not used originally from the backbone network CSPNeXt of RTMDet, and apply the spatial attention mechanism to the C2 layer features; the spatial attention mechanism includes: grouping the feature maps, using GroupNorm to perform spatial attention processing on the grouped feature maps, and re-merging the processed feature maps and fusing them with the C3 layer features;

[0017] The MGD distillation learning method is introduced to further construct the waterlogging detection model. In the distillation process, the RTMDet-m model with small parameters and computational complexity is used as the student model, and the RTMDet-x model with large parameters and computational complexity is used as the teacher model. Some pixels in the neck layer feature map of the RTMDet-m model are randomly selected for masking, and a simple module consisting of a convolutional layer and an activation layer is used to process the masked features. The mean square error between the processed student features and the neck layer feature map of the RTMDet-x model is calculated, and the error is multiplied by a coefficient and added to the total loss function to guide the student model to generate feature representations similar to those of the teacher model.

[0018] Furthermore, the training set is input into the waterlogging detection model training to obtain the model weight file, including:

[0019] The input image is resized to the specified size, and data enhancement is performed using random scale transformation, random cropping, color space enhancement, random horizontal flipping, Mosaic, and Mixup.

[0020] Load the training set;

[0021] Set the number of training times and learning rate parameters to start training the model;

[0022] Calculate the classification loss, detection box loss, mask loss and distillation loss and perform back propagation to update the weights. After the training is completed, store the final model weights to obtain the weight file; the loss function is expressed as:

[0023] Loss = L cls +L bbox +L mask +αL distill

[0024]

[0025] In the formula, Loss represents the loss value, L cls represents the classification loss, L bbox represents the detection box regression loss, L mask represents the pixel segmentation mask loss; i is the index of the anchor box, p i * Indicates whether the i-th anchor frame contains a target. When the i-th anchor frame contains a target, p i * Equal to 1; when the i-th anchor frame is the background, p i * is 0; p i represents the probability value of the target in the i-th anchor box; v i Represents the parameterized vector of the center point coordinates and width and height of the i-th anchor box prediction, v i * is the parameterized vector of the i-th label bounding box; n c is the number of classifications. Here there are only two categories: water accumulation and background. n ij is the number of pixels belonging to the i-th class and predicted to be the j-th class, n ii Represents the number of pixels that belong to the i-th class and are predicted to be the i-th class, that is, the number of correct predictions for the i-th class, n i Represents the total number of detection boxes predicted for the i-th type of damage; N cls 、N bbox are constants used for normalization; N level is the number of feature fusion layers used; t i is the neck feature of the teacher network, s i The neck part of the student network features; L mask represents the mask loss, Ldistill Represents distillation loss.

[0026] Furthermore, the trained waterlogging detection model is deployed into the inference framework, including:

[0027] The model weights are converted into onnx format, the Onnxruntime-gpu framework is deployed on the server, and the waterlogging detection model is detected under the Onnxruntime-gpu framework.

[0028] Furthermore, the obtained weight file is used to detect frames of urban camera surveillance video, and the detection results of single frames are corrected in combination with real-time rainfall information, including:

[0029] Based on the set frame rate, frames are extracted from the real-time video stream of the urban camera surveillance system, and the extracted frames are input into the trained model for waterlogging detection;

[0030] The location, contour information and confidence of the waterlogged area in the detection results are extracted, and the detections with confidence below the threshold are filtered out according to the set score threshold.

[0031] Calculate the ratio of the detected water area to the entire image area, remove small puddles with an area ratio less than 0.05, and load the remaining detection frames and the corresponding classification confidence scores and water mask masks onto the original image to obtain a single-frame detection result image;

[0032] The detection results are further screened and corrected using the rainfall data from the automatic station near the camera. If the 6-hour cumulative rainfall at the automatic station is less than 28 mm and waterlogging is detected in the image, it is determined that the model detection is incorrect and the erroneous results are removed.

[0033] Furthermore, judging whether waterlogging occurs at a corresponding video point based on the number of frames in which waterlogging is detected includes:

[0034] The frames extracted from the real-time video stream are quickly detected, and the number of frames in which waterlogging is detected is counted. If 4 frames out of 5 consecutive frames show waterlogging, it is determined that waterlogging has indeed occurred at the video point, and an early warning is issued.

[0035] According to a second technical solution of the present invention, a video waterlogging recognition simulation device is provided, the device comprising:

[0036] A data set construction module is configured to obtain a plurality of pictures containing waterlogging, annotate the waterlogging contour of each picture, create an annotated data set, and divide the annotated data set into a training set and a test set;

[0037] The model building module is configured to improve RTMDet to build a waterlogging detection model; wherein the improvements to RTMDet include: fusing the C2 layer features of the backbone network to the C3 layer features after using the spatial attention mechanism; introducing MGD distillation learning, and using RTMDet-x as a teacher model to guide the training of the RTMDet-m student model;

[0038] The model training module is configured to input the training set into the waterlogging detection model training to obtain the model weight file;

[0039] A model deployment module, configured to deploy the trained waterlogging detection model to the inference framework;

[0040] A result correction module is configured to detect frames of the urban camera surveillance video using the obtained weight file, and correct the detection result of a single frame in combination with the real-time rainfall information;

[0041] The waterlogging determination module is configured to determine whether waterlogging occurs at a corresponding video point based on the number of frames in which waterlogging is detected.

[0042] According to a third technical solution of the present invention, a video waterlogging recognition simulation system is provided, the system comprising: a memory for storing a computer program; and a processor for executing the computer program to implement the method described above.

[0043] According to a fourth technical solution of the present invention, a non-transitory computer-readable storage medium storing instructions is provided. When the instructions are executed by a processor, the method described above is executed.

[0044] The present invention has at least the following beneficial effects:

[0045] 1. The present invention determines whether waterlogging has occurred by detecting the video stream, thereby overcoming the problem that single picture detection is prone to detection errors and effectively reducing the misjudgment rate.

[0046] 2. The present invention retains the rich texture information of the underlying backbone network by introducing the C2 layer features that are not used in the original model, making it easier to detect water accumulation and improving the detection accuracy of the model. The features of the C2 layer are extracted through the attention mechanism and then superimposed on the C3 layer features, which does not increase the computational complexity of the subsequent modules of the model and has almost no effect on the reasoning speed of the model.

[0047] 3. By using the MGD distillation method, the model of the present invention maintains the original parameters, calculation amount and detection speed while achieving the accuracy of RTMDet-x, whose parameters and calculation amount are more than three times that of the original model.

[0048] 4. The present invention eliminates some erroneous detection results through model detection of waterlogged area proportion control and combined automatic station rainfall, which can effectively reduce the false alarm rate.

[0049] 5. The present invention further improves the detection speed by deploying the improved model into the reasoning framework, so that the method integrates accuracy and real-time performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 A flow chart of a video waterlogging recognition simulation method according to an embodiment of the present invention is shown.

[0051] Figure 2 A structural diagram of an improved RTMDet model according to an embodiment of the present invention is shown.

[0052] Figure 3 A structural diagram of the C2 layer attention mechanism according to an embodiment of the present invention is shown.

[0053] Figure 4 A structural diagram of MGD knowledge distillation according to an embodiment of the invention is shown.

[0054] Figure 5 A structural diagram of a video waterlogging recognition simulation device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0055] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0056] The specific implementation of the present invention is further described in detail below in conjunction with the drawings and examples.

[0057] The present invention provides a video waterlogging recognition simulation method, please refer to Figure 3 , is a flow chart of a video waterlogging recognition and simulation method, the video waterlogging recognition and simulation method includes the following steps S1 to S6, which are described in detail as follows.

[0058] S1. Select urban surveillance images with waterlogging, annotate the waterlogging contours in detail, and create a labeled dataset in COCO format, which is divided into a training set and a test set.

[0059] Step S1 specifically includes:

[0060] We selected about 10,000 pictures of waterlogging from the city's video surveillance system, and then used the Labelme annotation tool to annotate these pictures in detail.

[0061] Specifically, use the polygon annotation tool to circle the outlines of all the waterlogged areas in each image, and select "waterlogging" as the classification label in the category section. After the annotation is completed, save the generated annotation file in json format.

[0062] After the annotation is completed, all JSON files and the corresponding original image files are divided into training sets and test sets in a ratio of 9:1. Specifically, 90% of the data is randomly selected as the training set, and the remaining 10% of the data is used as the test set.

[0063] Merge all json files in the training set into a json file in COCO format. COCO format is a dataset format commonly used in object detection tasks, which can effectively organize and store images and their annotation information. Similarly, process the test set and merge all json files in it into a json file in COCO format. In this way, two COCO format dataset files are generated, one for model training and the other for model testing and verification. Such data preparation provides a solid foundation for subsequent model training and performance evaluation, ensuring the integrity and consistency of the data.

[0064] S2. Construct a waterlogging detection model based on the improved RTMDet.

[0065] like Figure 2As shown in the figure, the improved RTMDet model consists of a backbone network (Backbone), a feature fusion network (Neck) and a detection head (Head). The backbone network CSPNeXt extracts features of different scales (C1, C2, C3, C4, C5) of the input image (inut) through 5 stages, and then fuses the features of the C2 layer to the features of the C3 layer using the spatial attention mechanism, and selects the features of (C3, C4, C5) to enter the feature fusion network PAFPN through lateral connections to fuse features of different scales to obtain (P5, P4, P3). PAFPN includes a top-down path aggregation network for transferring high-level features to low-level layers and fusing adjacent layers, and a bottom-up feature pyramid network for transferring low-level features back to high-level layers and fusing features of different scales. Each layer is followed by a 3*3 convolution to eliminate the aliasing effect caused by upsampling. Then the detection head generates categories (cls), detection boxes (boxes) and masks (masks) based on feature maps of different scales. Convolution weights are shared between different layers, while BN is calculated independently to reduce the amount of calculation without reducing accuracy. Mask generation is implemented by the kernel prediction head and the mask feature head. The mask feature head consists of 4 convolutional layers and extracts 8-channel mask features from multiple layers of features. The kernel prediction head predicts a 169-dimensional vector for each instance, which is decomposed into 3 dynamic convolution kernels to generate the water accumulation target mask by interacting with the mask features and coordinate features.

[0066] The improvements of RTMDet include fusing the C2 layer features in the backbone network to the C3 layer features after processing them through the spatial attention mechanism, introducing MGD distillation learning, and using RTMDet-x as the teacher model to guide the training of the RTMDet-m student model.

[0067] In order to enhance the performance of the model in waterlogging detection, the present invention improves the backbone network of RTMDet. Specifically, the underlying C2 layer features are introduced in the feature extraction process, and processed through the spatial attention mechanism to merge them with the higher-level C3 layer features. The structural diagram is shown in Figure 3 As shown, the details are as follows:

[0068] During the model feature extraction process, the low-level C2 features of the backbone network retain rich detail information, including important texture features, which are crucial for accurately detecting waterlogged areas in waterlogging. The original RTMDet model only used higher-level C3, C4, and C5 features, which contain more advanced semantic information but lack sensitivity to detail information. To make up for this deficiency, the lower-level C2 features are introduced to utilize their rich detail information to enhance the model's detection capabilities.

[0069] Specifically, the unused C2 layer features are first extracted from the backbone network CSPNeXt of RTMDet. These C2 features contain a lot of low-level detail information, such as the texture and shape features of water. Then, the spatial attention mechanism is applied to these C2 features. The specific implementation of the spatial attention mechanism is as follows: first, the feature maps are grouped to reduce the amount of calculation, and then GroupNorm (GN) is used on the grouped feature maps for spatial attention processing. The feature maps processed by spatial attention can better highlight important detail information while suppressing irrelevant information. Finally, the processed C2 feature maps are re-merged and fused with the C3 layer features to integrate detail information and semantic information at a higher level.

[0070] This improved method has significant advantages. By introducing the C2 feature, the model can better capture the details of the waterlogged area in the waterlogging and improve the detection accuracy. At the same time, the computational complexity of the spatial attention mechanism is small, which will not significantly increase the computational burden of the model and has almost no effect on the reasoning speed of the model. Therefore, while maintaining efficient reasoning performance, the present invention can more accurately detect the waterlogged area, significantly improving the effect of waterlogging detection.

[0071] The introduction of MGD distillation learning is as follows:

[0072] In order to improve the performance of the RTMDet-m model without increasing computing resources and reducing detection speed, the MGD distillation learning method is introduced. As a student model, the RTMDet-m model has 250 million parameters and a computing power of 2 billion floating-point operations per second, which makes it efficient in an environment with limited computing resources, but also limits its feature representation ability. The present invention adopts the RTMDet-x model as a teacher model. The RTMDet-x model has 750 million parameters and a computing power of 6 billion floating-point operations per second, which can provide richer feature information and effectively guide the training of the RTMDet-m model.

[0073] like Figure 4 As shown, the specific distillation process is as follows:

[0074] First, the neck layer feature map of the RTMDet-m model is masked. The neck layer includes three feature layers: P5, P4, and P3. In the masking operation, a certain proportion of pixels in the feature map are randomly selected, and the values ​​of these pixels are set to zero or a very small value (such as 1e-6) to simulate information loss. The mask ratio is set to 10% to 30%. This operation introduces the perturbation of missing information, which enhances the robustness of the RTMDet-m model in the face of incomplete information, enabling it to learn and generalize better while maintaining computational efficiency.

[0075] Secondly, feature processing is performed. The masked feature map passes through a feature processing module, which includes a convolution layer and an activation layer. The convolution layer is used to extract the masked features, and the activation layer (such as ReLU) performs nonlinear transformation to repair information loss. The convolution layer is configured with a 3×3 convolution kernel and a step size of 1 to ensure fine feature extraction. The activation layer further enhances the expressiveness of the feature map and improves the information recovery effect, ensuring that the RTMDet-m model can effectively recover and utilize the masked feature information without increasing the amount of computation.

[0076] Then, the distillation loss is calculated. The processed feature map is compared with the neck layer feature map (P5_x, P4_x and P3_x) of the RTMDet-x model layer by layer, and the mean square error (MSE) is calculated. MSE is used to quantify the consistency between the RTMDet-m model and the RTMDet-x model in the feature map. The specific calculation formula is:

[0077]

[0078] Among them, x i and i are the feature values ​​of the student model and the teacher model at the i-th pixel position, respectively, and N is the total number of pixels in the feature map. The overall feature map loss is obtained by accumulating the MSE values ​​of each layer of the feature map.

[0079] Then, the overall feature map loss is multiplied by a scaling factor α and added to the total loss function. The scaling factor is set to 0.0000005 to balance the impact of feature map loss with other loss terms.

[0080] Through this mask generative distillation (MGD) method, the RTMDet-m model can fully utilize the feature information of the RTMDet-x model without increasing computing resources and reducing detection speed, thereby significantly improving the accuracy and robustness of the model.

[0081] S3. Input the training set into the waterlogging detection model training to obtain the model weight file.

[0082] First, perform data enhancement on the training set. Data labeling is a very tedious task, and model training often relies on a large amount of data. Data enhancement can be used to expand data and make limited data sets more useful. The data enhancement operations used in this model include:

[0083] Random scale transformation, random cropping, color space enhancement, random horizontal flipping: These data enhancement operations are randomly performed on the training set data with a probability of 0.5.

[0084] Mosaic: Randomly select four images from the training dataset and crop and scale them to fit the resolution of the target composite image. The processed four images are spliced ​​into a new composite image in a grid pattern, and the waterlogging annotation information in these images is mapped to the new image to ensure accurate annotation. The resulting composite image contains the target object and background information from the four original images, increasing the diversity and complexity of the training data.

[0085] Mixup: Randomly select two images and their corresponding labels from the training dataset. Randomly generate a mixing coefficient λ, which is used to determine the weighted ratio of the two images and labels. Then, calculate the new mixed image and label, generate a new image by linearly combining the two images according to the mixing coefficient λ, and weight the labels with the same ratio. Finally, add the generated mixed image and label as a new training sample to the training dataset.

[0086] After setting the parameters such as the number of training times and learning rate, start training the model, as follows:

[0087] First, train the RTMDet-x model. Set the number of training iterations to 60 epochs, batch_size to 8, and the initial learning rate to 0.000125. In the early stage of training, all the data enhancement methods mentioned are used. When the model is trained to 30 epochs, Mosaic and Mixup data enhancements are no longer used, and other enhancement methods are used as usual. The LinearLR method is used for the first 30 epochs of the learning rate adjustment; the Cosine Annealing LR algorithm is used for the next 30 epochs, and the minimum learning rate is set to the initial learning rate multiplied by 0.05.

[0088] After the RTMDet-x model training is completed, load the RTMDet-x model weight file to start the knowledge distillation phase to train the RTMDet model. The initial learning rate is changed to 0.01, and the other parameters remain unchanged. During training, the output results of the RTMDet-m detection head and the label are used for loss calculation: the classification of damage and parts uses the cross entropy loss function; the detection box regression uses the smooth_L1 loss function; the mask uses the binary cross entropy loss function. In addition to the loss function generated by the RTMDet-m detection head, the distillation loss generated during the distillation process must also be calculated. After the loss is added, reverse gradient propagation is performed to update the model weights.

[0089] In the process of training the model, the classification loss, detection box loss, mask loss and distillation loss are calculated and back-propagation is performed to update the weights. After the training is completed, the final model weights are stored to obtain the weight file; the loss function is expressed as:

[0090] Loss = L cls +L bbox +L mask +αL distill

[0091]

[0092] In the formula, Loss represents the loss value, L cls represents the classification loss, L bbox represents the detection box regression loss, L mask represents the pixel segmentation mask loss; i is the index of the anchor box, p i * Indicates whether the i-th anchor frame contains a target. When the i-th anchor frame contains a target, p i * Equal to 1; when the i-th anchor frame is the background, p i * is 0; p i represents the probability value of the target in the i-th anchor box; v i Represents the parameterized vector of the center point coordinates and width and height of the i-th anchor box prediction, v i * is the parameterized vector of the i-th label bounding box; n c is the number of classifications. Here there are only two categories: water accumulation and background. n ij is the number of pixels belonging to the i-th class and predicted to be the j-th class, n ii Represents the number of pixels that belong to the i-th class and are predicted to be the i-th class, that is, the number of correct predictions for the i-th class, n i Represents the total number of detection boxes predicted for the i-th type of damage; N cls 、N bbox are constants used for normalization; N level is the number of feature fusion layers used; t i is the neck feature of the teacher network, s i The neck part of the student network features; L mask represents the mask loss, L distill Represents distillation loss.

[0093] S4. Deploy the trained detection model into the inference framework to accelerate the detection process.

[0094] Step S4 is specifically as follows:

[0095] In order to achieve efficient model reasoning and deployment, the trained model weights need to be converted to the ONNX (Open Neural Network Exchange) format first. ONNX is an open format designed to provide cross-platform model interoperability, allowing models to be transferred and shared between different deep learning frameworks. By converting the model weights to the ONNX format, you can ensure that the model maintains consistent reasoning performance and results in different computing environments. The specific process of this step includes using the conversion tools or libraries provided by ONNX to export the model weights trained by the original framework into an ONNX format file, which can describe the model's structure, parameters, and computational graph information.

[0096] Then, the Onnxruntime-gpu framework is deployed on the server side to achieve efficient model inference. Onnxruntime-gpu is an optimized inference engine developed by Microsoft, which is specifically designed to run deep learning models in the ONNX format. It can use the powerful computing power of the GPU (graphics processing unit) to accelerate the model's inference process and significantly improve computing efficiency and speed. Using the API of the Onnxruntime-gpu framework to load the converted ONNX model, the model can perform inference efficiently, thereby achieving fast detection, which can be applied to frame detection in video streams.

[0097] S5. Use the obtained weight file to detect frames of the urban camera surveillance video, and correct the detection result of a single frame in combination with real-time rainfall information.

[0098] In the urban waterlogging monitoring system, image frames are first extracted from the real-time video stream. The frame rate of the video stream is set to 1 frame / second. This setting balances processing efficiency and computational burden, so that only one frame of image is extracted per second, which can ensure the representativeness of the data and reduce the computational pressure. Each frame of the extracted image is sent to the improved RTMDet model for detection. From the detection results, the bounding box, contour and confidence score of each waterlogged area are extracted. The confidence threshold is set to filter the part of the detection results with confidence lower than the threshold. This step helps to eliminate the low-confidence prediction of the model for certain areas and ensure that the retained results have high reliability. Subsequently, the pixel area of ​​each detection area is calculated and compared with the total area of ​​the entire image to obtain the proportion of waterlogged area. Detection areas with an area ratio of less than 0.05 are regarded as small puddles rather than waterlogged areas and are therefore excluded. The remaining valid detection boxes, classification confidence scores and waterlogged masks are superimposed on the original image to generate the final detection result image.

[0099] Further accuracy improvement depends on combining the detection results with the rainfall data of the automatic station. First, the video point and the automatic station need to be matched. The matching method is to take the automatic station closest in spatial distance to form a matching relationship with the video point. Obtain the cumulative rainfall in the last 6 hours through the API interface or database query. Since the rainfall data of the automatic station stored in the database is half an hour, and the video point data is real-time, it is necessary to round the time data of the video point to the nearest half an hour time point to the actual shooting time, and then align the automatic station time to obtain the rainfall data of the automatic station that matches the video point in space and the video image in time. If the rainfall recorded by the automatic station is less than 28mm, but the detection result shows that waterlogging exists, this may be a signal of a false alarm of the model. Because when the rainfall is below this threshold, there is usually no significant waterlogging phenomenon. Therefore, these false alarm detection results will be marked and removed from the final output. By combining the rainfall data of the automatic station, the detection results can be accurately corrected, improving the overall accuracy and reliability of the waterlogging monitoring system.

[0100] S6. Count the number of frames in which waterlogging is detected. If waterlogging is detected in 4 frames out of 5 consecutive frames, it is determined that waterlogging has indeed occurred at the video point, and an early warning is issued.

[0101] In order to further improve the reliability of the detection results, a continuous frame analysis strategy is adopted. Specifically, a time window is set, including the most recent 5 consecutive frames. If the cumulative number of waterlogging frames detected in this time window reaches or exceeds 4 frames, the system will determine that waterlogging has indeed occurred at the video point (i.e., the area captured by the camera). This judgment is based on the proportion of the detected waterlogging frames to the total number of frames in the window, that is, more than 80% of the detection results are waterlogging. This strategy effectively reduces false alarms caused by single-frame misjudgments or occasional situations.

[0102] Once it is confirmed that waterlogging is detected in at least 4 out of 5 consecutive frames of the video point, the system will automatically trigger the early warning mechanism. At this time, the system will generate an alarm message and send it to the relevant emergency response department or management personnel. Through this process, the system can ensure the accuracy of the early warning and avoid unnecessary alarms due to single-frame misjudgment, thereby effectively supporting emergency response and handling of waterlogging incidents.

[0103] The embodiment of the present invention also provides a video waterlogging recognition simulation device, such as Figure 5 As shown, the device comprises:

[0104] The data set construction module 501 is configured to obtain a plurality of pictures containing waterlogging, annotate the waterlogging contours of each picture, create an annotated data set, and divide the annotated data set into a training set and a test set;

[0105] The model building module 502 is configured to improve RTMDet to build a waterlogging detection model; wherein the improvement of RTMDet includes: fusing the C2 layer features of the backbone network to the C3 layer features after using the spatial attention mechanism; introducing MGD distillation learning, and using RTMDet-x as a teacher model to guide the training of the RTMDet-m student model;

[0106] The model training module 503 is configured to input the training set into the waterlogging detection model training to obtain the model weight file;

[0107] A model deployment module 504 is configured to deploy the trained waterlogging detection model into an inference framework;

[0108] The result correction module 505 is configured to detect frames of the urban camera surveillance video using the obtained weight file, and correct the detection result of a single frame in combination with the real-time rainfall information;

[0109] The waterlogging determination module 506 is configured to determine whether waterlogging occurs at a corresponding video point based on the number of frames in which waterlogging is detected.

[0110] In some embodiments, the data set construction module is further configured to:

[0111] Based on multiple pictures containing waterlogging, the waterlogged areas in the pictures are annotated to obtain an annotation file, which contains the category, position and contour information of the target in the image; the annotation file and the corresponding original picture file are divided into a training set and a test set according to a set ratio, and then the data set is converted into a unified format.

[0112] In some embodiments, the model building module is further configured to:

[0113] Extract the C2 layer features that were not used originally from the backbone network CSPNeXt of RTMDet, and apply the spatial attention mechanism to the C2 layer features; the spatial attention mechanism includes: grouping the feature maps, using GroupNorm to perform spatial attention processing on the grouped feature maps, and re-merging the processed feature maps and fusing them with the C3 layer features;

[0114] The MGD distillation learning method is introduced to further construct the waterlogging detection model. In the distillation process, the RTMDet-m model with small parameters and computational complexity is used as the student model, and the RTMDet-x model with large parameters and computational complexity is used as the teacher model. Some pixels in the neck layer feature map of the RTMDet-m model are randomly selected for masking, and a simple module consisting of a convolutional layer and an activation layer is used to process the masked features. The mean square error between the processed student features and the neck layer feature map of the RTMDet-x model is calculated, and the error is multiplied by a coefficient and added to the total loss function to guide the student model to generate feature representations similar to those of the teacher model.

[0115] In some embodiments, the model training module is further configured to:

[0116] The input image is resized to the specified size, and data enhancement is performed using random scale transformation, random cropping, color space enhancement, random horizontal flipping, Mosaic, and Mixup.

[0117] Load the training set;

[0118] Set the number of training times and learning rate parameters to start training the model;

[0119] Calculate the classification loss, detection box loss, mask loss and distillation loss and perform back propagation to update the weights. After the training is completed, store the final model weights to obtain the weight file; the loss function is expressed as:

[0120] Loss = L cls +L bbox +L mask +αL distill

[0121]

[0122] In the formula, Loss represents the loss value, L cls represents the classification loss, L bbox represents the detection box regression loss, L mask represents the pixel segmentation mask loss; i is the index of the anchor box, p i * Indicates whether the i-th anchor frame contains a target. When the i-th anchor frame contains a target, p i * Equal to 1; when the i-th anchor frame is the background, p i * is 0; p i represents the probability value of the target in the i-th anchor box; v i Represents the parameterized vector of the center point coordinates and width and height of the i-th anchor box prediction, v i* is the parameterized vector of the i-th label bounding box; n c is the number of classifications. Here there are only two categories: water accumulation and background. n ij is the number of pixels belonging to the i-th class and predicted to be the j-th class, n ii Represents the number of pixels that belong to the i-th class and are predicted to be the i-th class, that is, the number of correct predictions for the i-th class, n i Represents the total number of detection boxes predicted for the i-th type of damage; N cls 、N bbox are constants used for normalization; N level is the number of feature fusion layers used; t i is the neck feature of the teacher network, s i The neck part of the student network features; L mask represents the mask loss, L distill Represents distillation loss.

[0123] In some embodiments, the model deployment module is further configured to:

[0124] The model weights are converted into onnx format, the Onnxruntime-gpu framework is deployed on the server, and the waterlogging detection model is detected under the Onnxruntime-gpu framework.

[0125] In some embodiments, the result correction module is further configured to:

[0126] Based on the set frame rate, frames are extracted from the real-time video stream of the urban camera surveillance system, and the extracted frames are input into the trained model for waterlogging detection;

[0127] The location, contour information and confidence of the waterlogged area in the detection results are extracted, and the detections with confidence below the threshold are filtered out according to the set score threshold.

[0128] Calculate the ratio of the detected water area to the entire image area, remove small puddles with an area ratio less than 0.05, and load the remaining detection frames and the corresponding classification confidence scores and water mask masks onto the original image to obtain a single-frame detection result image;

[0129] The detection results are further screened and corrected using the rainfall data from the automatic station near the camera. If the 6-hour cumulative rainfall at the automatic station is less than 28 mm and waterlogging is detected in the image, it is determined that the model detection is incorrect and the erroneous results are removed.

[0130] In some embodiments, the waterlogging determination module is further configured to:

[0131] The frames extracted from the real-time video stream are quickly detected, and the number of frames in which waterlogging is detected is counted. If 4 frames out of 5 consecutive frames show waterlogging, it is determined that waterlogging has indeed occurred at the video point, and an early warning is issued.

[0132] It should be noted that the device described in this embodiment and the method described previously belong to the same technical concept, have the same technical principles, and can achieve the same beneficial effects, so they will not be repeated here.

[0133] An embodiment of the present invention further provides a video waterlogging recognition simulation system, the system comprising: a memory for storing a computer program; and a processor for executing the computer program to implement the method described in any of the above embodiments.

[0134] An embodiment of the present invention further provides a non-transitory computer-readable storage medium storing instructions, and when the instructions are executed by a processor, the method described in any of the above embodiments is executed.

[0135] The above implementation modes are only used to illustrate the present invention, but not to limit the present invention. Ordinary technicians in the relevant technical field can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, all equivalent technical solutions also belong to the scope of the present invention. The patent protection scope of the present invention should be defined by the claims.

Claims

1. A video waterlogging recognition simulation method, characterized in that: The method comprises: Acquire multiple pictures containing waterlogging, annotate the waterlogging contours of each picture, create an annotated data set, and divide the annotated data set into a training set and a test set; Improve RTMDet to build a waterlogging detection model; the improvements to RTMDet include: fusing the C2 layer features of the backbone network to the C3 layer features after using the spatial attention mechanism; introducing MGD distillation learning, and using RTMDet-x as a teacher model to guide the training of the RTMDet-m student model; Input the training set into the waterlogging detection model training to obtain the model weight file; Deploy the trained waterlogging detection model to the inference framework; The obtained weight file is used to detect the frames of the urban camera surveillance video, and the detection results of the single frame are corrected in combination with the real-time rainfall information; Based on the number of frames in which waterlogging is detected, it is determined whether waterlogging occurs at the corresponding video point.

2. The video waterlogging recognition simulation method according to claim 1, characterized in that: A plurality of pictures containing waterlogging are obtained, and after waterlogging contours are annotated for each picture, annotated data sets are prepared, and the annotated data sets are divided into a training set and a test set, including: Based on multiple pictures containing waterlogging, the waterlogged areas in the pictures are annotated to obtain an annotation file, which contains the category, position and contour information of the target in the image; the annotation file and the corresponding original picture file are divided into a training set and a test set according to a set ratio, and then the data set is converted into a unified format.

3. The video waterlogging recognition simulation method according to claim 1, characterized in that: Improvements were made to RTMDet to build a waterlogging detection model, including: Extract the C2 layer features that were not used originally from the backbone network CSPNeXt of RTMDet, and apply the spatial attention mechanism to the C2 layer features; the spatial attention mechanism includes: grouping the feature maps, using GroupNorm to perform spatial attention processing on the grouped feature maps, and re-merging the processed feature maps and fusing them with the C3 layer features; The MGD distillation learning method is introduced to further construct the waterlogging detection model. In the distillation process, the RTMDet-m model with small parameters and computational complexity is used as the student model, and the RTMDet-x model with large parameters and computational complexity is used as the teacher model. Some pixels in the neck layer feature map of the RTMDet-m model are randomly selected for masking, and a simple module consisting of a convolutional layer and an activation layer is used to process the masked features. The mean square error between the processed student features and the neck layer feature map of the RTMDet-x model is calculated, and the error is multiplied by a coefficient and added to the total loss function to guide the student model to generate feature representations similar to those of the teacher model.

4. The video waterlogging recognition simulation method according to claim 3, characterized in that: Input the training set into the waterlogging detection model training to obtain the model weight file, including: The input image is resized to the specified size, and data enhancement is performed using random scale transformation, random cropping, color space enhancement, random horizontal flipping, Mosaic, and Mixup. Load the training set; Set the number of training times and learning rate parameters to start training the model; Calculate the classification loss, detection box loss, mask loss and distillation loss and perform back propagation to update the weights. After the training is completed, store the final model weights to obtain the weight file; the loss function is expressed as: Loss=L cls +L bbox +L mask +αL distill In the formula, Loss represents the loss value, L cls represents the classification loss, L bbox represents the detection box regression loss, L mask represents the pixel segmentation mask loss; i is the index of the anchor box, p i * Indicates whether the i-th anchor frame contains a target. When the i-th anchor frame contains a target, p i * Equal to 1; when the i-th anchor frame is the background, p i * is 0; p i represents the probability value of the target in the i-th anchor box; v i Represents the parameterized vector of the center point coordinates and width and height of the i-th anchor box prediction, v i * is the parameterized vector of the i-th label bounding box; n c is the number of classifications. Here there are only two categories: water accumulation and background. n ij is the number of pixels belonging to the i-th class and predicted to be the j-th class, n ii Represents the number of pixels that belong to the i-th class and are predicted to be the i-th class, that is, the number of correct predictions for the i-th class, n i Represents the total number of detection boxes predicted for the i-th type of damage; N cls 、N bbox are constants used for normalization; N level is the number of feature fusion layers used; t i is the neck feature of the teacher network, s i The neck part of the student network features; L mask represents the mask loss, L distill Represents distillation loss.

5. The video waterlogging recognition simulation method according to claim 1, characterized in that: Deploy the trained waterlogging detection model to the inference framework, including: The model weights are converted into onnx format, the Onnxruntime-gpu framework is deployed on the server, and the waterlogging detection model is detected under the Onnxruntime-gpu framework.

6. The video waterlogging recognition simulation method according to claim 1, characterized in that: The obtained weight file is used to detect the frames of the urban camera surveillance video, and the detection results of the single frame are corrected in combination with the real-time rainfall information, including: Based on the set frame rate, frames are extracted from the real-time video stream of the urban camera surveillance system, and the extracted frames are input into the trained model for waterlogging detection; The location, contour information and confidence of the waterlogged area in the detection results are extracted, and the detections with confidence below the threshold are filtered out according to the set score threshold. Calculate the ratio of the detected water area to the entire image area, remove small puddles with an area ratio less than 0.05, and load the remaining detection frames and the corresponding classification confidence scores and water mask masks onto the original image to obtain a single-frame detection result image; The detection results are further screened and corrected using the rainfall data from the automatic station near the camera. If the 6-hour cumulative rainfall at the automatic station is less than 28 mm and waterlogging is detected in the image, it is determined that the model detection is incorrect and the erroneous results are removed.

7. The video waterlogging recognition simulation method according to claim 1, characterized in that: Based on the number of frames where waterlogging is detected, it is determined whether waterlogging occurs at the corresponding video point, including: The frames extracted from the real-time video stream are quickly detected, and the number of frames in which waterlogging is detected is counted. If 4 frames out of 5 consecutive frames show waterlogging, it is determined that waterlogging has indeed occurred at the video point, and an early warning is issued.

8. A video waterlogging recognition simulation device, characterized in that: The device comprises: A data set construction module is configured to obtain a plurality of pictures containing waterlogging, annotate the waterlogging contour of each picture, create an annotated data set, and divide the annotated data set into a training set and a test set; The model building module is configured to improve RTMDet to build a waterlogging detection model; wherein the improvements to RTMDet include: fusing the C2 layer features of the backbone network to the C3 layer features after using the spatial attention mechanism; introducing MGD distillation learning, and using RTMDet-x as a teacher model to guide the training of the RTMDet-m student model; The model training module is configured to input the training set into the waterlogging detection model training to obtain the model weight file; A model deployment module, configured to deploy the trained waterlogging detection model to the inference framework; A result correction module is configured to detect frames of the urban camera surveillance video using the obtained weight file, and correct the detection result of a single frame in combination with real-time rainfall information; The waterlogging determination module is configured to determine whether waterlogging occurs at a corresponding video point based on the number of frames in which waterlogging is detected.

9. A video waterlogging recognition simulation system, characterized by: The system comprises: Memory for storing computer programs; A processor, configured to execute the computer program to implement the method according to any one of claims 1 to 7. 10 . A non-transitory computer-readable storage medium storing instructions, which, when executed by a processor, perform the method according to claim 1 .

Images