River abnormity early warning method and system based on deep learning image processing

Abstract

The invention discloses a river abnormity early warning method and system based on deep learning image processing. The method comprises the following steps: acquiring first information and second information; preprocessing the first information to obtain preprocessed data; training a preset multi-modal anomaly detection model according to the preprocessed data, wherein the multi-modal anomaly detection model comprises a CNN feature extraction module, an LSTM feature extraction module, a graph convolution module, a feature fusion module and a spatio-temporal behavior prediction module; and preprocessing the second information, inputting the preprocessed second information into a trained multi-modal anomaly detection model, and outputting a river anomaly early warning result. According to the method, global monitoring and accurate anomaly detection of the river environment can be realized, potential risks can be dynamically analyzed and predicted, the adaptive capacity of the system in a complex environment is remarkably improved, and more intelligent and automatic technical support is provided for river safety management.

Description

Technical Field

[0001] The present invention relates to the field of abnormality warning technology based on a neural network model, and in particular to a river abnormality warning method and system based on deep learning image processing. Background Art

[0002] Extreme weather events frequently cause floating debris accumulation, river bank collapse, abnormal water levels, and other phenomena. These problems not only cause serious damage to the river's ecological environment, but may also cause secondary disasters such as floods, traffic disruptions, and infrastructure damage. Traditional river monitoring methods mostly rely on manual patrols and static video monitoring. Although they can provide certain monitoring guarantees, they are unable to monitor and efficiently manage dynamic abnormal conditions in real time due to limited coverage, slow response speed, and lack of intelligent analysis and processing capabilities, resulting in greater safety hazards.

[0003] In recent years, artificial intelligence, especially computer vision technology based on deep learning, has made significant progress. As an innovative method, multimodal anomaly detection models can effectively combine the dynamic relationship between space and time and have demonstrated excellent performance in many fields. Multimodal anomaly detection models can analyze the behavior of dynamic targets (such as floating objects, river bank cracks, etc.) in real time in river environments, overcoming the limitations of traditional methods in dynamic target detection.

[0004] Most existing methods focus on static target detection and classification, lacking effective analysis of dynamic features in time series, resulting in the inability to accurately identify real-time dynamic anomalies in river environments. Summary of the invention

[0005] The present invention proposes a river anomaly early warning method and system based on deep learning image processing to solve the problem that the prior art cannot accurately identify real-time dynamic anomalies in a river environment.

[0006] The present invention achieves the above-mentioned purpose through the following technical solutions:

[0007] The present invention provides a river anomaly early warning method based on deep learning image processing, comprising:

[0008] Acquire first information and second information, wherein the first information includes a historical multimodal data set of a river area, and the second information includes a real-time multimodal data set of the river area, wherein the multimodal data set includes video data, hydrological parameters, and environmental parameters monitored in the river area;

[0009] Preprocessing the first information to obtain preprocessed data;

[0010] A preset multimodal anomaly detection model is trained according to the preprocessed data, and the multimodal anomaly detection model includes a CNN feature extraction module, an LSTM feature extraction module, a graph convolution module, a feature fusion module and a spatiotemporal behavior prediction module, wherein the CNN feature extraction module is used to extract features from the video data through a pretrained CNN network, the LSTM feature extraction module is used to extract features from hydrological parameters and environmental parameters, the graph convolution module is used to model the spatiotemporal features output by the CNN feature extraction module using a graph convolution network, the feature fusion module is used to splice the output of the LSTM feature extraction module and the output of the graph convolution module, and the spatiotemporal behavior prediction module is used to fuse features through several fully connected layers to output the spatial position and behavior prediction results of the target;

[0011] The second information is preprocessed and input into a trained multimodal anomaly detection model to output a river anomaly warning result.

[0012] Furthermore, a multimodal dataset is obtained, including:

[0013] Acquire multispectral video data of river areas;

[0014] Obtain the detection data of micro hydrological monitoring stations, including turbidity detection data, pH value measurement data, and flow rate data at the upstream of water conservancy facilities and in sensitive areas where floating objects gather;

[0015] The environmental parameters monitored by the environmental sensor include precipitation, wind speed, temperature and humidity.

[0016] Furthermore, the pre-processing step comprises:

[0017] Performing denoising processing on video data in the data to be processed based on a Gaussian filtering algorithm, wherein the data to be processed is the first information or the second information;

[0018] The denoised data is enhanced based on the contrast-limited adaptive histogram equalization algorithm;

[0019] The illumination normalization model based on convolutional neural network is used to perform illumination normalization on the enhanced data;

[0020] Based on the adaptive Retinex decomposition method, the data after illumination normalization is processed to eliminate reflection interference;

[0021] Normalizing the flow velocity data in the data to be processed based on a minimum-maximum normalization method;

[0022] Normalizing the turbidity data based on a standardized method;

[0023] The environmental parameter pair is normalized so that the environmental parameter, the video data and the hydrological parameter are at the same scale.

[0024] Further, training a preset multimodal anomaly detection model according to the preprocessed data includes:

[0025] Build a multimodal anomaly detection model;

[0026] The multimodal anomaly detection model is trained using the PyTorch framework, with an initial learning rate of 0.01, a cosine annealing strategy for dynamic adjustment, a batch size of 256, and an Adam optimizer for optimization;

[0027] The optimized loss functions include: , ,

[0028] in, is the bounding box loss, is the spatial-temporal feature loss, N is the total number of targets, C is the number of target categories, is the true category distribution of target i, is the probability value that the target i predicted by the multimodal anomaly detection model belongs to category k, Area of ​​Overlap represents the area of ​​the overlap between the predicted bounding box and the true bounding box, and Area of ​​Union represents the total area of ​​the predicted bounding box and the true bounding box;

[0029] Data augmentation strategies were added during training, including randomly rotating images in the range of -15° to 15°, randomly adjusting brightness in the range of 0.8 to 1.2 times, and randomly cropping the edges of images to increase data diversity. The model was trained for a total of 100 epochs. The validation set was evaluated at the end of each epoch. Training was stopped when the average accuracy reached more than 90%.

[0030] Furthermore, the calculation formula of the graph convolution module is as follows:

[0031] ,

[0032] in, is the spatiotemporal adjacency matrix, represents the spatiotemporal connections between nodes in the graph, is the convolution kernel weight of the lth layer, is the activation function.

[0033] Furthermore, the feature fusion module is used to splice the output of the LSTM feature extraction module and the output of the graph convolution module, and the feature fusion module is also used to:

[0034] Input the concatenated features into a fully connected layer and output a feature vector;

[0035] Input the feature vector into the Softmax activation function to obtain a weight vector;

[0036] The weight vector is multiplied by the output of the LSTM feature extraction module and the output of the graph convolution module respectively, and then the features are concatenated.

[0037] Furthermore, it also includes abnormal events of abnormal warning in the abnormal warning results of the river. The system triggers the warning mechanism in real time, generates alarm information and pushes it to the monitoring center, and records video clips of the abnormal events.

[0038] Furthermore, it also includes summarizing and analyzing the abnormal warning results of the river to generate a monitoring report including the frequency, occurrence time and regional distribution of abnormal events.

[0039] Furthermore, it also includes optimizing the multimodal anomaly detection model through incremental learning in combination with the newly collected data, and using a version management tool to record the iterative optimization process of the multimodal anomaly detection model.

[0040] The present invention also provides a system for a river anomaly early warning method based on deep learning image processing, comprising:

[0041] Video monitoring equipment, including multi-spectral intelligent monitoring equipment deployed in the river area according to the river terrain characteristics and monitoring key areas:

[0042] Micro hydrological monitoring stations are located 50 meters upstream of water conservancy facilities and in areas sensitive to floating debris accumulation;

[0043] Environmental sensors, including precipitation sensors, wind speed sensors, temperature sensors, and humidity sensors deployed in the river area according to river terrain characteristics and key monitoring areas;

[0044] A data acquisition and storage system, wherein the data acquisition and storage system is connected to the video monitoring equipment, the micro hydrological monitoring station, and the environmental sensor, respectively, and the data acquisition and storage system is used to acquire and store the acquired video data, hydrological parameters, and environmental parameters;

[0045] An acquisition module, the acquisition module is used to acquire information collected by the data acquisition and storage system, the information including first information and second information, the first information including a historical multimodal data set of the river area, the second information including a real-time multimodal data set of the river area, the multimodal data set including video data, hydrological parameters and environmental parameters monitored in the river area;

[0046] A preprocessing module, the preprocessing module is used to preprocess the first information to obtain preprocessed data;

[0047] A training module, wherein the training module is used to train a preset multimodal anomaly detection model according to the preprocessed data, wherein the multimodal anomaly detection model includes a CNN feature extraction module, an LSTM feature extraction module, a graph convolution module, a feature fusion module and a spatiotemporal behavior prediction module, wherein the CNN feature extraction module is used to extract features from the video data through a pretrained CNN network, the LSTM feature extraction module is used to extract features from hydrological parameters and environmental parameters, the graph convolution module is used to model the spatiotemporal features output by the CNN feature extraction module using a graph convolution network, the feature fusion module is used to splice the output of the LSTM feature extraction module with the output of the graph convolution module, and the spatiotemporal behavior prediction module is used to fuse features through several fully connected layers to output the spatial position and behavior prediction results of the target;

[0048] An early warning module is used to pre-process the second information and input it into a trained multi-modal anomaly detection model to output a river anomaly early warning result.

[0049] The beneficial effects of the present invention are:

[0050] The present invention proposes a river anomaly warning method and system based on deep learning image processing. By combining deep learning target detection technology with a multimodal anomaly detection model, multimodal data fusion and physical constraint optimization, an efficient and intelligent river anomaly warning system has been developed. The system can not only realize global monitoring and accurate anomaly detection of the river environment, but also dynamically analyze and predict potential risks, significantly improve the system's adaptability in complex environments, and provide more intelligent and automated technical support for river safety management. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 This is a flow chart of a river anomaly early warning method based on deep learning image processing of the present invention.

[0052] Figure 2 Schematic diagram of the system architecture in an embodiment of the present invention.

[0053] Figure 3 It is a structural diagram of the multimodal anomaly detection model of the present invention. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0055] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0057] In the description of the present invention, it should be understood that the terms "upper", "lower", "inside", "outside", "left", "right", etc. indicate directions or positional relationships based on the directions or positional relationships shown in the accompanying drawings, or are directions or positional relationships in which the product of the invention is usually placed when in use, or are directions or positional relationships commonly understood by those skilled in the art. These directions or positional relationships are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore should not be understood as a limitation on the present invention.

[0058] Furthermore, the terms “first”, “second”, etc. are merely used for distinguishing descriptions and should not be understood as indicating or implying relative importance.

[0059] In the description of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms such as "setting" and "connection" should be understood in a broad sense. For example, "connection" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or an indirect connection through an intermediate medium, or it can be the internal communication of two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0060] The specific implementation modes of the present invention are described in detail below in conjunction with the accompanying drawings.

[0061] like Figure 1 As shown, a flow chart of a river abnormality early warning method based on deep learning image processing of the present invention is shown.

[0062] The present invention provides a river anomaly early warning method based on deep learning image processing, comprising:

[0063] S1: Acquire first information and second information, wherein the first information includes a historical multimodal data set of a river area, and the second information includes a real-time multimodal data set of the river area, wherein the multimodal data set includes video data, hydrological parameters, and environmental parameters monitored in the river area;

[0064] S2: preprocessing the first information to obtain preprocessed data;

[0065] S3: training a preset multimodal anomaly detection model according to the preprocessed data, the multimodal anomaly detection model comprising a CNN feature extraction module, an LSTM feature extraction module, a graph convolution module, a feature fusion module and a spatiotemporal behavior prediction module, the CNN feature extraction module being used to extract features from the video data through a pretrained CNN network, the LSTM feature extraction module being used to extract features from hydrological parameters and environmental parameters, the graph convolution module being used to model the spatiotemporal features output by the CNN feature extraction module using a graph convolution network, the feature fusion module being used to splice the output of the LSTM feature extraction module with the output of the graph convolution module, the spatiotemporal behavior prediction module being used to fuse features through a number of fully connected layers to output the spatial position and behavior prediction results of the target;

[0066] S4: pre-process the second information and input it into a trained multi-modal anomaly detection model to output a river anomaly warning result.

[0067] like Figure 2 As shown, a system for a river abnormality early warning method based on deep learning image processing includes:

[0068] Video monitoring equipment, including multi-spectral intelligent monitoring equipment deployed in the river area according to the river terrain characteristics and monitoring key areas:

[0069] Micro hydrological monitoring stations are located 50 meters upstream of water conservancy facilities and in areas sensitive to floating debris accumulation;

[0070] Environmental sensors, including precipitation sensors, wind speed sensors, temperature sensors, and humidity sensors deployed in the river area according to river terrain characteristics and key monitoring areas;

[0071] A data acquisition and storage system, wherein the data acquisition and storage system is connected to the video monitoring equipment, the micro hydrological monitoring station, and the environmental sensor, respectively, and the data acquisition and storage system is used to acquire and store the acquired video data, hydrological parameters, and environmental parameters;

[0072] An acquisition module, the acquisition module is used to acquire information collected by the data acquisition and storage system, the information including first information and second information, the first information including a historical multimodal data set of the river area, the second information including a real-time multimodal data set of the river area, the multimodal data set including video data, hydrological parameters and environmental parameters monitored in the river area;

[0073] A preprocessing module, the preprocessing module is used to preprocess the first information to obtain preprocessed data;

[0074] A training module, wherein the training module is used to train a preset multimodal anomaly detection model according to the preprocessed data, wherein the multimodal anomaly detection model includes a CNN feature extraction module, an LSTM feature extraction module, a graph convolution module, a feature fusion module and a spatiotemporal behavior prediction module, wherein the CNN feature extraction module is used to extract features from the video data through a pretrained CNN network, the LSTM feature extraction module is used to extract features from hydrological parameters and environmental parameters, the graph convolution module is used to model the spatiotemporal features output by the CNN feature extraction module using a graph convolution network, the feature fusion module is used to splice the output of the LSTM feature extraction module with the output of the graph convolution module, and the spatiotemporal behavior prediction module is used to fuse features through several fully connected layers to output the spatial position and behavior prediction results of the target;

[0075] An early warning module is used to pre-process the second information and input it into a trained multi-modal anomaly detection model to output a river anomaly early warning result.

[0076] The specific implementation steps of the present invention are as follows:

[0077] Step 1: Deploy multi-spectral intelligent monitoring equipment in the river area to collect multi-source perception data in real time. According to the river terrain characteristics (water surface width, flow velocity distribution, shoreline curvature) and key monitoring areas (areas with high incidence of floating objects, areas around water conservancy facilities, and areas prone to bank collapse), a three-level monitoring network is constructed:

[0078] 1. Optical perception layer

[0079] Deploy 4 multi-spectral smart cameras (model HikvisionDS-2CD4A26FWD-IZS), each of which includes: visible light module: resolution 2560×1440, horizontal viewing angle 110°, equipped with polarization filter (eliminating the interference of mirror reflection on the water surface); infrared thermal imaging module: resolution 640×512, thermal sensitivity ≤50mK (detecting the heat source of illegal ships at night); built-in IMU inertial measurement unit (compensating for equipment jitter, dynamic blur index <0.3). Installation parameters: Use RTK surveying instrument (model Huace Navigation i70) for precise positioning, geographic coordinate error <2cm; installation height 3.5 meters, spacing dynamically adjusted according to the width of the river (adjacent field of view overlap rate ≥15%); verify the installation angle through the laser rangefinder (model LeicaDISTOD2), and the pitch angle is set to 25°±2°.

[0080] 2. Hydrological perception layer

[0081] A micro hydrological monitoring station (model HydrolabMS5) was deployed, with technical parameters: sampling frequency 1Hz, flow velocity monitoring (range 0-5m / s, accuracy ±0.02m / s); turbidity detection (range 0-4000NTU, resolution 1NTU); pH value measurement (range 0-14, accuracy ±0.1). Installation location: 50 meters upstream of water conservancy facilities (to sense sudden changes in water flow in advance); sensitive areas for floating objects to gather (to establish visual-hydrological data association).

[0082] 3. Environmental Perception Layer

[0083] Environmental sensors (including weather stations and water surface monitoring equipment) are deployed to collect the following environmental parameters:

[0084] Precipitation (range 0-300mm / h, accuracy ±0.2mm / h);

[0085] Wind speed (range 0-60m / s, accuracy ±0.1m / s);

[0086] Temperature (range -40°C to 85°C, accuracy ±0.2°C);

[0087] Humidity (range 0-100%, accuracy ±2%);

[0088] These environmental parameters are collected in real time by environmental sensors and synchronously transmitted to the data processing system through wireless transmission technology (such as LoRa or 5G), combining hydrological and video data to form a multimodal data set.

[0089] 4. Data acquisition and storage system

[0090] Construct a spatiotemporal synchronous acquisition network: GPS taming clock (model EndRunTempusLX) unified time synchronization, time synchronization error ≤ 1ms; video stream: 30fpsH.265 encoding, resolution 2560×1440 (day mode) / 640×512 (night mode); hydrological data: through LoRa wireless transmission, establish a timestamp mapping relationship with the video stream; environmental data: synchronize with the video stream through wireless transmission to achieve spatiotemporal synchronization of multimodal data. Storage solution: video data: NVR (model HikvisionDS-7716NXI-K4) storage, 4TBRAID5 array (model Seagate Cool Hawk ST4000VX015), continuous recording for 120 hours; hydrological time series data: stored in InfluxDB time series database, sampling interval 1 second; environmental data: stored in MongoDB database, sampling interval 1 second; metadata association table: record the flow rate, turbidity, pH value, wind speed, precipitation and other environmental parameters corresponding to each frame of the image. When the hydrological parameters suddenly change (turbidity change rate > 5% / s or flow velocity gradient > 0.1m / s²), the 60fps high-speed sampling mode is automatically triggered, and motion-significant frames are extracted based on the optical flow analysis method (OpenCV Farneback algorithm), and static redundant frames are eliminated. And a dynamic sampling strategy is adopted: when the hydrological parameters suddenly change (turbidity change rate > 5% / s or flow velocity gradient > 0.1m / s²), the 60fps high-speed sampling mode is automatically triggered, and motion-significant frames are extracted based on the optical flow analysis method (OpenCV Farneback algorithm), and static redundant frames are eliminated. After the sampling is completed, the video frame is bound to the hydrological parameters using a customized annotation tool to generate a six-dimensional annotation vector: [category ID, x_center, y_center, w, h, associated flow velocity, local turbidity, pH value, wind speed, precipitation], and the total number of frames sampled is: 2,400 frames (1,600 frames of visible light + 600 frames of infrared + 200 frames of polarization state).

[0091] Step 2: Perform preprocessing operations such as denoising, image enhancement, and illumination normalization on the collected raw video data to generate a high-quality image dataset suitable for the training needs of deep learning models. First, in order to reduce the impact of noise in the video on model training, the Gaussian filter algorithm is used to denoise each frame of the image. The standard deviation of the Gaussian filter is set to 1.0, and the filter window size is 5×5. The formula is as follows:

[0092] ,

[0093] in, is the weight of the filter; is the standard deviation, which controls the degree of blur; x, y are the pixel position offsets.

[0094] Next, in order to improve the image contrast and brightness uniformity, the contrast-limited adaptive histogram equalization (CLAHE) technology is used for enhancement processing. CLAHE divides the image into multiple small blocks and equalizes the histogram of each small block to limit excessive contrast enhancement. The main parameters are: grid size: 8×8; contrast limit parameter: 2.0. After image enhancement, the original detailed features under low light conditions (such as the edges of floating objects) are clearer, providing more reliable input data for the target detection model. In addition, in order to eliminate the feature interference caused by reflections on the river surface and uneven lighting, an illumination normalization model based on a convolutional neural network (CNN) is constructed. The network structure of the model includes:

[0095] Input layer: accepts RGB images of size 256×256×3;

[0096] Convolutional layer: three convolutional layers, using 3×3, 5×5 and 3×3 convolution kernels respectively, and the activation function is ReLU;

[0097] Pooling layer: The maximum pooling window size is 2×2;

[0098] Output layer: Generates the illumination normalization factor matrix .

[0099] The formula for illumination normalization is:

[0100] ,

[0101] in, is the normalized pixel value; is the original pixel value; is the illumination factor matrix; A constant to prevent the denominator from being zero.

[0102] Specular reflection interference on the water surface is an important challenge in river monitoring, especially under strong light. In order to eliminate this interference, the present invention adopts an adaptive Retinex decomposition method. Based on the Retinex theory, the method decomposes the image into reflection components and illumination components, and removes the influence of specular reflection by estimating the illumination component.

[0103] The core idea of ​​Retinex decomposition is to transform the input image into Decomposed into two parts: reflected image (i.e., true reflections from the surface of an object) and the illuminated image (i.e., light intensity). This decomposition can be expressed by the following formula:

[0104] ,

[0105] in, is the original image, is the reflected image, For lighting images.

[0106] In order to remove the effect of specular reflection, we need to estimate the illumination image. The adaptive Retinex method performs reflection decomposition through the following steps:

[0107] Initial image decomposition: First, use a Gaussian filter to smooth the image to obtain a low-frequency illumination component image:

[0108] ,

[0109] in, is a Gaussian filter, Represents the convolution operation. This low-frequency image serves as the lighting component and includes the effects of water surface reflection and lighting.

[0110] Reflection component calculation: By comparing the ratio of the input image to the illumination image, the reflection component is obtained:

[0111] ,

[0112] Enhance reflection component: In order to improve the details and contrast of the image, the reflection component is enhanced.

[0113] To ensure that the fusion of hydrological and environmental parameters (such as flow rate, turbidity, precipitation, wind speed, etc.) with video data is more accurate, the following normalization processing is performed:

[0114] Flow velocity normalization: The collected flow velocity data (unit: m / s) is processed by the minimum-maximum normalization method:

[0115] ,

[0116] in, is the normalized flow rate value, and are the minimum and maximum values ​​of the flow rate data respectively.

[0117] Turbidity normalization: Normalize the turbidity data (unit: NTU) using the standardization method:

[0118] ,

[0119] in, is the normalized turbidity value, is the mean of the turbidity data, is the standard deviation.

[0120] Normalization of environmental parameters: Normalize environmental parameters such as precipitation and wind speed to ensure that these parameters can be integrated with video data and hydrological data at the same scale. For example, the normalization formula for wind speed is:

[0121] ,

[0122] in, is the normalized value, and are the minimum and maximum values ​​of the wind speed data respectively.

[0123] After preprocessing, each frame of the image is quality checked to remove frames with high blur or low contrast to ensure that all images meet the training requirements. Ultimately, the generated image dataset contains about 50,000 frames of high-quality samples, which have been processed by denoising, enhancement, illumination normalization, and reflection elimination, providing a reliable training data foundation for the deep learning model.

[0124] Step 3: Using the preprocessed data, the multimodal anomaly detection model jointly analyzes river images, hydrological parameters and environmental parameters to optimize the network structure to adapt to the complex scene characteristics of the river environment. The specific steps include:

[0125] Step 3.1: Convert the format of the preprocessed image dataset and the corresponding annotation file to ensure that the data is adapted to the model input requirements, and divide the data into training sets and validation sets to ensure data diversity and balanced distribution, providing a stable training foundation. For the operation of the multimodal anomaly detection model, the feature vector obtained after each frame of the image is processed by CNN will be input into the multi-graph convolutional network model as a node feature: the features obtained after each frame of the image is extracted by CNN will be used as graph nodes; the connection between the graph nodes represents the spatiotemporal adjacency relationship, and the adjacent nodes in time are connected through the edges of the graph. At the same time, the hydrological parameters (such as flow rate, turbidity, etc.) and environmental parameters (such as precipitation, wind speed, etc.) related to the video data are converted into time series features through LSTM. After obtaining the multimodal features, the features of different modes are fused through the feature fusion module, and the final detection results are output through the spatiotemporal behavior prediction module.

[0126] like Figure 3 As shown in Figure 2, the composition of the multimodal anomaly detection model is as follows:

[0127] CNN feature extraction module (image processing): Use a pre-trained CNN network to extract features from each frame of the image. The output of the network is the deep feature map of the image, which will be used as the input node features of the graph convolution module. Resnet50 is used in this embodiment.

[0128] Graph Convolutional Module (GCN): Use graph convolutional networks to model the deep features of images and capture the changes of targets in space and time. The network models the deep features of video frames through graph convolution and models the dynamics of targets in time through spatiotemporal convolution. The calculation formula is:

[0129]

[0130] in, is the spatiotemporal adjacency matrix, which represents the spatiotemporal connections between nodes in the graph. is the convolution kernel weight of the lth layer, is the activation function.

[0131] LSTM feature extraction module (parameter processing): Use the LSTM network to extract features of hydrological parameters and environmental parameters, capture the trend of various parameters changing over time, and output their features for subsequent steps.

[0132] Feature fusion module: The features of the graph convolution module and the features output by the LSTM module are concatenated. After concatenation, they pass through a fully connected layer of the same length and output a feature vector of the same length. The vector passes through the Softmax activation function to obtain a weight vector. The weight vector is multiplied with the features of the graph convolution module and the LSTM features respectively, and then the features are concatenated to achieve adaptive fusion of features of different modalities.

[0133] Spatiotemporal behavior prediction module: The fused features are passed through several fully connected layers to output the spatial position of the target (such as bounding box coordinates) and behavior prediction results (such as the trajectory of floating objects, changes in water level, etc.).

[0134] Step 3.2: Model training. The model is trained using the PyTorch framework. The initial learning rate is 0.01, and the cosine annealing strategy is used for dynamic adjustment. The batch size is 256. The Adam optimizer is used. During the training process, for the multimodal anomaly detection model, the following loss function is optimized:

[0135] Bounding box loss: Use Generalized IoU (GIoU) ​​to measure the overlap between the predicted box and the true box:

[0136] ,

[0137] Spatial-temporal feature loss:

[0138] ,

[0139] Where N is the total number of targets, C is the number of target categories, is the true category distribution of target i, is the probability value predicted by the multimodal anomaly detection model that target i belongs to category k.

[0140] In order to improve the generalization ability of the model, data augmentation strategies were added during training, including randomly rotating the image within the range of -15° to 15°; randomly adjusting the brightness (in the range of 0.8 to 1.2 times); and randomly cropping the edge of the image to increase data diversity. The model training was performed for a total of 100 epochs, and the validation set was evaluated at the end of each epoch. When the mean average precision (mAP) reached more than 90%, the training was stopped.

[0141] Step 3.3: Model verification and optimization. In the verification phase, the verification set data is input into the multimodal anomaly detection model frame by frame, and the bounding box coordinates, category label and confidence of each target are output. The following indicators are used to evaluate the model performance: Precision: the proportion of targets predicted to be positive that are actually positive; Recall: the proportion of targets that are actually positive that are correctly predicted to be positive; mAP (mean Average Precision): an average indicator for comprehensive evaluation of detection accuracy. Finally, the optimized model is exported to ONNX format for easy deployment in hardware devices.

[0142] Step 4: After combining the optimized multimodal anomaly detection model, it is deployed to the monitoring system to achieve real-time detection and dynamic behavior analysis of river waters. The monitoring system receives the video stream of the high-definition camera through the RTSP protocol. The multimodal anomaly detection model is exported in ONNX format and loaded and run through the TensorRT framework for frame-by-frame target detection. The inference time is optimized to less than 15ms per frame, which can support video stream processing at 60 frames per second. On this basis, the continuous multi-frame image sequence (default 16 frames) is input into the multimodal anomaly detection model module for spatiotemporal feature extraction. The multimodal anomaly detection model combines multimodal input (video data and hydrological and environmental parameters) to perform multi-dimensional analysis of the target's motion trajectory, speed change, and shape characteristics, and accurately identify abnormal behaviors. The detection results include the target's location (bounding box coordinates), category label, confidence, and behavior analysis results generated by the multimodal anomaly detection model (such as trajectory deviation score or anomaly level score). The results are transmitted to the monitoring center in real time through the WebSocket protocol, and the detection and analysis information (such as the target's real-time classification label, predicted trajectory, and abnormal behavior warning) is superimposed on the local display screen. The system has a preset buffer mechanism that can automatically adjust the image resolution to 640×640 pixels when the input frame rate fluctuates or the network is delayed, ensuring the efficient operation of the multimodal anomaly detection model module. For detected abnormal events (such as obvious accumulation of floating objects or rapid rise of water levels), the system will trigger an alarm mechanism, notify the management personnel through SMS, backend system or email, and store the corresponding multi-frame video sequence in the database for subsequent analysis and tracing, ensuring the stability and accuracy of the system in complex environments.

[0143] Step 5: The system combines the real-time inference results of the deployed multimodal anomaly detection model to extract and analyze the features of the targets in the river water monitoring screen to identify potential abnormal conditions, including floating objects accumulation, water level exceeding the standard, and riverbank collapse. In each frame of the image, the system extracts the location information, category and confidence of the detected target, and generates behavioral feature data in combination with multi-frame target trajectory analysis. The detected target features are compared with the preset normal mode. For example, in the case of water level exceeding the standard, the system calculates the vertical distance between the water level mark line and the water surface boundary. When the distance is less than the set threshold (such as 0.2 meters), it is determined to be a water level abnormality. For the identified abnormal events, the system triggers the early warning mechanism in real time, generates alarm information and pushes it to the monitoring center, and records the video clips of the abnormal events for subsequent management and analysis.

[0144] Step 6: The system regularly summarizes and analyzes the detection results, and generates monitoring reports including the frequency, occurrence time and regional distribution of abnormal events to guide subsequent safety management measures. Combined with the newly collected data, the target detection model is optimized through incremental learning, and the training data set is expanded to adapt to the needs of new scenarios. The model optimization adopts a fine-tuning strategy to update some layer parameters while maintaining the original weights, reducing training time and computing resource consumption. After verification, the optimized model replaces the old model and is deployed in the monitoring system to ensure that the system's detection performance in different lighting conditions, weather changes and dynamic environments continues to improve. In addition, the system uses version management tools to record the iterative optimization process of the model to ensure that each model upgrade has a basis and can be traced back to ensure the long-term stable operation and adaptability of the monitoring system.

[0145] In summary, the present invention can accurately detect abnormal targets in river waters and realize real-time abnormal condition identification and early warning through the close combination of deep learning target detection model and advanced image processing technology. In particular, based on the optimized multimodal anomaly detection model, multimodal data fusion and advanced image preprocessing strategies (including adaptive Retinex decomposition method to remove water surface reflection interference, illumination normalization, CLAHE image enhancement, etc.), the present invention significantly improves the adaptability of the system in complex river environments, ensuring that various dynamic abnormal scenes such as floating objects, water level anomalies, and riverbank collapse can be effectively identified. By combining the real-time collected hydrological data (such as flow rate, water level, turbidity, etc.) with the multimodal anomaly detection model module, the present invention further enhances the ability to analyze the interaction between water flow and floating objects, so that the system can respond to water flow changes in real time and accurately identify abnormal behaviors caused by water level changes and flow velocity fluctuations. The spatiotemporal modeling capability of the multimodal anomaly detection model enables the system to simultaneously process spatial and temporal dynamic features, improving the accuracy of predicting target behaviors in complex scenes. In these scenarios, the system demonstrated high robustness and real-time performance, and was able to cope with environmental challenges such as strong reflection interference, lighting changes, rainy and foggy weather, improving target detection accuracy and reliability, and providing strong technical support for river abnormality monitoring and early warning.

[0146] The present invention provides a set of intelligent solutions for river safety management needs, combining automatic early warning mechanism with continuously optimized target detection model, and successfully realizes the full process automation of river water monitoring. The system not only significantly reduces the cost of manual inspections, but also effectively improves monitoring accuracy, significantly reduces safety hazards and management costs. The system's efficient and real-time monitoring capabilities give it broad application prospects in the fields of river safety monitoring, disaster prevention and mitigation, and water conservancy project management, providing strong technical support for improving the safety management efficiency of the river environment.

[0147] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A river anomaly early warning method based on deep learning image processing, characterized in that: include: Acquire first information and second information, wherein the first information includes a historical multimodal data set of a river area, and the second information includes a real-time multimodal data set of the river area, wherein the multimodal data set includes video data, hydrological parameters, and environmental parameters monitored in the river area; Preprocessing the first information to obtain preprocessed data; A preset multimodal anomaly detection model is trained according to the preprocessed data, and the multimodal anomaly detection model includes a CNN feature extraction module, an LSTM feature extraction module, a graph convolution module, a feature fusion module and a spatiotemporal behavior prediction module, wherein the CNN feature extraction module is used to extract features from the video data through a pretrained CNN network, the LSTM feature extraction module is used to extract features from hydrological parameters and environmental parameters, the graph convolution module is used to model the spatiotemporal features output by the CNN feature extraction module using a graph convolution network, the feature fusion module is used to splice the output of the LSTM feature extraction module and the output of the graph convolution module, and the spatiotemporal behavior prediction module is used to fuse features through several fully connected layers to output the spatial position and behavior prediction results of the target; The second information is preprocessed and input into a trained multimodal anomaly detection model to output a river anomaly warning result.

2. The method for early warning of river anomalies based on deep learning image processing according to claim 1 is characterized in that: Get multimodal datasets, including: Acquire multispectral video data of river areas; Obtain the detection data of micro hydrological monitoring stations, including turbidity detection data, pH value measurement data, and flow rate data at the upstream of water conservancy facilities and in sensitive areas where floating objects gather; The environmental parameters monitored by the environmental sensor include precipitation, wind speed, temperature and humidity.

3. The river anomaly early warning method based on deep learning image processing according to claim 2 is characterized in that: The pre-processing step comprises: Performing denoising processing on video data in the data to be processed based on a Gaussian filtering algorithm, wherein the data to be processed is the first information or the second information; The denoised data is enhanced based on the contrast-limited adaptive histogram equalization algorithm; The illumination normalization model based on convolutional neural network is used to perform illumination normalization on the enhanced data; Based on the adaptive Retinex decomposition method, the data after illumination normalization is processed to eliminate reflection interference; Normalizing the flow velocity data in the data to be processed based on a minimum-maximum normalization method; Normalizing the turbidity data based on a standardized method; The environmental parameter pair is normalized so that the environmental parameter, the video data and the hydrological parameter are at the same scale.

4. The method for river anomaly warning based on deep learning image processing according to claim 1 is characterized in that: Training a preset multimodal anomaly detection model according to the preprocessed data includes: Build a multimodal anomaly detection model; The multimodal anomaly detection model is trained using the PyTorch framework, with an initial learning rate of 0.01, a cosine annealing strategy for dynamic adjustment, a batch size of 256, and an Adam optimizer for optimization; The optimized loss functions include: ， ， in, is the bounding box loss, is the spatial-temporal feature loss, N is the total number of targets, C is the number of target categories, is the true category distribution of target i, is the probability value that the target i predicted by the multimodal anomaly detection model belongs to category k, Area of ​​Overlap represents the overlapping area of ​​the predicted bounding box and the true bounding box, and Area of ​​Union represents the total area of ​​the predicted bounding box and the true bounding box; Data augmentation strategies were added during training, including randomly rotating images in the range of -15° to 15°, randomly adjusting brightness in the range of 0.8 to 1.2 times, and randomly cropping the edges of images to increase data diversity. The model was trained for a total of 100 epochs. The validation set was evaluated at the end of each epoch. Training was stopped when the average accuracy reached more than 90%.

5. The method for early warning of river anomalies based on deep learning image processing according to claim 1 is characterized in that: The calculation formula of the graph convolution module is as follows: ， in, is the spatiotemporal adjacency matrix, represents the spatiotemporal connections between nodes in the graph, is the convolution kernel weight of the lth layer, is the activation function.

6. The river anomaly early warning method based on deep learning image processing according to claim 1 is characterized in that: The feature fusion module is used to splice the output of the LSTM feature extraction module and the output of the graph convolution module, and the feature fusion module is also used to: Input the concatenated features into a fully connected layer and output a feature vector; Input the feature vector into the Softmax activation function to obtain a weight vector; The weight vector is multiplied by the output of the LSTM feature extraction module and the output of the graph convolution module respectively, and then the features are concatenated.

7. The river anomaly early warning method based on deep learning image processing according to claim 1 is characterized in that: It also includes triggering an early warning mechanism in real time according to abnormal events of abnormal warning in the abnormal early warning results of the river, generating alarm information and pushing it to a monitoring center, and recording video clips of abnormal events at the same time.

8. The method for river anomaly early warning based on deep learning image processing according to claim 1 is characterized in that: It also includes summarizing and analyzing the abnormal warning results of the river and generating a monitoring report including the frequency, occurrence time and regional distribution of abnormal events.

9. The method for river anomaly warning based on deep learning image processing according to claim 1 is characterized in that: The method also includes optimizing the multimodal anomaly detection model through incremental learning in combination with the newly collected data, and recording the iterative optimization process of the multimodal anomaly detection model using a version management tool.

10. A system for the river anomaly early warning method based on deep learning image processing as described in any one of claims 1 to 9, characterized in that: include: Video monitoring equipment, including multi-spectral intelligent monitoring equipment deployed in the river area according to the river terrain characteristics and monitoring key areas: Micro hydrological monitoring stations are located 50 meters upstream of water conservancy facilities and in areas sensitive to floating debris accumulation; Environmental sensors, including precipitation sensors, wind speed sensors, temperature sensors, and humidity sensors deployed in the river area according to river terrain characteristics and key monitoring areas; A data acquisition and storage system, wherein the data acquisition and storage system is connected to the video monitoring equipment, the micro hydrological monitoring station, and the environmental sensor, respectively, and the data acquisition and storage system is used to acquire and store the acquired video data, hydrological parameters, and environmental parameters; An acquisition module, the acquisition module is used to acquire information collected by the data acquisition and storage system, the information including first information and second information, the first information including a historical multimodal data set of the river area, the second information including a real-time multimodal data set of the river area, the multimodal data set including video data, hydrological parameters and environmental parameters monitored in the river area; A preprocessing module, the preprocessing module is used to preprocess the first information to obtain preprocessed data; A training module, wherein the training module is used to train a preset multimodal anomaly detection model according to the preprocessed data, wherein the multimodal anomaly detection model includes a CNN feature extraction module, an LSTM feature extraction module, a graph convolution module, a feature fusion module and a spatiotemporal behavior prediction module, wherein the CNN feature extraction module is used to extract features from the video data through a pretrained CNN network, the LSTM feature extraction module is used to extract features from hydrological parameters and environmental parameters, the graph convolution module is used to model the spatiotemporal features output by the CNN feature extraction module using a graph convolution network, the feature fusion module is used to splice the output of the LSTM feature extraction module with the output of the graph convolution module, and the spatiotemporal behavior prediction module is used to fuse features through several fully connected layers to output the spatial position and behavior prediction results of the target; An early warning module is used to pre-process the second information and input it into a trained multi-modal anomaly detection model to output a river anomaly early warning result.

Images