System and method for monitoring and identifying structural damage of open flow cavern based on video stream fusion

Abstract

The invention discloses an open flow cavern structure damage monitoring and identifying system and method based on video stream fusion. The method comprises the following steps: acquiring video stream data of a water flow state of an open flow cavern; extracting a first video stream feature and a first optical stream feature of the video stream data by using a preliminarily trained deep learning neural network model; respectively inputting the first video stream feature and the first optical stream feature into a trained deep learning neural network model for feature extraction to obtain a corresponding second video stream feature and a corresponding second optical stream feature; and fusing and splicing the second video stream feature and the second optical stream feature to obtain a fused feature vector, and inputting the fused feature vector into a trained deep learning neural network model to carry out water flow state classification prediction so as to identify the damage condition of the open flow cavern structure. According to the invention, the relation between the water flow state and the corresponding failure state can be identified more accurately.

Description

Technical Field

[0001] The invention relates to the technical field of water conservancy project monitoring, and in particular to an open flow cavern structure damage monitoring and identification system and method based on video stream fusion. Background Art

[0002] Open flow caverns are underground structures widely used in water conservancy projects. Their design and application mainly depend on specific mission requirements. In water conservancy projects, open flow caverns are used to control floods in dams and ensure the safe operation of reservoirs. At the same time, open flow caverns can also be used as drain holes in reservoirs to discharge excess water in reservoirs. By installing open flow caverns, environmental protection and hydrogeological workers can better monitor and control the flow of groundwater and understand important information such as the distribution, flow direction, and flow rate of groundwater. Open flow caverns can also be used as part of flood control and waterlogging prevention facilities to prevent excessive erosion of river banks and reduce the occurrence of floods by controlling water flow speed and flow. At the same time, open flow caverns also play an important role in agricultural irrigation, traffic drainage, and urban planning. In short, open flow caverns have a wide range of application scenarios and diverse purposes, which can meet the needs of different fields. Its importance lies in its ability to effectively control and manage water flows, protect the environment, and promote sustainable development. Since open flow caverns are usually built in the open air or underground, the destruction of their structures will lead to problems such as water splashing and river flooding, which will have a serious impact on the safe operation of water conservancy projects. Therefore, it is particularly important to monitor and identify the structural damage of open flow caverns.

[0003] Open flow caverns generally need to monitor and manage their water flow state. On the one hand, water resources are rationally allocated and used by monitoring the hydrological parameters of the water flow. On the other hand, it is considered that there is a close relationship between the damage of hydraulic structures and the flow state. The inlet and outlet, side walls and bottom of open flow caverns are most susceptible to scouring and erosion by water flow, causing deformation of the inlet and outlet, erosion and wear damage to the surface of the cavern; when the water flows through the bends and slope change points of the cavern, vortices and shock waves are easily formed, causing cracks, peeling and deformation problems on the side walls and bottom of the cavern. The roughness, angle changes, and unevenness of the surface of the hydraulic structure caused by these damages will affect the movement state of the water flow. The change in flow state will also affect the structure of the hydraulic structure again. When the water flow changes from one flow state to another, it may cause the hydraulic structure to be suddenly impacted or vibrated, causing structural damage. Under certain flow states, the water flow may be unstable, such as vortexes and backflows. These phenomena will increase the impact force of the water flow and the pressure on the hydraulic structure, thereby causing damage to the structure. For example, when cavitation-induced erosion occurs on the downstream wall where the flow pattern changes suddenly, the unevenness of the wall often makes the cavitation damage of the water flow more serious. In view of this, the monitoring of water flow pattern is particularly important in the detection and identification of structural damage in open flow caverns.

[0004] At present, the monitoring of water flow state generally uses water level meters, electronic water gauges and other equipment to monitor water level changes. This method has the disadvantages of low efficiency and susceptibility to external environmental influences. Its monitoring methods are backward, costly and time-sensitive. Using cameras for flow state monitoring can intuitively observe and record water flow states, provide real-time image data, and facilitate monitoring and analysis. Cameras are rarely used in flood discharge safety monitoring systems, and generally need to be combined with other monitoring methods. The field application of cameras in hydraulic flow state monitoring needs further research.

[0005] With the rapid development of deep learning neural networks and their migration applications in various fields in recent years, computer vision and image processing technology have also begun to be applied to the post-processing process of hydraulic monitoring. This technology not only overcomes many of the old shortcomings of flow state monitoring, but also provides new ideas and methods for the field of hydraulic monitoring, and has broad application prospects. Summary of the invention

[0006] The present invention aims to solve one of the technical problems in the related art at least to a certain extent.

[0007] To this end, the present invention proposes an open flow cavern structure damage monitoring and identification system based on video stream fusion, which can monitor the flow state of water in the open flow cavern through a camera, and output the video stream data of the flow state for training the deep learning neural network, so that it can more accurately identify the relationship between the water flow state and the corresponding damage form.

[0008] Another object of the present invention is to propose a method for monitoring and identifying structural damage of open flow caverns based on video stream fusion.

[0009] To achieve the above-mentioned purpose, the present invention proposes a method for monitoring and identifying structural damage of an open flow cavern based on video stream fusion, comprising:

[0010] The camera is used to collect the video stream data of the flow state of the open flow cavern and transmit the video stream data back to the computer through the network cable;

[0011] A computer, used for preprocessing the video stream data and inputting the preprocessed data into a server;

[0012] On the server side, a deep neural network model for classification trained using the initial data set is stored, and the preprocessed data is converted into video features and optical flow features using the model, which are then input into the deep neural network model for classification for classification prediction to obtain water flow pattern prediction classification results, so as to identify the damage of the open flow cavern structure based on the water flow pattern prediction classification results.

[0013] The open flow cavern structure damage monitoring and identification system based on video stream fusion according to the embodiment of the present invention may also have the following additional technical features:

[0014] In one embodiment of the present invention, the server is further used to obtain the optical flow feature, including:

[0015] analyzing difference data between consecutive frames in a video sequence of video stream data;

[0016] Calculate the motion vector of each pixel point in the video based on the difference data to obtain an optical flow field;

[0017] The optical flow field is encoded into a feature vector to obtain an optical flow feature.

[0018] In one embodiment of the present invention, the server is further used to extract the optical flow features using the Farneback optical flow method, including:

[0019] Centered on each pixel in the image, based on all pixels in the neighborhood, the grayscale value of the pixel is approximated using the binomial:

[0020] f(x,y)=r 1 +r 2 X+r 3 y+r 4 x 2 +r 5 y 2 +r 6 xy

[0021] Rewritten using matrices and vectors:

[0022] f(X)=X T AX+b T X+c

[0023] Where X = (x, y) T ; c = r 1 ;

[0024] In one embodiment of the present invention, the deep learning neural network model is a 3DResNet50 network model; the server side is also used to train the deep learning neural network model, including:

[0025] Get the video stream dataset;

[0026] Dividing the video stream into a training data set and a test data set according to a preset ratio;

[0027] The training data set is divided into small batches and input into the 3DResNet50 network model to enter the network through the 3D image input layer-fusion layer of the model, and then passes through a plurality of additional layers consisting of a convolution layer-batch normalization layer-linear correction layer, and finally outputs the result through a fully connected layer-exponential normalization layer to train the 3DResNet50 network model;

[0028] The test data set is input into the 3DResNet50 network model for testing to obtain the trained 3DResNet50 network model.

[0029] In one embodiment of the present invention, the video stream data set includes corresponding video data obtained by camera monitoring under various working conditions; the server side is also used for:

[0030] Extract video flow features and optical flow features from video data simultaneously;

[0031] Input the video stream features and optical flow features from the two 3D image input layers to the fusion layer at the same time to obtain the fused feature data;

[0032] The fused feature data is input into the 3DResNet50 network model for prediction and classification to output the water flow state classification results.

[0033] To achieve the above object, the present invention proposes a method for monitoring and identifying structural damage of an open flow cavern based on video stream fusion, comprising:

[0034] Obtain video stream data of water flow patterns in open flow caverns;

[0035] Extracting a first video stream feature and a first optical flow feature of the video stream data using a preliminarily trained deep learning neural network model;

[0036] Inputting the first video stream feature and the first optical flow feature into the trained deep learning neural network model for feature extraction to obtain corresponding second video stream features and second optical flow features;

[0037] The second video stream feature and the second optical flow feature are fused and spliced ​​to obtain a fused feature vector, and the fused feature vector is input into a trained deep learning neural network model to perform water flow pattern classification prediction to identify the damage of the open flow cavern structure.

[0038] The open flow cavern structure damage monitoring and identification system and detection method based on video stream fusion of the embodiment of the present invention can automatically monitor, analyze and process water flow data, and has the advantages of real-time monitoring, high efficiency and energy saving, data visualization and easy maintenance. It can ensure the structural safety of the open flow cavern and ensure the long-term safe operation of the water conservancy hub.

[0039] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0041] Figure 1 is a structural diagram of an open flow cavern structure damage monitoring and identification system based on video stream fusion according to an embodiment of the present invention;

[0042] Figure 2 is a logic diagram of open flow cavern structure damage monitoring and identification based on video stream fusion according to an embodiment of the present invention;

[0043] Figure 3 is a schematic application scenario diagram according to an embodiment of the present invention;

[0044] Figure 4 is a flow chart of a method for monitoring and identifying structure damage of an open flow cavern based on video stream fusion according to an embodiment of the present invention;

[0045] Figure 5 is a schematic diagram of the ResNet50 neural network structure according to an embodiment of the present invention;

[0046] Figure 6 is a schematic diagram of data fusion according to an embodiment of the present invention;

[0047] Figure 7 is a loss function variation diagram according to an embodiment of the present invention;

[0048] Figure 8 is a confusion matrix diagram according to an embodiment of the present invention. DETAILED DESCRIPTION

[0049] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0050] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0051] The following describes a system and method for monitoring and identifying structural damage in an open flow cavern based on video stream fusion according to an embodiment of the present invention with reference to the accompanying drawings.

[0052] Figure 1 is a structural diagram of a system for monitoring and identifying structure damage of an open flow cavern based on video stream fusion according to an embodiment of the present invention. Figure 1 As shown, including:

[0053] The camera 100 is used to collect video stream data of the flow state of the open flow cavern and transmit the video stream data back to the computer 200 via a network cable;

[0054] The computer 200 is used to pre-process the video stream data and input the pre-processed data into the server 300;

[0055] The server side 300 stores a deep neural network model for classification that is trained using an initial data set, and uses the model to convert preprocessed data into video features and optical flow features, which are then input into the deep neural network model for classification for classification prediction to obtain a water flow pattern prediction classification result, so as to identify the damage of the open flow cavern structure based on the water flow pattern prediction classification result.

[0056] It can be understood that the present invention selects video stream as monitoring data, which can provide intuitive and real-time information, and has the characteristics of rich information, convenient visualization, easy data acquisition, and repeatable traceability analysis; optical flow features are extracted from video data, and by fusing video features and optical flow features, the accuracy, real-time and utilization of data are improved, and the interpretability, analysis angle and flow of data are enhanced. Data fusion can also provide more comprehensive and accurate data support for decision-making, thereby improving decision-making efficiency; the fused data stream is imported into the deep learning neural network. Due to the powerful learning and adaptive capabilities of deep learning, a neural network model for classification can be quickly established in a short time, and it can continuously train itself under the condition of continuous input of subsequent data, and grow into a neural network model with powerful flow classification and recognition capabilities.

[0057] Specifically, the camera of the present invention is used to shoot the flow characteristics of the open flow cavern flow channel during discharge, and transmit the video stream data back to the computer through the network cable; the data is preprocessed on the computer, including cutting the long video into several small sections and converting the video data format. The computer finally inputs the data into the server side. The server side has a deep learning neural network model for classification trained with an initial data set. The model converts the input video stream data into video features and optical flow features and inputs them into the neural network for classification prediction. After each classification prediction, the system will judge the degree of structural damage corresponding to the flow video and give processing suggestions. If the flow video corresponding to the serious degree of structural damage is monitored, the system will issue an early warning and notify the staff as a priority.

[0058] In one embodiment of the present invention, the camera should have stable performance, adaptability to a variety of harsh environments, and high pixel characteristics to obtain a reliable and stable source of video data; the camera should be installed above a location where the hydraulic structure is prone to damage: such as a set distance downstream of the aeration sill, the beginning and end of the cavern gradient section, and the entrance and exit of the cavern.

[0059] In one embodiment of the present invention, the trained deep neural model has the ability of self-learning and continuous improvement. For the classification results of the model, the correctly predicted samples are then selected through manual inspection methods to form a training data set, and the model is trained again to improve its prediction accuracy. As more video stream data is obtained through monitoring, the classification ability of the model will continue to increase during training, and the accuracy of flow state recognition will become higher and higher, which can effectively and efficiently achieve real-time monitoring of the flow state and enhance the functionality and durability of the system.

[0060] This system can determine the degree of damage to hydraulic structures by identifying the flow pattern in the cavern, and predict in advance whether the building is in a safe operating state, so as to take corresponding engineering measures to avoid accidents and ensure the safety of hydraulic structures. It solves the problem of detecting and identifying structural damage under open flow caverns, and combines the flow pattern monitoring system with the deep learning neural network, greatly improving the recognition efficiency of the monitoring system for videos, saving manpower, material resources and time costs, and realizing the intelligent development of hydraulic monitoring. Its overall implementation is as follows: Figure 2 shown.

[0061] It is understandable that when the flow channel or side wall of the open flow cavern is damaged to varying degrees, the corresponding flow pattern will also change. By monitoring the flow surface and timely discovering abnormal flow conditions, the operating status of the building can be predicted and evaluated, so as to take corresponding engineering measures and avoid accidents in advance.

[0062] The present invention will first briefly introduce the application scenarios of the present invention. Multiple cameras are installed at different positions on the top of the cavern, such as at a set distance downstream of the aeration sill, at the beginning and end of the cavern gradient section, and at the entrance and exit of the cavern. The camera is facing the flow surface, and the multiple cameras are connected to the computer through a network switch via a network cable. The computer is arranged in a safety monitoring room, which is a certain distance away from the open flow cavern to ensure a dry environment. The monitored video data is preliminarily preprocessed on the computer and then transmitted from the computer to the remote server. The neural network model on the server predicts and classifies the video data. Considering the problem of model training accuracy, there must be a certain amount of samples with incorrect predictions in the predicted and classified samples. The misclassified samples should be selected through manual inspection by on-site personnel, and the correctly predicted sample data set should be re-imported into the model to train the neural network, thereby improving the accuracy of model prediction.

[0063] It is understandable that the camera needs to be in a harsh environment for a long time, so it needs to have the ability to adapt to harsh environments, such as waterproof and dustproof, high temperature resistant, low temperature resistant, etc., to ensure the stability and durability of the equipment. The camera needs to be able to reliably and stably perform the task of monitoring water flow, so it needs to have high clarity, remote control, large storage space, night vision function, infrared light function and other characteristics to ensure that it can meet the actual needs of hydraulic engineering detection. The communication network cable is connected to the output end of the camera, and the line should be reasonably planned, connected from the air supply hole and ventilation shaft, and connected to the computer in the safety monitoring room after exiting the hole. The computer is connected to the rear server end through the transmission line. Figure 3 shown.

[0064] The server side is also used to obtain optical flow features, including: analyzing the difference data between consecutive frames in the video sequence of the video stream data; calculating the motion vector of the pixel point of each image in the video based on the difference data to obtain the optical flow field; encoding the optical flow field into a feature vector to obtain the optical flow feature.

[0065] The server side is also used to extract the optical flow features using the Farneback optical flow method, including: taking each pixel in the image as the center and based on all pixels in the neighborhood, using binomial approximation to the gray value of the pixel:

[0066]

[0067] Rewritten using matrices and vectors:

[0068] f(X)=X T AX+b T X+c (2)

[0069] Where X = (x, y) T ; c = r 1;

[0070] The deep learning neural network model is a 3DResNet50 network model; the server side is also used to train the deep learning neural network model, including: obtaining a video stream data set; dividing the video stream into a training data set and a test data set according to a preset ratio; dividing the training data set into small batches and inputting the 3DResNet50 network model to enter the network through the model's 3D image input layer-fusion layer, and then passing through multiple additional layers composed of convolutional layers-batch normalization layers-linear correction layers, and finally outputting the results through a fully connected layer-exponential normalization layer to train the 3DResNet50 network model; inputting the test data set into the 3DResNet50 network model for testing to obtain a trained 3DResNet50 network model.

[0071] The video stream data set of the embodiment of the present invention includes corresponding video data obtained through camera monitoring under various working conditions; the server side is also used to: simultaneously extract video stream features and optical flow features from the video data; simultaneously input the video stream features and optical flow features from two 3D image input layers to the fusion layer to obtain fused feature data; input the fused feature data into the 3DResNet50 network model for predictive classification to output water flow state classification results.

[0072] According to the open flow cavern structure damage monitoring and identification system based on video stream fusion of the present invention, for the classification result of the model, the samples with correct prediction are selected through manual inspection to form a training data set, and the model is trained again to improve its prediction accuracy. As more video stream data is obtained through monitoring, the classification ability of the model will continue to increase during training, and the recognition accuracy of the flow state will become higher and higher, which can effectively and efficiently achieve real-time monitoring of the flow state and enhance the functionality and durability of the system.

[0073] In order to implement the above embodiment, Figure 4 As shown, this embodiment also provides a method for monitoring and identifying structural damage of an open flow cavern based on video stream fusion, including:

[0074] S1, obtaining video stream data of water flow patterns in open flow caverns;

[0075] S2, extracting a first video stream feature and a first optical flow feature of the video stream data using a preliminarily trained deep learning neural network model;

[0076] S3, respectively inputting the first video stream feature and the first optical flow feature into a trained deep learning neural network model for feature extraction to obtain corresponding second video stream features and second optical flow features;

[0077] S4, fusing and splicing the second video stream feature and the second optical flow feature to obtain a fused feature vector, and inputting the fused feature vector into a trained deep learning neural network model to perform water flow pattern classification prediction to identify the damage of the open flow cavern structure.

[0078] It is understandable that the method of data fusion + deep model is adopted in the present invention. The 3D ResNet50 network based on video and optical flow feature fusion fuses the video stream features and the optical flow features to obtain a richer feature expression. A common method is to send the video stream features and the optical flow features into the 3DResNet50 network for secondary feature extraction, and then splice the features extracted twice to obtain a set of fused feature vectors containing video and optical flow information.

[0079] For the data in the input layer, the method of fusing video flow features with optical flow features is chosen. Optical flow is a method for describing the motion of pixels in an image. In video processing, optical flow can be used for tasks such as estimating object motion and behavior recognition. Optical flow feature extraction is to obtain the optical flow field by calculating the motion vector of each pixel in the video. These optical flow fields can be further encoded into feature vectors to represent the motion features in the video.

[0080] In the neural network provided by the method, the Farneback optical flow method is used to extract optical flow features. The theoretical basis of the Farneback method is the image polynomial expansion. Image polynomial expansion is an accurate image approximation method. Its principle is to use a polynomial to approximate the gray value of each pixel based on the pixels in its neighborhood, with each pixel as the center. After a polynomial expansion of an image, multiple polynomials equal to the number of pixels will be obtained. Each polynomial acts on a pixel, and the independent variable in the polynomial is the coordinate of the pixel point. Since the image is two-dimensional, the binomial can be selected as the expansion of the image. After the image is binomially expanded, the pixel is no longer represented by a gray value, but by a binomial containing coordinate information. Given two adjacent radar echo images at a given time, after binomial expansion, the relationship between the pixel point displacement and the binomial coefficient is derived by transforming the binomial form and setting constraints, and the displacement of each pixel point can be estimated.

[0081] Centered on each pixel in the image, a binomial is used to approximate the grayscale value of the pixel based on all pixels in its neighborhood:

[0082] f(x,y)=r 1 +r 2 X+r 3 y+r 4 x 2 +r 5 y 2 +r6 xy

[0083] It can be rewritten using matrices and vectors as follows:

[0084] f(X)=X T AX+b T X+c

[0085] Where: X = (x, y) T ; c = r 1 ;

[0086] The coefficient (r) in formula (1) 1 ,r 2 ...r 6 ) can be obtained by weighted least squares method. The weight reflects the contribution of points at different distances in the neighborhood to the fitting point. The weight function uses Gaussian function. The closer the distance to the fitting point, the greater the weight. The weight decreases rapidly as the distance increases. Through binomial expansion, the relationship between the grayscale value and coordinates of the pixel point is obtained. Substituting the coordinates of a pixel point into this relationship, the grayscale value can be calculated in reverse. This converts the pixel point from a simple grayscale value to a spatial category, and the introduction of coordinate information provides a prerequisite for subsequent tracking of pixel movement.

[0087] Video stream feature extraction is the process of converting video data into feature vectors. By encoding the video data using programming software, a set of feature vectors can be obtained, which can represent the motion, color, shape and other features of various objects in the video. Compared with single video data as input, the simultaneous fusion of video data and optical flow data increases the richness of information and improves the robustness of the algorithm. Video data usually contains rich visual information, such as color, shape, motion, etc., while optical flow data provides the spatiotemporal motion and changes of this information. By fusing these two types of data, more comprehensive and accurate visual information can be obtained, improving the neural network's ability to understand and perceive complex scenes.

[0088] When monitoring and identifying water flow, optical flow data can help neural networks detect and track the flow form and flow direction in the flow channel, while video data can provide richer environmental information to help neural networks make more accurate identification decisions. At the same time, the calculation of optical flow data can also provide a supplementary description of video data features, thereby increasing the accuracy of neural network classification and identification.

[0089] The neural network model used in the embodiment of the present invention is a 3D ResNet50 network based on the fusion of video and optical flow features. The 3D ResNet50 network is a powerful deep learning model suitable for tasks such as video classification, behavior recognition, and motion estimation. It makes full use of the characteristics of three-dimensional convolution and residual structure, so that the network has stronger feature extraction capabilities and classification accuracy.

[0090] It is understandable that the basic idea of ​​ResNet (Deep Residual Network) is to build a deep neural network by introducing residual connection. Residual connection refers to adding a direct connection in the network to pass the input data directly to the output, thus spanning multiple neural network layers. This connection method allows the network to learn the detailed features of the input data more easily, thereby improving the performance of the network. In the ResNet structure, a method called "shortcut connection" is used. This connection method allows information to be passed directly from the input to the output without any other processing or parameters. This connection method can be regarded as a "shortcut" method, which can reduce the difficulty of network training and help improve the performance of the network. The ResNet50 neural network structure is as follows Figure 5 shown.

[0091] In addition, in theory, for the problem of "accuracy decreases as the network deepens", ResNet provides two options, namely identity mapping and residual mapping. If the network has reached the optimal state, if the network is further deepened, the residual mapping will be pushed to 0, and only the identity mapping will remain. In this way, the network is theoretically always in the optimal state, and the network performance will not decrease as the depth increases. In general, ResNet can effectively improve the performance and training effect of deep neural networks through residual connections and shortcut connections.

[0092] The embodiment of the present invention uses the neural network as a classification and recognition network for indoor water flow detection in open flow caves, and is used in conjunction with corresponding systems. It has the characteristics of strong recognition ability, continuous learning and improvement of its own algorithm, long service life, strong durability, etc. Due to the characteristics of the flow state of indoor water flow in open flow caves, the water flow makes a long-term relative motion relative to the background environment. For the detection task of moving objects, the use of optical flow features can greatly reduce the amount of calculation, because the optical flow calculation can be regarded as the calculation of the motion vector field of the image pixel points, and there is no need to classify or regress each pixel point like the segmentation task. Therefore, by fusing video and optical flow features, the calculation efficiency can be improved while ensuring accuracy; due to the complexity of the indoor environment of open flow caves, the fusion of video and optical flow features can utilize more information, thereby reducing the influence of background and errors and improving the robustness of the algorithm.

[0093] Furthermore, considering that open flow cavern water flow detection has the characteristics of a long time period and a large amount of monitoring data, the fusion of video stream and optical flow can make full use of the spatiotemporal information in the video, enabling the algorithm to better understand and utilize the video content, which is beneficial to the learning and improvement of the algorithm; at the same time, the fusion of video and optical flow features can use more data to train the model, thereby enhancing the generalization ability of the model, which helps to improve the performance of the model on unseen data, thereby giving it a wider application value and a longer use cycle.

[0094] Furthermore, in order to verify the effectiveness of the system and method, an improvement was made on the existing flood discharge tunnel model in the laboratory, and a platform for open flow cavern structural damage monitoring based on video stream was built. By modifying the flow channel morphology with materials such as plaster or plasticine, seven different structural damage types were simulated according to the degree of damage to the flow channel, the degree of cavitation and the size of the scouring pit.

[0095] The corresponding video data are obtained through camera monitoring under seven working conditions, and the video data are converted into a file format readable by programming software. Before neural network training, the data sets under the seven working conditions are divided into training set and test set in a ratio of 4:1.

[0096] The purpose of multimodal fusion is to utilize the complementarity between different modalities to obtain a more comprehensive and accurate information representation. It can improve the performance and generalization ability of machine learning models, so it has been widely used in many application fields. Video stream features and optical flow features are extracted from the acquired video data at the same time. The video stream format size is 224×224×20×1, and the optical flow format size is 224*224*20*2. They are input from two 3D image input layers to the fusion layer at the same time. The fused feature data format size is 224*224*20*3, and then input into the ResNet50 network for prediction and classification. The data fusion is as follows: Figure 6 shown.

[0097] During the model training process, the training data set is divided into small batches and input into the neural network. It enters the network through the 3D image input layer-fusion layer, and then passes through multiple additional layers consisting of convolution layer-batch normalization layer-linear correction layer, and finally outputs through the fully connected layer-exponential normalization layer. Each iteration will continuously update the weight value of the convolution layer, optimize the prediction ability of the model, and make the loss function smaller and smaller. After 10,000 iterations of the training model, the loss value converges to a very small value. At this time, the model performance has reached a certain level and can be used for classification prediction tasks. The loss function changes as follows: Figure 7 shown.

[0098] The confusion matrix diagram obtained by classifying and predicting the test set data is as follows: Figure 8 As shown in the figure, the model's recognition accuracy for videos under seven working conditions reaches 100%, and there are no recognition errors or recognition failures.

[0099] The model of the embodiment of the present invention is a ResNet50 network model for classification tasks. The streaming video stream data captured by the camera is used for model classification and prediction. The data set contains seven categories, of which 80% of the data is used for training and 20% of the data is used for testing. After the training and testing process of the neural network, the accuracy of the model on the test set is 100%, indicating that the model performs well in the classification task of water flow state and can perform efficient and reliable predictions on classification tasks.

[0100] This example verifies that this method has strong robustness and can be sustainably applied and developed; this system has strong practicality and reliability, and can be migrated and applied to the structural damage monitoring and identification of various open flow caverns in real scenarios, and can effectively and efficiently predict the degree of structural damage corresponding to the water flow state, thereby ensuring the structural safety of the open flow caverns and the safe operation of the water conservancy hub.

[0101] According to the open-flow cavern structure damage monitoring and identification method based on video stream fusion according to the embodiment of the present invention, video stream is selected as monitoring data, which can provide intuitive and real-time information, and has the characteristics of rich information, convenient visualization, easy data acquisition, repeatable traceability analysis, etc.; optical flow features are extracted from video data, and by fusing video features and optical flow features, the accuracy, real-time and utilization of data are improved, and the interpretability, analysis angle and flow of data are enhanced. Data fusion can also provide more comprehensive and accurate data support for decision-making, thereby improving decision-making efficiency; the fused data stream is imported into the deep learning neural network. Due to the powerful learning ability and adaptive ability of deep learning, a neural network model for classification can be quickly established in a short time, and it can continuously train itself under the condition of continuous input of subsequent data, and grow into a neural network model with powerful flow state classification and recognition ability. It can more accurately identify the relationship between water flow state and corresponding damage form. It can automatically monitor, analyze and process water flow data, and has the advantages of real-time monitoring, high efficiency and energy saving, data visualization and easy maintenance, etc. It can ensure the structural safety of open-flow caverns and ensure the long-term safe operation of water conservancy hubs.

[0102] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0103] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

Claims

1. A system for monitoring and identifying structural damage of open flow caverns based on video stream fusion, characterized in that: include: The camera is used to collect the video stream data of the flow state of the open flow cavern and transmit the video stream data back to the computer through the network cable; A computer, used for preprocessing the video stream data and inputting the preprocessed data into a server; On the server side, a deep neural network model for classification trained using the initial data set is stored, and the preprocessed data is converted into video features and optical flow features using the model, which are then input into the deep neural network model for classification for classification prediction to obtain water flow pattern prediction classification results, so as to identify the damage of the open flow cavern structure based on the water flow pattern prediction classification results.

2. The system according to claim 1, characterized in that The server side is also used to obtain the optical flow features, including: analyzing difference data between consecutive frames in a video sequence of video stream data; Calculate the motion vector of each pixel point of each image in the video based on the difference data to obtain an optical flow field; The optical flow field is encoded into a feature vector to obtain an optical flow feature.

3. The system according to claim 2, characterized in that The server side is also used to extract the optical flow features using the Farneback optical flow method, including: Centered on each pixel in the image, based on all pixels in the neighborhood, the grayscale value of the pixel is approximated using the binomial: f(x,y)=r1+r2X+r3y+r4x 2 +r5y 2 +r6xy Rewritten using matrices and vectors: f(X)=X T AX+b T X+c Where X = (x, y) T ;c=r1; 4. The system according to claim 1, characterized in that The deep learning neural network model is a 3DResNet50 network model; The server side is also used to train the deep learning neural network model, including: Get the video stream dataset; Dividing the video stream into a training data set and a test data set according to a preset ratio; The training data set is divided into small batches and input into the 3DResNet50 network model to enter the network through the 3D image input layer-fusion layer of the model, and then passes through a plurality of additional layers consisting of a convolution layer-batch normalization layer-linear correction layer, and finally outputs the result through a fully connected layer-exponential normalization layer to train the 3DResNet50 network model; The test data set is input into the 3DResNet50 network model for testing to obtain the trained 3DResNet50 network model.

5. The system according to claim 4, characterized in that The video stream data set includes corresponding video data obtained by camera monitoring under various working conditions; the server side is also used for: Extract video flow features and optical flow features from video data simultaneously; Input the video stream features and optical flow features from the two 3D image input layers to the fusion layer at the same time to obtain the fused feature data; The fused feature data is input into the 3DResNet50 network model for prediction and classification to output the water flow state classification results.

6. A method for monitoring and identifying structural damage in open flow caverns based on video stream fusion, characterized in that: include: Obtain video stream data of water flow patterns in open flow caverns; Extracting a first video stream feature and a first optical flow feature of the video stream data using a preliminarily trained deep learning neural network model; Inputting the first video stream feature and the first optical flow feature into the trained deep learning neural network model for feature extraction to obtain the corresponding second video stream feature and the second optical flow feature; The second video stream feature and the second optical flow feature are fused and spliced ​​to obtain a fused feature vector, and the fused feature vector is input into a trained deep learning neural network model to perform water flow pattern classification prediction to identify the damage of the open flow cavern structure.

7. The method according to claim 6, characterized in that Obtaining the optical flow feature includes: analyzing difference data between consecutive frames in a video sequence of video stream data; Calculate the motion vector of each pixel point of each image in the video based on the difference data to obtain an optical flow field; The optical flow field is encoded into a feature vector to obtain an optical flow feature.

8. The method according to claim 7, characterized in that The optical flow features are extracted using the Farneback optical flow method, including: Centered on each pixel in the image, based on all pixels in the neighborhood, the grayscale value of the pixel is approximated using the binomial: f(x,y)=r1+r2X+r3y+r4x 2 +r5y 2 +r6xy Rewritten using matrices and vectors: f(X)=X T AX+b T X+c Where X = (x, y) T ;c=r1; 9. The method according to claim 6, characterized in that The deep learning neural network model is a 3DResNet50 network model; training the deep learning neural network model includes: Get the video stream dataset; Dividing the video stream into a training data set and a test data set according to a preset ratio; The training data set is divided into small batches and input into the 3DResNet50 network model to enter the network through the 3D image input layer-fusion layer of the model, and then passes through a plurality of additional layers consisting of a convolution layer-batch normalization layer-linear correction layer, and finally outputs the result through a fully connected layer-exponential normalization layer to train the 3DResNet50 network model; The test data set is input into the 3DResNet50 network model for testing to obtain the trained 3DResNet50 network model.

10. The method according to claim 9, characterized in that The video stream data set includes corresponding video data obtained by camera monitoring under various working conditions; the method further includes: Extracting a third video flow feature and a third optical flow feature simultaneously from the video data; Input the third video stream feature and the third optical flow feature from the two 3D image input layers to the fusion layer at the same time to obtain fused feature data; The fused feature data is input into the 3DResNet50 network model for prediction and classification to output the water flow state classification results.

Images