Deep learning-based spillway accumulation body blockage identification method

Abstract

The invention discloses a spillway accumulation body blockage identification method based on deep learning, and belongs to the technical field of hydraulic engineering safety detection and intelligent operation and maintenance. According to the invention, a possible rock block accumulation condition is simulated through an indoor experiment, and a set of image acquisition system is established to shoot spillway discharge chute blockage caused by rock blocks in different forms. Besides, an improved DeepLabV3 + model is adopted as an image semantic segmentation algorithm, the improved model can meet the detection requirements in efficiency and precision, and a model performance analysis experiment shows the effectiveness of the established method. Furthermore, the trained model is integrated into an open platform, so that the model is promoted to be integrated into the equipment terminal to realize automatic detection. The method provided by the invention is mainly applied to an engineering operation and maintenance stage, and a tedious process of traditional manual inspection is avoided.

Description

Technical Field

[0001] The present invention relates to the technical field of water conservancy project safety detection and intelligent operation and maintenance, and in particular to a method for identifying spillway accumulation blockage based on deep learning. Background Art

[0002] As a component structure of water conservancy projects, spillways play a key role in discharging excess floodwaters that exceed the reservoir water level and preventing dam damage. Generally speaking, spillways must have a flood discharge capacity equivalent to the structural grade to ensure discharge before and during the flood season. However, due to the long-term disrepair of the project construction and the lack of management personnel. Especially for some small reservoirs, the project safety information cannot be obtained in time during the operation period, which may lead to delays in reporting spillway blockage danger information.

[0003] Engineering management units will regularly inspect some structures, but it is difficult to provide quick information about dangerous situations. In addition, manual inspections are inefficient, prone to errors, and will increase management costs. Therefore, traditional manual inspections are untimely and costly.

[0004] With the development of engineering construction and management, information and intelligent technology provide a convenient way for engineering management. Some computer vision technologies have shown great potential in the field of structural safety and health detection, which can replace traditional manual detection to establish an efficient and flexible management method. Therefore, it is of great significance to invent a method and system that can use computer vision technology to realize the identification of spillway accumulation hazards. Summary of the invention

[0005] The purpose of the present invention is to propose a method for identifying spillway accumulation blockage based on deep learning to solve the shortcomings of existing spillway accumulation blockage detection technology. The present invention includes three aspects: data acquisition experiment, deep learning algorithm and integrated system. Based on the deep learning model, a flexible and fast detection method is provided for spillway blockage hazard detection caused by block accumulation. The established image segmentation model is integrated into the "AI open source platform" to establish a spillway block blockage analysis platform.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] A spillway accumulation blockage identification method based on deep learning, including the following contents:

[0008] Establish an image dataset: Establish a spillway chute physical model in the laboratory, and artificially set up piled stones to simulate spillway blockage in different situations; in the process of simulating spillway blockage, use a camera to shoot the blockage situation in the chute physical model, collect image photos, and construct an image dataset;

[0009] Establish an image segmentation model: Establish an image segmentation model based on deep learning to identify whether there are rocks in the images in the image dataset;

[0010] Establish an analysis platform: Integrate the constructed image segmentation model into the "AI open source platform" to establish a spillway rock blockage analysis platform.

[0011] Preferably, the establishing of the image data set includes the following contents:

[0012] A physical model of the spillway chute was built in the laboratory, and stones of different numbers, shapes, sizes and colors were placed in the chute as blockages to simulate different situations of spillway blockage;

[0013] A camera was used to record the stone placement images during each spillway blockage simulation process. After removing low-quality and duplicate images, image annotation tools were used to complete image annotation, thereby forming a spillway blockage simulation image dataset, which was divided into a training set and a test set.

[0014] Preferably, the establishing of the image segmentation model includes the following contents:

[0015] Create a DeepLabV3+ model in the Pytorch framework and improve the DeepLabV3+ model. The specific contents are as follows:

[0016] In the encoding network structure, spatial dilated pyramid pooling is used to complete the convolution operation of the image, and low-level features and high-level features are output; wherein the low-level features represent the most basic color and texture features of the image, and the low-level features are spliced ​​with the output of the high-level features in the decoding network structure, and the final pixel-level segmentation result is obtained through convolution and upsampling;

[0017] MobileNetV2 is used as the backbone of the model. The number of channels is expanded through point-by-point convolution, so that the model can extract features in a higher-dimensional space. A linear bottleneck layer is introduced to ensure the effectiveness of neurons in processing diverse features.

[0018] CE Loss is used as the loss function of the model, and the SGD function with momentum is used as the optimizer. The specific function is expressed as:

[0019] V dw =βV dw +(1-β)dw

[0020] V db =βV db +(1-β)db

[0021] W=W-αV dw , b=b-αVdb

[0022] Among them, V dw With V db Represent the current weight gradient and offset gradient respectively; β represents the parameter that affects the degree of smooth gradient descent; W and b represent the weight and bias of the model respectively;

[0023] Input the images in the training set into the improved DeepLabV3+ model, save the model files after each training, and select the training results with the highest performance for testing; use the classification accuracy and average intersection-over-union ratio as the evaluation indicators of the model image segmentation performance. The expressions of classification accuracy and average intersection-over-union ratio are as follows:

[0024]

[0025] Among them, Acc represents the classification accuracy; mIoU represents the mean intersection over union ratio; n represents the number of different types in the image; TP represents the number of pixels that belong to the positive example and the model segmentation result is also the number of pixels that are positive examples; TN represents the number of pixels that belong to the negative example and the model segmentation result is also the number of pixels that are negative examples; FN represents the number of pixels that belong to the positive example and the model segmentation result is the number of pixels that are negative examples; FP represents the number of pixels that belong to the negative example and the model segmentation result is the number of pixels that are positive examples;

[0026] After the training is completed, a trained image segmentation model based on deep learning is obtained.

[0027] Preferably, the establishment of the analysis platform includes the following contents:

[0028] Save the network structure and parameters of the trained deep learning-based image segmentation model in binary format, and load the model through the "AI Open Source Platform"; after the model is bound to the platform engine, complete the image upload and analysis by selecting the appropriate analysis mode.

[0029] Compared with the prior art, the present invention provides a spillway accumulation blockage identification method based on deep learning, which has the following beneficial effects:

[0030] (1) The present invention simulates the blockage of the spillway caused by the accumulation of rocks through indoor experiments. The experimental method and equipment are simple and a large amount of dangerous image data can be obtained.

[0031] (2) The present invention uses the MobileNetV2 network to replace the conventional Xception network, and the model is more lightweight and meets the requirements of instant detection;

[0032] (3) The present invention uses deep learning technology to detect whether there is a dangerous situation of rock accumulation in the spillway, avoiding tedious manual inspections and improving the level of automation of engineering management;

[0033] (4) Based on the method proposed in the present invention, a corresponding spillway accumulation blockage identification system can be established, which can then be combined with a camera terminal to achieve less-manned management of the project, and the proposed method can provide a reference for the detection of other dangerous situations. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is an overall flow chart of a spillway accumulation blockage identification method based on deep learning proposed by the present invention;

[0035] Figure 2 This is the indoor experimental image data collection scenario mentioned in Example 1 of the present invention;

[0036] Figure 3 The collected photos and the annotated results mentioned in Embodiment 1 of the present invention;

[0037] Figure 4 The MobileNetV2 model structure mentioned in Example 1 of the present invention;

[0038] Figure 5 The deep learning model structure mentioned in Embodiment 1 of the present invention;

[0039] Figure 6 The loss function changes of different models mentioned in the experimental examples of the present invention in the training set;

[0040] Figure 7 The image segmentation effects of different model test sets mentioned in the experimental examples of the present invention;

[0041] Figure 8 It is the system interface of the integrated deep learning model mentioned in the experimental example of the present invention;

[0042] Fig. 9 An interface for uploading images to be analyzed by the system mentioned in the experimental example of the present invention;

[0043] Fig.10 The system mentioned in the experimental example of the present invention performs the analysis task results. DETAILED DESCRIPTION

[0044] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0045] See also Figure 1 The present invention proposes a spillway accumulation blockage identification method based on deep learning, which includes the following contents:

[0046] (1) Through indoor experiments, simulate the danger of blockage in the spillway caused by rock accumulation and establish an image data set. A physical model of the spillway chute was established in the laboratory, and rock accumulation was artificially set to simulate blockage in different situations of the spillway. In this process, a camera was used to capture images of the blockage in the chute. The specific implementation steps include:

[0047] During the collection process, a camera was used to collect images of local locations in the chute. A total of 1,135 RGB images in JPG format were collected with a resolution of 6,000 pixels × 4,000 pixels. To ensure the strong applicability of the data, different types of blockages were selected. Stones of different numbers, shapes, sizes, and colors were placed manually in the chute, and the camera was used to record the images of each stone placement. After removing low-quality and duplicate images, image annotation was completed in the LabelMe platform. The images captured include a single stone, multiple parallel stones, and stone stacking. After deleting duplicate and blurred images, 1,000 images were retained as the data set to be segmented.

[0048] (2) Establish a deep learning image segmentation model to identify whether there are rocks in the image. The specific implementation steps include:

[0049] The improved DeepLabV3+ model is used to complete the semantic segmentation task of the image. The collected images are input into the improved DeepLabV3+ network to obtain the predicted image segmentation results.

[0050] The improved DeepLabV3+ model established uses the MobileNetV2 network with lightweight features to replace the Xception network in the conventional model. The high-level semantic features and low-level semantic features of the spillway blockage image are processed by a deep convolutional neural network. The channel dimension reduction calculation of the spatial atrous pyramid pooling (ASPP) module and point-by-point convolution is used to input the decoding module of the model. In order to ensure consistency with the low-level semantic features, the output of the encoder is processed by 4 times upsampling. Finally, a 3×3 convolution kernel and upsampling are used to restore the image size consistent with the model input. After the weight parameters of the established model are determined in the training set, the entire model structure and parameters are output and saved in a binary file, and then the performance evaluation in the test set is completed.

[0051] (3) Integrate the algorithm into the system and establish a spillway rock blockage analysis platform. The specific implementation steps include:

[0052] Based on the "Hikvision Super Brain Platform", the model is loaded and saved in the "AI Open Source Platform" to upload and analyze images. The trained deep learning model saves the network structure and parameters in binary format, and the model is loaded through the Hikvision Super Brain Platform.

[0053] The technical details and implementation process of the present invention are described in detail below in conjunction with the accompanying drawings.

[0054] Embodiment 1:

[0055] A method for identifying spillway accumulation blockage based on deep learning, the method comprising: simulating the spillway blockage caused by block accumulation through indoor experiments, thereby using a camera to take images to establish a data set; preprocessing the spillway blockage images, and further inputting them into the established semantic segmentation model to obtain a binary file including the model structure and weights; based on the Ultrain platform, integrating the trained model into the analysis platform, the specific contents are as follows:

[0056] 1. Creation of Dataset

[0057] Since there are limited data sets of spillway blockage in actual projects, this embodiment simulates spillway blockage caused by rock accumulation through indoor experiments to create a data set. Figure 1 For the indoor experimental image data collection scene, the camera is placed in the chute through a tripod; stones of different numbers, shapes, sizes and colors are placed in the chute manually, and the camera is used to record the image of each stone placement; after removing low-quality and repeated images, the image is annotated in the LabelMe platform; all the stone accumulation bodies are labeled as "Stone", and the rest are background pixels. Figure 2 The collected images and the annotated images are shown in Figure 2. Subsequently, the dataset is divided into a training set and a test set in a ratio of 8:2.

[0058] 2. Creation of Deep Learning Model

[0059] This embodiment first creates a DeepLabV3+ model in the Pytorch framework. As one of the most effective semantic segmentation models, DeepLabv3+ is an encoder-decoder structural model. In the encoder network structure, the spatial atrous pyramid pooling (ASPP) is used to complete the convolution operation of the image. This process can expand the receptive field of the convolution kernel and does not lead to an increase in parameters. ASPP avoids the problem of excessive downsampling rate in the encoder that may occur in other encoder-decoder structural models, resulting in the problem of ignoring too much information. The image features are subjected to 4 convolutions with different expansion rates and 1 average pooling operation through the ASPP module to obtain 5 outputs, and then the 5 outputs are spliced ​​and fused to obtain the output of high-level features using a 1×1 convolution operation. In addition, after entering the encoder, the image will output a low-level feature, indicating the most basic color and texture features of the image. The low-level features are concatenated with the output of the high-level features in the Decoder, and the final pixel-level segmentation results are obtained through convolution and upsampling.

[0060] In the conventional DeepLabv3+ model, the Xception model is used as the backbone, which uses deep separable convolution to replace the multi-scale convolution network. Since the information of the image in the channel dimension and the dimension can be decoupled, the deep separable convolution performs feature transformation in two dimensions respectively. That is, the number of convolution kernels of the filter is consistent with the number of image channels, and the convolution kernel participates in the calculation on the corresponding channel. Therefore, after the data is input into the separable convolution, the filter first completes the convolution operation in the channel dimension to create a feature matrix, and then performs feature fusion through 1×1 point-by-point convolution. Separable convolution is a lightweight operation that can reduce network parameters while ensuring training accuracy. The Xception architecture used in the classic DeepLabv3+ model is a linear stack of deep separable convolution layers based on residual connections, which has a large number of model parameters and low efficiency.

[0061] In this invention, the conventional DeepLabv3+ model is improved, and MobileNetV2 is used as the backbone of the model. The network structure of MobileNetV2 is as follows: Figure 3As shown in the figure, its structure is an inverted residual structure. The image data is first expanded in channel dimension using 1×1 convolution, and then a 1×1 convolution operation is used after passing through a 3×3 separable convolution layer to obtain an output with the same channel dimension as the original. In addition, since the activation function (ReLU6) is removed in the last layer of the model, the diversity of data features is retained and the network's expressive power is improved. Compared with the conventional residual structure, MobileNetV2 first expands the number of channels through point-by-point convolution, allowing the model to extract features in a higher-dimensional space. At the end of the model, MobileNetV2 introduces a linear bottleneck layer to ensure the effectiveness of neurons in processing diverse features and prevent the loss of too much information. In addition, the inverted residual structure uses a separable convolution operation, which performs convolution operations independently in the channel dimension and the spatial dimension, significantly reducing the parameters required by the model.

[0062] Considering the immediacy of spillway blockage detection, a lightweight MobileNetV2 model is used to replace the original Xception model. Figure 4 This is the structure diagram of the model used. The input image is processed with high-level semantic features and low-level semantic features through a deep convolutional neural network. The channel dimension reduction calculation is input to the decoding module through the ASPP module and point-by-point convolution. In order to ensure consistency with the low-level semantic features, the encoder output is processed by 4 times upsampling. Finally, a 3×3 convolution kernel and upsampling are used to restore it to the same size as the model input image.

[0063] The model's loss function uses CE Loss as the loss function. The optimizer uses the SGD function with momentum to avoid repeated oscillations in the optimal solution when the local gradient is large:

[0064] V dw =βV dw +(1-β)dw

[0065] V db =βV db +(1-β)db

[0066] W=W-αV dw ,b=b-αV db

[0067] Among them, V dw With V db They represent the current weight gradient and offset gradient respectively, β is a parameter that affects the degree of smooth gradient descent and is taken as 0.9. W and b represent the weight and bias of the model respectively.

[0068] The constructed model is trained in the data set, the model files of each training are retained, and the training results with the highest performance are selected for testing; the classification accuracy (Acc) and mean intersection over union (mIoU) are used as evaluation indicators of the model image segmentation performance. The expressions of Acc and mIoU are as follows:

[0069]

[0070] Where n represents the number of different types in the image, TP represents the number of pixels that are positive examples and the model segmentation results are also positive examples. TN represents the number of pixels that are negative examples and the model segmentation results are also negative examples. FN represents the number of pixels that are positive examples and the model segmentation results are negative examples. FP represents the number of pixels that are negative examples and the model segmentation results are positive examples.

[0071] Experimental example:

[0072] The structure of DeepLabV3+ adopted in the present invention is compared with two other commonly used backbone models in the same data set. Figure 5 The loss function changes of the three models in the training set are shown. It can be seen that the model with ResNet50 as the backbone obtains the lowest loss function value, and the Xception model has the largest error after the backbone training. The MobileNetV2 model used as the backbone of DeepLabV3+ has basically stabilized after the loss value is reduced to 0.0041 during training. The model sizes obtained after the training of the three models are 418M, 308M, and 44.7M respectively. The Xception model training requires the most resources and time, and the model size obtained by MobileNetV2 is one order of magnitude lower than the other two models. Figure 6 The segmentation effects of different models in the test set.

[0073] The performance indicators of different backbone models in the test set are shown in Table 1. When the backbone adopts MobileNetV2, Acc and mIoU are improved by 2.87% and 12.61% respectively compared with the conventional DeepLabV3+ model. The Acc and mIoU obtained by the model using ResNet50 as the backbone are improved by 3.99% and 14.80% respectively compared with the conventional DeepLabV3+ model. From the perspective of image segmentation time, the MobileNetV2 model adopted in the present invention has the least average time to perform segmentation tasks in 200 images, and the accuracy obtained is not much different from that of the ResNet50 model. Therefore, it is reasonable to consider using the MobileNetV2 model as the backbone of DeepLabV3+ from the perspective of lightweight, which has a faster segmentation speed while being able to obtain acceptable accuracy, and is more suitable for integration into device terminals.

[0074] Table 1 Model performance indicators

[0075]

[0076] 3. Spillway blockage detection system

[0077] The trained deep learning model saves the network structure and parameters in binary format, and the model is loaded through the Hikvision Super Brain platform. Figure 7 This is the system interface after uploading the model. After binding the model to the engine, the detection task can be performed by selecting the appropriate analysis mode. Figure 8 Configure the engine task in the image import analysis mode. Click "Start Analysis" to execute the image detection task. Fig. 9 It is the interface of the image analysis system.

[0078] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. For ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles described in the present invention, and these improvements and modifications should also be regarded as the protection scope of the present invention.

Claims

1. A spillway accumulation blockage identification method based on deep learning, characterized in that: Includes the following: Establish an image dataset: Establish a spillway chute physical model in the laboratory, and artificially set up piled stones to simulate spillway blockage in different situations; in the process of simulating spillway blockage, use a camera to shoot the blockage situation in the chute physical model, collect image photos, and construct an image dataset; Establish an image segmentation model: Establish an image segmentation model based on deep learning to identify whether there are rocks in the images in the image dataset; Establish an analysis platform: Integrate the constructed image segmentation model into the "AI open source platform" to establish a spillway rock blockage analysis platform.

2. The method for identifying spillway accumulation blockage based on deep learning according to claim 1, characterized in that: The establishment of the image data set includes the following contents: A physical model of the spillway chute was built in the laboratory, and stones of different numbers, shapes, sizes and colors were placed in the chute as blockages to simulate different situations of spillway blockage; A camera was used to record the stone placement images during each spillway blockage simulation process. After removing low-quality and duplicate images, image annotation tools were used to complete image annotation, thereby forming a spillway blockage simulation image dataset, which was divided into a training set and a test set.

3. The method for identifying spillway accumulation blockage based on deep learning according to claim 2, characterized in that: The image segmentation model is established as follows: Create a DeepLabV3+ model in the Pytorch framework and improve the DeepLabV3+ model. The specific contents are as follows: In the encoding network structure, spatial dilated pyramid pooling is used to complete the convolution operation of the image, and low-level features and high-level features are output; wherein the low-level features represent the most basic color and texture features of the image, and the low-level features are spliced ​​with the output of the high-level features in the decoding network structure, and the final pixel-level segmentation result is obtained through convolution and upsampling; MobileNetV2 is used as the backbone of the model. The number of channels is expanded through point-by-point convolution, so that the model can extract features in a higher-dimensional space. A linear bottleneck layer is introduced to ensure the effectiveness of neurons in processing diverse features. CE Loss is used as the loss function of the model, and the SGD function with momentum is used as the optimizer. The specific function is expressed as: V dw =βV dw +(1-β)dw V db =βV db +(1-β)db W=W-αV dw ,b=b-αV db Among them, V dw With V db Represent the current weight gradient and offset gradient respectively; β represents the parameter that affects the degree of smooth gradient descent; W and b represent the weight and bias of the model respectively; Input the images in the training set into the improved DeepLabV3+ model, save the model files after each training, and select the training results with the highest performance for testing; use the classification accuracy and average intersection-over-union ratio as the evaluation indicators of the model image segmentation performance. The expressions of classification accuracy and average intersection-over-union ratio are as follows: Wherein, Acc represents classification accuracy; mIoU represents mean intersection over union; n represents the number of different types in the image; TP represents the number of pixels that belong to positive examples and the model segmentation results are also positive examples; TN represents the number of pixels that belong to negative examples and the model segmentation results are also negative examples; FN represents the number of pixels that belong to positive examples and the model segmentation results are negative examples; FP represents the number of pixels that belong to negative examples and the model segmentation results are positive examples; After the training is completed, a trained image segmentation model based on deep learning is obtained.

4. The method for identifying spillway accumulation blockage based on deep learning according to claim 1, characterized in that: The establishment of the analysis platform includes the following contents: Save the network structure and parameters of the trained deep learning-based image segmentation model in binary format, and load the model through the "AI Open Source Platform"; after the model is bound to the platform engine, complete the image upload and analysis by selecting the appropriate analysis mode.

Images