A computer vision-based large shaft creep detection method for a hydroelectric generator set

Abstract

The application discloses a water turbine generator set large shaft peristalsis detection method based on computer vision, comprising the following steps: S1, collecting water wheel room monitoring video data containing the water turbine generator set large shaft under three states; S2, making a sample set based on the collected video data, and labeling the sample in categories and regions of interest; S3, constructing a water turbine generator set large shaft peristalsis detection neural network based on an improved YOLOv5s-RNN; S4, constructing a training set, a verification set and a test set; S5, inputting the training set into the constructed neural network to train the model, then inputting the test set into the trained deep learning network to perform performance evaluation after adjusting the hyperparameters of the model by using the verification set; S6, saving the model and performing model deployment, using the trained improved YOLOv5s-RNN model to perform water turbine set large shaft peristalsis detection, and issuing an alarm signal when the large shaft of the unit appears peristalsis; the application does not need to install additional hardware devices, is stable, accurate and real-time, and reduces maintenance cost.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of large shaft creep detection of hydroelectric generating set, and particularly relates to a large shaft creep detection method of hydroelectric generating set based on computer vision. BACKGROUND

[0002] The large shaft creep phenomenon of hydroelectric generating set refers to that when the guide vane should be in the full-closed position, the guide vane is not closed tightly, thereby causing water leakage. When the water leakage reaches a certain degree, the runner is slowly rotated, that is, the unit is creeping. When the large shaft creep of hydroelectric generating set occurs, if it cannot be found in time, potential major equipment damage risk will occur, and long-time large shaft creep will cause dry friction and burn the bearing bush. At present, the large shaft creep detection of hydroelectric generating set mainly adopts two methods of non-contact electrical pulse detection and mechanical friction detection.

[0003] The non-contact hydroelectric generating set creep detection method relies on the hydroelectric generating set governor and speed device, and realizes the creep detection function through the cooperation of the probe and the toothed disc. The measurement method needs to install multiple photoelectric probes at the fixed position outside the speed tooth belt, the distance from the probe to the tooth belt is about 1.5 mm, and the adjustment accuracy requirement is high. In addition, due to the small space of the water wheel room, it causes certain inconvenience to the later maintenance and repair of the unit.

[0004] The contact type hydroelectric generating set creep detection method is that after the unit is stopped, the friction wheel is extended from the detection device and contacts the surface of the large shaft. When the large shaft creeps, the friction disc is deflected through the friction force. When the deflection angle reaches a certain angle, the creep device sends a creep alarm signal. Since it is a contact measurement, when the large shaft creep monitoring device is frequently put into and taken out, the device position will be deviated, and the contact with the large shaft will be poor, thereby causing inaccurate measurement, and even the large shaft creep cannot be detected. Moreover, the large shaft creep monitoring device has a complex structure, and it is difficult to install and operate and maintain. SUMMARY

[0005] The present application aims at overcoming the above-mentioned deficiencies, and provides a large shaft creep detection method of hydroelectric generating set based on computer vision, so as to solve the problems in the background art.

[0006] In order to solve the above technical problems, the technical scheme adopted by the present application is as follows: a large shaft creep detection method of hydroelectric generating set based on computer vision, which comprises the following steps:

[0007] S1, collecting water wheel room monitoring video data containing the large shaft of the hydroelectric generating set in three states; wherein the three states are: normal operation state of the hydroelectric generator; unit large shaft non-creep state of the hydroelectric generator in the stopped state; unit large shaft creep state of the hydroelectric generator in the stopped state;

[0008] S2, sample set making based on the collected video data and sample category labeling and region of interest labeling;

[0009] S3, constructing a hydro-turbine generator set large shaft creep detection neural network based on improved YOLOv5s-RNN; wherein the improved YOLOv5s is used to learn the video frame features in the sample and realize target detection, and the RNN is used to learn the video features of the sample;

[0010] S4, constructing a training set, a validation set and a test set; wherein the training set is used to train the neural network, the validation set is used to adjust the model hyperparameters, and the test set is used to test the model performance;

[0011] S5, inputting the training set into the constructed neural network to train the model, and then adjusting the model hyperparameters using the validation set, and inputting the test set into the trained deep learning network to evaluate the performance;

[0012] S6, saving the model and deploying the model, using the trained improved YOLOv5s-RNN model to detect the hydro-turbine generator set large shaft creep, and issuing an alarm signal when the large shaft of the unit appears to creep.

[0013] Preferably, the step S2 specifically comprises the following steps:

[0014] S21, dividing or intercepting the collected hydro-turbine generator set water wheel room monitoring video data under three states into short video sequences of N frames of video frames, and regarding the short video sequences of N frames of length as a sample; finally, a total of running samples, exit samples, creep samples are collected;

[0015] S22, manually labeling each sample, using the numbers 0, 1 and 2 to represent the sample categories, wherein 0 represents running, 1 represents exiting, and 2 represents creeping; using the image calibration tool Lableimg to label the target region of each frame of image of the sample, and boxing the unit main shaft in the candidate box.

[0016] Preferably, in step S3, the improved YOLOv5s-RNN model constructed is specifically:

[0017] S31, the improved YOLOv5s-RNN model is composed of input (video sequence of length N frames), image initialization feature extraction layer, backbone network feature extraction module, neck structure feature fusion module, head structure target detection module and recurrent neural network (RNN) module;

[0018] The image initialization feature extraction layer is used for pre-processing the image, and the image is scaled to a certain size; the structure of the backbone network includes a Conv module (composed of a Conv2d, a BatchNorm2d and a SiLu activation function), a C3 module (composed of three Conv modules and a Bottleneck module using a residual connection), and an SPPF module (spatial pyramid pooling) for extracting shallow graphical features and deep semantic features of the image and obtaining feature maps of different scales; the neck structure includes a Conv module, a C3 module, a Concat operation, and an Upsample operation, has an up-sampling branch and a down-sampling branch, realizes better fusion of shallow graphical features and deep semantic features, and obtains feature maps of different scales; the head structure is composed of three 1*1 convolutions corresponding to three detection layers of different scales, each detection layer is responsible for predicting the category and position information of an object, and through multi-scale detection, different sizes of objects can be better adapted; the RNN is composed of an input layer, a hidden layer and an output layer, the calculation of the hidden layer depends not only on the input of the current video frame, but also on the hidden state of the previous video frame, and is used for capturing the dependency in the video sequence and understanding the context information of the video sequence.

[0019] S32, using the constructed water turbine unit large shaft creep detection neural network based on the improved YOLOv5s-RNN to detect the water turbine unit large shaft creep, adopting the following steps:

[0020] S321, using the image initialization feature extraction layer to perform preliminary feature extraction on the sample video frame;

[0021] S322, using the backbone feature extraction network in the improved YOLOv5s, including the Conv module of the first layer of the backbone network feature extraction module, the Conv module of the second layer of the backbone network feature extraction module, the C3 module of the third layer of the backbone network feature extraction module, the Conv module of the fourth layer of the backbone network feature extraction module, the C3 module of the fifth layer of the backbone network feature extraction module, the Conv module of the sixth layer of the backbone network feature extraction module, the C3 module of the seventh layer of the backbone network feature extraction module, the Conv module of the eighth layer of the backbone network feature extraction module, the C3 module of the ninth layer of the backbone network feature extraction module, and the SPPF module of the tenth layer of the backbone network feature extraction module, to perform image feature extraction from shallow to deep on the sample video frame to obtain image features of multiple scales;

[0022] S323, using the neck structure in the improved YOLOv5s, including the Conv module of the first layer of the neck network feature fusion module of the up-sampling branch, the Upsample module of the second layer of the neck network feature fusion module of the up-sampling branch, the Concat module of the third layer of the neck network feature fusion module of the up-sampling branch, the C3 module of the fourth layer of the neck network feature fusion module of the up-sampling branch, the Conv module of the fifth layer of the neck network feature fusion module of the up-sampling branch, the Upsample module of the sixth layer of the neck network feature fusion module of the up-sampling branch, the Concat module of the seventh layer of the neck network feature fusion module of the up-sampling branch, the C3 module of the first layer of the neck network feature fusion module of the down-sampling branch, the Conv module of the second layer of the neck network feature fusion module of the down-sampling branch, the Concat module of the third layer of the neck network feature fusion module of the down-sampling branch, the C3 module of the fourth layer of the neck network feature fusion module of the down-sampling branch, the Conv module of the fifth layer of the neck network feature fusion module of the down-sampling branch, the Concat module of the sixth layer of the neck network feature fusion module of the down-sampling branch, and the C3 module of the seventh layer of the neck network feature fusion module of the down-sampling branch, to realize better fusion of features of different scales;

[0023] In the Concat module of the third layer of the neck network feature fusion module of the up-sampling branch, the C3 module of the seventh layer of the backbone network feature extraction module is realized to perform a splicing fusion operation on the extracted features; in the Concat module of the seventh layer of the neck network feature fusion module of the up-sampling branch, the C3 module of the fifth layer of the backbone network feature extraction module is realized to perform a splicing fusion operation on the extracted features; in the Concat module of the third layer of the neck network feature fusion module of the down-sampling branch, the Conv module of the fifth layer of the neck network feature fusion module of the up-sampling branch is realized to perform a splicing fusion operation on the extracted features; in the Concat module of the sixth layer of the neck network feature fusion module of the down-sampling branch, the Conv module of the first layer of the neck network feature fusion module of the up-sampling branch is realized to perform a splicing fusion operation on the extracted features;

[0024] S324, the C3 module of the first layer of the down-sampling branch of the neck network feature fusion module, the C3 module of the fourth layer of the down-sampling branch of the neck network feature fusion module, and the C3 module of the seventh layer of the down-sampling branch of the neck network feature fusion module are used as inputs of three different scales of the head structure target detection module in the improved YOLOv5s, and three corresponding decoding layers (1*1 convolution modules) are used for decoding to obtain candidate box position information, target categories and confidence;

[0025] S325, the features obtained by the C3 module of the neck network feature fusion module down-sampling branch seventh layer are input into a full connection layer, and then input into a hidden layer and an output layer to obtain video feature encoding of each sample.

[0026] Preferably, in the step S3, the improvement of YOLOv5s is realized in the following way:

[0027] All Conv modules in the original network are replaced by attention mechanism Conv modules, which pay attention to channel attention and spatial attention at the same time, and pay attention to important information from two dimensions of features and space, and the function is as follows:

[0028]

[0029] Wherein, represents the input of the attention mechanism Conv module, represents the element-wise multiplication operation, represents the channel attention output, represents the spatial attention output, represents the channel attention mapping, represents the spatial attention mapping.

[0030]

[0031] Wherein, represents the sigmoid activation function, represents the multi-layer perception network, represents the average pooling operation, represents the maximum pooling operation, and represents the weight of the multi-layer perception, represents the output of the average pooling, represents the output of the maximum pooling.

[0032]

[0033] Wherein, represents the convolution operation with the convolution kernel size of , represents the output of the average pooling, represents the output of the maximum pooling. represents that the feature map obtained by dimensionally splicing the outputs of the average pooling and the maximum pooling is taken as the input of the convolution.

[0034] Preferably, the step S4 divides the training set, the validation set and the test set in the following way:

[0035] For the samples with completed annotation, one running sample, one exit sample, one peristalsis sample; the three-class sample set is divided into training set, validation set and test set in the ratio of 7:2:1 by random division;

[0036] The number of training set samples is: ;

[0037] The number of validation set samples is: ;

[0038] The number of test set samples is: .

[0039] Preferably, the step S5 specifically comprises the following steps:

[0040] S51, set the hyperparameters required for training, including epoch, batch_size, learning rate, training optimizer and loss function; wherein the loss function of the improved YOLOv5s-RNN model is:

[0041]

[0042] In the formula, , , , represents the loss weight, represents the boundary box loss calculated using the IoU loss, represents the confidence loss calculated using the binary cross entropy, represents the target detection loss calculated using the binary cross entropy, represents the water turbine unit large shaft state prediction loss calculated using the cross entropy loss;

[0043] S52, set the model evaluation index:

[0044] Two widely used evaluation indexes for classification tasks, namely accuracy (Accuarcy) and F1-score, are selected as the evaluation indexes; the number of classes of the data set for water turbine unit large shaft peristalsis detection is {k=1,2,3}, when the kth class is regarded as a positive sample (Positive), all other classes except the kth class belong to negative samples (Negative); then for the kth class,

[0045]

[0046]

[0047]

[0048]

[0049] Wherein, TP represents successfully predicting positive samples as positive samples, TN represents successfully predicting negative samples as negative samples, FP represents wrongly predicting negative samples as positive samples, and FN represents wrongly predicting positive samples as negative samples, represents the precision when the kth class is regarded as a positive sample, represents the recall when the kth class is regarded as a positive sample.

[0050] Taking the unweighted average of each class as the final evaluation index,

[0051]

[0052]

[0053] S53, initialize the network parameters, and input the training set into the improved YOLOv5s-RNN model constructed, start the training process, update the model parameters according to the calculated loss through the gradient descent algorithm until the training is completed, then freeze the model parameters, input the verification set into the improved YOLOv5s-RNN model trained, optimize and adjust the model through the evaluation index, finally, input the test set into the improved YOLOv5s-RNN model optimized to evaluate the performance of the model.

[0054] Preferably, in step S6, when the improved YOLOv5s-RNN model trained is used for large shaft creep detection of a hydro-turbine unit, the following steps are adopted:

[0055] S61, save the model (including model architecture and parameters) when the evaluation index of the model is optimal in the verification process;

[0056] S62, write a real-time video data reading script to realize uninterrupted reading of video data, and use the image initialization layer to preprocess the sample video frame;

[0057] S63, call the saved improved YOLOv5s-RNN model to perform real-time detection of the large shaft creep of the hydro-turbine unit, and return the large shaft detection result and the predicted state of the hydro-turbine unit.

[0058] The present application has the following advantages:

[0059] 1. The present application solves the problems of traditional creep detection devices, such as inaccurate adjustment of the distance between the probe and the toothed belt, dangerous installation position, and inconvenient inspection, etc. The present application method does not need to install additional hardware devices, and is stable, accurate, real-time, and reduces maintenance cost.

[0060] 2. This invention obtains video containing the turbine shaft only through a monitoring camera installed in the water turbine room. Based on computer vision technology, it realizes real-time detection and alarm of the turbine generator shaft creep, reducing potential hazards and equipment damage caused by turbine shaft creep.

[0061] 3. This invention addresses the multi-view, multi-scale problem of generator main shaft appearing in video sequences by designing an improved YOLOv5s model based on an attention mechanism to achieve real-time detection and recognition of generator main shaft; this invention also achieves video feature learning based on an RNN model, thereby enabling real-time detection of generator main shaft creep.

[0062] 4. This invention uses an improved YOLOv5s algorithm based on an attention mechanism to achieve adaptive detection of the generator set's main shaft. Secondly, it uses an RNN network to learn the video features of the video sequence, thereby realizing the detection of the generator set's main shaft's operating status. The method proposed in this invention has high detection accuracy and can intelligently detect and track the operating status of the generator set's main shaft in real time, enabling early detection and alarm of generator set main shaft creep. Attached Figure Description

[0063] Figure 1 This is a monitoring image of a waterwheel room;

[0064] Figure 2 This is a flowchart illustrating a computer vision-based method for detecting the main shaft creep of a hydro turbine unit.

[0065] Figure 3 This is a diagram of a hydro-generator shaft creep detection system based on YOLOv5s-RNN as the main network.

[0066] Figure 4 This invention provides an improved YOLOv5s network framework diagram;

[0067] Figure 5 This is a diagram of the RNN network framework of the present invention;

[0068] Figure 6 This is a schematic diagram of the Conv module based on the attention mechanism of the present invention. Detailed Implementation

[0069] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0070] Example 1: As Figure 2 As shown in the figure, this embodiment provides a computer vision-based method for detecting the creep of a turbine main shaft, which includes the following steps:

[0071] S1, collect water wheel room monitoring video data containing the large shaft of the hydroelectric generator set in three states. The three states are ① normal operation state of the hydroelectric generator; ② hydroelectric generator shutdown, unit large shaft not creeping state; ③ hydroelectric generator shutdown, unit large shaft creeping state.

[0072] S2, based on the collected video data, sample set is made, and the samples are classified and labeled.

[0073] S3, construct a hydroelectric generator set large shaft creeping detection neural network based on improved YOLO v5s-RNN. Among them, the improved YOLOv5s is used to learn the video frame features in the sample and realize target detection, and the RNN is used to learn the video features of the sample.

[0074] S4, construct training set, validation set and test set. The training set is used to train the neural network, the validation set is used to adjust the model hyperparameters, and the test set is used to test the model performance.

[0075] S5, input the training set into the constructed neural network to train the model, and then adjust the model hyperparameters using the validation set. After that, input the test set into the trained deep learning network to evaluate the model.

[0076] S6, save the model and deploy the model, and when the unit large shaft creeps, an alarm signal is sent.

[0077] Further, step S2 includes:

[0078] S21, divide or intercept the collected water wheel room monitoring video data of the hydroelectric generator set in three states (running, exiting, and creeping) into short video sequences with N=10 frames, and regard the short video sequences with length N=10 frames as a sample. Finally, a total of =1000 running samples, =1000 exit samples, =1000 creeping samples.

[0079] S22, manually label each sample, and use the RectLabel image labeling tool to classify and label the sample category (use numbers 0, 1, and 2 to represent the sample category, where 0 represents running, 1 represents exiting, and 2 represents creeping); use the image calibration tool Lableimg to label the target area of each frame of the sample (as shown in Figure 1 ), and frame the unit main shaft in the candidate box.

[0080] Further, step S3 includes:

[0081] Construct a hydroelectric generator set large shaft creeping detection neural network based on improved YOLOv5s-RNN. Its network structure is as followsFigure 3 As shown, it can be divided into four parts: (1) improving the backbone network in YOLOv5s (for multi-scale feature extraction); (2) improving the neck structure in YOLOv5s (for multi-scale feature fusion); (3) improving the head structure in YOLOv5s (for object detection); (4) as shown Figure 5 The RNN module shown is used for video sequence encoding.

[0082] like Figure 4 As shown, the forward propagation process of the turbine shaft creep detection neural network based on the improved YOLOv5s-RNN is as follows: The video frames in the sample first pass through the image initialization layer, are adjusted to a square size using padding, and then proportionally scaled to 416*416*3 before being used as input to the backbone network. They then pass sequentially through the first layer (Conv module with dimensions (320, 320, 64)), the second layer (Conv module with dimensions (160, 160, 128)), the third layer (C3 module with dimensions (160, 160, 128)), and the fourth layer (C3 module with dimensions (80, 8)). The system uses a Conv module with dimensions (0, 256), a C3 module with dimensions (80, 80, 256) in the fifth layer of the backbone, a Conv module with dimensions (40, 40, 512) in the sixth layer of the backbone, a C3 module with dimensions (40, 40, 512) in the seventh layer of the backbone, a Conv module with dimensions (20, 20, 1024) in the eighth layer of the backbone, a C3 module with dimensions (20, 20, 1024) in the ninth layer of the backbone, and an SPPF module with dimensions (20, 20, 1024) in the tenth layer of the backbone to extract image features from sample video frames from shallow to deep, thereby obtaining image features at multiple scales.

[0083] The output features of the backbone tenth layer SPPF module with a size of (20, 20, 1024) enter the neck structure, first pass through an up-sampling with a size of (40, 40, 512), are spliced with the output features of the backbone seventh layer C3 module with a size of (40, 40, 512) to obtain features with a size of (40, 40, 1024), and then sequentially pass through the neck up-sampling branch fourth layer C3 module with a size of (40, 40, 512), the neck up-sampling branch fifth layer Conv module with a size of (40, 40, 256), and the neck up-sampling branch sixth layer Upsample module with a size of (80, 80, 256), are spliced with the output features of the backbone fifth layer C3 module with a size of (80, 80, 256) to obtain features with a size of (80, 80, 512), and are taken as the input of the neck down-sampling branch, pass through the neck down-sampling branch first layer C3 module with a size of (80, 80, 256), the neck down-sampling branch second layer Conv module with a size of (40, 40, 256), are spliced with the output features of the neck up-sampling branch fifth layer Conv module with a size of (40, 40, 256) to obtain features with a size of (40, 40, 512), and then pass through the neck down-sampling branch fourth layer C3 module with a size of (40, 40, 512), the neck down-sampling branch fifth layer Conv module with a size of (20, 20, 512), are spliced with the output features of the neck up-sampling branch first layer Conv module with a size of (20, 20, 512) to obtain features with a size of (20, 20, 1024), and finally pass through the neck down-sampling branch seventh layer C3 module with a size of (20, 20, 1024).

[0084] The three different size features, including the output features of the neck down-sampling branch first layer C3 module, the output features of the neck down-sampling branch fourth layer C3 module, and the output features of the neck down-sampling branch seventh layer C3 module, are decoded by the head structure to generate the candidate box upper left corner coordinates (x, y) and the candidate box width w and height h, the target class and the confidence.

[0085] The output features of the neck down-sampling branch seventh layer C3 module with a size of (20, 20, 1024) are first input into an input layer, a hidden layer, and an output layer of an RNN network with a size of {1024, 256, 3} via a fully connected layer with a size of (20*20*1024, 20*1024) to “flatten” the feature map into a vector for video feature learning.

[0086] To improve the prediction accuracy of the model, all Conv modules in the original network are replaced by attention mechanism Conv modules, as shown in Figure 6 The attention mechanism Conv module pays attention to spatial attention and channel attention at the same time, and focuses on important information from the feature and spatial dimensions respectively, and its function is as follows:

[0087]

[0088] wherein, represents the input of the attention mechanism Conv module, represents the corresponding element multiplication operation, represents the channel attention output, represents the spatial attention output, represents the channel attention mapping, represents the spatial attention mapping;

[0089]

[0090] wherein, represents the sigmoid activation function, represents the multi-layer perception network, represents the average pooling operation, represents the maximum pooling operation, and represents the weight of the multi-layer perception, represents the output of the average pooling, represents the output of the maximum pooling.

[0091]

[0092] wherein, represents the convolution operation with a convolution kernel size of represents the output of the average pooling, represents the output of the maximum pooling. represents that the feature map obtained by dimensionally splicing the outputs of the average pooling and the maximum pooling is taken as the input of the convolution. Further, step S4 comprises:

[0093] For the 1000 running samples, 1000 exit samples and 1000 peristalsis samples that have been labeled, the three-class sample set samples are divided into a training set, a validation set and a test set in a random division manner at a ratio of 7:2:1.

[0094] The divided training set contains 2100 samples, the validation set contains 600 samples, and the test set contains 300 samples.

[0095] ​

[0096] Further, step S5 comprises:

[0097] S51, set the hyperparameters required for training epoch=1000, batch_size=64, learning rate is 0.0001, training optimizer is Adam, set the loss function as:

[0098]

[0099] wherein, , , , represents the loss weight, represents the boundary box loss calculated using the IoU loss, represents the confidence loss calculated using the binary cross entropy, represents the target detection loss calculated using the binary cross entropy, represents the turbine large shaft state prediction loss calculated using the cross entropy loss. In the embodiment =0.2, =0.2, =0.3, =0.3.

[0100] S52, set the model evaluation index, and the evaluation index selects two widely used evaluation indexes for classification tasks, namely accuracy (Accuarcy) and F1-score. Let the class number of the data set for the large shaft creep detection of the hydraulic turbine unit be {k=1, 2, 3}, when the kth class is regarded as a positive sample (Positive), all other classes except the kth class belong to negative samples (Negative). Then for the kth class,

[0101]

[0102]

[0103]

[0104]

[0105] wherein, TP represents successfully predicting a positive sample as a positive sample, TN represents successfully predicting a negative sample as a negative sample, FP represents incorrectly predicting a negative sample as a positive sample, and FN represents incorrectly predicting a positive sample as a negative sample, represents the precision rate when the kth class is regarded as a positive sample, represents the recall rate when the kth class is regarded as a positive sample.

[0106] The unweighted average value of each category is taken as the final evaluation index,

[0107]

[0108]

[0109] S53, initialize the network parameters, and input the training set into the improved YOLOv5s-RNN model constructed, start the training process, update the model parameters according to the calculated loss through the gradient descent algorithm, and stop until the training is completed. After freezing the model parameters, input the verification set into the trained improved YOLOv5s-RNN model, and optimize and adjust the model through the evaluation index, such as adjusting the initial learning rate, batch_size, etc. Finally, input the test set into the optimized improved YOLOv5s-RNN model to evaluate the performance of the model.

[0110] Further, step S6 comprises:

[0111] S61, save the model (including model architecture and parameters) at the optimal evaluation index in the verification process.

[0112] S62, write a real-time video data reading script to realize uninterrupted reading of video data, and use the image initialization layer to preprocess the sample video frame.

[0113] S63, call the saved YOLOv5s-RNN model to perform real-time detection of the large shaft creep of the hydro-turbine unit, and return the detection result and the predicted state of the large shaft of the hydro-turbine unit.

[0114] Preferably, the performance results of the basic model and the improved model can be obtained as shown in the following table.

[0115]

[0116] The above embodiments are only preferred technical solutions of the present application, and should not be regarded as limitations of the present application. The protection scope of the present application should be based on the technical solutions recited in the claims, including equivalent replacement solutions of the technical features recited in the claims. That is, equivalent replacement improvements within this scope are also within the protection scope of the present application.

Claims

1. A computer vision-based method for detecting the creep of the main shaft of a hydro-generator unit, characterized in that: It includes the following steps: S1. Collect monitoring video data of the turbine room containing the main shaft of the hydro-generator unit under three states: normal operation of the hydro-generator; hydro-generator stopped, main shaft not moving; hydro-generator stopped, main shaft moving. S2. Based on the collected video data, create a sample set and label the samples with categories and regions of interest; S3. Construct a neural network for detecting the main shaft creep of a hydro-generator unit based on an improved YOLOv5s-RNN; wherein, the improved YOLOv5s is used to learn the video frame features in the samples and realize target detection, and the RNN is used to learn the video features of the samples. S4. Construct training, validation, and test sets; the training set is used to train the neural network, the validation set is used to tune the model's hyperparameters, and the test set is used to test the model's performance. S5. Input the training set into the constructed neural network to train the model, then use the validation set to adjust the model's hyperparameters, and finally input the test set into the trained deep learning network for performance evaluation. S6. Save and deploy the model. Use the trained improved YOLOv5s-RNN model to detect the creep of the turbine shaft. When creep occurs in the turbine shaft, an alarm signal is issued. The loss function of the improved YOLOv5s-RNN model is: ； In the formula, , , , Indicates the loss weight. This represents the bounding box loss calculated using IoU loss. This represents the confidence loss calculated using binary cross-entropy. This represents the target detection loss calculated using binary cross-entropy. This represents the predicted loss of the turbine shaft state calculated using cross-entropy loss; In step S3, the improvements to YOLOv5s are implemented as follows: Replace all Conv modules in the original network with Conv modules using an attention mechanism.

2. The method for detecting the main shaft creep of a hydro-generator unit based on computer vision according to claim 1, characterized in that: The attention mechanism Conv module simultaneously focuses on channel attention and spatial attention, focusing on important information from both feature and spatial dimensions, respectively. Its function is as follows: ； in, This represents the input to the Conv module of the attention mechanism. This indicates an element-wise multiplication operation. Indicates channel attention output, This indicates spatial attention output. This represents channel attention mapping. Represents spatial attention mapping; ； in, This represents the sigmoid activation function. This represents a multilayer perceptron network. This indicates the average pooling operation. This indicates a max pooling operation. and The weights of the multilayer perceptron are represented. This represents the output of average pooling. This represents the output of max pooling; ； in, Indicates the kernel size as Convolution operation, This represents the output of average pooling. This represents the output of max pooling. This means that the feature map obtained by concatenating the outputs of average pooling and max pooling is used as the input of the convolution.

3. The method for detecting the main shaft creep of a hydro-generator unit based on computer vision according to claim 1, characterized in that: Step S2 specifically includes the following steps: S21. Divide or extract the collected monitoring video data of the turbine generator unit under three states into short video sequences of N frames each, and treat each N-frame short video sequence as a sample; finally, a total of One running sample, One exit sample, One peristaltic sample; S22. Manually label each sample, using the numbers 0, 1, and 2 to represent the sample category, where 0 represents running, 1 represents exiting, and 2 represents crawling; use the image labeling tool Labelimg to label the target region of each frame of the sample image, and frame the main axis of the unit within the candidate box.

4. The method for detecting the main shaft creep of a hydro-generator unit based on computer vision according to claim 1, characterized in that: In step S3, the improved YOLO v5s-RNN model constructed is as follows: S31. The improved YOLOv5s-RNN model consists of an input image initialization feature extraction layer, a backbone network feature extraction module, a neck structure feature fusion module, a head structure target detection module, and a recurrent neural network (RNN) module. The input is a video sequence of length N frames. The image initialization feature extraction layer is used to preprocess the image, scaling it to a certain size. The backbone network consists of Conv, C3, and SPPF modules, used to extract shallow graphic features and deep semantic features of the image, and to obtain feature maps at different scales. The Conv module consists of a Conv2d, a BatchNorm2d, and a SiLu activation function. The C3 module consists of three Conv modules and a Bottleneck module using residual connections. The neck structure consists of Conv, C3, Concat operation, and Upsample. The operation has one upsampling branch and one downsampling branch, which can better integrate shallow graphic features and deep semantic features and obtain feature maps at different scales. The head structure consists of three 1*1 convolutions, corresponding to three detection layers at different scales. Each detection layer is responsible for predicting the category and location information of the object. Through multi-scale detection, it can better adapt to objects of different sizes. The RNN consists of an input layer, a hidden layer and an output layer. The calculation of its hidden layer depends not only on the input of the current video frame, but also on the hidden state of the previous video frames. It is used to capture the dependencies in the video sequence and understand the contextual information of the video sequence. S32. The turbine shaft creep detection neural network based on the improved YOLOv5s-RNN is used to perform turbine shaft creep detection, and the following steps are taken: S321. Use the image initialization feature extraction layer to perform preliminary feature extraction on the sample video frames; S322. Using the improved YOLOv5s backbone feature extraction network, which includes the Conv module of the first layer of the backbone feature extraction module, the Conv module of the second layer of the backbone feature extraction module, the C3 module of the third layer of the backbone feature extraction module, the Conv module of the fourth layer of the backbone feature extraction module, the C3 module of the fifth layer of the backbone feature extraction module, the Conv module of the sixth layer of the backbone feature extraction module, the C3 module of the seventh layer of the backbone feature extraction module, the Conv module of the eighth layer of the backbone feature extraction module, the C3 module of the ninth layer of the backbone feature extraction module, and the SPPF module of the tenth layer of the backbone feature extraction module, image feature extraction is performed on the sample video frames from shallow to deep to obtain image features at multiple scales; S323. Using the improved neck structure in YOLOv5s, including the Conv module of the first layer of the upsampling branch of the neck network feature fusion module, the Upsample module of the second layer of the upsampling branch of the neck network feature fusion module, the Concat module of the third layer of the upsampling branch of the neck network feature fusion module, the C3 module of the fourth layer of the upsampling branch of the neck network feature fusion module, the Conv module of the fifth layer of the upsampling branch of the neck network feature fusion module, the Upsample module of the sixth layer of the upsampling branch of the neck network feature fusion module, the Concat module of the seventh layer of the upsampling branch of the neck network feature fusion module, the C3 module of the first layer of the downsampling branch of the neck network feature fusion module, the Conv module of the second layer of the downsampling branch of the neck network feature fusion module, the Concat module of the third layer of the downsampling branch of the neck network feature fusion module, the C3 module of the fourth layer of the downsampling branch of the neck network feature fusion module, the Conv module of the fifth layer of the downsampling branch of the neck network feature fusion module, the Concat module of the sixth layer of the downsampling branch of the neck network feature fusion module, and the C3 module of the seventh layer of the downsampling branch of the neck network feature fusion module, to achieve better fusion of features at different scales; Specifically, the Concat module in the third layer of the upsampling branch of the neck network feature fusion module performs a concatenation and fusion operation with the C3 module in the seventh layer of the backbone network feature extraction module; the Concat module in the seventh layer of the upsampling branch of the neck network feature fusion module performs a concatenation and fusion operation with the C3 module in the fifth layer of the backbone network feature extraction module; the Concat module in the third layer of the downsampling branch of the neck network feature fusion module performs a concatenation and fusion operation with the Conv module in the fifth layer of the upsampling branch of the neck network feature fusion module; and the Concat module in the sixth layer of the downsampling branch of the neck network feature fusion module performs a concatenation and fusion operation with the Conv module in the first layer of the upsampling branch of the neck network feature fusion module. S324. The features obtained by the C3 module of the first layer of the downsampling branch of the neck network feature fusion module, the C3 module of the fourth layer of the downsampling branch of the neck network feature fusion module, and the C3 module of the seventh layer of the downsampling branch of the neck network feature fusion module are used as inputs to the head structure target detection module in the improved YOLOv5s at three different scales, and are decoded using three corresponding 1*1 convolutional modules to obtain candidate box position information, target category and confidence. S325. The features obtained by the C3 module of the seventh layer of the downsampling branch of the neck network feature fusion module are passed through a fully connected layer and used as the input of the RNN at a certain time. After passing through the hidden layer and the output layer, the video feature code of each sample is obtained.

5. The method for detecting the main shaft creep of a hydro-generator unit based on computer vision according to claim 1, characterized in that: Step S4 divides the training set, validation set, and test set as follows: For the completed annotations One running sample, One exit sample, One crawling sample; the three-class sample set is randomly divided into training set, validation set and test set in a ratio of 7:2:1; The number of samples in the training set is: ; The number of samples in the validation set is: ; The number of test set samples is: .

6. The method for detecting the main shaft creep of a hydro-generator unit based on computer vision according to claim 1, characterized in that: Step S5 specifically includes the following steps: S51. Set the hyperparameters required for training, including epoch, batch_size, learning rate, and training optimizer; S52. Set model evaluation indicators: Two widely used evaluation metrics for classification tasks were selected: accuracy and F1-score. The dataset used for detecting the main shaft creep of the hydroelectric turbine has {k=1,2,3} classes. When class k is considered a positive sample, all other classes are considered negative samples. Therefore, for class k... ； ； ； ； Where TP indicates that a positive sample was successfully predicted as a positive sample, TN indicates that a negative sample was successfully predicted as a negative sample, FP indicates that a negative sample was incorrectly predicted as a positive sample, and FN indicates that a positive sample was incorrectly predicted as a negative sample. This represents the precision when the k-th class is considered a positive sample. This represents the recall rate when the k-th class is considered a positive sample; The unweighted average of each category was used as the final evaluation metric. ； ； S53. Initialize the network parameters and input the training set into the constructed improved YOLOv5s-RNN model to start the training process. Based on the calculated loss, update the model parameters using the gradient descent algorithm until training is complete. Then, freeze the model parameters, input the validation set into the trained improved YOLOv5s-RNN model, optimize and adjust the model using evaluation metrics, and finally, input the test set into the optimized improved YOLOv5s-RNN model to evaluate the model's performance.

7. The method for detecting the main shaft creep of a hydro-generator unit based on computer vision according to claim 1, characterized in that: In step S6, when using the trained improved YOLOv5s-RNN model to detect the main shaft creep of the turbine unit, the following steps are taken: S61. Save the model architecture and parameters when the evaluation index is optimal during the model validation process; S62. Write a real-time video data reading script to achieve uninterrupted reading of video data, and use the image initialization layer to preprocess the sample video frames; S63. Call the saved improved YOLOv5s-RNN model to perform real-time detection of turbine shaft creep, and return the turbine shaft detection results and predicted status.

Images