Hydropower station state automatic inspection device and inspection method thereof

Abstract

A hydropower station state automatic inspection device disclosed by the present invention comprises a rectangular machine body and a control center, a driving mechanism is arranged in the machine body, the driving mechanism is connected with a gear box and a ground climbing mechanism, the ground climbing mechanism is arranged on the outer side wall of the machine body, the gear box is arranged at the top end of the machine body, and the gear box is connected with a flight mechanism. A holder is arranged on the side, close to the gearbox, of the machine body, and a camera is installed on the holder and wirelessly connected with the control center. The invention further discloses a hydropower station state inspection method, and the hydropower station state automatic inspection device and the inspection method thereof provided by the invention solve the problems of low scene recognition precision and repeated training in the prior art.

Description

Technical Field

[0001] The invention belongs to the technical field of health monitoring of equipment in a hydropower station, relates to an automatic inspection device for pipeline status in a hydropower station, and also relates to an inspection method using the automatic inspection device for pipeline status in a hydropower station. Background Art

[0002] As the largest component structure in old hydropower stations, pipelines undertake important functions such as oil, gas and water transportation. They are also the focus of daily operation and maintenance and inspection of old hydropower stations. It is of great significance to ensure the safe and reliable operation of pipelines in hydropower stations. At present, the inspection of pipelines and meters on pipelines in old power stations mainly relies on manual inspection. The inspection content includes pipeline leakage, meter readings, pointer readings and pipeline swing monitoring. Due to the large number of pipeline equipment in hydropower stations, including multi-media pipelines such as oil, water and gas, as well as various types of pipelines such as thick pipes, thin pipes and elbows. In addition, the need for 24-hour operation of hydropower stations leads to high inspection load and high work intensity for operation and maintenance personnel on pipelines. At the same time, the inspection effect of different inspectors depends on their personal experience and ability, and their inspection effect has a certain degree of randomness. In recent years, with the development of new technologies, some newly built hydropower stations have adopted robots to replace manual inspections, and with image processing and analysis algorithms, they can realize all-weather inspection of hydropower station pipelines. The use of robot inspection has become the mainstream of future hydropower station inspections.

[0003] At present, there are two main types of inspection robots used in new power stations, including wheeled robots and rail-mounted robots. However, for new hydropower stations, the use scenarios of robots are taken into consideration in the design and planning, including the design of tracks and doors. However, for old power stations, the use of existing wheeled robots and rail-mounted robots mainly faces the following difficulties: (1) The use of robots was not considered during the construction of old hydropower stations. The use of existing wheeled and rail-mounted robots requires the reconstruction of existing door openings, factory buildings and other scenes, which is a huge construction workload; (2) Regardless of whether it is a wheeled or rail-mounted robot, it mainly inspects indoor environments such as the power generation layer, turbine layer, waterwheel room and simple outdoor environments such as switch stations, and cannot inspect the pipeline status of facilities such as dams and spillways; (3) The visual processing algorithms carried by current robots are not designed for existing hydropower station facilities and equipment. They all use a single neural network with large-scale neurons for multi-scene recognition, resulting in low recognition accuracy for a certain scene, and need to be retrained for different scenes, and the large-scale neural network has weak migration capabilities. Summary of the invention

[0004] The first purpose of the present invention is to provide an automatic inspection device for the status of a hydropower station, which solves the problems of low scene recognition accuracy and repeated training in the prior art.

[0005] The second object of the present invention is to provide an inspection method using an automatic inspection device for the status of a hydropower station.

[0006] The first technical solution adopted by the present invention is an automatic inspection device for the status of a hydropower station, including a rectangular body and a control center. A driving mechanism is provided in the body, and the driving mechanism is respectively connected to a gear box and a crawling mechanism. The crawling mechanism is provided on the outer wall of the body, and the gear box is provided on the top of the body. The gear box is connected to a flying mechanism. A pan-tilt platform is provided on the side of the body close to the gear box, and a camera is installed on the pan-tilt platform. The camera is connected to the control center wirelessly.

[0007] The first technical solution of the present invention is also characterized in that:

[0008] The driving mechanism includes two motors, and the crawling mechanism includes four legs. Two legs are symmetrically installed on both side walls of the machine body, and the two motors are respectively connected to the two legs through a transmission shaft.

[0009] The flight mechanism includes four rotors, which are respectively connected to a gear box through transmission shafts, and the four rotors are evenly distributed along the center of the gear box.

[0010] The second technical solution adopted by the present invention is a hydropower station status inspection method, which adopts a hydropower station status automatic inspection device, and the specific steps are as follows:

[0011] S1, use the camera to capture the target scene image and build a twin-residual neural network model;

[0012] S2, transmits the target scene image captured in S1 to the twin-residual neural network model through the control center, trains the twin-residual neural network model to classify the scene images into digital meter scene images, pointer meter scene images and pipeline scene images;

[0013] S3, performing scene image recognition on the digital meter scene image, the pointer meter scene image and the pipeline scene image respectively, outputting the scene image recognition result, and completing the status monitoring of the hydropower station.

[0014] The second technical solution of the present invention is also characterized in that:

[0015] The specific steps for building the twin-residual neural network model in S1 are:

[0016] The twin architecture is formed by the upper and lower convolutional neural networks, and the residual learning module is added to the upper and lower convolutional neural network structure to complete the construction of the twin-residual neural network model;

[0017] The specific steps for training the twin-residual neural network model for image classification in S2 are:

[0018] S2.2.1. Constructing the Dataset

[0019] Assume that there are m scenes to be classified, each scene has n target image samples, and the number of target image sample pairs to be formed is N; the number of target image sample pairs is divided into the number of target image positive sample pairs and the number of target image negative sample pairs. The number of target image positive sample pairs constructed is the same as the number of target image negative samples, that is, the number of positive sample pairs and negative sample pairs is N / 2. The labels corresponding to the number of target image positive sample pairs and the number of target image negative sample pairs are defined as:

[0020]

[0021] Where: y is the true label of the sample pair; is the Dirac delta function, k1 and k2 are different image samples. When k1 and k2 belong to the same scene, their label value is defined as 1, and when they do not belong to the same scene, their label value is 0.

[0022] S2.2.2, construct loss function

[0023] A loss function based on the cross entropy function is constructed, and the overfitting risk of the twin residual neural network model is reduced by adding the L2 weight decay term to the loss function. The final loss function J is expressed as:

[0024] J(k1,k2)=y(k1,k2)logh θ (k1,k2)+(1-y(k1,k2))log(1-h θ (k1,k2))+λθ T θ(2);

[0025] In the formula, y(k1,k2) is the sample label; h θ (x1, x2) is the sample pair similarity output by the model; θ is the weight of the residual twin neural network; λ∈[0,∞) is the L2 regularization coefficient;

[0026] Secondly, during the training process, the data is learned by minimizing the loss function by introducing the stochastic gradient descent optimization algorithm. At the same time, the momentum term is added to minimize the objective function, and the loss function is minimized by stochastic gradient descent with momentum term. The specific formula is as follows:

[0027]

[0028] In the formula, the subscript i represents the i-th iteration step in model training; η represents the learning rate; μ represents the momentum parameter; Δθ represents the parameter change in each training step. represents the function gradient;

[0029] S2.2.3, construct similarity measurement function

[0030] The convolutional features of the target image are identified by the upper and lower convolutional neural networks respectively, and the convolutional features of the target image are further processed by the fully connected network. Finally, the scene classification is realized by the similarity metric function. The specific formula is:

[0031] Similarity = ξ(f θ (k1,k2)) (4);

[0032] In the formula, f θ (·) is the twin-residual neural network model for extracting features; ξ is the similarity measurement function, and θ is the neural network weight;

[0033] For high-precision classification of different scenarios, a similarity measurement function based on the Langmuir distance is constructed. The Langmuir distance d can be described as:

[0034]

[0035] Where, X up and X down The convolution features of the target image obtained by the upper and lower layers of the neural network respectively, and P is the dimension of the convolution features in the fully connected layer.

[0036] The specific steps of recognizing the digital meter scene image in S3 are:

[0037] a1, multi-scale decomposition of standard digital meter scene image

[0038] The standard digital meter images of numbers 0 to 9 are represented in multi-scale space, and a standard data multi-scale space sample library is established. The multi-scale space representation process of digital meter images is as follows:

[0039] The Gaussian basis function is used to smooth the standard image of the digital meter. The specific formula is as follows:

[0040]

[0041] Where L(x,y,σ) is the smoothed image, I(x,y) is the standard data image, G(x,y,σ) is the Gaussian basis function, and σ is the scale parameter. Represents a convolution operation.

[0042] Secondly, the smoothed image is downsampled. The specific formula is as follows:

[0043]

[0044] In the formula, I downsample (σ+1) represents the sampled image, and s represents the downsampling window size;

[0045] Repeat iterative smoothing and sampling m times to obtain digital meter images with different resolutions and sizes, and construct a multi-scale pyramid of standard digital meter images, with a total of m layers;

[0046] a2, similarity calculation of digital meter images of different scales

[0047] The digital meter scene image captured by the camera is decomposed into multiple scales to construct a multi-scale pyramid of the digital meter scene image. Then, the multi-scale pyramid of the digital meter scene image is similar to the multi-scale pyramid of the standard digital meter image from 0 to 9, and the digits of the monitored digital meter scene image are recognized by similarity measurement. The cosine distance is used to realize the similarity calculation between images of different scales. The specific formula is as follows:

[0048]

[0049] In the formula, cos(d σ ) is the similarity between the standard image with scale σ and the digital meter scene image, and Ω′ k σ are the kth elements of the standard image and the digital meter scene image at scale σ, respectively; M is the dimension of the standard image and the digital meter scene image at scale σ;

[0050] By calculating the cosine distance between the standard image and the digital meter scene image at different scales, and by integrating the cosine distances at different scales, the recognition of the numbers in the digital meter scene image is realized. The specific formula is as follows:

[0051]

[0052] Where m is the number of pyramid layers and Q is the similarity factor.

[0053] The specific steps of identifying the pointer meter scene image in S3 are:

[0054] The camera is used to perform high-precision recognition of the image contours of different pointer-type meter scenes, depict the contours of pointer-type meters, measure the angles of existing dial scales, and calculate the correspondence between scales and angles based on the proportional relationship, namely:

[0055]

[0056] In the formula, δ is the angle corresponding to the unit scale, Φ is the maximum scale value used by the instrument, and Ω is the angle corresponding to the instrument panel;

[0057] The LSD line detection algorithm is used to identify the pointer position, and the current offset angle of the pointer is calculated through the pixel points on the line. The specific formula is expressed as:

[0058]

[0059] Where x and y are the horizontal and vertical pixel coordinates of the point p on the current pointer, respectively, and m and n are the horizontal and vertical pixel coordinates of the center O of the meter, respectively. The reading pointed by the pointer can be expressed as:

[0060] num = δ(θ-α) (12);

[0061] Where num is the reading indicated by the pointer and α is the angle of the initial position profile of the meter.

[0062] The specific steps for identifying pipeline scenarios in S3 are:

[0063] b1, identifying leaks in pipeline scene images;

[0064] b2, identify pipeline vibration in pipeline scene image.

[0065] b1 The specific steps are:

[0066] A YOLO v5 neural network model is constructed to identify pipeline leakage. The backbone network of the YOLO v5 neural network model adopts the Darknet53 network, and the YOLO v5 neural network model adopts the Mini-batch training strategy.

[0067] b2 The specific steps are:

[0068] c1, feature point detection

[0069] The SIFT feature detection algorithm is selected to detect the feature points of the pipeline scene image. The Gaussian function is used to convolve with the pipeline scene image to obtain the multi-scale space of the image, and the multi-scale difference space of the image is obtained through the adjacent scale images. With the pixel as the center, the brightness of the 5×5 neighborhood of the pixel and the 5×5 pixels of the upper and lower scales are compared to select the pixel with the largest brightness as the image feature point, and all the pixels of the image are traversed through the pipeline scene image to form an image feature point set.

[0070] C2, feature point detection

[0071] Select a 16×16 feature point area, calculate the gradient direction histogram of pixels in each 4×4 interval in the area in 8 directions of 0°, 45°, 90°, 135°, 180°, 225°, 270° and 315°, and make an accumulation value for each gradient direction to get a seed point. Each feature point is composed of 4×4 seed points, so each seed point has vector information in 8 directions, and a total of 128 feature points are generated. Feature information is a descriptor vector of a feature point; among them, the calculation formula of the gradient g and direction θ of the feature point is:

[0072] θ(x,y)=arctan((L(x,y+1)-L(x,y-1)) / (L(x+1,y)-L(x-1,y))) (14);

[0073] Where L is the differential space where each feature point is located, x and y are the pixel coordinates of the image pixel in the horizontal and vertical directions respectively;

[0074] The matching between feature points is achieved by calculating the cosine distance between the feature point descriptors of different frame images. The cosine distance cos(d) calculation is specifically expressed as:

[0075]

[0076] In the formula, Ω k,l and Ω k,r are the descriptor vectors of the feature points of the left and right view images, respectively, and k is the dimension of the descriptor vector;

[0077] In the matching process, by calculating the similarity between the feature points of different frames and the feature points of the initial frame image, the error of the later structural vibration that needs to be accumulated frame by frame is avoided, and the high-precision measurement of the vibration of the management structure of the old hydropower station is achieved. The specific expression is as follows:

[0078] s t =loc(t)-loc(t0) (16);

[0079] In the formula, s t is the pipeline vibration displacement, loc(t) is the position of the feature point on the pipeline at time t, and loc(t0) is the position of the feature point on the pipeline at the initial time. Finally, the vibration of the pipeline structure is monitored by calculating the position change of the pipeline feature point in the video image sequence in real time, and the status of the pipeline structure is monitored according to the change of the vibration amplitude and the vibration frequency.

[0080] Compared with the prior art, the present invention has the following beneficial effects:

[0081] (1) The automatic inspection device and inspection method for the status of pipelines in a hydropower station of the present invention provide a rotor quadruped robot that can adapt to multiple scenarios such as narrow spaces, uneven roads, and stairs, without the need for large-scale facility and equipment modifications to existing old hydropower stations.

[0082] (2) The automatic inspection device and method for the status of pipelines in a hydropower station of the present invention, based on a rotor quadruped robot, can not only inspect internal pipelines such as turbine layers and generator layers, but also inspect pipelines in large external facilities such as dams and factory buildings, thus realizing comprehensive monitoring of the pipelines of the entire old hydropower station based on a single device.

[0083] (3) Compared with the existing monitoring method based on a single large-scale neural network model, the automatic inspection device and inspection method for the pipeline status of a hydropower station of the present invention proposes a video image analysis method for the inspection of the pipeline status of a hydropower station according to the inspection requirements of the hydropower station, which has the advantages of high monitoring accuracy and strong migration capability. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] Figure 1 It is a structural schematic diagram of the automatic inspection device for the status of a hydropower station of the present invention;

[0085] Figure 2 It is a schematic diagram of the installation of the camera of the present invention;

[0086] Figure 3 It is a schematic diagram of the flow of the hydropower station status inspection method of the present invention;

[0087] Figure 4 It is a schematic diagram of the architecture of the twin-residual neural network model of the present invention;

[0088] Figure 5 It is a schematic diagram for identifying the counting value of the digital meter of the present invention;

[0089] Figure 6 It is a schematic diagram of the principle of multi-scale image similarity calculation of the present invention;

[0090] Figure 7 It is a schematic diagram for identifying the outline of a pointer-type meter of the present invention;

[0091] Figure 8 It is a schematic diagram for identifying the sector inclination angle of a pointer type meter of the present invention;

[0092] Fig. 9 It is a schematic diagram of the positioning of the reading of the pointer type meter of the present invention;

[0093] Fig.10 It is a schematic diagram of the architecture of the YOLO v5 neural network model of the present invention;

[0094] Fig.11 It is a schematic diagram of constructing a 16x16 window of image feature points of the present invention;

[0095] Fig.12 It is a schematic diagram of the 128-dimensional descriptor of the present invention.

[0096] In the picture, 1. Body, 2. Gearbox, 3. Rotor, 4. Legs, 5. Motor, 6. Control center, 7. Gimbal, 8. Camera. DETAILED DESCRIPTION

[0097] The present invention will be further explained below in conjunction with the accompanying drawings and specific embodiments.

[0098] Example 1

[0099] The automatic inspection device for the status of a hydropower station provided by the present invention is as follows: Figure 1 and Figure 2 As shown, it includes a rectangular body 1 and a control center 6, a driving mechanism is provided in the body 1, the driving mechanism is respectively connected to a gear box 2 and a crawling mechanism, the crawling mechanism is arranged on the outer wall of the body 1, the gear box 2 is arranged on the top of the body 1, the gear box 2 is connected to a flying mechanism, a gimbal 7 is connected to the front side of the body 1 close to the gear box 2, a camera 8 is installed on the gimbal 7, the camera 8 is connected to the control center 6 by wireless, the driving mechanism includes two motors 5, the crawling mechanism includes four legs 4, two legs 4 are symmetrically installed on each side wall of the body 1, two motors are respectively connected to the two legs 4 through transmission shafts, the flying mechanism includes four rotors 3, the four rotors are respectively connected to the gear box 2 through transmission shafts, and the four rotors 3 are evenly distributed along the center of the gear box 2.

[0100] Among them, the motor 5 is the energy and power system of the inspection device, which provides power for the inspection device to walk and fly; the gear box 2 is the power transmission mechanism of the inspection device, which is responsible for transmitting power to the rotor 3 and the leg 4; the control center 6 is the brain of the inspection device, which is responsible for receiving, processing and sending external commands; the camera 8 is an image acquisition device, which is used to capture images of monitoring objects such as pipelines and pointers; the images captured by the camera 8 are sent to the control center; the gimbal 7 is the supporting mechanism of the camera, which is used to adjust the inclination angle of the camera.

[0101] In terms of the design of the inspection device, in order to reduce the weight of the inspection device and consider the actual needs of non-simultaneous movement of the foot 4 and the rotor 3, two motors are installed on the front and rear of the body 1, and the motor 5 is connected to the rotor 3 and the foot 4 through the gear box 2. A differential is designed in the gear box 2. By changing the speed and direction of the two motors differently, the inspection device can move on the ground and fly in the air. The power transmission between the rotor 3, the foot 4 and the motor 5 is realized through gears, and the power is transmitted to the foot 4 through gears. The foot 4 is connected to the body 1 with bolts.

[0102] Example 2

[0103] The present invention provides a method for inspecting the status of a hydropower station, such as Figure 1-12 As shown, the specific steps of using the automatic inspection device for the status of a hydropower station are as follows:

[0104] S1, using the motor 5 to drive the rotor 3 and the leg 4 to move, controlling them to a suitable angle, using the camera 8 to capture the target scene image, and building a twin-residual neural network model;

[0105] S2, the target scene image captured in S1 is transmitted to the twin-residual neural network model through the control center 6, and the twin-residual neural network model is trained to classify the scene images into digital meter scene images, pointer meter scene images and pipeline scene images;

[0106] S3, performing scene image recognition on the digital meter scene image, the pointer meter scene image and the pipeline scene image respectively, outputting the scene image recognition result, and completing the status monitoring of the hydropower station.

[0107] Among them, in S3, mainly based on the classification results of pipeline structure, digital meter, and pointer meter scene images in S2, corresponding status monitoring methods are proposed for different scenarios. This step mainly includes digital meter recognition, pointer meter recognition, pipeline structure leakage, vibration monitoring, etc.

[0108] Example 3

[0109] On the basis of Example 2, Figure 1-12 As shown in Figure 1, the specific steps for building the twin-residual neural network model in S1 are:

[0110] Considering the unbalanced collection of sample images in multiple scenes and the problem that existing simple neural network training is prone to forgetting and missing, a twin residual neural network architecture is proposed. This network architecture consists of two parts, the upper and lower convolutional neural networks, to form a twin architecture. At the same time, in order to solve the problem that the gradient of the existing convolutional neural network is easy to disappear, a residual learning module is added to the upper and lower convolutional neural network structure to complete the construction of the twin-residual neural network model. The model is used to further decompose the learned image convolution features, and the upper and lower fully connected network features are compared to realize the classification of hydropower station pipeline structure, digital meter and pointer meter scenes.

[0111] The specific steps for training the twin-residual neural network model for image classification in S2 are:

[0112] S2.2.1, construct the dataset;

[0113] Different from the existing single-layer neural network architecture that learns one image at a time, the twin-residual neural network needs to input image data into the upper and lower networks in the form of image sample pairs. Assuming that there are m scenes to be classified, each scene has n target image samples, and the number of target image sample pairs that need to be composed is N; the number of target image sample pairs is divided into the number of target image positive sample pairs and the number of target image negative sample pairs. The number of target image positive sample pairs constructed is the same as the number of target image negative samples, that is, the number of positive sample pairs and negative sample pairs is N / 2. The labels corresponding to the number of target image positive sample pairs and the number of target image negative sample pairs are defined as:

[0114]

[0115] Where: y is the true label of the sample pair; is the Dirac delta function, k1 and k2 are different image samples. When k1 and k2 belong to the same scene, their label value is defined as 1, and when they do not belong to the same scene, their label value is 0.

[0116] S2.2.2, construct loss function;

[0117] In order to reduce the conflict between network recognition efficiency and accuracy, a loss function based on the cross entropy function is constructed, and the overfitting risk of the twin residual neural network model is reduced by adding the L2 weight decay term to the loss function. The final loss function J is expressed as:

[0118] J(k1,k2)=y(k1,k2)logh θ (k1,k2)+(1-y(k1,k2))log(1-h θ (k1,k2))+λθ T θ(2);

[0119] In the formula, y(k1,k2) is the sample label; h θ (x1, x2) is the sample pair similarity output by the model; θ is the weight of the residual twin neural network; λ∈[0,∞) is the L2 regularization coefficient;

[0120] Secondly, during the training process, the stochastic gradient descent optimization algorithm is introduced to minimize the loss function to learn the data. At the same time, in order to increase the convergence of the model during the training process, the momentum term is added to minimize the objective function. The loss function is minimized by stochastic gradient descent with momentum term. The specific formula is as follows:

[0121]

[0122] In the formula, the subscript i represents the i-th iteration step in model training; η represents the learning rate; μ represents the momentum parameter; Δθ represents the parameter change in each training step. represents the function gradient;

[0123] S2.2.3, construct similarity measurement function;

[0124] The convolutional features of the target image are identified by the upper and lower convolutional neural networks respectively, and the convolutional features of the target image are further processed by the fully connected network. Finally, the scene classification is realized by the similarity metric function. The specific formula is:

[0125] Similarity = ξ(f θ (k1,k2)) (4);

[0126] In the formula, f θ (·) is the twin-residual neural network model for extracting features; ξ is the similarity measurement function, and θ is the neural network weight;

[0127] For high-precision classification of different scenarios, a similarity measurement function based on the Langmuir distance is constructed. The Langmuir distance d can be described as:

[0128]

[0129] Where, X up and X down The convolution features of the target image obtained by the upper and lower layers of the neural network respectively, and P is the dimension of the convolution features in the fully connected layer.

[0130] At this point, by training the twin-residual neural network model, high-precision classification of multi-scene images of pipeline structures, digital meters, and pointer meters can be achieved.

[0131] The specific steps of recognizing the digital meter scene image in S3 are:

[0132] Digital meters are one of the common meters in old hydropower stations. During daily inspections, maintenance personnel are required to regularly record the readings on digital meters. However, there are many digital meters in old power stations, and maintenance personnel need to spend a lot of energy to record the contents of digital meters. Based on this, the present invention proposes a digital meter recognition algorithm based on multi-scale image similarity measurement according to the characteristics of digital meters. The specific contents are as follows:

[0133] a1, multi-scale decomposition of standard digital meter scene image;

[0134] Considering the characteristics of image multi-scale space that can realize the description of image macro and micro features, the standard digital meter images of numbers 0 to 9 are represented in multi-scale space, and a standard data multi-scale space sample library is established. The multi-scale space representation process of digital meter images is as follows:

[0135] Considering the linear invariance of the Gaussian basis function, the Gaussian basis function is used to smooth the standard image of the digital meter. The specific formula is as follows:

[0136]

[0137] Where L(x,y,σ) is the smoothed image, I(x,y) is the standard data image, G(x,y,σ) is the Gaussian basis function, and σ is the scale parameter. Represents a convolution operation.

[0138] Secondly, the smoothed image is downsampled. The specific formula is as follows:

[0139]

[0140] In the formula, I downsample (σ+1) represents the sampled image, and s represents the downsampling window size;

[0141] Repeat iterative smoothing and sampling m times to obtain digital meter images with different resolutions and sizes, and construct a multi-scale pyramid of standard digital meter images, with a total of m layers;

[0142] a2, similarity calculation of digital meter images of different scales;

[0143] The digital meter scene image captured by the camera (8) is decomposed into multiple scales to construct a multi-scale pyramid of the digital meter scene image. Then, similarity is calculated between the multi-scale pyramid of the digital meter scene image and the multi-scale pyramid of the standard digital meter image from 0 to 9. The digits of the monitored digital meter scene image are recognized by similarity measurement. The cosine distance is used to realize the similarity calculation between images of different scales. The specific formula is as follows:

[0144]

[0145] In the formula, cos(d σ ) is the similarity between the standard image with scale σ and the digital meter scene image, and are the kth elements of the standard image and the digital meter scene image at scale σ, respectively; M is the dimension of the standard image and the digital meter scene image at scale σ;

[0146] By calculating the cosine distance between the standard image and the digital meter scene image at different scales, and by integrating the cosine distances at different scales, the recognition of the numbers in the digital meter scene image is realized. The specific formula is as follows:

[0147]

[0148] Where m is the number of pyramid layers and Q is the similarity factor.

[0149] The specific steps of identifying the pointer meter scene image in S3 are:

[0150] Pointer meters are one of the common meters in old hydropower stations and often appear on electromechanical equipment. The daily maintenance of old power stations requires recording the readings of pointer meters to reflect the operating status of the electromechanical equipment.

[0151] Before using the inspection device for inspection, the camera 8 is used to perform high-precision recognition of the image contours of different pointer-type meter scenes, such as Figure 7 As shown in the figure, the outline of the pointer meter is depicted, and we get Figure 8, and measure the angle corresponding to the existing dial scale, and calculate the corresponding relationship between the scale and the angle according to the proportional relationship, that is:

[0152]

[0153] In the formula, δ is the angle corresponding to the unit scale, Φ is the maximum scale value used by the instrument, and Ω is the angle corresponding to the instrument panel;

[0154] Secondly, considering the characteristics of pointer-type meters that the pointer presents a straight line at different times, and taking into account the factors of recognition accuracy and efficiency, the LSD line detection algorithm is used to identify the pointer position. The principle of the LSD line detection algorithm is to obtain the pixel point set of the straight line through local analysis of the image, verify and solve it through assumed parameters, and further merge the pixel point set with the error control set to adaptively control the number of false detections.

[0155] The results of pointer recognition using the LSD line detection algorithm are as follows: Fig. 9 As shown, the current offset angle of the pointer is calculated through the pixel points on the straight line. The specific formula is expressed as:

[0156]

[0157] Where x and y are the horizontal and vertical pixel coordinates of the point p on the current pointer, respectively, and m and n are the horizontal and vertical pixel coordinates of the center O of the meter, respectively. The reading pointed by the pointer can be expressed as:

[0158] num = δ(θ-α) (12);

[0159] Where num is the reading indicated by the pointer and α is the angle of the initial position profile of the meter.

[0160] The specific steps for identifying pipeline scenarios in S3 are:

[0161] The pipeline structure is one of the most common structures in hydropower stations. The status monitoring of the pipeline structure is the most important part of the daily inspection of old hydropower stations. In this invention, considering the requirements of daily hydropower stations for pipeline inspection, a pipeline leakage monitoring and vibration monitoring strategy based on video images is proposed. The specific contents are as follows:

[0162] b1, identifying leaks in pipeline scene images;

[0163] b2, identify pipeline vibration in pipeline scene image.

[0164] b1 The specific steps are:

[0165] When using the YOLO v5 neural network to identify leaks, first collect photos of leaks during the operation of the hydropower station to obtain an image dataset; and considering the small number of image samples and the large difference in the number of samples of different leaks, enhance the image dataset through image cropping, multi-scale decomposition, rotation, noise addition and other angles; secondly, use annotation tools to label the dataset images, i.e. oil leakage, water leakage, etc.; secondly, divide the dataset images into 80% training sets, train the YOLO v5 neural network, and use 20% test sets to test the model. Finally, use the trained YOLO v5 neural network to identify whether there are leaks in the image. Among them, the deep learning model structure of YOLO v5 is as follows: Fig.10 shown.

[0166] Compared with existing image target recognition algorithms based on SSD, RNN, etc., the advantages of the YOLO v5 neural network model used in this invention are mainly reflected in the following aspects:

[0167] First, the backbone network of the YOLO v5 neural network model uses the Darknet53 network, which has fewer learning parameters and computational complexity, and has higher recognition efficiency while maintaining accuracy;

[0168] Second, the YOLO v5 neural network model uses an image pyramid data sample learning strategy, which has a strong ability to identify small targets in large scenes;

[0169] Third, the YOLO v5 neural network model adopts the Mini-batch training strategy, that is, the number of images read from the data set for training in one batch does not need to be set very large, and a relatively good training effect can be achieved using a GPU during training.

[0170] In actual use, the YOLO v5 neural network model is trained through the collected leakage images, and the trained neural network is used to realize pipeline leakage monitoring.

[0171] b2 The specific steps are:

[0172] Vibration monitoring of pipelines is one of the most important tasks in daily inspections of old hydropower stations, especially for electromechanical equipment that has been in service for a long time. At present, vibration inspections of pipeline structures are mainly carried out using handheld devices. Considering the large number of hydropower stations to be managed, the long time required for a single inspection, and the need for multiple inspections, a vibration monitoring strategy for old hydropower stations based on video images is proposed.

[0173] In the specific test, considering the characteristics of pipeline structure with many mutation points and rich image texture, the pipeline structure vibration measurement is realized by constructing an image feature matching algorithm. The measurement process is mainly divided into two contents:

[0174] c1, feature point detection;

[0175] Feature point detection is the core of feature matching algorithm, and its function is to select pixel points in the texture-rich area of ​​the image as feature points. In the present invention, Harris, SIFT, SURF and other image feature detection algorithms are compared from the perspectives of computational efficiency, accuracy and lighting robustness, and SIFT feature detection algorithm is selected to detect pipeline scene image feature points, and Gaussian function is used to convolve with pipeline scene image to obtain image multi-scale space, and image multi-scale differential space is obtained through adjacent scale images; with pixel as the center, by comparing the brightness of pixel 5×5 neighborhood and upper and lower scale 5×5 pixels, the pixel with the largest brightness is selected as the image feature point, and all pixels of the image are traversed through the pipeline scene image to form an image feature point set.

[0176] c2, feature point detection;

[0177] The function of feature point matching is to determine the position change of feature points in the video sequence, which is mainly achieved through feature point descriptors. Specifically, a 16×16 area with feature points is selected, and the gradient direction histogram of pixels in each 4×4 interval in the area in 8 directions of 0°, 45°, 90°, 135°, 180°, 225°, 270° and 315° is calculated, and an accumulation value is made for each gradient direction to obtain a seed point. Each feature point is composed of 4×4 seed points, and each seed point has vector information in 8 directions, generating feature information of 128 feature points in total, which is a descriptor vector of a feature point; among them, the calculation formula of the gradient g and direction θ of the feature point is:

[0178]

[0179] θ(x,y)=arctan((L(x,y+1)-L(x,y-1)) / (L(x+1,y)-L(x-1,y))) (14);

[0180] Where L is the differential space where each feature point is located, x and y are the pixel coordinates of the image pixel in the horizontal and vertical directions respectively;

[0181] In terms of feature point descriptor vector matching, in order to achieve high-precision matching of image feature points in video sequences, the present invention proposes a feature point descriptor vector measurement method based on the cosine function, that is, matching between feature points is achieved by calculating the cosine distance between feature point descriptors of different frame images. The calculation of the cosine distance cos(d) is specifically expressed as:

[0182]

[0183] In the formula, Ω k,l and Ω k,r are the descriptor vectors of the feature points of the left and right view images, respectively, and k is the dimension of the descriptor vector;

[0184] In the matching process, by calculating the similarity between the feature points of different frames and the feature points of the initial frame image, the error of the later structural vibration that needs to be accumulated frame by frame is avoided, and the high-precision measurement of the vibration of the management structure of the old hydropower station is achieved. The specific expression is as follows:

[0185] s t =loc(t)-loc(t0) (16);

[0186] In the formula, s t is the pipeline vibration displacement, loc(t) is the position of the feature point on the pipeline at time t, and loc(t0) is the position of the feature point on the pipeline at the initial time. Finally, the vibration of the pipeline structure is monitored by calculating the position change of the pipeline feature point in the video image sequence in real time, and the status of the pipeline structure is monitored according to the change of the vibration amplitude and the vibration frequency.

[0187] The automatic inspection device and inspection method of the hydropower station status provided by the present invention, at the equipment level, designs a quadruped robot inspection device with rotors, which can simultaneously adapt to large scenes such as dams and powerhouses as well as small scenes such as pointers and meters. At the monitoring method level, according to the inspection requirements of the hydropower station, corresponding monitoring algorithms are proposed for the pipeline structure and the pointers and meters on the pipeline. It can realize the automatic inspection of the safety pipelines of old hydropower stations, providing support for ensuring the safe service of old hydropower stations.

Claims

1. Automatic inspection device for hydropower station status, characterized in that: The invention comprises a rectangular body (1) and a control center (6), wherein a driving mechanism is arranged inside the body (1), wherein the driving mechanism is respectively connected to a gear box (2) and a crawling mechanism, wherein the crawling mechanism is arranged on the outer wall of the body (1), wherein the gear box (2) is arranged on the top of the body (1), wherein the gear box (2) is connected to a flying mechanism, wherein a gimbal (7) is arranged on a side of the body (1) close to the gear box (2), wherein a camera (8) is installed on the gimbal (7), and wherein the camera (8) is connected to the control center (6) via a wireless connection.

2. The automatic inspection device for the status of a hydropower station according to claim 1 is characterized in that: The driving mechanism comprises two motors (5), the crawling mechanism comprises four legs (4), two legs (4) are symmetrically mounted on each of the two side walls of the machine body (1), and the two motors are respectively connected to the two legs (4) via a transmission shaft.

3. The automatic inspection device for the status of a hydropower station according to claim 2 is characterized in that: The flight mechanism comprises four rotors (3), the four rotors are respectively connected to a gear box (2) via a transmission shaft, and the four rotors (3) are evenly distributed along the center of the gear box (2).

4. A method for inspecting the status of a hydropower station, using the automatic inspection device for the status of a hydropower station as claimed in claim 3, characterized in that: The specific steps are as follows: S1, using the camera (8) to capture an image of a target scene and build a twin-residual neural network model; S2, transmitting the target scene image captured in S1 to the twin-residual neural network model through the control center (6), training the twin-residual neural network model to classify the scene images into digital meter scene images, pointer meter scene images and pipeline scene images; S3, performing scene image recognition on the digital meter scene image, the pointer meter scene image and the pipeline scene image respectively, outputting the scene image recognition result, and completing the status monitoring of the hydropower station.

5. The hydropower station status inspection method according to claim 4, characterized in that: The specific steps for building the twin-residual neural network model described in S1 are: The twin architecture is formed by the upper and lower convolutional neural networks, and the residual learning module is added to the upper and lower convolutional neural network structure to complete the construction of the twin-residual neural network model; The specific steps of training the twin-residual neural network model for image classification described in S2 are: S2.2.

1. Constructing the Dataset Assume that there are m scenes to be classified, each scene has n target image samples, and the number of target image sample pairs to be formed is N; the number of target image sample pairs is divided into the number of target image positive sample pairs and the number of target image negative sample pairs. The number of target image positive sample pairs constructed is the same as the number of target image negative samples, that is, the number of positive sample pairs and negative sample pairs is N / 2. The labels corresponding to the number of target image positive sample pairs and the number of target image negative sample pairs are defined as: Where: y is the true label of the sample pair; is the Dirac delta function, k1 and k2 are different image samples. When k1 and k2 belong to the same scene, their label value is defined as 1, and when they do not belong to the same scene, their label value is 0; S2.2.2, construct loss function A loss function based on the cross entropy function is constructed, and the overfitting risk of the twin residual neural network model is reduced by adding the L2 weight decay term to the loss function. The final loss function J is expressed as: J(k1,k2)=y(k1,k2)logh θ (k1,k2)+(1-y(k1,k2))log(1-h θ (k1,k2))+λθ T θ (2); In the formula, y(k1,k2) is the sample label; h θ (x1, x2) is the sample pair similarity output by the model; θ is the weight of the residual twin neural network; λ∈[0,∞) is the L2 regularization coefficient; Secondly, during the training process, the data is learned by minimizing the loss function by introducing the stochastic gradient descent optimization algorithm. At the same time, the momentum term is added to minimize the objective function, and the loss function is minimized by stochastic gradient descent with momentum term. The specific formula is as follows: In the formula, the subscript i represents the i-th iteration step in model training; η represents the learning rate; μ represents the momentum parameter; Δθ represents the parameter change in each training step. represents the function gradient; S2.2.3, construct similarity measurement function The convolutional features of the target image are identified by the upper and lower convolutional neural networks respectively, and the convolutional features of the target image are further processed by the fully connected network. Finally, the scene classification is realized by the similarity measurement function. The specific formula is: Similarity=ξ(f θ (k1,k2)) (4); In the formula, f θ (·) is the twin-residual neural network model for extracting features; ξ is the similarity measurement function, and θ is the neural network weight; For high-precision classification of different scenarios, a similarity measurement function based on the Langmuir distance is constructed. The Langmuir distance d can be described as: Where, X up and X down The convolution features of the target image obtained by the upper and lower layers of the neural network respectively, and P is the dimension of the convolution features in the fully connected layer.

6. The hydropower station status inspection method according to claim 5, characterized in that: The specific steps of recognizing the digital meter scene image in S3 are: a1, multi-scale decomposition of standard digital meter scene image The standard digital meter images of numbers 0 to 9 are represented in multi-scale space, and a standard data multi-scale space sample library is established. The multi-scale space representation process of digital meter images is as follows: The Gaussian basis function is used to smooth the standard image of the digital meter. The specific formula is as follows: Where L(x,y,σ) is the smoothed image, I(x,y) is the standard data image, G(x,y,σ) is the Gaussian basis function, and σ is the scale parameter. Represents the convolution operation; Secondly, the smoothed image is downsampled. The specific formula is as follows: In the formula, I downsample (σ+1) represents the sampled image, and s represents the downsampling window size; Repeat iterative smoothing and sampling m times to obtain digital meter images with different resolutions and sizes, and construct a multi-scale pyramid of standard digital meter images, with a total of m layers; a2, similarity calculation of digital meter images of different scales The digital meter scene image captured by the camera (8) is decomposed into multiple scales to construct a multi-scale pyramid of the digital meter scene image. Then, similarity is calculated between the multi-scale pyramid of the digital meter scene image and the multi-scale pyramid of the standard digital meter image from 0 to 9. The digits of the monitored digital meter scene image are recognized by similarity measurement. The cosine distance is used to realize the similarity calculation between images of different scales. The specific formula is as follows: In the formula, cos(d σ ) is the similarity between the standard image with scale σ and the digital meter scene image, and are the kth elements of the standard image and the digital meter scene image at scale σ, respectively; M is the dimension of the standard image and the digital meter scene image at scale σ; By calculating the cosine distance between the standard image and the digital meter scene image at different scales, and by integrating the cosine distances at different scales, the recognition of the numbers in the digital meter scene image is realized. The specific formula is as follows: Where m is the number of pyramid layers and Q is the similarity factor.

7. The hydropower station status inspection method according to claim 5, characterized in that: The specific steps of identifying the pointer meter scene image in S3 are: A camera (8) is used to perform high-precision recognition of the image contours of different pointer-type meter scenes, to depict the contours of the pointer-type meter, and to measure the angles corresponding to the existing dial scales. The corresponding relationship between the scales and the angles is calculated based on the proportional relationship, namely: In the formula, δ is the angle corresponding to the unit scale, Φ is the maximum scale value used by the instrument, and Ω is the angle corresponding to the instrument panel; The LSD line detection algorithm is used to identify the pointer position, and the current offset angle of the pointer is calculated through the pixel points on the line. The specific formula is expressed as: Where x and y are the horizontal and vertical pixel coordinates of the point p on the current pointer, respectively, and m and n are the horizontal and vertical pixel coordinates of the center O of the meter, respectively. The reading indicated by the pointer can be expressed as: num = δ(θ-α) (12); Where num is the reading indicated by the pointer and α is the angle of the initial position profile of the meter.

8. The hydropower station status inspection method according to claim 5, characterized in that: The specific steps of identifying pipeline scenes in S3 are: b1, identifying leaks in pipeline scene images; b2, identify pipeline vibration in pipeline scene image.

9. The hydropower station status inspection method according to claim 8, characterized in that: The specific steps of b1 are: A YOLO v5 neural network model is constructed to identify pipeline leakage. The backbone network of the YOLO v5 neural network model adopts the Darknet53 network, and the YOLO v5 neural network model adopts the Mini-batch training strategy.

10. The hydropower station status inspection method according to claim 8, characterized in that: The specific steps of b2 are: c1, feature point detection The SIFT feature detection algorithm is selected to detect the feature points of the pipeline scene image, and the Gaussian function is used to convolve with the pipeline scene image to obtain the image multi-scale space, and the image multi-scale difference space is obtained through adjacent scale images; Taking the pixel as the center, by comparing the brightness of the 5×5 neighborhood of the pixel and the 5×5 pixels above and below, the pixel with the largest brightness is selected as the image feature point, and all pixels of the image are traversed through the pipeline scene image to form an image feature point set; c2, feature point detection Select a 16×16 feature point area, calculate the gradient direction histogram of pixels in each 4×4 interval in the area in 8 directions of 0°, 45°, 90°, 135°, 180°, 225°, 270° and 315°, and make an accumulation value for each gradient direction to get a seed point. Each feature point is composed of 4×4 seed points, so each seed point has vector information in 8 directions, and a total of 128 feature points are generated. Feature information is a descriptor vector of a feature point; among them, the calculation formula of the gradient g and direction θ of the feature point is: θ(x,y)=arctan((L(x,y+1)-L(x,y-1)) / (L(x+1,y)-L(x-1,y))) (14); Where L is the differential space where each feature point is located; x and y are the pixel coordinates of the image pixel in the horizontal and vertical directions respectively; The matching between feature points is achieved by calculating the cosine distance between the feature point descriptors of different frame images. The cosine distance cos(d) calculation is specifically expressed as: In the formula, Ω k,l and Ω k,r are the descriptor vectors of the feature points of the left and right view images, respectively, and k is the dimension of the descriptor vector; In the matching process, by calculating the similarity between the feature points of different frames and the feature points of the initial frame image, the error of the later structural vibration that needs to be accumulated frame by frame is avoided, and the high-precision measurement of the vibration of the management structure of the old hydropower station is achieved. The specific expression is as follows: S t =loc(t)-loc(t0) (16); In the formula, s t is the pipeline vibration displacement, loc(t) is the position of the feature point on the pipeline at time t, and loc(t0) is the position of the feature point on the pipeline at the initial time. Finally, the vibration of the pipeline structure is monitored by calculating the position change of the pipeline feature point in the video image sequence in real time, and the status of the pipeline structure is monitored according to the change of the vibration amplitude and the vibration frequency.

Images