Strip mine area slope abnormal event detection method based on power transmission line video data

Abstract

The invention discloses a strip mine area slope abnormal event detection method based on power transmission line video data. The method comprises the following steps: defining a slope abnormal event and a discrimination standard thereof; collecting video data, de-noising and smoothing a background image by using a bilateral filtering algorithm, and expanding a data set; adopting VGG16 and ResNet101 to extract complex background features in parallel, and detecting and outputting coordinates of an interference target frame; performing background filling on the target frame area by using an auto-encoder model, and recovering a background image without interferents; after the image contrast is enhanced, inputting the image into a self-adaptive edge extraction module, adjusting a threshold value according to local brightness, generating a binarized edge image, extracting a contour, comparing the contour with a reference image pixel by pixel, and detecting an abnormal event according to a discrimination standard; and if an alarm is detected, carrying out manual checking to confirm whether the alarm is abnormal or not. The method can efficiently detect the abnormal events of the power transmission line slope in the strip mine area, and has practical significance.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and slope abnormal event monitoring, and in particular to a method for detecting abnormal events on slopes in open-pit mines based on transmission line video data. Background Art

[0002] Unlike overhead transmission lines in general power grids, the harsh environment in open-pit mines poses a huge threat to the normal operation of transmission lines. Due to restrictions on geological environment and land acquisition, some lines intersect with the production operation areas of the mines, and accidents often occur. In addition, there are also some abnormal events such as sudden collapse events, which can easily lead to casualties and huge economic losses.

[0003] The current detection of slope anomaly events in open-pit mines mainly relies on geological surveys and land feature information to evaluate slope stability. This type of method is relatively complex and cannot survey slope conditions in real time. In addition, open-pit mines are equipped with a large number of transmission line monitoring. It is a relatively simple way to detect slope anomaly events based on transmission line video images, but there is currently no applicable method for detecting slope anomaly events based on transmission line video images. Summary of the invention

[0004] Considering the lack of a video image detection method for power transmission lines under complex backgrounds, and the inability to survey special abnormal events such as abnormal slope collapse, the technical problem to be solved by the present invention is to provide a method for defining abnormal events in open-pit mine slopes, set a criterion for distinguishing abnormal events in open-pit mine slopes based on video images; use an improved Faster R-CNN suitable for complex scene feature extraction to realize the detection of interference factors in open-pit mine slope images; eliminate the target area of ​​interference factors, use an autoencoder model for background filling, and restore the background image from which the interference has been removed; improve the contrast of the image, input an adaptive edge extraction module, obtain a binary edge extraction image, perform contour extraction, and compare it pixel by pixel with the reference image, and determine whether an abnormal slope event occurs according to the slope abnormal event distinction standard, so as to effectively realize the method for detecting abnormal events in the slope of power transmission lines in open-pit mines.

[0005] The technical solution adopted by the present invention is: a method for detecting abnormal events on slopes in open-pit mines based on transmission line video data, the method comprising the following steps:

[0006] S1. Define abnormal events of slopes in open-pit mines, and set abnormal event identification criteria of slopes in open-pit mines based on video images;

[0007] S2. Collect video data of abnormal events on the slopes of open-pit mines. The video data includes original color images of power transmission lines in open-pit mines and gray images of power transmission lines in open-pit mines. Use bilateral filtering algorithm to denoise and smooth the images of slopes in the mines. Expand the data set of slopes in open-pit mines by data enhancement methods such as synthesis, adjustment of brightness and contrast, and addition of Gaussian noise.

[0008] S3, using VGG16 (16-layer convolutional neural network) and ResNet101 (101-layer residual network) parallel networks to extract features from complex mining area slope images, and adding a feature fusion layer as a region extraction network for improving the Faster R-CNN network to detect interference targets in open-pit mining area slope images and output the target frame coordinates;

[0009] S4, according to the target frame coordinates obtained in S3, the target frame area of ​​the slope detection interference factor is eliminated, the autoencoder model is used to perform background filling, and the mining area slope image with the interference removed is restored;

[0010] S5, improving the contrast of the image, inputting the adaptive edge extraction module, adaptively adjusting the local threshold according to the brightness of different areas of the image, obtaining a binary edge extraction image, then performing contour extraction and comparing pixel by pixel with the reference image, and determining whether a slope abnormal event occurs according to the slope abnormal event discrimination standard;

[0011] S6. When an alarm message appears, manual verification is carried out to further confirm whether an abnormal slope event occurs in the power transmission line area of ​​the open-pit mine; if so, the emergency response mechanism is activated, including notifying relevant personnel, dispatching maintenance equipment, and conducting on-site investigation and processing; if not, the false alarm information is recorded, and the data set and model training are updated to improve the system detection accuracy and reduce the false alarm rate.

[0012] Preferably, in step S1, the open-pit mine slope abnormal event refers to the open-pit mine area having the risk of slope collapse, facing the threat of collapse or imminent collapse, and its characteristics on the video image are manifested in two aspects: one is the change of the edge of the mountain contour; the other is the change of the pixel value of the mountain area, and the pixel value of the slope collapse area is obviously different from that of the non-collapsed area. If both features are met at the same time, it is determined to be a slope abnormality, and the slope abnormality event discrimination condition can be set as:

[0013] (1) The edge contour of the mountain on the image changes;

[0014] (2) The hue and saturation of some areas in the mountain image are clearly different from those of the surrounding areas, and the pixel values ​​in this area fluctuate significantly.

[0015] Preferably, step S2 includes processing the gray image of the power transmission line in the open-pit mine area by using a bilateral filtering algorithm. The weight of the bilateral filtering takes into account the pixel Euclidean distance and the radiation difference in the pixel range domain at the same time, and can better preserve the pixel value on the edge of the slope detection image. The specific process is as follows: first input the image size, that is, the image with (pixel × pixel) of M*N, and use a sliding window of (pixel × pixel) or n*n size to traverse. The pixel value output each time depends on the neighborhood of the convolved pixel. Then, the pixel value P(i, j) at the (i, j) position in the bilateral filtering can be represented by the pixel value f(s, t) in the neighborhood and its weight coefficient w(i, j, s, t):

[0016]

[0017] Wherein, i and s represent the i-th row and s-th row of the pixel array, respectively; j and t represent the j-th column and t-th column of the pixel array, respectively.

[0018] Among them, the weight coefficient w(i,j,s,t) depends on the spatial domain kernel d and the range kernel r, which can be expressed as:

[0019]

[0020] In the formula, σ d , σ r Respectively represent the standard deviation of the Gaussian kernel selected in the spatial domain and value range.

[0021] Preferably, the step S2 further includes expanding the open-pit mine slope data set, and the specific method is as follows:

[0022] (1) Use image processing equipment to synthesize buckets, trucks, bulldozers, excavators, and suspended foreign objects into the shooting scene, and synthesize image data of personnel risks in dangerous areas such as mobile substations to expand the data set;

[0023] (2) Gaussian noise is added and brightness and contrast are adjusted to simulate weather changes. Geometric transformation is performed at the same time, and the synthetic images are used together as a dataset for slope detection of power transmission lines in open-pit mines.

[0024] Anchor boxes are annotated for interference factors in slope detection in the dataset.

[0025] Preferably, in step S3, the process of improving the Faster R-CNN network is as follows:

[0026] S31: Input the image of the power transmission line in the open-pit mine into the deep convolutional network, which extracts the image features and outputs the feature map of the image, i.e., feature maps, which are used as the input for the next stage. The structural features of the deep convolutional network are as follows:

[0027] (1) Since VGG16 (16-layer convolutional neural network) can obtain fine features of small-sized images, ResNet101 (101-layer residual network) can obtain fine features of large-sized images. The parallel network of VGG16 and ResNet101 is used to obtain more detailed image feature maps;

[0028] (2) Add a feature fusion layer, which is mainly obtained by fusing the bottom and top feature maps of the VGG16 network and the bottom and top feature maps of the ResNet101 network;

[0029] S32: Input the slope image feature map obtained by the feature extraction network into the region proposal network RPN (region proposal network) to generate the target candidate frame. The main implementation process includes the following two parts:

[0030] (1) Determine whether all preset anchor boxes contain the target, and classify them as positive samples or negative samples according to the presence or absence of the target;

[0031] (2) Correct the positive sample anchor box through bounding box regression to generate more accurate candidate boxes;

[0032] S33: Perform a regional interest pooling operation, combine the feature maps extracted in step S31 with the candidate box information generated in step S32, extract the feature map proposal feature maps of the candidate box, and input them into the subsequent fully connected layer;

[0033] S34: Complete the classification operation and use the candidate box feature map extracted in S33 to determine the specific category of the image. At the same time, the final position of the target detection box is accurately located again through bounding box regression.

[0034] Preferably, in step S4, the target frame area of ​​the slope detection interference factor is removed according to the target frame coordinates obtained in S3, and the autoencoder model is used to perform background filling. The specific steps are as follows:

[0035] S41: removing the detected slope detection interference factor target frame to obtain a picture lacking a piece of pixel information;

[0036] S42: Fill in the missing information with pixels. The implementation steps are as follows:

[0037] (1) Use a large number of complete images and randomly cover some pixels to train the autoencoder model. The encoder module of the model compresses the input image into a low-dimensional feature vector, and the decoder module restores the feature vector to a complete image. The encoding process of the original data X, that is, input layer → hidden layer, is expressed as:

[0038] h=G θ1 (X) = σ(W1x+b1);

[0039] The decoding process hidden layer → input layer is expressed as:

[0040]

[0041] Where W1 and W2 are the linear change parameters of model training, and b1 and b2 are the nonlinear activation offsets of the model.

[0042] (2) Input the image with missing information into the autoencoder model to predict and complete the missing information.

[0043] Preferably, in step S5, the contrast of the image is improved and an adaptive edge extraction module is input, and the module is implemented as follows:

[0044] S51: converting the original open-pit mine area power transmission line color image in the open-pit mine area slope abnormal event video data into a grayscale image, reading it in a single channel, and performing denoising;

[0045] S52: Use the Sobel operator to calculate the gradient of the input image, thereby calculating the edge amplitude and angle of the image, performing similar pixel matching, suppressing high-similarity edge information, and performing non-maximum suppression along the gradient direction;

[0046] S53: Using the moving average method to calculate the local threshold of the gradient image after non-maximum suppression, that is, calculating the local threshold mean of the image, so as to adaptively calculate the threshold according to the brightness distribution of different regions of the image. The calculation method is as follows:

[0047] adaptive = mean - C * σ;

[0048] Where mean is the average value of the pixel values ​​in the neighborhood of the image, C is a constant used to adjust the threshold sensitivity, and σ represents the standard deviation of the pixel values ​​in the neighborhood.

[0049] S54: Perform connected domain analysis according to the image adaptive threshold to detect the edges of objects in the image.

[0050] Preferably, in step S5, the local threshold is adaptively adjusted according to the brightness of different areas of the image to obtain a binary edge extraction image, specifically using the hollowing out internal point method, the operation is as follows: if there is a black point in the image and its 8 adjacent points are all black, then the point is deleted, otherwise it is considered that the point is at the edge of the image and needs to be retained, and each pixel in the image is processed in turn, and what is left at the end is the outline of the image.

[0051] Preferably, in step S5, contour extraction is performed and pixel-by-pixel comparison is performed with the reference image, and whether a slope abnormality event occurs is determined according to a slope abnormality event discrimination standard, including:

[0052] The reference image is obtained by inputting the adaptive edge module into the picture taken without the accident, and the pixel point is compared with the contour extraction picture one by one. If the pixel change of the binary image exceeds the threshold value θ0, it is judged to meet the slope abnormal event judgment condition one; the pixel value of the original image (unprocessed video frame image) is compared with the previous frame original image, and the pixel value of the mountain area has obvious fluctuations exceeding the threshold value θ r It is determined that the slope abnormal event judgment condition 2 is met. The threshold setting method is as follows:

[0053] (1) In order to eliminate the influence of vegetation and weather factors on the extraction of mountain edge contours, the binary images obtained by processing the morning, noon and evening video frames of spring, summer, autumn and winter are compared with the reference image, and the maximum change value is taken as the value of the threshold θ0;

[0054] (2) Appropriately reduce the threshold θ based on engineering experience r , in order to achieve early warning of slope collapse events.

[0055] Preferably, in step S6, when an alarm message appears, a manual check is performed to check whether an abnormal slope event such as slope collapse has occurred.

[0056] After the relevant steps are executed, on-site operation and maintenance personnel can promptly detect the occurrence of abnormal slope events based on early warning information.

[0057] Beneficial effects of the present invention:

[0058] (1) The present invention solves the problem of low detection accuracy and difficulty in feature extraction of transmission line slope images under complex backgrounds;

[0059] (2) The present invention integrates edge extraction algorithm and target detection to realize image background change detection, and effectively detects abnormal events on the slope of open-pit mines based on transmission line video data;

[0060] (3) The present invention combines the detection requirements in special scenarios of open-pit mines with conventional detection algorithms, which has practical significance.

[0061] On the whole, the present invention provides a method for defining abnormal events in open-pit mine slopes, sets a standard for distinguishing abnormal events in open-pit mine slopes based on video images; uses an improved Faster R-CNN suitable for complex scene feature extraction to detect interference factors in open-pit mine slope images; eliminates the target area of ​​interference factors, uses an autoencoder model for background filling, and restores the background image from which the interference has been removed; improves the contrast of the image, inputs an adaptive edge extraction module, obtains a binary edge extraction image, and then performs contour extraction and compares it with the reference image pixel by pixel, and determines whether an abnormal event in the slope occurs according to the standard for distinguishing abnormal events in the slope, which can effectively detect abnormal events in the slope of power transmission lines in open-pit mines; if so, activates an emergency response mechanism, including notifying relevant personnel, dispatching maintenance equipment, and conducting on-site investigation and processing; if not, records false alarm information, updates the data set and model training, so as to improve the detection accuracy of the system and reduce the false alarm rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 is a flow chart of a method for detecting abnormal events on slopes in open-pit mines based on transmission line video data according to an embodiment of the present invention;

[0063] Figure 2 Schematic diagram of slope data set expansion in an embodiment of the present invention; a is a schematic diagram of a normal state without slope anomaly; b is a schematic diagram of regional landslide under slope anomaly; c is a schematic diagram of target misidentification under detection interference; d is a schematic diagram of simulated weather influence under sample expansion;

[0064] Figure 3 Schematic diagram of the network structure of the improved Faster R-CNN of the present invention;

[0065] Figure 4 1 is a comparison diagram of the background filling effect of an embodiment of the present invention; a is a schematic diagram of a false alarm area for interference target detection and identification; b is a schematic diagram of eliminating interference and performing background filling;

[0066] Figure 5 is a schematic diagram of an autoencoder module according to an embodiment of the present invention;

[0067] Figure 6 1 is a slope anomaly binarization detection result of an embodiment of the present invention, a is a schematic diagram of a reference image, and b is a slope anomaly edge extraction diagram. DETAILED DESCRIPTION

[0068] The present invention will be further described below in conjunction with the accompanying drawings.

[0069] In order to make those skilled in the art better understand the scheme of the present invention, the technical scheme in the embodiment of the present invention will be clearly described below in conjunction with the drawings in the embodiment of the present invention. The embodiment described is only a part of the embodiment of the present invention, not all the embodiments. Based on the embodiment of the present invention, all other embodiments obtained by ordinary technicians in the field without making creative work belong to the protection scope of the present invention.

[0070] In the embodiment of the present invention, Figure 1 As shown, a method for detecting abnormal events on slopes in open-pit mines based on transmission line video data comprises the following steps:

[0071] S1. Define abnormal events of slopes in open-pit mines, and set abnormal event identification criteria of slopes in open-pit mines based on video images;

[0072] In the embodiment of the present invention, in step S1, the open-pit mine slope abnormal event refers to the open-pit mine area having the risk of slope collapse, facing the threat of collapse or imminent collapse, and its characteristics on the video image are manifested in two aspects: one is the change of the edge of the mountain contour; the other is the change of the pixel value of the mountain area, and the pixel value of the slope collapse area is obviously different from that of the non-collapsed area. If both features are met at the same time, it is determined to be a slope abnormality, and the slope abnormality event discrimination condition can be set as:

[0073] (1) The edge contour of the mountain on the image changes;

[0074] (2) The hue and saturation of some areas in the mountain image are clearly different from those of the surrounding areas, and the pixel values ​​in this area fluctuate significantly.

[0075] S2. Collect video data of abnormal events on the slopes of open-pit mines. The video data includes original color images of power transmission lines in open-pit mines and gray images of power transmission lines in open-pit mines. Use bilateral filtering algorithm to denoise and smooth the images of slopes in mines. Expand the data set of slopes in open-pit mines by data enhancement methods such as synthesis, adjusting brightness and contrast, and adding Gaussian noise.

[0076] In the embodiment of the present invention, in step S2, a camera is installed on a tower of a power transmission line in an open-pit mine area to confirm that the field of view of the camera shooting area is wide and unobstructed.

[0077] In the embodiment of the present invention, a bilateral filtering algorithm is used to process the gray image of the power transmission line in the open-pit mine area. The weight of the bilateral filtering takes into account the pixel Euclidean distance and the radiation difference in the pixel range domain at the same time, and can better preserve the pixel value on the edge of the slope image. The specific implementation process is as follows: first, an image with a size (pixel × pixel) of M*N is input, and a sliding window of size (pixel × pixel) n*n is used for traversal. The pixel value output each time depends on the neighborhood of the convolved pixel. The pixel value P(i, j) at the position (i, j) in the bilateral filtering can be represented by the pixel value f(s, t) in the neighborhood and its weight coefficient w(i, j, s, t):

[0078]

[0079] Wherein, i and s represent the i-th row and s-th row of the pixel array, respectively; j and t represent the j-th column and t-th column of the pixel array, respectively.

[0080] Among them, the weight coefficient w(i,j,s,t) depends on the spatial domain kernel d and the range kernel r, which can be expressed as:

[0081]

[0082] In the formula, σ d , σ r Respectively represent the standard deviation of the Gaussian kernel selected in the spatial domain and value range.

[0083] In the embodiment of the present invention, Figure 2 As shown in the figure, a is a schematic diagram of no slope anomaly under normal conditions; b is a schematic diagram of regional landslide under slope anomaly; c is a schematic diagram of target misidentification under detection interference; d is a schematic diagram of simulated weather influence under sample expansion. The open-pit mine slope dataset is expanded, and the specific method is as follows:

[0084] (1) Use image processing equipment to synthesize buckets, trucks, bulldozers, excavators, and suspended foreign objects into the shooting scene, and synthesize image data of personnel risks in dangerous areas such as mobile substations to expand the data set;

[0085] (2) Gaussian noise is added and brightness and contrast are adjusted to simulate weather changes. Geometric transformation is performed at the same time, and the synthetic images are used together as a dataset for slope detection of power transmission lines in open-pit mines.

[0086] Anchor boxes are annotated for interference factors in slope detection in the dataset.

[0087] S3, using VGG16 and ResNet101 parallel networks to extract features from complex mining area slope images, and adding a feature fusion layer as a region extraction network to improve the Faster R-CNN network, detect interference targets in open-pit mining area slope images and output the target frame coordinates;

[0088] In a specific embodiment, in step S3, Figure 3 As shown in the figure, the implementation process of improving the Faster R-CNN network is as follows:

[0089] S31: Input the image of the power transmission line in the open-pit mine into the deep convolutional network, which extracts the image features and outputs the feature map of the image, i.e., feature maps, which are used as the input for the next stage. The structural features of the deep convolutional network are as follows:

[0090] (1) Since VGG16 can obtain fine features of small-sized images and ResNet101 can obtain fine features of large-sized images, the parallel network of VGG16 and ResNet101 is used to obtain a more refined image feature map;

[0091] (2) Add a feature fusion layer, which is mainly obtained by fusing the bottom and top feature maps of the VGG16 network and the bottom and top feature maps of the ResNet101 network;

[0092] S32: Input the slope image feature map obtained by the feature extraction network into the region candidate network RPN to generate the target candidate box. The main implementation process includes the following two parts:

[0093] (1) Determine whether all preset anchor boxes contain the target, and classify them as positive or negative samples based on the presence or absence of the target;

[0094] (2) Correct the positive sample anchor box through bounding box regression to generate more accurate candidate boxes;

[0095] S33: Perform a regional interest pooling operation, combine the feature maps extracted in step S31 with the candidate box information generated in step S32, extract the feature maps of the candidate box (proposal feature maps), and input them into the subsequent fully connected layer;

[0096] S34: Complete the classification operation and use the candidate box feature map extracted in S33 to determine the specific category of the image. At the same time, the final position of the target detection box is accurately located again through bounding box regression.

[0097] In a specific embodiment, the trained improved Faster R-CNN target detection model is used to perform target detection on a new detection image, and the coordinates of the target box are recorded at the same time.

[0098] S4, remove the target frame area of ​​the slope detection interference factor, use the autoencoder model to fill in the background, and restore the background image with the interference removed;

[0099] In the embodiment of the present invention, in step S4, the target frame area of ​​the slope detection interference factor is removed according to the target frame coordinates obtained in S3, and the autoencoder model is used to fill the background, such as Figure 4 As shown in the figure, a is a schematic diagram of the false positive area of ​​interference target detection and identification; b is a schematic diagram of removing interference and filling the background. The self-encoder module is as follows Figure 5 The specific steps are as follows:

[0100] S41: removing the detected slope detection interference factor target frame to obtain a picture lacking a piece of pixel information;

[0101] S42: Fill in the missing information with pixels. The implementation steps are as follows:

[0102] (1) Use a large number of complete images and randomly cover some pixels to train the autoencoder model. The encoder module of the model compresses the input image into a low-dimensional feature vector, and the decoder module restores the feature vector to a complete image. The encoding process of the original data X, that is, input layer → hidden layer, is expressed as:

[0103] h=G θ1 (X) = σ(W1x+b1);

[0104] The decoding process hidden layer → input layer is expressed as:

[0105]

[0106] Where W1 and W2 are the linear change parameters of model training, and b1 and b2 are the nonlinear activation offsets of the model.

[0107] (2) Input the image with missing information into the autoencoder model to predict and complete the missing information.

[0108] S5, improving the contrast of the image, inputting the adaptive edge extraction module, adaptively adjusting the local threshold according to the brightness of different areas of the image, obtaining a binary edge extraction image, then performing contour extraction and comparing pixel by pixel with the reference image, and determining whether a slope abnormal event occurs according to the slope abnormal event discrimination standard;

[0109] In a specific embodiment, in step S5, the contrast of the image is improved and an adaptive edge extraction module is input. The implementation process of the module is as follows:

[0110] S51: converting the original open-pit mine area power transmission line color image in the open-pit mine area slope abnormal event video data into a grayscale image, reading it in a single channel, and performing denoising;

[0111] S52: Use the Sobel operator to calculate the gradient of the input image, thereby calculating the edge amplitude and angle of the image, performing similar pixel matching, suppressing high-similarity edge information, and performing non-maximum suppression along the gradient direction;

[0112] S53: Using the moving average method to calculate the local threshold of the gradient image after non-maximum suppression, that is, calculating the local threshold mean of the image, so as to adaptively calculate the threshold according to the brightness distribution of different regions of the image. The calculation method is as follows:

[0113] adaptive = mean - C * σ;

[0114] Where, mean is the average value of the pixel values ​​in the neighborhood of the image, C is a constant used to adjust the threshold sensitivity, and σ represents the standard deviation of the pixel values ​​in the neighborhood;

[0115] S54: Perform connected domain analysis according to the image adaptive threshold to detect the edges of objects in the image.

[0116] In a specific embodiment, the local threshold is adaptively adjusted according to the brightness of different areas of the image to obtain a binary edge extraction image. Specifically, the internal point hollowing method is used, and the operation is as follows: if there is a black point in the image and its 8 adjacent points are all black, then the point is deleted, otherwise it is considered that the point is at the edge of the image and needs to be retained. Each pixel in the image is processed in turn, and what is left at the end is the outline of the image.

[0117] In a specific embodiment, contour extraction is performed and pixel-by-pixel comparison is performed with the reference image, and whether a slope abnormal event occurs is determined according to the slope abnormal event discrimination standard, including: the reference image is obtained by inputting a photographed image without an accident into an adaptive edge module, and is compared with the contour extraction image pixel by pixel. If the pixel change of the binary image exceeds the threshold value θ0, it is determined that the slope abnormal event discrimination condition 1 is met; the original image (unprocessed video frame image) is compared with the previous frame original image in pixel value, and the pixel value of the mountain area has a significant fluctuation exceeding the threshold value θ r It is determined that the slope abnormal event judgment condition 2 is met. The threshold setting method is as follows:

[0118] (1) In order to eliminate the influence of vegetation and weather factors on the extraction of mountain edge contours, the binary images obtained by processing the morning, noon and evening video frames of spring, summer, autumn and winter are compared with the reference image, and the maximum change value is taken as the value of the threshold θ0;

[0119] (2) Appropriately reduce the threshold θ based on engineering experience r , in order to achieve early warning of slope collapse events.

[0120] S6. When an alarm message appears, a manual check is carried out to further confirm whether any abnormal slope events occur in the power transmission line area of ​​the open-pit mine.

[0121] In a specific embodiment, Figure 6 As shown, a is a schematic diagram of the reference image, and b is an abnormal slope edge extraction diagram. In step S6, when an alarm message appears, a manual check is performed to check whether an abnormal slope event such as slope collapse has occurred; if so, an emergency response mechanism is activated, including notifying relevant personnel, dispatching maintenance equipment, and implementing on-site investigation and processing; if not, false alarm information is recorded, and the data set and model training are updated to improve the system detection accuracy and reduce the false alarm rate.

[0122] After the relevant steps are executed, on-site operation and maintenance personnel can promptly detect the occurrence of abnormal slope events based on early warning information.

[0123] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0124] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and cannot be construed as limitations on the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A method for detecting abnormal events on slopes in open-pit mines based on transmission line video data, characterized in that: The method comprises the following steps: S1. Define abnormal events of slopes in open-pit mines, and set abnormal event identification criteria of slopes in open-pit mines based on video images; S2. Collect video data of abnormal events on the slopes of open-pit mines. The video data includes original color images of power transmission lines in open-pit mines and gray images of power transmission lines in open-pit mines. Use bilateral filtering algorithm to denoise and smooth the images of slopes in the mines. Expand the dataset of slopes in open-pit mines by synthesis, adjusting brightness and contrast, and adding Gaussian noise. S3, using VGG16 and ResNet101 parallel networks to extract features from complex mining area slope images, and adding a feature fusion layer as a region extraction network to improve the Faster R-CNN network, detect interference targets in open-pit mining area slope images and output the target frame coordinates; S4, according to the target frame coordinates obtained in S3, the target frame area of ​​the slope detection interference factor is eliminated, the autoencoder model is used to perform background filling, and the mining area slope image with the interference removed is restored; S5, improving the contrast of the image, inputting the adaptive edge extraction module, adaptively adjusting the local threshold according to the brightness of different areas of the image, obtaining a binary edge extraction image, then performing contour extraction and comparing pixel by pixel with the reference image, and determining whether a slope abnormal event occurs according to the slope abnormal event discrimination standard; S6. When an alarm message appears, manual verification is carried out to further confirm whether an abnormal slope event occurs in the power transmission line area of ​​the open-pit mine; if so, the emergency response mechanism is activated, including notifying relevant personnel, dispatching maintenance equipment, and conducting on-site investigation and processing; if not, the false alarm information is recorded, and the data set and model training are updated to improve the system detection accuracy and reduce the false alarm rate.

2. The method for detecting abnormal events on slopes in open-pit mines based on transmission line video data according to claim 1, characterized in that: In step S1, the open-pit mine slope abnormal event refers to the open-pit mine area with the risk of slope collapse, facing the threat of collapse or imminent collapse. Its characteristics on the video image are manifested in two aspects: one is the change of the edge of the mountain contour; the other is the change of the pixel value of the mountain area. The pixel value of the slope collapse area is significantly different from that of the non-collapsed area. If both features are met at the same time, it is determined to be a slope abnormality. The slope abnormality event discrimination condition can be set as: (1) The edge contour of the mountain on the image changes; (2) The hue and saturation of some areas in the mountain image are clearly different from those of the surrounding areas, and the pixel values ​​in this area fluctuate significantly.

3. The method for detecting abnormal events on slopes in open-pit mines based on transmission line video data according to claim 1, characterized in that: The step S2 includes processing the gray image of the power transmission line in the open-pit mine area by using a bilateral filtering algorithm. The weight of the bilateral filtering takes into account both the pixel Euclidean distance and the radiation difference in the pixel range domain, and can save the pixel value on the edge of the slope detection image. The specific process is as follows: First, input the image size, that is, the image with pixel × pixel size of M*N, and use a sliding window of pixel × pixel size, that is, n*n, to traverse. The pixel value of each output depends on the neighborhood of the convolved pixel. The pixel value P(i,j) at position (i,j) in the bilateral filter can be represented by the pixel value f(s,t) in the neighborhood and its weight coefficient w(i,j,s,t): Wherein, i and s represent the i-th row and s-th row of the pixel array, respectively; j and t represent the j-th column and t-th column of the pixel array, respectively; Among them, the weight coefficient w(i,j,s,t) depends on the spatial domain kernel d and the range kernel r, which can be expressed as: In the formula, σ d , σ r Respectively represent the standard deviation of the Gaussian kernel selected in the spatial domain and value range.

4. The method for detecting abnormal events on slopes in open-pit mines based on transmission line video data according to claim 1, characterized in that: In step S2, the open-pit mine slope data set is expanded, and the specific method is as follows: (1) Use image processing equipment to synthesize buckets, trucks, bulldozers, excavators, and suspended foreign objects into the shooting scene, and synthesize image data of personnel risks in dangerous areas such as mobile substations to expand the data set; (2) Adding Gaussian noise, adjusting brightness and contrast to simulate weather changes, and performing geometric transformations, together with synthetic images, are used as a dataset for slope detection of power transmission lines in open-pit mines; Anchor boxes are annotated for interference factors in slope detection in the dataset.

5. The method for detecting abnormal events on slopes in open-pit mines based on transmission line video data according to claim 1, characterized in that: In step S3, the process of improving the Faster R-CNN network is as follows: S31: Input the image of the power transmission line in the open-pit mine into the deep convolutional network, which extracts the image features and outputs the feature maps of the image as the input for the next stage. The structural features of the deep convolutional network are as follows: (1) Since VGG16 can obtain fine features of small-sized images and ResNet101 can obtain fine features of large-sized images, the parallel network of VGG16 and ResNet101 is used to obtain a more refined image feature map; (2) Add a feature fusion layer, which is mainly obtained by fusing the bottom and top feature maps of the VGG16 network and the bottom and top feature maps of the ResNet101 network; S32: Input the slope image feature map obtained by the feature extraction network into the region candidate network RPN to generate the target candidate box. The main implementation process includes the following two parts: (1) Determine whether all preset anchor boxes contain the target, and classify them as positive samples or negative samples according to the presence or absence of the target; (2) Correct the positive sample anchor box through bounding box regression to generate more accurate candidate boxes; S33: Perform a regional interest pooling operation, combine the feature maps extracted in step S31 with the candidate box information generated in step S32, extract the feature map proposal feature maps of the candidate box, and input them into the subsequent fully connected layer; S34: Complete the classification operation and use the candidate box feature map extracted in S33 to determine the specific category of the image; at the same time, accurately locate the final position of the target detection box through bounding box regression again.

6. The method for detecting abnormal events on slopes in open-pit mines based on transmission line video data according to claim 1, characterized in that: In step S4, the target frame area of ​​the slope detection interference factor is removed according to the target frame coordinates obtained in S3, and the background is filled using the autoencoder model. The specific steps are as follows: S41: removing the detected slope detection interference factor target frame to obtain a picture lacking a piece of pixel information; S42: Fill in the missing information with pixels. The implementation steps are as follows: (1) Use a large number of complete images and randomly cover some pixels to train the autoencoder model. The encoder module of the model compresses the input image into a low-dimensional feature vector, and the decoder module restores the feature vector to a complete image. The encoding process of the original data X, that is, input layer → hidden layer, is expressed as: h=G θ1 (X)=σ(W1x+b1); The decoding process hidden layer → input layer is expressed as: Where W1 and W2 are the linear change parameters of model training, and b1 and b2 are the nonlinear activation offsets of the model; (2) Input the image with missing information into the autoencoder model to predict and complete the missing information.

7. The method for detecting abnormal events on slopes in open-pit mines based on transmission line video data according to claim 1, characterized in that: In step S5, the contrast of the image is improved and input into an adaptive edge extraction module, the implementation process of which is as follows: S51: converting the original open-pit mine area power transmission line color image in the open-pit mine area slope abnormal event video data into a grayscale image, reading it in a single channel, and performing denoising; S52: Use the Sobel operator to calculate the gradient of the input image, thereby calculating the edge amplitude and angle of the image, performing similar pixel matching, suppressing high-similarity edge information, and performing non-maximum suppression along the gradient direction; S53: Using the moving average method to calculate the local threshold of the gradient image after non-maximum suppression, that is, calculating the local threshold mean of the image, so as to adaptively calculate the threshold according to the brightness distribution of different regions of the image. The calculation method is as follows: adaptive = mean - C * σ; Where, mean is the average value of the pixel values ​​in the neighborhood of the image, C is a constant used to adjust the threshold sensitivity, and σ represents the standard deviation of the pixel values ​​in the neighborhood; S54: Perform connected domain analysis according to the image adaptive threshold to detect the edges of objects in the image.

8. The method for detecting abnormal events on slopes in open-pit mines based on transmission line video data according to claim 1, characterized in that: In step S5, the local threshold is adaptively adjusted according to the brightness of different areas of the image to obtain a binary edge extraction image. Specifically, the internal point hollowing method is used, and the operation is as follows: if there is a black point in the image and its 8 adjacent points are all black, then the point is deleted, otherwise it is considered that the point is at the edge of the image and needs to be retained. Each pixel in the image is processed in turn, and what is left at the end is the outline of the image.

9. The method for detecting abnormal events on slopes in open-pit mines based on transmission line video data according to claim 1, characterized in that: In step S5, contour extraction is performed and pixel-by-pixel comparison is performed with the reference image, and whether a slope abnormality event occurs is determined according to the slope abnormality event discrimination standard, including: The reference image is obtained by inputting the adaptive edge module into the picture taken without the accident, and the pixel point is compared with the contour extraction picture one by one. If the pixel change of the binary image exceeds the threshold value θ0, it is judged to meet the slope abnormal event judgment condition one; the pixel value of the original image is compared with the original image of the previous frame, and the pixel value of the mountain area has obvious fluctuations exceeding the threshold value θ r , it is determined that the slope abnormal event judgment condition 2 is met; the threshold setting method is as follows: (1) In order to eliminate the influence of vegetation and weather factors on the extraction of mountain edge contours, the binary images obtained by processing the morning, noon and evening video frames of spring, summer, autumn and winter are compared with the reference image, and the maximum change value is taken as the value of the threshold θ0; (2) Appropriately reduce the threshold θ based on engineering experience r , in order to achieve early warning of slope collapse events.

10. The method for detecting abnormal events on slopes in open-pit mines based on transmission line video data according to claim 1, characterized in that: In step S6, when an alarm message appears, a manual check is performed to check whether an abnormal slope event such as slope collapse occurs; if so, an emergency response mechanism is activated, including notifying relevant personnel, dispatching maintenance equipment, and conducting on-site investigation and processing; if not, false alarm information is recorded, and the data set and model training are updated to improve the system detection accuracy and reduce the false alarm rate.

Images