Power operation construction ladder handling detection method, system and storage medium

Abstract

The present invention discloses a method, system and storage medium for detecting the handling of ladders in power operation construction. The method comprises: obtaining real-time images or videos of the construction site; performing target detection on the real-time images or videos based on a pre-trained target detection model to determine whether there are target features; the target features include: personnel and ladders; if there are target features in the real-time images or videos, the positions and numbers of the target features in the images or videos are calculated, and the region intersection method is used to determine whether there are any violations; and the violation status, personnel positions, ladder positions and the number of personnel are displayed in real time. The present invention can perform fast and high-precision detection on construction personnel and construction ladders, and can also determine whether there are any violations in handling.

Description

Technical Field

[0001] The present invention relates to the field of electric power construction detection technology, and more specifically to a method, system and storage medium for detecting ladder handling during electric power construction based on deep convolution. Background Art

[0002] With the rapid development of the national economy, the scope of infrastructure construction is also expanding. At present, China has the world's largest power grid, and the UHV power transmission project has achieved the highest, farthest, largest and most advanced in the world in terms of voltage level, power transmission distance, power transmission capacity and technology. However, it also leads to the increasing workload and difficulty of the corresponding power operation tasks.

[0003] The scene of carrying construction ladders often occurs at power operation sites. Whether the ladders are carried and used in accordance with standards affects the progress of power operations and the safety of workers. At present, Faster-RCNN and YOLO are mostly used for target detection. Faster-RCNN implements a two-stage network plus RPN, which has high detection accuracy, but increases the amount of calculation. YOLO uses a 7*7 network for target bounding box regression, which is fast but low in accuracy.

[0004] Therefore, how to provide a detection method, system and storage medium for quickly and accurately detecting whether there is any illegal ladder transportation at a construction site is an urgent problem that needs to be solved by technical personnel in this field. Summary of the invention

[0005] In view of this, the present invention provides a method, system and storage medium for detecting the transportation of construction ladders in power operations, which can quickly and accurately detect construction personnel and construction ladders, and at the same time determine whether there are any transportation violations.

[0006] In order to achieve the above object, the present invention adopts the following technical solution:

[0007] A method for detecting ladder handling in electric power construction based on deep convolution, comprising:

[0008] Get real-time images or videos of the construction site;

[0009] Performing target detection on the real-time image or video based on a pre-trained target detection model to determine whether there are target features; the target features include: personnel and ladders;

[0010] If there are target features in the real-time image or video, the position and number of the target features in the image or video are calculated, and a region intersection method is used to determine whether there is a violation;

[0011] Violations, personnel locations, ladder locations, and the number of personnel are displayed in real time.

[0012] Furthermore, the construction and training process of the target detection model is as follows:

[0013] Acquire historical images and videos of the construction site, and segment the videos to obtain an original picture set;

[0014] After expanding and enhancing the original image data set, the positions of the ladder and the personnel are marked to obtain a training set;

[0015] A deep convolution integration module and a feature fusion module are introduced into the YOLOV5s network structure to obtain the target detection model; the deep convolution integration module is used to modify the number of convolution layers of the YOLOV5s network structure; the feature fusion module is used to perform feature extraction, fusion and normalization processing on the input construction image;

[0016] The target detection model is trained and verified using the training set to obtain the final target detection model.

[0017] Furthermore, the execution process of the deep convolution integration module is:

[0018] The convolution of the YOLOV5s network structure is divided into three layers, wherein the convolution kernel size of the first layer is d×d, the convolution kernel size of the second layer is 1×d, and the convolution kernel size of the third layer is d×1;

[0019] Add the three layers of convolution kernels at the corresponding positions according to the following formula to obtain an equivalent kernel that produces the same output;

[0020]

[0021] Among them, I represents a matrix, K (1) and K (2) represents two 2D kernels with compatible sizes, Represents the sum operation at the corresponding position.

[0022] Furthermore, the execution process of the feature fusion module is as follows:

[0023] The image features of the input construction image are extracted, the high-level features and low-level features of the image features are fused, and then the singular data in the fused features are eliminated.

[0024] Furthermore, the process of the target detection model performing target detection on the real-time image or video is as follows:

[0025] Inputting the real-time image or video into the target detection model;

[0026] Extracting target features from the real-time image or video;

[0027] The FPN+PAN structure is used to upsample and downsample the extracted target features, and the kmean algorithm is used to classify the anchors to generate candidate boxes;

[0028] Performing non-maximum suppression on the candidate box to obtain the best labeling box position;

[0029] The width and depth of the target detection model are adjusted, wherein when the width is adjusted, the output channel of the target detection model satisfies the following formula:

[0030] C2≠anchors×(nc+5) (2);

[0031]

[0032] Where C2 represents the output channel size, anchors represents the number of anchor boxes, which defaults to 3, and nc represents the number of classes to be detected. If formula (2) is satisfied, the output channel size is calculated according to formula (3). Where width in formula (3) represents the set network width, which takes a value between 0 and 2.

[0033] Furthermore, the use of the regional intersection method to determine whether there is a violation includes:

[0034] Extract the labeled areas of the ladder and the personnel in the real-time image or video, and first make an intersection between the personnel area and the ladder area. If the intersection is 0, exclude it, and if the intersection is not 0, keep it;

[0035] Extend the rectangular area where the ladder is located to the personnel area that intersects with it, and calculate the intersection-over-union ratio (IoU1) between the personnel area and the ladder area according to the following formula;

[0036]

[0037] The above formula represents the ladder area Ladder s and Person area s Compared with the union and intersection of , the subscript s indicates that the calculation is performed in the extended area;

[0038] If the intersection ratio is greater than 0.2, it will be recorded, and the situation where the number of people is less than 2 will be regarded as a violation.

[0039] Furthermore, during the training phase of the target detection model, the global intersection-over-union loss function shown below is used to learn the training set:

[0040]

[0041] Among them, A represents the original box for annotation; B represents the predicted box output by the model; C represents the smallest box surrounding A and B; IoU2 represents the intersection-over-union ratio between the original box A and the predicted box B; IoU G represents the global intersection-union ratio.

[0042] It can be known from the above technical solutions that compared with the prior art, the present invention discloses a method for detecting the handling of ladders in power operations based on deep convolution, constructs a new target detection model, and trains the model in the cloud, and carries the trained target detection model to the edge device or the monitoring terminal computer to realize target detection of staff and construction ladders, detect the number of personnel, determine whether the handling of ladders is illegal, and present the results to the display device in real time. The model also has the characteristics of fast speed and small size, strong scalability, and is relatively easy to modify and add. In addition to being able to detect images, the present invention can also directly perform target detection on videos, reduce manpower input, and improve power operation efficiency.

[0043] The present invention also provides a power operation construction ladder handling detection system based on deep convolution, comprising:

[0044] Image acquisition module, used to obtain real-time images or videos of the construction site;

[0045] A target detection module, used to perform target detection on the real-time image or video based on a pre-trained target detection model to determine whether there are target features; the target features include: people and ladders;

[0046] a violation judgment module, for calculating the position and number of the target features in the real-time image or video when the target features exist in the image or video, and using a region intersection method to judge whether there is a violation; and

[0047] The display module is used to display the violation status, personnel location, ladder location and number of personnel in real time.

[0048] Furthermore, the target detection model includes a YOLOV5s network structure, a deep convolution integration module and a feature fusion module;

[0049] The deep convolution integration module is used to modify the number of convolutional layers of the YOLOV5s network structure;

[0050] The feature fusion module is used to perform feature extraction, fusion and normalization processing on the input construction image.

[0051] The present invention also provides a storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the above-mentioned method for detecting ladder transportation during electric power construction based on deep convolution are executed. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0053] Figure 1 The accompanying drawing is a flow chart of a method for detecting ladder handling during electric power construction based on deep convolution provided by the present invention;

[0054] Figure 2 The accompanying drawing is a schematic diagram of the execution process of the deep convolution integration module provided by the present invention;

[0055] Figure 3 The accompanying drawing is a schematic diagram of calculating the global intersection-over-union ratio provided by the present invention;

[0056] Figure 4 The accompanying drawing is a structural schematic diagram of the power operation construction ladder handling detection system based on deep convolution provided by the present invention. DETAILED DESCRIPTION

[0057] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] like Figure 1 As shown, an embodiment of the present invention discloses a method for detecting ladder handling in electric power construction based on deep convolution, comprising:

[0059] S1. Obtain real-time images or videos of the construction site;

[0060] S2, based on the pre-trained target detection model, perform target detection on the real-time image or video to determine whether there are target features; target features include: people and ladders;

[0061] S3. If there are target features in the real-time image or video, the position and number of the target features in the image or video are calculated, and the region intersection method is used to determine whether there is a violation;

[0062] S4. Real-time display of violation status, personnel location, ladder location and number of personnel.

[0063] The above steps are further explained below.

[0064] The acquisition of the image or video in S1 may be an image captured and transmitted back by the flight platform, specifically, an image acquired by an image acquisition system mounted in the above-mentioned flight platform, or acquired by a monitoring camera or a handheld shooting device. The image may be a 2D image or a 3D image. A 2D image may be an image acquired by a 2D image acquisition system, for example, an image acquired by a 2D camera in an image acquisition system; a 3D image may be an image acquired by a 3D image acquisition system, for example, an image acquired by a 3D camera in an image acquisition system. The above-mentioned camera may also be referred to as an image acquisition device, a camera, a video camera, etc. In a possible embodiment, the above-mentioned image acquisition system may be equipped with a 2D camera and a 3D camera at the same time. In a possible embodiment, the acquired image is one or more images. If video data is acquired, the video data needs to be segmented to obtain multiple video frame images, and then subsequent target detection is performed.

[0065] S2. Perform target detection on real-time images or videos based on the pre-trained target detection model to determine whether there are target features; target features include: people and ladders.

[0066] Among them, the construction and training process of the target detection model is:

[0067] S21. Obtain historical images and videos of the construction site, and segment the videos to obtain an original picture set.

[0068] You can collect images or videos of the construction site. To expand the image set, you can also search for images of workers carrying ladders on the Internet. If you collect video data, you need to segment it to get images. Then perform data enhancement processing such as rotation, cropping, translation, noise addition, and exposure on the collected images to get the original image set. The original image set consists of illegal and non-illegal images.

[0069] S22. After expanding and enhancing the original image data set, the positions of the ladders and personnel are annotated to obtain a training set. After obtaining the original image set, all images are manually annotated to obtain a position annotation file of the construction personnel and the ladder positions.

[0070] S23. Introduce a deep convolution integration module and a feature fusion module into the YOLOV5s network structure to obtain a target detection model; the deep convolution integration module is used to modify the number of convolution layers of the YOLOV5s network structure; the feature fusion module is used to perform feature extraction, fusion and normalization on the input construction image.

[0071] The present invention uses feature fusion + convolution integration for target detection on the construction site for the first time. Based on the YOLOV5s network structure, a feature fusion module and a deep convolution integration module are added to obtain a new model YOLO deep convolution integration model (YDCI). According to the stacking number of layers and combination effect of the deep convolution integration module and the feature fusion module, YDCIP and YDCIR with different focuses under YDCI are obtained. YDCIP has a higher precision rate, while YDCIR has a higher recall rate. However, in general, both models are more accurate than YOLOV5s, and one can be selected for use. The present invention effectively solves the problem of low classification accuracy of the basic YOLOV5s for fuzzy targets and small difference objects, and effectively improves the recognition accuracy of ladders in power inspections.

[0072] S24. Use the training set to train and verify the target detection model to obtain the final target detection model.

[0073] The image to be detected is input into the trained target detection model (YDCIP or YDCIR model) to extract and identify the features of the image. Specifically, the features of the image are extracted through the convolution calculation in the target detection model, and the features of the image are identified through feature matching. If the matched similarity is greater than or equal to a preset threshold, such as IoU greater than or equal to 0.5, it can be considered that the detected target overlaps with the actual target. If the matched IoU is less than the preset threshold, such as less than 0.5, it can be considered that no target features are detected in the image. If there is no target feature, no processing is performed.

[0074] More specifically, Figure 2 As shown in Figure 2, the execution process of the deep convolution integration module in S3 is as follows:

[0075] The convolution of the YOLOV5s network structure is divided into three layers, where the convolution kernel size of the first layer is d×d, the convolution kernel size of the second layer is 1×d, and the convolution kernel size of the third layer is d×1;

[0076] Add the three layers of convolution kernels at the corresponding positions according to the following formula to obtain an equivalent kernel that produces the same output;

[0077]

[0078] Among them, I represents a matrix, K (1) and K (2)represents two 2D kernels with compatible sizes, Represents the sum operation at the corresponding position.

[0079] In a specific embodiment, a 3×3 convolution kernel is used for reasoning, and the deep convolution integration module divides the convolution of the YOLOV5s network structure reasoning into three layers. The first layer has a convolution kernel size of 3×3, the second layer has a convolution kernel size of 1×3, and the third layer has a convolution kernel size of 3×1. The three convolution kernels are then fused together through the ReLu function, so the convolution calculation amount is still 3×3, and the calculation amount is almost not increased.

[0080] The execution process of the feature fusion module is:

[0081] The image features of the input construction image are extracted, the high-level features and low-level features of the image features are fused, and then the singular data in the fused features are eliminated.

[0082] In a specific embodiment, after the target detection model training is completed in S2, the specific process of detecting the target is as follows:

[0083] S25, inputting the real-time image or video into the target detection model;

[0084] S26, extracting target features from the real-time image or video;

[0085] S27, use the FPN+PAN structure to upsample and downsample the extracted target features, and use the kmean algorithm to classify the anchors to generate candidate boxes;

[0086] S28, performing non-maximum suppression on the candidate box to obtain the best annotation box position;

[0087] S29, adjusting the width and depth of the target detection model, wherein when the width is adjusted, the output channel of the target detection model satisfies the following formula:

[0088] C2≠anchors×(nc+5) (2);

[0089]

[0090] Where C2 represents the output channel size, anchors represents the number of anchor boxes, which defaults to 3, and nc represents the number of classes to be detected. If formula (2) is satisfied, the output channel size is calculated according to formula (3). Where width in formula (3) represents the set network width, which takes a value between 0 and 2.

[0091] S3. If there are target features in the real-time image or video, the position and number of the target features in the image or video are calculated, and the region intersection method is used to determine whether there is a violation. In this embodiment, when the pre-trained YDCIP or YDCIR model detects the presence of target features in the image, it means that there are people or construction ladders at the location of the image, and the number of detection personnel is counted to determine whether there is a violation. If the personnel are not all present, or there is only one person on the ladder, it is determined to be a violation.

[0092] The regional intersection method is used to determine whether there are any violations, including:

[0093] S31, extracting the labeled areas of the ladder and the personnel in the real-time image or video, first performing an intersection between the personnel area and the ladder area, and excluding those whose intersection is 0, and retaining those whose intersection is not 0;

[0094] S32, extending the rectangular frame area where the ladder is located to the personnel area that intersects with it, and calculating the intersection-over-union ratio IoU1 between the personnel area and the ladder area according to the following formula;

[0095]

[0096] The above formula represents the ladder area Ladder s and Person area s Compared with the union and intersection of , the subscript s indicates that the calculation is performed in the extended area;

[0097] If the intersection ratio is greater than 0.2, it will be recorded, and the situation where the number of people is less than 2 will be regarded as a violation.

[0098] Based on this, in one embodiment, during the training of the target detection model, a global IoU loss function may be determined based on the above IoU, and the training set may be learned using the global IoU loss function, such as Figure 3 As shown in Figure 2, the specific process of obtaining the global intersection-over-union loss function is as follows:

[0099] Get the coordinates of the two vertices of the original box A (x A1 ,y A1 ) and (x A2 ,y A2 ), the coordinates of the two diagonal vertices of the prediction box B (x B1 ,y B1 ) and (x B2 ,y B2 ), take the outermost vertices of A and B to form the smallest box C that surrounds A and B, as shown below. The vertex coordinate formula of C is as follows:

[0100] x C1 =min(x A1 , xB1 );

[0101] y C1 =max(y A1 ,y B1 );

[0102] x C2 =max(x A2 , x B2 );

[0103] y C2 =min(y A2 ,y B2 );

[0104] The intersection-over-union ratio of A and B and the area of ​​C are calculated. The global intersection-over-union ratio is the intersection-over-union ratio of A and B minus the area of ​​C minus the ratio of the union area of ​​A and B to the area of ​​C. Finally, the global intersection-over-union loss function is as follows:

[0105]

[0106] Among them, A represents the original box for annotation; B represents the predicted box output by the model; C represents the smallest box surrounding A and B; IoU2 represents the intersection-over-union ratio between the original box A and the predicted box B; IoU G represents the global intersection-union ratio.

[0107] like Figure 4 As shown, an embodiment of the present invention further provides a power operation construction ladder handling detection system based on deep convolution, comprising:

[0108] Image acquisition module, used to obtain real-time images or videos of the construction site;

[0109] The target detection module is used to detect targets in real-time images or videos based on a pre-trained target detection model to determine whether there are target features; target features include: people and ladders;

[0110] A violation judgment module, for calculating the position and number of target features in the real-time image or video when there are target features in the image or video, and using a region intersection method to judge whether there is a violation; and

[0111] The display module is used to display the violation status, personnel location, ladder location and number of personnel in real time.

[0112] Among them, the image acquisition module includes an image acquisition unit, a data preprocessing unit, a data enhancement unit and a training unit.

[0113] The image acquisition unit includes but is not limited to drones, construction site cameras, filming equipment, etc., which shoot the construction scene in real time to obtain pictures or videos of the construction scene.

[0114] The data preprocessing unit cuts the video to obtain images for training or testing. If the images are already images, no processing is performed.

[0115] The data enhancement unit performs operations such as flipping, cutting, scaling, adding noise, and exposing the image.

[0116] The training unit trains the target detection model based on the training set. The target detection model can select YDCIP or YDCIR.

[0117] The target detection module uses the trained target detection model to identify and detect the real-time images collected by the image acquisition module to determine whether there are target features. When there are target features in the real-time image or video, the violation judgment module calculates the position and number of target features in the image or video, and uses the regional intersection method to determine whether there are violations. The violation status, personnel position, ladder position and number of personnel are transmitted to the display module in real time for display, so that the staff can know the working conditions of the construction site in real time, so as to stop the violations in time.

[0118] An embodiment of the present invention also provides a storage medium having a computer program stored thereon. The computer program can be executed by at least one processor. When the computer program is executed by the processor, the steps of the above-mentioned method for detecting ladder transportation during power operation construction based on deep convolution are executed.

[0119] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0120] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for detecting ladder handling in electric power construction based on deep convolution, characterized in that: include: Get real-time images or videos of the construction site; Performing target detection on the real-time image or video based on a pre-trained target detection model to determine whether there are target features; The target features include: people and ladders; If there are target features in the real-time image or video, the position and number of the target features in the image or video are calculated, and a region intersection method is used to determine whether there is a violation; Real-time display of violation status, personnel location, ladder location, and number of personnel; The construction and training process of the target detection model is as follows: Acquire historical images and videos of the construction site, and segment the videos to obtain an original picture set; After expanding and enhancing the original image set, the positions of the ladders and personnel are marked to obtain a training set; A deep convolution integration module and a feature fusion module are introduced into the YOLOV5s network structure to obtain the target detection model; the deep convolution integration module is used to modify the number of convolution layers of the YOLOV5s network structure; the feature fusion module is used to perform feature extraction, fusion and normalization processing on the input construction image; the execution process of the deep convolution integration module is: The convolution of the YOLOV5s network structure is divided into three layers, wherein the convolution kernel size of the first layer is d×d, the convolution kernel size of the second layer is 1×d, and the convolution kernel size of the third layer is d×1; Add the three layers of convolution kernels at the corresponding positions according to the following formula to obtain an equivalent kernel that produces the same output; Among them, I represents a matrix, K (1) and K (2) represents two 2D kernels with compatible sizes, represents the sum operation at the corresponding position; The execution process of the feature fusion module is as follows: Extract the image features of the input construction image, fuse the high-level features and low-level features of the image features, and then eliminate the singular data in the fused features; Using the training set to train and verify the target detection model to obtain the final target detection model; The process of the target detection model performing target detection on the real-time image or video is as follows: Inputting the real-time image or video into the target detection model; Extracting target features from the real-time image or video; The FPN+PAN structure is used to upsample and downsample the extracted target features, and the kmean algorithm is used to classify the anchors to generate candidate boxes; Performing non-maximum suppression on the candidate box to obtain the best labeling box position; The width and depth of the target detection model are adjusted, wherein when the width is adjusted, the output channel of the target detection model satisfies the following formula: C2≠anchors×(nc+5) (2); Where C2 represents the output channel size, anchors represents the number of anchor boxes, which defaults to 3, and nc represents the number of classes to be detected. If formula (2) is satisfied, the output channel size is calculated according to formula (3). Where width in formula (3) represents the set network width, which is between 0 and 2. The use of the regional intersection method to determine whether there is a violation includes: Extract the labeled areas of the ladder and the personnel in the real-time image or video, and first make an intersection between the personnel area and the ladder area. If the intersection is 0, exclude it, and if the intersection is not 0, keep it; Extend the rectangular area where the ladder is located to the personnel area that intersects with it, and calculate the intersection-over-union ratio (IoU1) between the personnel area and the ladder area according to the following formula; The above formula represents the ladder area Ladder s and Person area s Compared with the union and intersection of , the subscript s indicates that the calculation is performed in the extended area; If the intersection ratio is greater than 0.2, it will be recorded, and the situation where the number of people is less than 2 will be regarded as a violation.

2. According to a method for detecting ladder handling in electric power construction based on deep convolution in claim 1, it is characterized in that: During the training phase of the target detection model, the training set is learned using the global intersection-over-union loss function shown below: Among them, A represents the original box for annotation; B represents the predicted box output by the model; C represents the smallest box surrounding A and B; IoU2 represents the intersection-over-union ratio between the original box A and the predicted box B; IoU G represents the global intersection-union ratio.

3. A power operation construction ladder handling detection system based on deep convolution, characterized in that: The method adopts the deep convolution-based detection method for ladder handling in electric power construction as described in any one of claims 1 to 2, comprising: Image acquisition module, used to obtain real-time images or videos of the construction site; A target detection module is used to perform target detection on the real-time image or video based on a pre-trained target detection model to determine whether there are target features; the target features include: people and ladders; a violation judgment module, for calculating the position and number of the target features in the real-time image or video when the target features exist in the image or video, and using a region intersection method to judge whether there is a violation; and The display module is used to display the violation status, personnel location, ladder location and number of personnel in real time.

4. According to the deep convolution-based power work construction ladder handling detection system of claim 3, it is characterized in that: The target detection model includes a YOLOV5s network structure, a deep convolution integration module and a feature fusion module; The deep convolution integration module is used to modify the number of convolutional layers of the YOLOV5s network structure; The feature fusion module is used to perform feature extraction, fusion and normalization processing on the input construction image.

5. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, the steps of the method for detecting ladder transportation in electric power construction based on deep convolution as described in any one of claims 1 to 2 are executed.

Images