High-altitude electric power operation violation identification method

Abstract

The invention relates to the technical field of target detection, in particular to a high-altitude electric power operation violation recognition method, which comprises the following steps: acquiring a high-definition image and a low-quality image of an electric power operation site through an unmanned aerial vehicle, and making an electric power site high-definition data set; constructing an image enhancement network based on a local information enhancement module and a global information enhancement module; building a violation recognition model based on the improved YOLOv8; selecting a proper optimizer and a loss function; and training and optimizing the constructed model. And converting the optimal model after training optimization into a front-end RKNN model, and deploying the front-end RKNN model to the front end of the unmanned aerial vehicle. Real-time detection is realized by means of a camera at the front end, and a detection result is sent to the study and judgment unit to study and judge whether operators break rules or not. The invention provides a high-altitude electric power operation violation identification method, which considers construction of an operation data set and enhancement of a low-quality image, and provides an improved YOLOv8-based violation identification network model to improve the detection accuracy to the greatest extent.

Description

Technical Field

[0001] The invention relates to the technical field of target detection, and in particular to a method for identifying violations of high-altitude power operations. Background Art

[0002] Aerial work sites usually involve complex environments, such as cross-construction and three-span situations. The working space is small and changeable, which brings high technical requirements to drone flights. Drone operations face great challenges in these environments, especially in bad weather conditions such as rain, dim light, and strong light. Pilots are prone to problems such as difficulty focusing and unstable flight. In addition, violations of regulations in aerial operations often pose great safety risks, but due to the dynamic changes of the target objects and the limited computing resources of edge devices, it is difficult to identify violations in real time through existing technologies. This results in the inability to detect safety risks in a timely manner, and it is necessary to rely on manual judgment to identify potential dangers, which has the hidden dangers of delayed judgment and insufficient accuracy.

[0003] Therefore, how to improve the accuracy and real-time performance of identifying illegal behaviors in high-altitude operations and lower the operating threshold has become an urgent problem to be solved. Summary of the invention

[0004] Purpose of the invention: In order to overcome the deficiencies of the prior art, the present invention provides a method for identifying violations in high-altitude power operations, which solves the problem of low accuracy in identifying violations in high-altitude operations.

[0005] Technical solution: The present invention provides a method for identifying violations of high-altitude power operations, the method comprising:

[0006] Collect high-definition images and low-quality images of the high-altitude power operation site through a drone, and generate a high-definition data set of the power site based on the high-definition images and the low-quality images. The high-definition data set of the power site is obtained by annotating the high-definition images with targets to obtain a first label file corresponding to the image;

[0007] The low-quality image is processed by using a first processing model and a second processing model. The first processing method corresponding to the first processing model includes: introducing a grouping branch in the first three convolutional layers in the dynamic contrast enhancement network structure, and processing the original low-quality image by the first processing method to finally obtain a corresponding first feature map; the second processing method corresponding to the second processing model includes: firstly performing a convolution operation on the original low-quality image, then extracting image features through a multi-head attention module, and finally, respectively generating a corresponding adjustment matrix and transformation value through multi-layer perception and linear transformation, wherein the adjustment matrix is ​​used to perform color correction on the image features, and the transformation value is used to adjust the illumination of the image features, thereby obtaining a corresponding second feature map, and fusing the first feature map and the second feature map to obtain a final optimized image;

[0008] Annotate the optimized image to obtain a corresponding second label file, obtain an optimized data set based on the second label file combined with the corresponding image, and use the optimized data set to train the first processing model and the second processing model until a final training result is achieved;

[0009] Combining the power field high-definition data set and the optimized data set to form a final data set, and formally dividing the final data set into a training set, a validation set, and a test set;

[0010] Constructing a violation recognition model based on improved YOLOv8, wherein the improved YOLOv8 violation recognition model comprises: replacing the backbone network in the YOLOv8 network structure with a Transformer model, replacing the C2f module in the YOLOv8 network structure with a C2f-CBDC module, and finally, the model outputs the result and label data corresponding to the violation recognition;

[0011] Using the training set to train the improved YOLOv8 violation recognition model until convergence;

[0012] Use the test set to evaluate the trained model, judge the effect based on the overall accuracy, and output the final model when the accuracy is stable and improved; if the expected accuracy is not achieved, retrain;

[0013] Convert the optimal model after training and optimization into the front-end RKNN model and deploy it to the front-end of the drone;

[0014] Rely on the front-end camera to achieve real-time detection, and send the detection results to the analysis unit to determine whether the relevant operators have violated regulations, and save the results.

[0015] Further, including:

[0016] The dynamic contrast enhancement network structure includes 7 convolutional layers, and channel separation operations are implemented in the first 3 convolutional layers, grouping branches are added, and in each grouping branch, 1×1 convolution is used for dimensionality increase, and the number of feature channels of the first 3 convolutional layers is halved, and restored to the original number of channels after splicing with the corresponding grouping branches; and the first 6 convolutional layers use ReLU activation function, and the 7th convolutional layer uses Tanh activation function.

[0017] Further, including:

[0018] The dynamic contrast enhancement network structure further includes an illumination adjustment layer, which is used to adjust the illumination range of the image features output from the convolution layer, and then output the first feature map. The illumination adjustment layer represents the relationship between different features based on an estimation formula, and the estimation formula is expressed as:

[0019]

[0020] Among them, I represents the current image feature; a is the set threshold parameter, which is the same as the input image feature dimension. Based on this formula, all pixels in the image are normalized, and the graph formed by the mapping relationship is used as the corresponding color channel to achieve the range adjustment of the illumination.

[0021] Further, including:

[0022] The second processing model includes: a convolution module, an encoding module, a first-layer normalization module, a multi-head attention module, a second-layer normalization module, a multi-layer perception module and a linear transformation module. The original low-quality image is input into the convolution module. The convolution module includes a 3×3 convolution layer to obtain a first feature map with 64 channels. After the dimensionality conversion of the encoding module, the first feature map is sequentially input into the first-layer normalization module and the multi-head attention module, and the corresponding second feature is extracted through the module. The second feature passes through the second normalization module and then the multi-layer perception module to generate a 3×3 adjustment matrix, and is linearly transformed with the second feature output by the multi-head attention module to obtain a 1×1 transformation value.

[0023] Further, including:

[0024] The fusing of the first feature map and the second feature map to obtain a final optimized image specifically includes:

[0025] The second feature map is input into a convolutional layer with a size of 1×1 and a channel number of 1 to obtain a third feature, the third feature is multiplied by the first feature to obtain a fourth feature, the fourth feature is exponentially calculated with the third feature, and a final optimized image is output.

[0026] Further, including:

[0027] The use of the Transformer model to replace the backbone network in the YOLOv8 network structure includes: using the EfficientFormerV2 model in the Transformer model as the backbone network of the model.

[0028] Further, including:

[0029] The C2f-CBDC module is used to replace the C2f module in the YOLOv8 network structure, including:

[0030] Block1, Block2 and Block3 in the EfficientFormerV2 model are all connected to a convolutional layer of the Neck part of the YOLOv8 network model, and the fast spatial pyramid pooling layer SPPF is connected to a convolutional layer of the Neck part of the YOLOv8 network model. The C2f module of the Neck part of the YOLOv8 network model is replaced by a C2f-CBDC module. The C2f-CBDC module is a C2f module with a CBDC attention module added to the C2f module. The CBDC attention module integrates the CBAM attention module and deformable convolution. The CBAM attention module combines channel attention and spatial attention to refine feature maps in two different dimensions.

[0031] Further, including:

[0032] The CBDC attention module combines the CBAM attention module and deformable convolution, specifically including:

[0033] Through the channel attention module CAM, the input feature map is subjected to maximum pooling and average pooling operations to compress the information of each channel and generate two weight vectors. These two weight vectors are mapped through a shared network to obtain a weight matrix for each channel and multiplied element by element with the original input feature map, thereby enhancing the feature response of important channels and suppressing redundant or irrelevant channel information.

[0034] The spatial attention module SAM focuses on the features of the spatial dimension based on the output of the channel attention module. In this module, the maximum pooling and average pooling operations are applied to all channels at each position respectively to generate two two-dimensional feature maps. Then, after the deformable convolution operation, the channel information is fused, and the spatial attention matrix is ​​multiplied element by element with the output feature map of the channel attention module.

[0035] Further, including:

[0036] The optimal model after training and optimization is converted into a front-end RKNN model and deployed to the front-end of the drone, including:

[0037] Use Python scripts to convert the trained image enhancement model and object detection model into a model format suitable for the RKNN framework. The conversion process includes the reconstruction of the model structure and parameters to ensure that the model can run efficiently in the RKNN engine.

[0038] After the conversion is completed, a series of optimization operations are performed on the converted RKNN model, including adaptive hybrid quantization, which aims to improve its execution efficiency on embedded platforms by reducing the model's computational workload and storage usage while maintaining its accuracy level as much as possible;

[0039] Through multiple experiments, we verified the impact of different quantization methods on model performance and conducted a detailed analysis of the detection accuracy of each category to ensure that the model can still maintain a high recognition accuracy in different scenarios;

[0040] The optimized RKNN model is deployed to the front end of the drone. During the deployment process, the model will be loaded into the hardware platform on the drone and the inference process will be accelerated through the integrated neural network processing unit NPU.

[0041] Further, including:

[0042] The front-end camera is used to realize real-time detection, and the detection results are sent to the judgment unit to judge whether the relevant operators have violated the rules and regulations, and the results are saved, which specifically includes:

[0043] The inspection drone starts reading the real-time video stream and performs uninterrupted frame sampling detection;

[0044] The drone relies on the violation recognition algorithm in the RKNN model to detect the construction tower and guide the drone to fly towards the tower;

[0045] After flying above the construction tower, the drone begins to fly around the tower based on the preset longitude and latitude.

[0046] When a worker is detected on the construction tower, the analysis and judgment unit determines whether the worker is working in accordance with the regulations and whether the worker pixel frame is equipped with the prescribed safety measures based on the detection results;

[0047] Once a violation is identified, the detection and analysis results will be saved and the data will be transmitted back to the backend system, which will further analyze and process the data.

[0048] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0049] The present invention proposes an image processing method based on two processing modes. The method can be trained only with low-quality images in an environment without paired data, which greatly reduces the difficulty of training. In addition, the method of the present invention fully mines the details and overall features in the image through the combination of the first processing model and the second processing model. In the first processing model, the grouping branch and the dilated convolution layer are used to improve the sensitivity of the network to details, so that the enhancement process can be more refined, the detail information in the image is effectively retained, and the color distortion and detail loss problems common in traditional methods are avoided. In the second processing model, combined with the multi-head attention mechanism of the Transformer network, the overall context information in the image can be captured, and the overall contrast and brightness distribution of the image can be improved, thereby effectively solving the common problems of over-enhancement of bright areas and insufficient enhancement of dark areas, and further improving the visual effect of the image. In summary, the low-light image enhancement method proposed in the present invention, by virtue of its innovative design of training without paired data and combining details with overall information, not only greatly reduces the difficulty of training and data requirements, but also achieves a more refined and flexible image enhancement effect, and has a wide range of application prospects and excellent practical value.

[0050] In the power operation environment, the present invention improves the basic framework of YOLOv8: first, EfficientFormerV2 is selected as the feature extraction network to improve the algorithm's ability to capture overall information. Secondly, the C2f-CBDC module is used to replace the original C2f module. This module integrates the CBAM attention module and deformable convolution, which can effectively reduce the scale of the model while ensuring the performance of the model, and dynamically adjust the receptive field, focusing on the features of the detailed targets, significantly reducing background interference, and improving the feature extraction capability. Finally, a feature fusion pyramid structure is used to better integrate and utilize features of different scales. In summary, after adopting this technical solution, the performance of target detection has been comprehensively improved, especially in the power operation environment, it can more accurately identify and locate targets, reduce the impact of background noise, and improve the model's adaptability to multi-scale targets. . BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 is a flow chart of a method for identifying violations of power operations according to an embodiment of the present invention;

[0052] Figure 2 is a general structure diagram of an image enhancement network of a first processing model and a second processing model according to an embodiment of the present invention;

[0053] Figure 3 is a network structure diagram of the first processing model according to an embodiment of the present invention;

[0054] Figure 4It is a general structural diagram of the improved yolov8 network model described in the embodiment of the present invention;

[0055] Figure 5 yes Figure 3 The structure diagram of the C2f-CBDC module in the network structure unit;

[0056] Figure 6 yes Figure 4 Internal module structure diagram of the CBDC attention module. DETAILED DESCRIPTION

[0057] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] The present invention provides a method for identifying violations of high-altitude power operations. The method takes into account the construction of an operation data set and the enhancement of low-quality images. At the same time, an improved YOLOv8-based violation recognition network model is proposed to maximize the detection accuracy, including:

[0059] Collect high-definition images and low-quality images of the high-altitude power operation site through a drone, and generate a high-definition data set of the power site based on the high-definition images and the low-quality images. The high-definition data set of the power site is obtained by annotating the high-definition images with targets to obtain a first label file corresponding to the image;

[0060] The low-quality image is processed by using a first processing model and a second processing model. The first processing method corresponding to the first processing model includes: introducing a grouping branch in the first three convolutional layers in the dynamic contrast enhancement network structure, and processing the original low-quality image by the first processing method to finally obtain a corresponding first feature map; the second processing method corresponding to the second processing model includes: firstly performing a convolution operation on the original low-quality image, then extracting image features through a multi-head attention module, and finally, respectively generating a corresponding adjustment matrix and transformation value through multi-layer perception and linear transformation, wherein the adjustment matrix is ​​used to perform color correction on the image features, and the transformation value is used to adjust the illumination of the image features, thereby obtaining a corresponding second feature map, and fusing the first feature map and the second feature map to obtain a final optimized image;

[0061] Annotate the optimized image to obtain a corresponding second label file, obtain an optimized data set based on the second label file combined with the corresponding image, and use the optimized data set to train the first processing model and the second processing model until a final training result is achieved;

[0062] Combining the power field high-definition data set and the optimized data set to form a final data set, and formally dividing the final data set into a training set, a validation set, and a test set;

[0063] Constructing a violation recognition model based on improved YOLOv8, wherein the improved YOLOv8 violation recognition model comprises: replacing the backbone network in the YOLOv8 network structure with a Transformer model, replacing the C2f module in the YOLOv8 network structure with a C2f-CBDC module, and finally, the model outputs the result and label data corresponding to the violation recognition;

[0064] Using the training set to train the improved YOLOv8 violation recognition model until convergence;

[0065] Use the test set to evaluate the trained model, judge the effect based on the overall accuracy, and output the final model when the accuracy is stable and improved; if the expected accuracy is not achieved, retrain;

[0066] Convert the optimal model after training and optimization into the front-end RKNN model and deploy it to the front-end of the drone;

[0067] Rely on the front-end camera to achieve real-time detection, and send the detection results to the analysis unit to determine whether the relevant operators have violated regulations, and save the results.

[0068] The dynamic contrast enhancement network structure includes 7 convolutional layers, and channel separation operations are implemented in the first 3 convolutional layers, grouping branches are added, and in each grouping branch, 1×1 convolution is used for dimensionality increase, and the number of feature channels of the first 3 convolutional layers is halved, and restored to the original number of channels after splicing with the corresponding grouping branches; and the first 6 convolutional layers use ReLU activation function, and the 7th convolutional layer uses Tanh activation function.

[0069] The dynamic contrast enhancement network structure further includes an illumination adjustment layer, which is used to adjust the illumination range of the image features output from the convolution layer, and then output the first feature map. The illumination adjustment layer represents the relationship between different features based on an estimation formula, and the estimation formula is expressed as:

[0070]

[0071] Among them, I represents the current image feature; a is the set threshold parameter, which is the same as the input image feature dimension. Based on this formula, all pixels in the image are normalized, and the graph formed by the mapping relationship is used as the corresponding color channel to achieve the range adjustment of the illumination.

[0072] The second processing model includes: a convolution module, an encoding module, a first-layer normalization module, a multi-head attention module, a second-layer normalization module, a multi-layer perception module and a linear transformation module. The original low-quality image is input into the convolution module. The convolution module includes a 3×3 convolution layer to obtain a first feature map with 64 channels. After the dimensionality conversion of the encoding module, the first feature map is sequentially input into the first-layer normalization module and the multi-head attention module, and the corresponding second feature is extracted through the module. The second feature passes through the second normalization module and then the multi-layer perception module to generate a 3×3 adjustment matrix, and is linearly transformed with the second feature output by the multi-head attention module to obtain a 1×1 transformation value.

[0073] The fusing of the first feature map and the second feature map to obtain a final optimized image specifically includes:

[0074] The second feature map is input into a convolutional layer with a size of 1×1 and a channel number of 1 to obtain a third feature, the third feature is multiplied by the first feature to obtain a fourth feature, the fourth feature is exponentially calculated with the third feature, and a final optimized image is output.

[0075] Further, including:

[0076] The use of the Transformer model to replace the backbone network in the YOLOv8 network structure includes: using the EfficientFormerV2 model in the Transformer model as the backbone network of the model.

[0077] The C2f-CBDC module is used to replace the C2f module in the YOLOv8 network structure, including:

[0078] Block1, Block2 and Block3 in the EfficientFormerV2 model are all connected to a convolutional layer of the Neck part of the YOLOv8 network model, and the fast spatial pyramid pooling layer SPPF is connected to a convolutional layer of the Neck part of the YOLOv8 network model. The C2f module of the Neck part of the YOLOv8 network model is replaced by a C2f-CBDC module. The C2f-CBDC module is a C2f module with a CBDC attention module added to the C2f module. The CBDC attention module integrates the CBAM attention module and deformable convolution. The CBAM attention module combines channel attention and spatial attention to refine feature maps in two different dimensions.

[0079] The CBDC attention module combines the CBAM attention module and deformable convolution, specifically including:

[0080] Through the channel attention module CAM, the input feature map is subjected to maximum pooling and average pooling operations to compress the information of each channel and generate two weight vectors. These two weight vectors are mapped through a shared network to obtain a weight matrix for each channel and multiplied element by element with the original input feature map, thereby enhancing the feature response of important channels and suppressing redundant or irrelevant channel information.

[0081] The spatial attention module SAM focuses on the features of the spatial dimension based on the output of the channel attention module. In this module, the maximum pooling and average pooling operations are applied to all channels at each position respectively to generate two two-dimensional feature maps. Then, after the deformable convolution operation, the channel information is fused, and the spatial attention matrix is ​​multiplied element by element with the output feature map of the channel attention module.

[0082] The optimal model after training and optimization is converted into a front-end RKNN model and deployed to the front-end of the drone, including:

[0083] Use Python scripts to convert the trained image enhancement model and object detection model into a model format suitable for the RKNN framework. The conversion process includes the reconstruction of the model structure and parameters to ensure that the model can run efficiently in the RKNN engine.

[0084] After the conversion is completed, a series of optimization operations are performed on the converted RKNN model, including adaptive hybrid quantization, which aims to improve its execution efficiency on embedded platforms by reducing the model's computational workload and storage usage while maintaining its accuracy level as much as possible;

[0085] Through multiple experiments, we verified the impact of different quantization methods on model performance and conducted a detailed analysis of the detection accuracy of each category to ensure that the model can still maintain a high recognition accuracy in different scenarios;

[0086] The optimized RKNN model is deployed to the front end of the drone. During the deployment process, the model will be loaded into the hardware platform on the drone and the inference process will be accelerated through the integrated neural network processing unit NPU.

[0087] The front-end camera is used to realize real-time detection, and the detection results are sent to the judgment unit to judge whether the relevant operators have violated the rules and regulations, and the results are saved, which specifically includes:

[0088] The inspection drone starts reading the real-time video stream and performs uninterrupted frame sampling detection;

[0089] The drone relies on the violation recognition algorithm in the RKNN model to detect the construction tower and guide the drone to fly towards the tower;

[0090] After flying above the construction tower, the drone begins to fly around the tower based on the preset longitude and latitude.

[0091] When a worker is detected on the construction tower, the analysis and judgment unit determines whether the worker is working in accordance with the regulations and whether the worker pixel frame is equipped with the prescribed safety measures based on the detection results;

[0092] Once a violation is identified, the detection and analysis results will be saved and the data will be transmitted back to the backend system, which will further analyze and process the data.

[0093] In order to further describe the scheme of the present application more clearly, the following examples are supplemented:

[0094] like Figure 1 As shown, the present invention proposes a method for identifying violations of power operations based on drones and an improved YOLOv8 target detection algorithm. The specific implementation steps are as follows:

[0095] Step 1: Collect high-definition images of the power operation site through drones and create a high-definition data set of the power site;

[0096] The process of preparing the power field high-definition data set in step 1 is as follows:

[0097] Step 1.1: Collect high-definition and low-quality images of the power operation site through drones;

[0098] Step 1.2: Use the LabelImg annotation tool to annotate the high-definition image data and obtain the label file corresponding to the image; finally, use the Python script to crop and classify the annotated data to create a violation data set, and divide the data set into training set, validation set and test set in a random manner according to a fixed ratio.

[0099] Step 2: construct an image processing network based on the first processing module and the second processing module;

[0100] The overall architecture of the network is as follows: Figure 2 As shown, it mainly includes two parts:

[0101] In the first processing module, the present invention makes several key improvements on the basis of the original DCE-Net (Dynamic Contrast Enhancement Network) network to further improve the network performance, computing efficiency and adaptability in illumination adjustment. The following are the specific contents of the main improvements:

[0102] First, a grouping branch is introduced into the DCE-Net network structure. By separating the channels of the feature maps of the first three layers, more features can be obtained from the same batch of feature maps. This design helps to process and focus on the input features more finely. Each grouping branch contains three different types of convolutional layers, namely 1×1 convolutional layer, 3×3 dilated convolutional layer and 1×1 convolutional layer. The 1×1 convolutional layer plays a role of dimension increase and decrease in the network, and can effectively maintain the feature extraction efficiency of the dilated convolution without increasing too much computation; while the 3×3 dilated convolutional layer expands the receptive field, allowing the network to capture a wider range of contextual information and enhance the network's ability to focus on detailed features.

[0103] Secondly, in order to improve the range of illumination adjustment and further reduce the number of parameters and calculation time of the model, the dynamic contrast enhancement network structure also includes an illumination adjustment layer, which is used to adjust the illumination range of the image features output from the convolution layer, and then output the first feature map. The illumination adjustment layer represents the relationship between different features based on an estimation formula, and the estimation formula is expressed as:

[0104]

[0105] Among them, I represents the current image feature; a is the set threshold parameter, which is the same as the input image feature dimension. Based on this formula, all pixels in the image are normalized, and the graph formed by the mapping relationship is used as the corresponding color channel to achieve the range adjustment of illumination. This alternative method not only improves the computational efficiency of the model, but also enables better control of the adjustment range of illumination mapping, thereby improving the quality and adaptability of detail processing.

[0106] In the second processing module, the present invention uses a Transformer network for image enhancement, and captures the relationship between single pixels and peripheral pixels through a multi-head attention module to obtain rich overall feature information, thereby solving the problem of excessive enhancement of bright areas and insufficient enhancement of dark areas, and improving the overall contrast of the image. Specifically, first, a 3×3 convolution is performed on the input image to obtain a feature map with 64 channels. Then, after the dimensionality conversion of the encoding module, the feature map is input into the multi-head attention module, through which the overall features are extracted. Subsequently, a 3×3 adjustment matrix and a 1×1 transformation value are generated through a multi-layer perceptron MLP and a linear transformation. Among them, the adjustment matrix is ​​used to perform color correction on the three RGB color channels, and the transformation value is used for nonlinear adjustment of the image illumination, and finally, the final image optimization is completed.

[0107] Specifically, Figure 3 As shown, the first processing network of the present invention is composed of 7 convolutional layers, and a jump connection is introduced to retain the features extracted in stages. On this basis, the DCE-Net structure is improved, and a grouping branch is added by implementing a channel separation operation in the first 3 convolutional layers. In each grouping branch, 1×1 convolution is used for dimensionality increase, and dilated convolutions with different dilation rates are used to extract features, thereby ensuring the early nature of feature extraction and the continuity of feature information.

[0108] To avoid increasing network parameters and computational overhead, the number of feature channels in the first three layers is halved and restored to the original number of channels after being spliced ​​with the grouping branch. In the first processing model network, the first six layers use the ReLU activation function, which is responsible for feature extraction and training acceleration. The last layer uses the Tanh activation function to ensure the rationality of the output results and meet the requirements of the image enhancement task.

[0109] The second processing module module is as follows Figure 2As shown in the figure: First, a 3×3 convolution is performed on the input image to obtain a feature map with 64 channels. Then, after the dimension conversion of the encoding module, the feature map is input into the multi-head attention module, through which the overall features are extracted. Subsequently, a 3×3 color adjustment matrix and a 1×1 transformation value are generated through the multi-layer perceptron MLP and linear transformation. Among them, the adjustment matrix is ​​used to perform color correction on the three RGB color channels, and the transformation value is used for nonlinear adjustment of the overall image illumination.

[0110] Step 3: Build a traffic violation recognition model based on improved YOLOv8;

[0111] Taking into account the high similarity between categories in the process of drone flight detection, in order to enable the model to fully learn the global and local information in the image, the present invention has made improvements on the basic framework of YOLOv8. First, EfficientFormerV2 was selected as the backbone network to enhance the algorithm's ability to capture global information; secondly, the C2f-CBDC module was used to replace the original C2f module. This module integrates CBAM and deformable convolution, which can effectively reduce the scale of the model while ensuring model performance, and dynamically adjust the receptive field, focusing on the characteristics of local targets, significantly reducing background interference, and improving feature extraction capabilities. Finally, a feature fusion pyramid structure is adopted to better integrate and utilize features of different scales, thereby further improving the recognition accuracy of the model.

[0112] Step 3 improves the YOLOv8 network model as follows Figure 3 As shown, the specific process is as follows:

[0113] Step 3.1: Use the Transformer model to replace the original backbone network:

[0114] Since the Transformer model can capture the overall dependency between pixels by means of the self-attention mechanism, thereby efficiently acquiring overall information, the present invention introduces the Transformer into the backbone network.

[0115] Because the Transformer model can capture the overall dependencies between pixels with the help of the self-attention mechanism, it can efficiently obtain overall information.

[0116] Therefore, the present invention introduces Transformer into the backbone network. Specifically, Figure 4As shown, the present invention selects EfficientFormerV2 as the backbone network of the model, and its structure includes Stem, Subsample, feedforward network (FFN), attention feedforward network (AttenFFN) and attention downsampling (AttentionDownSample), among which Stem, Subsample and Attention DownSample modules are jointly responsible for the downsampling of feature maps to reduce redundant information; FFN is composed of point-by-point convolution and 3×3 depth-separable convolution, among which point-by-point convolution is mainly used for compression and expansion of feature channels, and depth-separable convolution focuses on the extraction of detail features. Finally, the model constructs the AttenFFN module by combining the Multi-Head Self-Attention (MHSA) mechanism with FFN, further improving the ability to capture overall information. This design enables the model to take into account both details and overall dependencies when processing images, thereby improving the overall detection and recognition performance.

[0117] In EfficientFormerV2, a multi-head self-attention downsampling module is designed that combines detail information. This module implements downsampling of detail information through strided convolution, while downsampling of overall information is completed by MHSA. In the MHSA framework, the query matrix is ​​used to determine the resolution of the output feature map. To this end, the query matrix is ​​downsampled by the pooling layer and the 1×1 convolution layer. Subsequently, the 1×1 convolution layer is combined by element-by-element addition to achieve fusion and interaction between different information. The detailed operation process is shown in formula (2).

[0118] L GQ =BN(Conv 1×1 (Pool 3×3，stride＝2 (Q)+Conv 3×3，stride＝2 (Q)));

[0120] Where: L GQ is the query matrix after two different downsampling strategies; BN is the batch normalization operation; Conv 1×1 It is a point-by-point convolution operation; Pool 3×3，stride＝2 Indicates an average pooling operation with a pooling kernel of 3×3 and a step size of 2; Conv 3×3，stride＝2 Represents a convolution operation with a pooling kernel of 3×3 and a stride of 2.

[0121] Specifically, the present invention selects EfficientFormerV2 as the backbone network of the model, and its structure is as follows Figure 2 In Block 1-4, the FFN (Feed Forward Neural Network) network is repeated 2, 2, 4, and 2 times respectively. In Block 3 and Block 4, the AttenFFN module is constructed by combining the Multi-Head Self-Attention (MHSA) mechanism with FFN, and it is added after the FFN network to further enhance the model's ability to capture the overall information of the target.

[0122] Step 3.2: The C2f-CBDC module is used to replace the original C2f module.

[0123] like Figure 5 As shown in the figure, as the number of network layers increases, the resolution of the input image gradually decreases, which may weaken the detection performance of the model. Moreover, since C2f uses a fixed receptive field, the feature extraction of this fixed structure is not ideal for targets with large scale changes.

[0124] To solve the above problems, the present invention adds a CBDC attention module on the basis of the original C2f module. The CBDC module combines the CBAM attention module with the deformable convolution module to dynamically increase the receptive field of the model while improving the model's attention to important feature channels, so that the convolution kernel can better adapt to the actual shape of the object. This improvement allows the module to focus more on the target area and effectively reduce background interference.

[0125] Specifically, in CBDC, the input feature map is first subjected to maximum pooling and average pooling operations through the channel attention module to compress the information of each channel and generate two weight vectors. These two weight vectors are mapped through a shared network to obtain a weight matrix for each channel, and are multiplied element-by-element with the original input feature map, thereby enhancing the feature response of important channels and suppressing redundant or irrelevant channel information. Next, the spatial attention module focuses on the features of the spatial dimension based on the output of the channel attention module. In this module, maximum pooling and average pooling operations are applied to all channels at each position respectively to generate two two-dimensional feature maps. Then, after a deformable convolution operation, the channel information is fused, and the spatial attention matrix is ​​multiplied element-by-element with the output feature map of the channel attention module, further enhancing the model's perception of key position information, thereby improving the model's positioning accuracy for the target area.

[0126] In summary, by adding deformable convolution and CBAM dual attention mechanism in the feature enhancement module, the model's target tracking ability can be effectively improved, and the target detection and recognition performance in complex environments can be improved.

[0127] Specifically, if Figure 3As shown in the figure, a CBDC attention module is added to the original C2f module. This module combines the CBAM attention module and deformable convolution, which can effectively reduce the scale of the model while ensuring the performance of the model, and dynamically adjust the receptive field, focusing on the detailed features of the target, significantly reducing background interference, and improving the feature extraction capability.

[0128] Specifically, in the CBDC module, first Figure 6 As shown in the left figure of , the input feature map is subjected to maximum pooling and average pooling operations through the channel attention module (CAM) to compress the information of each channel and generate two weight vectors. These two weight vectors are mapped through the shared network to obtain the weight matrix of each channel, and are multiplied element by element with the original input feature map, thereby enhancing the feature response of important channels and suppressing redundant or irrelevant channel information. Next, as Figure 6 As shown in the right figure in Figure , the spatial attention module (SAM) focuses on the features of the spatial dimension based on the output of the channel attention module. In this module, the maximum pooling and average pooling operations are applied to all channels at each position to generate two two-dimensional feature maps. Then, through the deformable convolution operation, the channel information is fused, and the spatial attention matrix is ​​multiplied element by element with the output feature map of the channel attention module, further enhancing the model's perception of key position information, thereby improving the model's positioning accuracy for the target area.

[0129] Step 3.3: Construct feature fusion pyramid.

[0130] The present invention proposes a feature fusion pyramid structure.

[0131] In YOLOv8, the FPN+PAN structure is used for feature fusion. Although the FPN+PAN structure achieves the complementarity of semantic information and positioning information, there is still room for improvement. On the one hand, feature fusion leads to an increase in the amount of calculation. In addition, the feature map will lose some of the original feature information after multiple ups and downsampling. On the other hand, in the process of feature fusion, it is easy to extract invalid feature information and ignore some detail information, resulting in reduced target detection accuracy.

[0132] Specifically, if Figure 3 As shown in the Neck structure, first, in order to make full use of the low-level feature maps to enrich the detailed information of the features, the number of channels of the feature maps is adjusted by introducing the 1×1 convolution operation, making the network more efficient when performing operations such as feature fusion and upsampling. Then, while fusing the feature maps at different scales, an element-by-element subtraction operation is performed to eliminate redundant background features and retain the target information. For example, when the size of the feature image of the mth layer is described as h m × m ×c mWhen, let the feature information of the i-th layer be f i ={x i , x i+1 , x i+2 ,…,x n}, the feature information of the i+1th layer should be f i+1 ={x i+1 , x i+2 , x i+3 ,…,x n}.

[0133] Usually, in the i to f i+1 In the forward propagation process, x i is ignored, but it may also contain detailed information of the target. Since some frames can obtain less detailed information of the target during the flight of the drone, they are easily ignored by the model during the forward propagation process. Therefore, element-by-element subtraction of features at different scales can eliminate redundant background features and retain target information. In short, since the module does not add additional calculations, it is lighter than FPN+PAN, has fewer parameters, and is more suitable for front-end requirements.

[0134] Step 4: Select a suitable optimizer and loss function for step 3;

[0135] The specific process is as follows: This paper compares the Adam optimizer and the SGD optimizer, and the stochastic gradient descent (Stochasti c Gradient Descent (SGD) is a basic optimization algorithm widely used in the training of deep learning models. SGD updates the model parameters according to the gradient of the current sample, so that the loss function is gradually reduced. However, the convergence speed of SGD is relatively slow, and the update process is easily disturbed by noise, which may lead to convergence instability because the gradient direction of each update is not always in the direction of the optimal solution. In contrast, the Adam (Adaptive Moment Estimation) optimizer is an adaptive learning rate optimization algorithm that combines the momentum method and the adaptive learning rate method, and can assign different weights to different gradients, thereby making the learning rate more stable during the training process, helping the model to converge to the optimal solution more quickly and smoothly. Therefore, the present invention selects the Adam optimizer for model training in order to achieve more efficient and stable convergence performance.

[0136] The bounding box regression loss function used in the present invention for training the model uses MPDIoU (Maximum Possible Distance Intersection over Union). Compared with the classic IoU, MPDIoU can more accurately evaluate the relative position and geometric relationship between objects by introducing the metric of the maximum possible distance between objects, avoiding some limitations of IoU. Especially when dealing with small objects, closely arranged objects or complex scenes, MPDIoU provides a more sophisticated and robust evaluation standard, which helps to improve the accuracy and stability of object detection tasks.

[0137] The formula for MPDIoU is as follows:

[0138]

[0139] In the formula, the superscripts a and b represent the annotation box and the prediction box respectively; the subscripts 1 and 2 represent the upper right corner and lower left corner of the corresponding box respectively; Represents the normalization operation on the diagonal to facilitate the comparison of the diagonal distance and IoU at the same scale; c1 is used to adjust the weight of the diagonal distance to better measure the impact of position offset on IoU.

[0140] Step 5: Train and optimize the model constructed in steps 2 and 3;

[0141] The specific process is as follows:

[0142] Step 5.1: Set training parameters, including maximum number of iterations, learning rate, optimizer momentum, and weight initialization;

[0143] Step 5.2: Take the low-quality image from the high-definition image collected in step 1.1 as the input of the image enhancement module and train it. Generate high-order curves. Then, use these high-order curves to adjust the dynamic range of the input image at the pixel level to obtain the enhanced image.

[0144] Step 5.3: Use the HD dataset to train the improved YOLOv8 network. After each round of training, adjust the weights according to the category difficulty until the training is completed and the best detection model is obtained;

[0145] Step 5.4: Use the test set to evaluate the trained model and judge the effect based on the overall accuracy. When the accuracy is stable and improved, output the final model; if it does not reach the expected accuracy, retrain.

[0146] Step 6: Convert the optimal model trained and optimized in step 5 into the front-end RKNN model and deploy it to the front-end of the drone.

[0147] The specific process is as follows:

[0148] Step 6.1: Use Python scripts to convert the trained image enhancement model and object detection model (usually in .pt format) into a model format suitable for the RKNN framework. The conversion process includes the reconstruction of the model structure and parameters to ensure that the model can run efficiently in the RKNN engine.

[0149] Step 6.2: After the conversion is completed, a series of optimization operations are performed on the converted RKNN model, including adaptive hybrid quantization. This process aims to improve the execution efficiency on the embedded platform by reducing the computational workload and storage usage of the model while maintaining its accuracy level as much as possible.

[0150] Step 6.3: Through multiple experiments, verify the impact of different quantization methods on model performance, and conduct a detailed analysis of the detection accuracy of each category to ensure that the model can still maintain a high recognition accuracy in different scenarios.

[0151] Step 6.4: Deploy the optimized RKNN model to the drone front end. During the deployment process, the model will be loaded into the hardware platform on the drone and the inference process will be accelerated through the integrated NPU (neural network processing unit).

[0152] Step 7: Real-time detection is achieved by using the front-end camera, and the detection results are sent to the analysis unit to determine whether the operator has violated regulations. The results are saved and sent back to the background.

[0153] The specific process is as follows:

[0154] The camera at the front end of the drone is responsible for acquiring the video stream of the surrounding environment in real time, and then inputting it into the image enhancement model for processing. The enhanced image will be passed to the target detection model for uninterrupted real-time detection. During this process, the model will quickly process each frame of the image and transmit it to the analysis unit. Based on the output of the model, the analysis unit determines whether the tower operator is correctly equipped with safety measures (such as helmets, etc.) in the pixel frame, thereby evaluating whether the operator has any illegal operation behavior and outputting the analysis results in real time.

[0155] Step 7.1: The inspector releases the inspection drone, which starts reading the real-time video stream and performs uninterrupted frame sampling detection;

[0156] Step 7.2: The drone detects the construction tower by using the violation recognition algorithm and guides the drone to fly towards the tower;

[0157] Step 7.3: After flying above the construction tower, the drone begins to fly around the tower based on the preset longitude and latitude;

[0158] Step 7.4: When a worker is detected on the construction tower, the judging unit judges whether the worker is working in accordance with the regulations and whether the worker pixel frame is equipped with the prescribed safety measures according to the detection results;

[0159] Step 7.5: Once a violation is identified, the detection and analysis results will be saved and the data will be sent back to the backend system. The backend system will further analyze and process the data, such as generating statistical reports, recording the time and location of the occurrence, and providing support for subsequent decision-making. This entire process realizes a closed loop from data collection, model reasoning to result feedback, allowing drones to perform violation detection tasks efficiently and accurately.

[0160] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0161] Obviously, those skilled in the art can make various changes and modifications to the embodiments of the present invention without departing from the spirit and scope of the embodiments of the present invention. Thus, if these modifications and variations of the embodiments of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for identifying violations of high-altitude power work, characterized in that: The method includes: Collect high-definition images and low-quality images of the high-altitude power operation site through a drone, and generate a high-definition data set of the power site based on the high-definition images and the low-quality images. The high-definition data set of the power site is obtained by annotating the high-definition images with targets to obtain a first label file corresponding to the image; The low-quality image is processed by using a first processing model and a second processing model. The first processing method corresponding to the first processing model includes: introducing a grouping branch in the first three convolutional layers in the dynamic contrast enhancement network structure, and processing the original low-quality image by the first processing method to finally obtain a corresponding first feature map; the second processing method corresponding to the second processing model includes: firstly performing a convolution operation on the original low-quality image, then extracting image features through a multi-head attention module, and finally, respectively generating a corresponding adjustment matrix and transformation value through multi-layer perception and linear transformation, wherein the adjustment matrix is ​​used to perform color correction on the image features, and the transformation value is used to adjust the illumination of the image features, thereby obtaining a corresponding second feature map, and fusing the first feature map and the second feature map to obtain a final optimized image; Annotate the optimized image to obtain a corresponding second label file, obtain an optimized data set based on the second label file combined with the corresponding image, and use the optimized data set to train the first processing model and the second processing model until a final training result is achieved; Combining the power field high-definition data set and the optimized data set to form a final data set, and formally dividing the final data set into a training set, a validation set, and a test set; Constructing a violation recognition model based on improved YOLOv8, wherein the improved YOLOv8 violation recognition model comprises: replacing the backbone network in the YOLOv8 network structure with a Transformer model, replacing the C2f module in the YOLOv8 network structure with a C2f-CBDC module, and finally, the model outputs the result and label data corresponding to the violation recognition; Using the training set to train the improved YOLOv8 violation recognition model until convergence; Use the test set to evaluate the trained model, judge the effect based on the overall accuracy, and output the final model when the accuracy is stable and improved; if the expected accuracy is not achieved, retrain; Convert the optimal model after training and optimization into the front-end RKNN model and deploy it to the front-end of the drone; Rely on the front-end camera to achieve real-time detection, and send the detection results to the analysis unit to determine whether the relevant operators have violated regulations, and save the results.

2. The method for identifying violations of high-altitude power work according to claim 1, characterized in that: The dynamic contrast enhancement network structure includes 7 convolutional layers, and channel separation operations are implemented in the first 3 convolutional layers, grouping branches are added, and in each grouping branch, 1×1 convolution is used for dimensionality increase, and the number of feature channels of the first 3 convolutional layers is halved, and restored to the original number of channels after splicing with the corresponding grouping branches; and the first 6 convolutional layers use ReLU activation function, and the 7th convolutional layer uses Tanh activation function.

3. The method for identifying violations of high-altitude power work according to claim 2, characterized in that: The dynamic contrast enhancement network structure further includes an illumination adjustment layer, which is used to adjust the illumination range of the image features output from the convolution layer, and then output the first feature map. The illumination adjustment layer represents the relationship between different features based on an estimation formula, and the estimation formula is expressed as: Among them, I represents the current image feature; a is the set threshold parameter, which is the same as the input image feature dimension. Based on this formula, all pixels in the image are normalized, and the graph formed by the mapping relationship is used as the corresponding color channel to achieve the range adjustment of the illumination.

4. The method for identifying violations of high-altitude power work according to claim 1 is characterized in that: The second processing model includes: a convolution module, an encoding module, a first-layer normalization module, a multi-head attention module, a second-layer normalization module, a multi-layer perception module and a linear transformation module. The original low-quality image is input into the convolution module. The convolution module includes a 3×3 convolution layer to obtain a first feature map with 64 channels. After the dimensionality conversion of the encoding module, the first feature map is sequentially input into the first-layer normalization module and the multi-head attention module, and the corresponding second feature is extracted through the module. The second feature passes through the second normalization module and then the multi-layer perception module to generate a 3×3 adjustment matrix, and is linearly transformed with the second feature output by the multi-head attention module to obtain a 1×1 transformation value.

5. The method for identifying violations of high-altitude power work according to claim 4 is characterized in that: The fusing of the first feature map and the second feature map to obtain a final optimized image specifically includes: The second feature map is input into a convolutional layer with a size of 1×1 and a channel number of 1 to obtain a third feature, the third feature is multiplied by the first feature to obtain a fourth feature, the fourth feature is exponentially calculated with the third feature, and a final optimized image is output.

6. The method for identifying violations of high-altitude power work according to claim 1, characterized in that: The use of the Transformer model to replace the backbone network in the YOLOv8 network structure includes: using the EfficientFormerV2 model in the Transformer model as the backbone network of the model.

7. The method for identifying violations of high-altitude power work according to claim 6 is characterized in that: The C2f-CBDC module is used to replace the C2f module in the YOLOv8 network structure, including: Block1, Block2 and Block3 in the EfficientFormerV2 model are all connected to a convolutional layer of the Neck part of the YOLOv8 network model, and the fast spatial pyramid pooling layer SPPF is connected to a convolutional layer of the Neck part of the YOLOv8 network model. The C2f module of the Neck part of the YOLOv8 network model is replaced by a C2f-CBDC module. The C2f-CBDC module is a C2f module with a CBDC attention module added to the C2f module. The CBDC attention module integrates the CBAM attention module and deformable convolution. The CBAM attention module combines channel attention and spatial attention to refine feature maps in two different dimensions.

8. The method for identifying violations of high-altitude power work according to claim 7, characterized in that: The CBDC attention module combines the CBAM attention module and deformable convolution, specifically including: Through the channel attention module CAM, the input feature map is subjected to maximum pooling and average pooling operations to compress the information of each channel and generate two weight vectors. These two weight vectors are mapped through a shared network to obtain a weight matrix for each channel and multiplied element by element with the original input feature map, thereby enhancing the feature response of important channels and suppressing redundant or irrelevant channel information. The spatial attention module SAM focuses on the features of the spatial dimension based on the output of the channel attention module. In this module, the maximum pooling and average pooling operations are applied to all channels at each position respectively to generate two two-dimensional feature maps. Then, after the deformable convolution operation, the channel information is fused, and the spatial attention matrix is ​​multiplied element by element with the output feature map of the channel attention module.

9. The method for identifying violations of high-altitude power work according to claim 1, characterized in that: The optimal model after training and optimization is converted into a front-end RKNN model and deployed to the front-end of the drone, including: Use Python scripts to convert the trained image enhancement model and object detection model into a model format suitable for the RKNN framework. The conversion process includes the reconstruction of the model structure and parameters to ensure that the model can run efficiently in the RKNN engine. After the conversion is completed, a series of optimization operations are performed on the converted RKNN model, including adaptive hybrid quantization, which aims to improve its execution efficiency on embedded platforms by reducing the model's computational workload and storage usage while maintaining its accuracy level as much as possible; Through multiple experiments, we verified the impact of different quantization methods on model performance and conducted a detailed analysis of the detection accuracy of each category to ensure that the model can still maintain a high recognition accuracy in different scenarios; The optimized RKNN model is deployed to the front end of the drone. During the deployment process, the model will be loaded into the hardware platform on the drone and the inference process will be accelerated through the integrated neural network processing unit NPU.

10. The method for identifying violations of high-altitude power work according to claim 1, characterized in that: The front-end camera is used to realize real-time detection, and the detection results are sent to the judgment unit to judge whether the relevant operators have violated the rules and regulations, and the results are saved, which specifically includes: The inspection drone starts reading the real-time video stream and performs uninterrupted frame sampling detection; The drone relies on the violation recognition algorithm in the RKNN model to detect the construction tower and guide the drone to fly towards the tower; After flying above the construction tower, the drone begins to fly around the tower based on the preset longitude and latitude. When a worker is detected on the construction tower, the analysis and judgment unit determines whether the worker is working in accordance with the regulations and whether the worker pixel frame is equipped with the prescribed safety measures based on the detection results; Once a violation is identified, the detection and analysis results will be saved and the data will be transmitted back to the backend system, which will further analyze and process the data.

Images