Anomaly detection method and device of transmission tower, electronic equipment and storage medium

Abstract

According to the anomaly detection method and device of the transmission tower, the electronic equipment and the storage medium provided by the embodiment of the invention, the to-be-detected image corresponding to the to-be-detected transmission tower is firstly acquired, and then the to-be-detected image is input to the pre-trained detection model to obtain the target semantic segmentation result; wherein the detection model is obtained by training a double-branch semantic segmentation network model with a semantic branch, a spatial branch and a bidirectional attention fusion module based on training data. According to the technical scheme, the acquired to-be-detected transmission tower image is input into the pre-trained detection model, the target semantic segmentation result is obtained, the abnormity of the transmission tower is automatically detected, complementation and enhancement of semantic features and spatial features of the transmission tower image are realized by using the bidirectional attention fusion module, and the detection accuracy is improved. Therefore, the efficiency and accuracy of transmission tower anomaly detection are improved.

Description

Technical Field

[0001] The present application relates to the technical field of power transmission, and in particular, to an abnormal detection method, device, electronic device, and storage medium for a transmission tower. Background Art

[0002] In the field of power transmission, transmission towers are key structures that support electric wires and ensure power transmission. Their integrity is crucial for the safety and reliability of the power system.

[0003] In the prior art, manual inspection is used to detect defects in transmission towers. However, the method of manual inspection has technical problems such as low detection efficiency, high cost, and safety risks. Summary of the Invention

[0004] Embodiments of the present application provide an abnormal detection method, device, electronic device, and storage medium for a transmission tower to solve the technical problems of low detection efficiency, high cost, and safety risks in the prior art for detecting defects in transmission towers.

[0005] In a first aspect, embodiments of the present application provide an abnormal detection method for a transmission tower, including:

[0006] Obtaining a to-be-detected image corresponding to the to-be-detected transmission tower;

[0007] Inputting the to-be-detected image into a pre-trained detection model to obtain a target semantic segmentation result, where the target semantic segmentation result is the damage information of each component in the to-be-detected transmission tower. The detection model is obtained by training a dual-branch semantic segmentation network model based on training data. The training data includes: a plurality of first images corresponding to the transmission tower and annotation information corresponding to the plurality of first images. The dual-branch semantic segmentation network model includes: a plurality of first basic modules in the semantic branch and a plurality of second basic modules in the spatial branch, and a bidirectional attention fusion module is arranged between every two adjacent first basic modules and every two adjacent second basic modules.

[0008] In a possible implementation manner, before inputting the to-be-detected image into the pre-trained detection model to obtain the target semantic segmentation result, the method further includes:

[0009] Obtaining a plurality of first images corresponding to the transmission tower and annotation information corresponding to the plurality of first images;

[0010] Training the dual-branch semantic segmentation network model according to the plurality of first images and the annotation information corresponding to the plurality of first images to obtain the detection model.

[0011] In a possible implementation manner, training the dual-branch semantic segmentation network model according to the multiple first images and the annotation information corresponding to the multiple first images to obtain the detection model includes:

[0012] For each first image, input the first image into the backbone network in the dual-branch semantic segmentation network model to obtain a second image after downsampling processing;

[0013] Input the second image into the semantic branch and the spatial branch respectively, and perform fusion processing of semantic features and spatial features through the bidirectional attention fusion module to obtain a third image corresponding to the semantic branch and a fourth image corresponding to the spatial branch respectively;

[0014] Input the third image into the pyramid pooling module in the dual-branch semantic segmentation network model to obtain a first semantic feature containing multi-scale information;

[0015] Input the fourth image and the first semantic feature into the feature fusion module in the dual-branch semantic segmentation network model to obtain first fusion information;

[0016] According to the semantic segmentation result corresponding to the first fusion information corresponding to each first image and the annotation information corresponding to each first image, adjust the parameters in the dual-branch semantic segmentation network model, and use the adjusted dual-branch semantic segmentation network model as the detection model.

[0017] In a possible implementation manner, the step of inputting the second image into the semantic branch and the spatial branch respectively, and performing fusion processing of semantic features and spatial features through the bidirectional attention fusion module to obtain a third image corresponding to the semantic branch and a fourth image corresponding to the spatial branch respectively includes:

[0018] According to the second image, the first first basic module of the semantic branch, the first second basic module of the spatial branch, and the first bidirectional attention fusion module, determine the second semantic feature and the first spatial feature corresponding to the second image;

[0019] According to the second semantic feature, the second first basic module of the semantic branch, the first spatial feature, the second second basic module of the spatial branch, and the second bidirectional attention fusion module, determine the third semantic feature and the second spatial feature;

[0020] According to the third semantic feature, the third first basic module, the second spatial feature, and the third second basic module, determine the third image corresponding to the semantic branch and the fourth image corresponding to the spatial branch.

[0021] In a possible implementation, determining the second semantic feature and the first spatial feature corresponding to the second image according to the second image, the first first basic module of the semantic branch, the first second basic module of the spatial branch, and the first bidirectional attention fusion module includes:

[0022] Input the second image into the first first basic module of the semantic branch to obtain a fifth image after convolution and downsampling processing;

[0023] Input the second image into the first second basic module of the spatial branch to obtain a sixth image after convolution processing;

[0024] Input the fifth image and the sixth image into the first bidirectional attention fusion module to obtain the second semantic feature corresponding to the fused fifth image and the first spatial feature corresponding to the sixth image.

[0025] In a possible implementation, determining the third semantic feature and the second spatial feature according to the second semantic feature, the second first basic module of the semantic branch, the first spatial feature, the second second basic module of the spatial branch, and the second bidirectional attention fusion module includes:

[0026] Input the second semantic feature and the fifth image into the second first basic module of the semantic branch to obtain a seventh image after convolution and downsampling processing;

[0027] Input the first spatial feature and the sixth image into the second second basic module of the spatial branch to obtain an eighth image after convolution processing;

[0028] Input the seventh image and the eighth image into the second bidirectional attention fusion module to obtain the third semantic feature corresponding to the fused seventh image and the second spatial feature corresponding to the eighth image.

[0029] In a possible implementation, determining the third image corresponding to the semantic branch and the fourth image corresponding to the spatial branch according to the third semantic feature, the third first basic module, the second spatial feature, and the third second basic module includes:

[0030] Input the third semantic feature and the seventh image into the third first basic module of the semantic branch to obtain a third image after convolution and downsampling processing;

[0031] Input the second spatial feature and the eighth image into the third second basic module of the spatial branch to obtain a fourth image after convolution processing.

[0032] In a second aspect, an abnormal detection device for a transmission tower provided by an embodiment of the present application includes:

[0033] An acquisition module, configured to acquire a to-be-detected image corresponding to a to-be-detected transmission tower;

[0034] A determination module, configured to input the to-be-detected image into a pre-trained detection model to obtain a target semantic segmentation result, where the target semantic segmentation result is damage information of each component in the to-be-detected transmission tower, and the detection model is obtained by training a dual-branch semantic segmentation network model based on training data, and the training data includes: a plurality of first images corresponding to the transmission tower and annotation information corresponding to the plurality of first images. The dual-branch semantic segmentation network model includes: a plurality of first basic modules in a semantic branch and a plurality of second basic modules in a spatial branch, and a bidirectional attention fusion module is arranged between every two adjacent first basic modules and every two adjacent second basic modules.

[0035] In a possible implementation manner, before inputting the to-be-detected image into the pre-trained detection model to obtain the target semantic segmentation result, the determination module is further configured to:

[0036] Acquire a plurality of first images corresponding to the transmission tower and annotation information corresponding to the plurality of first images;

[0037] Train the dual-branch semantic segmentation network model according to the plurality of first images and the annotation information corresponding to the plurality of first images to obtain the detection model.

[0038] In a possible implementation manner, for training the dual-branch semantic segmentation network model according to the plurality of first images and the annotation information corresponding to the plurality of first images to obtain the detection model, the determination module specifically is configured to:

[0039] For each first image, input the first image into a backbone network in the dual-branch semantic segmentation network model to obtain a second image after downsampling processing;

[0040] Input the second image into the semantic branch and the spatial branch respectively, and perform fusion processing of semantic features and spatial features through the bidirectional attention fusion module to obtain a third image corresponding to the semantic branch and a fourth image corresponding to the spatial branch respectively;

[0041] Input the third image into a pyramid pooling module in the dual-branch semantic segmentation network model to obtain a first semantic feature containing multi-scale information;

[0042] Input the fourth image and the first semantic feature into the feature fusion module in the dual-branch semantic segmentation network model to obtain first fusion information;

[0043] According to the semantic segmentation results corresponding to the first fusion information of each first image and the annotation information corresponding to each first image, adjust the parameters in the dual-branch semantic segmentation network model, and use the adjusted dual-branch semantic segmentation network model as the detection model.

[0044] In a possible implementation manner, for the step of inputting the second image into the semantic branch and the spatial branch respectively, and performing fusion processing on the semantic feature and the spatial feature through the bidirectional attention fusion module to obtain the third image corresponding to the semantic branch and the fourth image corresponding to the spatial branch respectively, the determination module is specifically configured to:

[0045] Determine the second semantic feature and the first spatial feature corresponding to the second image according to the second image, the first first basic module of the semantic branch, the first second basic module of the spatial branch, and the first bidirectional attention fusion module;

[0046] Determine the third semantic feature and the second spatial feature according to the second semantic feature, the second first basic module of the semantic branch, the first spatial feature, the second second basic module of the spatial branch, and the second bidirectional attention fusion module;

[0047] Determine the third image corresponding to the semantic branch and the fourth image corresponding to the spatial branch according to the third semantic feature, the third first basic module, the second spatial feature, and the third second basic module.

[0048] In a possible implementation manner, for the step of determining the second semantic feature and the first spatial feature corresponding to the second image according to the second image, the first first basic module of the semantic branch, the first second basic module of the spatial branch, and the first bidirectional attention fusion module, the determination module is specifically configured to:

[0049] Input the second image into the first first basic module of the semantic branch to obtain a fifth image after convolution and downsampling processing;

[0050] Input the second image into the first second basic module of the spatial branch to obtain a sixth image after convolution processing;

[0051] Input the fifth image and the sixth image into the first bidirectional attention fusion module to obtain the second semantic feature corresponding to the fused fifth image and the first spatial feature corresponding to the sixth image.

[0052] In a possible implementation, determining the third semantic feature and the second spatial feature according to the second semantic feature, the second first basic module of the semantic branch, the first spatial feature, the second second basic module of the spatial branch, and the second bidirectional attention fusion module, the determining module is specifically configured to:

[0053] Input the second semantic feature and the fifth image into the second first basic module of the semantic branch to obtain a seventh image after convolution and downsampling processing;

[0054] Input the first spatial feature and the sixth image into the second second basic module of the spatial branch to obtain an eighth image after convolution processing;

[0055] Input the seventh image and the eighth image into the second bidirectional attention fusion module to obtain the third semantic feature corresponding to the fused seventh image and the second spatial feature corresponding to the eighth image.

[0056] In a possible implementation, determining the third image corresponding to the semantic branch and the fourth image corresponding to the spatial branch according to the third semantic feature, the third first basic module, the second spatial feature, and the third second basic module, the determining module is specifically configured to:

[0057] Input the third semantic feature and the seventh image into the third first basic module of the semantic branch to obtain a third image after convolution and downsampling processing;

[0058] Input the second spatial feature and the eighth image into the third second basic module of the spatial branch to obtain a fourth image after convolution processing.

[0059] In a third aspect, an embodiment of the present application provides an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0060] The memory stores computer-executable instructions;

[0061] The processor executes the computer-executable instructions stored in the memory to implement the method described in the first aspect or any one of the above manners.

[0062] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored, and when the computer-executable instructions are executed by a processor, they are used to implement the method described in the first aspect or any one of the above manners

[0063] Fifth aspect, an embodiment of the present application provides a computer program. The computer program product includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium. When the at least one processor executes the computer program, the method described in the above first aspect or any one of the ways can be implemented.

[0064] The abnormal detection method, device, electronic device and storage medium for transmission towers provided by the embodiments of the present application. The method first obtains a to-be-detected image corresponding to a to-be-detected transmission tower, and then inputs the to-be-detected image into a pre-trained detection model to obtain a target semantic segmentation result, where the target semantic segmentation result is the damage information of each component in the to-be-detected transmission tower. The detection model is trained based on training data for a dual-branch semantic segmentation network model. The training data includes: a plurality of first images corresponding to the transmission tower and annotation information corresponding to the plurality of first images. The dual-branch semantic segmentation network model includes: a plurality of first basic modules in the semantic branch and a plurality of second basic modules in the spatial branch. A bidirectional attention fusion module is arranged between every two adjacent first basic modules and every two adjacent second basic modules. This technical solution inputs the obtained to-be-detected image of the transmission tower into a pre-trained detection model with a semantic branch, a spatial branch and a bidirectional attention fusion module to obtain a target semantic segmentation result, realizing the automatic detection of the abnormality of the transmission tower, reducing the cost and danger of the abnormal detection of the transmission tower, and using the bidirectional attention fusion module to achieve the complementarity and enhancement of the semantic features and spatial features of the transmission tower image, thereby improving the efficiency and accuracy of the abnormal detection of the transmission tower. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] The accompanying drawings herein are incorporated into the specification and constitute a part of this specification, showing embodiments consistent with the present application, and are used together with the specification to explain the principles of the present application.

[0066] Figure 1 It is a schematic framework diagram of the dual-branch semantic segmentation network model provided by the embodiments of the present application;

[0067] Figure 2 It is a schematic flowchart of the abnormal detection method for transmission towers provided by the embodiments of the present application Figure 1 ;

[0068] Figure 3 It is a schematic flowchart of the abnormal detection method for transmission towers provided by the embodiments of the present application Figure 2 ;

[0069] Figure 4 It is a schematic flowchart of the abnormal detection method for transmission towers provided by the embodiments of the present application Figure 3 ;

[0070] Figure 5 It is a schematic structural diagram of the bilateral attention fusion module in the detection model provided by the embodiments of the present application;

[0071] Figure 6 It is a schematic structural diagram of the pyramid pooling module in the detection model provided by the embodiments of the present application;

[0072] Figure 7 It is a schematic structural diagram of the feature fusion module in the detection model provided by the embodiments of the present application;

[0073] Figure 8 It is a schematic structural diagram of the abnormal detection device for transmission towers provided by the embodiments of the present application;

[0074] Figure 9 It is a schematic structural diagram of the electronic device provided by the embodiments of the present application.

[0075] Through the above-mentioned drawings, the specific embodiments of the present application have been shown, and there will be more detailed descriptions hereinafter. These drawings and text descriptions are not intended to limit the scope of the concept of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. Specific Embodiments

[0076] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Apparently, the described embodiments are some, but not all, of the embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without making creative efforts shall fall within the protection scope of the present application.

[0077] First, the terms in the embodiments of the present application are explained:

[0078] Bilateral Attention Fusion Module (BAFM): It is a module used in deep learning networks, aiming to improve the performance of the model by combining bilateral (or dual-channel) attention mechanisms.

[0079] Pyramid Pooling Module (PPM): It is a module in the field of computer vision, especially used in image segmentation tasks. Its main purpose is to capture context information at different scales in the image through multi-scale pooling operations, thereby helping the model better understand the global structure and local details of the image.

[0080] Seg Head: The module responsible for generating the final segmentation result, usually at the last layer of the neural network or near the output.

[0081] Basic Block: The basic module or unit that constitutes the neural network. Each basic block usually contains one or more operations among convolution, batch normalization, and activation functions. These operations work together to achieve a specific function and can be repeatedly stacked to build a more complex network.

[0082] Before introducing the embodiments of the present application, first, the application background of the embodiments of the present application is explained:

[0083] In the field of power transmission, transmission towers are key structures that support electric wires and ensure power transmission. Their integrity is crucial for the safety and reliability of the power system.

[0084] With the growth of the scale and the increase in the complexity of the power network, when the prior art uses manual inspection methods to detect defects in transmission towers, there are technical problems such as low detection efficiency, high cost, and safety risks.

[0085] In recent years, deep learning technology has made great breakthroughs in the field of computer vision, especially in the sub-field of semantic segmentation. Semantic segmentation methods can not only accurately classify each pixel in an image but also identify the contours and components of objects in complex environments. Through end-to-end training, deep neural networks have demonstrated powerful performance in image recognition and object detection tasks. Compared with early technologies, deep learning models are more advanced in feature extraction and can better handle challenges such as lighting changes and background noise.

[0086] In the application of using drones to inspect transmission towers, the application of deep learning technology has opened up a new way to solve existing problems. By using a semantic segmentation model, numerous structures and components on the tower can be accurately identified, including devices such as insulators, cross arms, bolts, etc., and even tiny defects and damages can be detected under complex environmental conditions. In addition, deep learning models can automatically learn from a large amount of inspection data and optimize themselves, which can improve the accuracy and efficiency of anomaly detection for transmission towers.

[0087] In view of the technical problems existing in the prior art, the inventors of the present application have the following ideas. Due to the technical problems of low detection efficiency, high cost and safety risks in manual inspection, an unmanned aerial vehicle (UAV) is used to obtain the image to be detected corresponding to the transmission tower to be detected, and then the image to be detected is input into a pre-trained detection model for processing to obtain the target semantic segmentation result, so that the abnormal information of the transmission tower in the image to be detected can be determined. The above detection model is established based on a dual-branch semantic segmentation network model. The dual-branch semantic segmentation network model includes a semantic branch, a spatial branch and a bidirectional attention fusion module arranged in the middle of the basic modules of the semantic branch and the spatial branch. The semantic branch, the spatial branch and the bidirectional attention fusion module are used to strengthen the information exchange between the two branches, realize the complementarity and enhancement of the image semantic features and spatial features, so that the category features and abnormal features of each component of the transmission tower in the image to be detected can be obtained more efficiently, effectively improving the detection efficiency of the transmission tower and reducing the cost and risk of abnormal detection of the transmission tower.

[0088] First, a brief description of the dual-branch semantic segmentation network model is given. Figure 1 The framework diagram of the dual-branch semantic segmentation network model provided by the embodiment of the present application is shown in Figure 1 As shown, the dual-branch semantic segmentation network model includes: a backbone network, a semantic branch module, a spatial branch module, BAFM, PPM, a feature fusion module and a segmentation head for result output.

[0089] Among them, the backbone network is the basis of the entire segmentation model and is responsible for extracting multi-level features from the input image. The semantic branch module is mainly responsible for extracting and focusing on the semantic information of the image. This module usually focuses on the global semantic context features to capture the overall semantic category information, such as the component category and damage information of the transmission tower. The spatial branch module focuses on capturing the spatial structure information in the image. It processes the spatial details of the image, such as the edges, shapes and positions of objects. BAFM is used for information fusion between the semantic branch and the spatial branch. By introducing a bidirectional attention mechanism, BAFM can adaptively perform weighted fusion between semantic information and spatial information, effectively improving the feature expression ability of the network, enabling different sources of information (such as space and semantics) to complement each other and optimizing the final segmentation result. PPM captures multi-scale context information by performing pooling operations on the feature map at different scales, which can help the network understand objects of different sizes and adaptively adjust the segmentation boundary in the image. The feature fusion module fuses the features from different modules (such as the semantic branch, the spatial branch, BAFM and PPM) to obtain a more comprehensive feature description, thereby improving the segmentation accuracy. The segmentation head is the last layer of the network and is used to map the fused feature map to the category label of each pixel to generate the final segmentation result.

[0090] Through the collaborative work of the above modules, the entire network architecture of the dual-branch semantic segmentation network model provided by the embodiments of this application can simultaneously capture semantic information and spatial information, perform multi-scale processing, fuse image semantic features and spatial features, and finally efficiently output accurate segmentation results.

[0091] Next, the technical solutions of this application will be described in detail through specific embodiments. It should be noted that the following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0092] Among them, the execution subject of this application can be an electronic device such as a drone, a background server, or a terminal device.

[0093] Figure 2 Flow schematic of the abnormal detection method for transmission towers provided by the embodiments of this application Figure 1 , such as Figure 2 shown, this method may include the following steps:

[0094] Step 21, obtain a to-be-detected image corresponding to a to-be-detected transmission tower.

[0095] In this step, a sensor device is used to obtain a to-be-detected image corresponding to a to-be-detected transmission tower.

[0096] Among them, the sensor device can be a high-definition camera or a drone equipped with a high-definition camera, etc.

[0097] In a possible implementation, the to-be-detected image may include image information of transmission towers under different angles and different lighting conditions obtained in real time or in advance.

[0098] Optionally, after obtaining the to-be-detected image, preprocessing (such as denoising, adjusting brightness, coordinate transformation, and normalization, etc.) can be performed on the to-be-detected image, so as to improve the image quality of the to-be-detected image and provide a better-quality image input for subsequent abnormal detection of transmission towers.

[0099] Step 22, input the to-be-detected image into a pre-trained detection model to obtain a target semantic segmentation result.

[0100] Among them, the target semantic segmentation result is the damage information of each component in the transmission tower to be detected. The detection model is obtained by training a dual-branch semantic segmentation network model based on training data, and the training data includes: a plurality of first images corresponding to the transmission tower and annotation information corresponding to the plurality of first images. The dual-branch semantic segmentation network model includes: a plurality of first basic modules in the semantic branch and a plurality of second basic modules in the spatial branch. A bidirectional attention fusion module is arranged between every two adjacent first basic modules and every two adjacent second basic modules.

[0101] In this step, the preprocessed image to be detected is input into a pre-trained detection model. After a series of convolution and feature fusion operations, the target semantic segmentation result of the image to be detected is output.

[0102] Among them, the target semantic segmentation result is the class label of each pixel point in the image to be detected and the corresponding damage information. The class label of each pixel point above is obtained by assigning each pixel in the input image to a specific class in the semantic segmentation task (such as components such as the tower body, tower base, wire suspension device, crossbeam, etc. in the transmission tower). Through the target semantic segmentation result, the damage conditions of each component (such as the above-mentioned tower body, tower base, etc.) on the transmission tower in the image can be identified.

[0103] The abnormal detection method for a transmission tower provided in an embodiment of the present application first obtains an image to be detected corresponding to the transmission tower to be detected, and then inputs the image to be detected into a pre-trained detection model to obtain a target semantic segmentation result. Among them, the target semantic segmentation result is the damage information of each component in the transmission tower to be detected. The detection model is obtained by training a dual-branch semantic segmentation network model based on training data, and the training data includes: a plurality of first images corresponding to the transmission tower and annotation information corresponding to the plurality of first images. The dual-branch semantic segmentation network model includes: a plurality of first basic modules in the semantic branch and a plurality of second basic modules in the spatial branch. A bidirectional attention fusion module is arranged between every two adjacent first basic modules and every two adjacent second basic modules. Through this technical solution, by inputting the obtained image of the transmission tower to be detected into a pre-trained detection model with a semantic branch, a spatial branch, and a bidirectional attention fusion module, a target semantic segmentation result is obtained, realizing the automatic detection of abnormalities in the transmission tower, reducing the cost and danger of abnormal detection of the transmission tower, and using the bidirectional attention fusion module to realize the complementarity and enhancement of the semantic features and spatial features of the transmission tower image. Therefore, the class features and abnormal features of each component of the transmission tower in the image to be detected can be obtained more efficiently, improving the efficiency and accuracy of abnormal detection of the transmission tower.

[0104] On the basis of the above embodiment, Figure 3Flow schematic of the abnormal detection method for transmission towers provided by the embodiments of the present application Figure 2 , as Figure 3 shown, before step 22, the method further includes the following steps:

[0105] Step 31: Obtain a plurality of first images corresponding to the transmission tower and annotation information corresponding to the plurality of first images;

[0106] In this step, a large number of transmission tower images are collected by using sensor devices to obtain the images corresponding to the transmission tower (i.e., the first images) and the annotation information corresponding to the transmission tower images.

[0107] Among them, the first images are various images of transmission towers obtained in advance in a large number of different environments and perspectives.

[0108] In addition, for each transmission tower image, the corresponding annotation information also needs to be obtained, that is, the category of each component in each transmission tower image and whether each component is damaged (such as broken, aged, etc.). The annotation information can be obtained by manual annotation or completed through some auxiliary tools to ensure that the category annotation of each pixel is accurate.

[0109] After obtaining the images corresponding to the transmission tower, preprocess the plurality of first images (for example, operations such as denoising, brightness adjustment, coordinate transformation, and normalization) to obtain better-quality first images.

[0110] Step 32: Train a dual-branch semantic segmentation network model according to the plurality of first images and the annotation information corresponding to the plurality of first images to obtain a detection model.

[0111] In this step, the obtained plurality of first images and the corresponding annotation information are input into the dual-branch semantic segmentation model for training. The dual-branch semantic model will learn the features and semantic information of the first images, adjust the parameters according to the pixel features of the first images and the labeled category information, and optimize the semantic segmentation performance so that it can effectively identify different components in the image and determine whether they are abnormal. After training, the model will have the ability to automatically analyze the image to be detected and output the semantic segmentation result, that is, obtain the detection model.

[0112] Among them, the plurality of first images can be a plurality of images after preprocessing.

[0113] The abnormal detection method for transmission towers provided by the embodiments of the present application first obtains a plurality of first images corresponding to the transmission towers and the annotation information corresponding to the plurality of first images, and then trains a dual-branch semantic segmentation network model according to the plurality of first images and the annotation information corresponding to the plurality of first images to obtain a detection model. This technical solution uses a plurality of first images corresponding to the transmission towers and the corresponding annotation information to provide comprehensive image information and image category features, improving the generalization ability of the detection model. Then, the plurality of first images and the corresponding annotation information are input into the dual-branch semantic segmentation model for training, fusing the semantic features and spatial features of the transmission towers in each image, enabling the model to accurately learn the category information and damage conditions of each component of the transmission towers in each image. The trained detection model can achieve the technical effect of quickly and efficiently detecting abnormalities in transmission towers.

[0114] Based on the above embodiments, Figure 4 is a flowchart of the abnormal detection method for transmission towers provided by the embodiments of the present application Figure 3 , as Figure 4 shown, the above step 32 may include the following steps:

[0115] Step 41: For each first image, input the first image into the backbone network of the dual-branch semantic segmentation network model to obtain a second image after downsampling processing;

[0116] In this step, each first image is input into the backbone network of the dual-branch semantic segmentation network, and each first image is downsampled to obtain a second image.

[0117] In a possible implementation, as Figure 1 shown, the backbone network is optimized based on the ResNet-18 network model. The initial module (English: Stem block) of the original ResNet-18 is improved to obtain the backbone network. The specific improvement method is to replace the 7x7 convolutional layer with two 3x3 convolutional layers with a stride of 2, which can downsample the first image to 1 / 8 of the resolution of the first image. The above improvement method not only reduces the number of parameters, but also helps to extract the features of the first image while enhancing the non-linear ability of the backbone network, thereby improving the processing efficiency of image features.

[0118] Step 42: Input the second image into the semantic branch and the spatial branch respectively, and perform fusion processing of semantic features and spatial features through a bidirectional attention fusion module to obtain a third image corresponding to the semantic branch and a fourth image corresponding to the spatial branch respectively;

[0119] In this step, the second image is input into the semantic branch and the spatial branch respectively. The semantic branch is mainly responsible for capturing the semantic features of the second image, and the spatial branch focuses on capturing the spatial layout or structural feature information in the second image. Then, the semantic features and spatial feature information are fused through a bidirectional attention fusion module to further optimize the complementarity of the semantic features and spatial features, and the third image corresponding to the fused semantic branch and the fourth image corresponding to the fused spatial branch are obtained respectively.

[0120] Optionally, step 42 is implemented as follows:

[0121] Step 1: Determine the second semantic feature and the first spatial feature corresponding to the second image according to the second image, the first first basic module of the semantic branch, the first second basic module of the spatial branch, and the first bidirectional attention fusion module;

[0122] In this step, the second image is input into the first first basic module of the semantic branch and the second second basic module of the spatial branch respectively, and then the processed second image feature information is input into the first bidirectional attention fusion module, so as to determine the second semantic feature and the first spatial feature corresponding to the second image.

[0123] In a possible implementation, as Figure 1 shown, the resolution of the second image processed by the basic module in the backbone network is 1 / 8 of the resolution of the first image. On the one hand, the basic module in the backbone network inputs the second image into the first first basic module in the semantic branch for processing, and the resolution of the obtained image is 1 / 16 of the first image. On the other hand, the basic module in the backbone network inputs the second image into the first second basic module in the spatial branch for processing, and the resolution of the obtained image is 1 / 8 of the first image. Then, the images processed by the semantic branch and the spatial branch are input into the BAFM for feature fusion to obtain the fused second semantic feature and the first spatial feature corresponding to the second image.

[0124] Among them, the semantic branch processes low-resolution images, the spatial branch processes high-resolution images, and information fusion is performed through the BAFM, which helps to achieve the complementarity and enhancement of the second image features.

[0125] Optionally, this step 1 is implemented as follows:

[0126] S1: Input the second image into the first first basic module of the semantic branch to obtain the fifth image after convolution and downsampling;

[0127] In this implementation, the second image is input into the first first basic module of the semantic branch to perform convolution and downsampling operations to obtain deep semantic information, and the fifth image is obtained after processing.

[0128] In a possible implementation, as Figure 1 shown, the image resolution of the fifth image obtained after the second image is processed by the first first basic module of the semantic branch is 1 / 16 of that of the first image.

[0129] S2. Input the second image into the first second basic module of the spatial branch to obtain a sixth image after convolution processing;

[0130] In this implementation, input the second image into the first second basic module of the spatial branch for convolution processing to obtain the processed sixth image.

[0131] In a possible implementation, the convolution of the first second basic module is a 3×3 convolution with a stride of 1, which retains the feature map with 1 / 8 resolution of the first image to supplement the spatial features of the semantic branch, and at the same time removes redundant convolution operations, thereby reducing the computational amount and the number of parameters of the network. The resolution of the processed sixth image is 1 / 8 of that of the first image.

[0132] S3. Input the fifth image and the sixth image into the first bidirectional attention fusion module to obtain the second semantic feature corresponding to the fused fifth image and the first spatial feature corresponding to the sixth image.

[0133] In this implementation, both the fifth image and the sixth image contain rich feature information. Since there are differences in the image features of the semantic branch and the spatial branch, the fifth image and the sixth image are input into the first bidirectional attention fusion module for feature fusion processing to obtain the second semantic feature corresponding to the fifth image and the first spatial feature corresponding to the sixth image.

[0134] Among them, the bidirectional attention fusion module is designed based on the attention mechanism. This module aims to strengthen the information exchange between the two branches, realize the complementarity and enhancement of semantic features and spatial features, and contribute to the improvement of the accuracy and robustness of the detection model.

[0135] In a possible implementation, Figure 5 is a schematic structural diagram of the bidirectional attention fusion module in the detection model provided by the embodiments of the present application. As Figure 5 shown, the BAFM includes two main paths: one is a low-resolution - high-resolution path from the semantic branch to the spatial branch, and the other is a high-resolution - low-resolution path from the spatial branch to the semantic branch.

[0136] In the high-resolution - low-resolution path (as Figure 5The dashed lines in ( ) indicate parts. The feature map of the spatial branch contains rich spatial detail information. The number of channels of the feature map of the spatial branch is expanded to the same as that of the semantic branch through downsampling operations, and then the spatial features and semantic features are directly added together for fusion.

[0137] In the low-resolution - high-resolution path (such as Figure 5 the solid lines in ( ) indicate parts), first, the low-resolution feature map of the semantic branch is upsampled by bilinear interpolation. Then, the upsampled feature map and the convolutional feature map of the spatial branch are added together and input into the attention mechanism module to obtain the fused features. Next, the fused features are input into the sigmoid function to obtain the relative attention weight α of the semantic branch, while the relative attention weight of the spatial branch is 1 - α. According to these weights, the feature maps of the two branches are multiplied point by point and then the results are added together. Finally, the result is added to the original image features of the spatial branch to obtain the fused result of the spatial branch.

[0138] The final fusion output of the spatial branch and the semantic branch can be expressed by the formula:

[0139] F′ H = F H +(F H ·(1 - α)+F L ·α)

[0140] F′ L = Downsample(F H )+F L

[0141] where F H and F L are the input features of the spatial branch (such as the features corresponding to the sixth image) and the input features of the semantic branch (such as the features corresponding to the fifth image) respectively, and F′ H and F′ L are the output features of the spatial branch (such as the first spatial feature) and the output features of the semantic branch (such as the second semantic feature) respectively. Downsample(*) represents the downsampling operation.

[0142] Step 2: Determine the third semantic feature and the second spatial feature according to the second semantic feature, the second first basic module of the semantic branch, the first spatial feature, the second second basic module of the spatial branch, and the second bidirectional attention fusion module;

[0143] In this step, the second semantic feature and the first spatial feature are respectively input into the second first basic module and the second second basic module of the semantic branch for processing, and the processed image features are input into the second bidirectional attention fusion module for feature fusion, so as to determine the third semantic feature and the second spatial feature.

[0144] Optionally, the second step may be implemented as follows:

[0145] S1. Input the second semantic feature and the fifth image into the second first basic module of the semantic branch to obtain the seventh image after convolution and downsampling processing;

[0146] In this implementation, input the second semantic feature and the fifth image into the second first basic module of the semantic branch, and perform convolution and downsampling operations in the second first basic module to obtain the seventh image.

[0147] In a possible implementation, as Figure 1 shown, the resolution of the seventh image obtained after being processed by the second first basic module is 1 / 32 of that of the first image.

[0148] S2. Input the first spatial feature and the sixth image into the second second basic module of the spatial branch to obtain the eighth image after convolution processing;

[0149] In this implementation, input the first spatial feature and the sixth image into the second second basic module of the spatial branch, and perform convolution operations in the second second basic module to obtain the eighth image.

[0150] In a possible implementation, as Figure 1 shown, the convolution in the second second basic module is a 3×3 convolution processing with a stride of 1, and the resolution of the eighth image obtained after processing is 1 / 8 of that of the first image.

[0151] S3. Input the seventh image and the eighth image into the second bidirectional attention fusion module to obtain the third semantic feature corresponding to the fused seventh image and the second spatial feature corresponding to the eighth image.

[0152] In this implementation, input the seventh image and the eighth image into the second bidirectional attention fusion module, and perform feature fusion processing according to the feature fusion method of the low-resolution - high-resolution path from the semantic branch to the spatial branch and the high-resolution - low-resolution path from the spatial branch to the semantic branch, to obtain the third semantic feature corresponding to the seventh image after feature fusion and the second spatial feature corresponding to the eighth image.

[0153] Step 3. Determine the third image corresponding to the semantic branch and the fourth image corresponding to the spatial branch according to the third semantic feature, the third first basic module, the second spatial feature, and the third second basic module.

[0154] In this step, the third semantic feature is input into the third first basic module for processing to obtain a third image, and the second spatial feature is input into the third second basic module for processing to obtain a fourth image.

[0155] In a possible implementation, the resolution of the third image is 1 / 64 of the resolution of the first image, and the resolution of the fourth image is 1 / 8 of the resolution of the first image.

[0156] Optionally, the third step can be implemented as follows:

[0157] S1. Input the third semantic feature and the seventh image into the third first basic module of the semantic branch to obtain the third image after convolution and downsampling processing;

[0158] In this implementation, input the third semantic feature and the seventh image into the third first basic module of the semantic branch, and perform convolution and downsampling processing in the first basic module to obtain the third image.

[0159] Among them, the third semantic feature is obtained after being processed by the second bidirectional attention fusion module, and the seventh image is obtained after being processed by the second first basic module of the semantic branch.

[0160] S2. Input the second spatial feature and the eighth image into the third second basic module of the spatial branch to obtain the fourth image after convolution processing.

[0161] In this implementation, input the second spatial feature and the eighth image into the third second basic module of the spatial branch, and perform convolution processing in the second basic module to obtain the fourth image.

[0162] Among them, the second spatial feature is obtained after being processed by the second bidirectional attention fusion module, and the eighth image is obtained after being processed by the second second basic module of the spatial branch.

[0163] Step 43. Input the third image into the pyramid pooling module in the dual-branch semantic segmentation network model to obtain the first semantic feature containing multi-scale information;

[0164] In this step, input the third image into the pyramid pooling module for processing, extract the feature information of different scales in the third image, and obtain the first semantic feature containing multi-scale information.

[0165] Among them, through the multi-level pooling operation of the pyramid pooling module, the semantic information of the image can be extracted from different scales. Through this multi-scale feature extraction, the dual-branch semantic segmentation network model can capture the context information of different scales, which helps to improve the detection ability of the detection model in the complex transmission tower image scene.

[0166] In a possible implementation, Figure 6 is a schematic structural diagram of the pyramid pooling module in the detection model provided by the embodiments of the present application. Combining Figure 1 and Figure 6 , using the feature map with a resolution of 1 / 64 of the first image as the input, first perform channel compression through a 1×1 convolution layer, and then use average pooling with strides of 2, 4, and 8 respectively (corresponding to Figure 6 the AvgPooling1, Avg Pooling2, and Avg Pooling3 modules in in ) to generate feature maps with resolutions of 1 / 128, 1 / 256, and 1 / 512 of the image, and splice the feature map of the third image (i.e., the original image input into the pyramid pooling module) with the global average pooling feature map to achieve the expansion of the size of the feature map of the third image. If the input feature is x in , then each scale pooling feature can be expressed by the following formula:

[0167]

[0168] where i represents different scales, C 1×1 is the 1×1 convolution operation, P k,s (*) is the pooling operation with a pooling kernel size of k and a stride of s, and P global (*) is the global pooling operation.

[0169] Among them, after obtaining the multi-scale feature images through the pooling operation , use bilinear interpolation for upsampling to keep the image size consistent, and then use 3×3 depth convolution for feature extraction, and continuously integrate context information of different scales in a bottom-up step-by-step residual manner. Among them is that the feature map of the third image is not upsampled and directly fused with the upsampled , and then a 3×3 convolution operation is performed. Subsequently, the output of the convolution operation is fused with the feature of the next scale. All output results can be expressed by the following formula:

[0170]

[0171] where C 3×3 is the 3×3 convolution operation, U(*) is the bilinear interpolation upsampling operation, represents addition and fusion. In order to obtain the relationship between each scale and reduce the dimension of the channel, splice each output feature , and then use 1×1 convolution for channel compression. After splicing all the features, and after all the features are spliced, fuse with the output feature of the third image to obtain the final output feature x out, which can be expressed by the formula:

[0172]

[0173] Among them, C 1×1 is a 1×1 convolution, concat(*) is concatenation, indicating the feature addition and fusion operation.

[0174] The above-mentioned final output feature x out is the first semantic feature containing multi-scale information.

[0175] Step 44: Input the fourth image and the first semantic feature into the feature fusion module in the dual-branch semantic segmentation network model to obtain the first fusion information;

[0176] In this step, input the fourth image and the first semantic feature into the feature fusion module, effectively combine the fourth image and the first semantic feature, and obtain the first fusion information of the image corresponding to the fourth image and the first semantic feature, thereby improving the accuracy of the detection model.

[0177] In a possible implementation, Figure 7 is the structural schematic diagram of the feature fusion module in the detection model provided by the embodiments of the present application. As Figure 7 shown, first, use the semantic branch to perform convolution and upsampling operations on the image with a lower resolution, and use the spatial branch to perform two consecutive convolution operations with a stride of 2 on the image with a higher resolution to realize the downsampling of the spatial branch, and also perform the first dot product operation with the image feature after the convolution operation of the semantic branch to realize the interaction of semantic information and spatial information. Then, perform the second dot product operation on the result of the first dot product operation and the image feature after the convolution operation of the semantic branch and then perform downsampling to obtain the downsampling processing result of the semantic branch. At the same time, perform the third dot product operation on the result of the first dot product operation and the image feature after the convolution operation of the spatial branch, and then perform the fourth dot product operation with the downsampling processing result of the semantic branch, and then pass through a layer of convolution, batch normalization, and rectified linear unit (ReLU) activation function to obtain the final fused first fusion information.

[0178] Among them, based on the feature fusion module, the feature fusion of the spatial branch and the semantic branch is realized, the rich spatial information in the high-resolution feature map is fused with the rich semantic information in the low-resolution map, the mutual complementation and enhancement of the image features are realized, and the comprehensive understanding of the image information can be effectively realized by using such a two-way feature complementary mechanism, improving the accuracy and robustness of the semantic segmentation task.

[0179] Step 45: Adjust the parameters in the dual-branch semantic segmentation network model according to the semantic segmentation results corresponding to the first fusion information of each first image and the annotation information corresponding to each first image, and use the adjusted dual-branch semantic segmentation network model as the detection model.

[0180] In this step, input the first fusion information corresponding to each first image into the segmentation head (as Figure 1 shown) for processing to obtain the semantic segmentation results corresponding to the first fusion information. Then, adjust the parameters in the dual-branch semantic segmentation network model according to the semantic segmentation results and the annotation information corresponding to each first image, and use the adjusted dual-branch semantic segmentation network model as the detection model.

[0181] Among them, the segmentation head mainly includes a 3×3 convolution, a batch normalization (BN) layer, a ReLU activation, and finally a 1×1 convolution for dimension compression. The main branch loss undertakes the main learning of the network, and the auxiliary loss is responsible for optimizing the learning. Different weights are set respectively, and the final loss is calculated as follows:

[0182] L f = L m + αL a

[0183] Among them, L f represents the final loss, L m represents the main branch loss, and L a represents the auxiliary loss.

[0184] In a possible implementation, the parameters can include the learning rate, the number of iterations, the loss, etc.

[0185] In a possible implementation, use multiple first images and the annotation information corresponding to the multiple first images to train the dual-branch semantic segmentation model, adjust the model parameters. The model training adopts the stochastic gradient descent method, the initial learning rate is set to 0.01, and the learning rate polynomial decay rate is 0.9. The learning rate formula is as follows:

[0186]

[0187] Among them, lr init is the initial learning rate, iter is the current number of iterations, and iter max is the total number of iterations.

[0188] In addition, using an auxiliary loss at an appropriate position in the dual-branch semantic segmentation network model can effectively solve the problem of backpropagation in the shallow layers of the network. Therefore, drawing on this method, this application adds a segmentation head after each stage in the proposed dual-branch to calculate the auxiliary loss. Considering the trade-off between computational load and efficiency, the dual-branch semantic segmentation network model adopted in the embodiments of this application adds an auxiliary loss function (such as the Online Hard Example Mining (OHEM) loss function) only after the first interaction fusion in the spatial branch (i.e., the first bidirectional attention fusion module) to optimize the training, so as to obtain the various parameters in the dual-branch semantic segmentation network model.

[0189] The abnormal detection method for transmission towers provided by the embodiments of this application first inputs each first image into the backbone network in the dual-branch semantic segmentation network model to obtain a second image after downsampling processing, then inputs the second image into the semantic branch and the spatial branch respectively, and performs fusion processing on the semantic features and spatial features through the bidirectional attention fusion module to obtain a third image corresponding to the semantic branch and a fourth image corresponding to the spatial branch respectively. Input the third image into the pyramid pooling module in the dual-branch semantic segmentation network model to obtain the first semantic feature containing multi-scale information, input the fourth image and the first semantic feature into the feature fusion module in the dual-branch semantic segmentation network model to obtain the first fusion information, and finally adjust the parameters in the dual-branch semantic segmentation network model according to the semantic segmentation results corresponding to the first fusion information of each first image and the annotation information corresponding to each first image, and use the adjusted dual-branch semantic segmentation network model as the detection model. This technical solution effectively improves the accuracy and robustness of semantic segmentation through multi-stage and multi-level feature extraction and fusion, combines the dual-branch structure and the bidirectional attention mechanism, and through precise parameter adjustment, makes the detection model also have good adaptability and flexibility in the application of abnormal detection of transmission towers.

[0190] The following is the device embodiment of this application, which can be used to execute the method embodiment of this application. For the details not disclosed in the device embodiment of this application, please refer to the method embodiment of this application.

[0191] Figure 8 It is a schematic structural diagram of the abnormal detection device for transmission towers provided by the embodiments of this application. As Figure 8 shown, the device includes:

[0192] An acquisition module 81, configured to acquire a to-be-detected image corresponding to a to-be-detected transmission tower;

[0193] A determination module 82 is configured to input an image to be detected into a pre-trained detection model to obtain a target semantic segmentation result, where the target semantic segmentation result is damage information of each component in the transmission tower to be detected. The detection model is obtained by training a dual-branch semantic segmentation network model based on training data. The training data includes: a plurality of first images corresponding to the transmission tower and annotation information corresponding to the plurality of first images. The dual-branch semantic segmentation network model includes: a plurality of first basic modules in the semantic branch and a plurality of second basic modules in the spatial branch. A bidirectional attention fusion module is provided between every two adjacent first basic modules and between every two adjacent second basic modules.

[0194] In a possible implementation manner, before inputting the image to be detected into the pre-trained detection model to obtain the target semantic segmentation result, the determination module 82 is further configured to:

[0195] Obtain a plurality of first images corresponding to the transmission tower and annotation information corresponding to the plurality of first images;

[0196] Train the dual-branch semantic segmentation network model according to the plurality of first images and the annotation information corresponding to the plurality of first images to obtain the detection model.

[0197] In a possible implementation manner, when training the dual-branch semantic segmentation network model according to the plurality of first images and the annotation information corresponding to the plurality of first images to obtain the detection model, the determination module is specifically configured to:

[0198] For each first image, input the first image into the backbone network in the dual-branch semantic segmentation network model to obtain a second image after downsampling processing;

[0199] Input the second image into the semantic branch and the spatial branch respectively, and perform fusion processing of semantic features and spatial features through the bidirectional attention fusion module to obtain a third image corresponding to the semantic branch and a fourth image corresponding to the spatial branch respectively;

[0200] Input the third image into the pyramid pooling module in the dual-branch semantic segmentation network model to obtain a first semantic feature containing multi-scale information;

[0201] Input the fourth image and the first semantic feature into the feature fusion module in the dual-branch semantic segmentation network model to obtain a first fusion information;

[0202] Adjust the parameters in the dual-branch semantic segmentation network model according to the semantic segmentation result corresponding to the first fusion information corresponding to each first image and the annotation information corresponding to each first image, and use the adjusted dual-branch semantic segmentation network model as the detection model.

[0203] In a possible implementation, the second image is input into the semantic branch and the spatial branch respectively, and the semantic features and spatial features are fused through a bidirectional attention fusion module to obtain the third image corresponding to the semantic branch and the fourth image corresponding to the spatial branch. The determination module 82 is specifically configured to:

[0204] Determine the second semantic feature and the first spatial feature corresponding to the second image according to the second image, the first first basic module of the semantic branch, the first second basic module of the spatial branch, and the first bidirectional attention fusion module;

[0205] Determine the third semantic feature and the second spatial feature according to the second semantic feature, the second first basic module of the semantic branch, the first spatial feature, the second second basic module of the spatial branch, and the second bidirectional attention fusion module;

[0206] Determine the third image corresponding to the semantic branch and the fourth image corresponding to the spatial branch according to the third semantic feature, the third first basic module, the second spatial feature, and the third second basic module.

[0207] In a possible implementation, to determine the second semantic feature and the first spatial feature corresponding to the second image according to the second image, the first first basic module of the semantic branch, the first second basic module of the spatial branch, and the first bidirectional attention fusion module, the determination module 82 is specifically configured to:

[0208] Input the second image into the first first basic module of the semantic branch to obtain the fifth image after convolution and downsampling processing;

[0209] Input the second image into the first second basic module of the spatial branch to obtain the sixth image after convolution processing;

[0210] Input the fifth image and the sixth image into the first bidirectional attention fusion module to obtain the second semantic feature corresponding to the fused fifth image and the first spatial feature corresponding to the sixth image.

[0211] In a possible implementation, to determine the third semantic feature and the second spatial feature according to the second semantic feature, the second first basic module of the semantic branch, the first spatial feature, the second second basic module of the spatial branch, and the second bidirectional attention fusion module, the determination module 82 is specifically configured to:

[0212] Input the second semantic feature and the fifth image into the second first basic module of the semantic branch to obtain the seventh image after convolution and downsampling processing;

[0213] Input the first spatial feature and the sixth image into the second second basic module of the spatial branch to obtain the eighth image after convolution processing;

[0214] Input the seventh image and the eighth image into the second bidirectional attention fusion module to obtain the third semantic feature corresponding to the fused seventh image and the second spatial feature corresponding to the eighth image.

[0215] In a possible implementation manner, according to the third semantic feature, the third first basic module, the second spatial feature, and the third second basic module, determine the third image corresponding to the semantic branch and the fourth image corresponding to the spatial branch. The determining module 82 is specifically configured to:

[0216] Input the third semantic feature and the seventh image into the third first basic module of the semantic branch to obtain the third image after convolution and downsampling processing;

[0217] Input the second spatial feature and the eighth image into the third second basic module of the spatial branch to obtain the fourth image after convolution processing.

[0218] The device provided by the embodiments of the present application can be used to execute the determination method in any of the above embodiments, and its implementation principle and technical effects are similar, which will not be elaborated here.

[0219] It should be noted that it should be understood that the division of each module of the above device is only a logical function division. In actual implementation, it can be fully or partially integrated into a physical entity, or physically separated. And these modules can all be implemented in the form of software called by processing elements; they can also all be implemented in the form of hardware; they can also be partially implemented in the form of software called by processing elements and partially implemented in the form of hardware. In addition, all or part of these modules can be integrated together or independently implemented. Here, the processing element can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each of the above modules can be completed by the integrated logic circuit in the processor element or the instructions in the form of software.

[0220] Figure 9 It is a schematic structural diagram of an electronic device provided by an embodiment of the present application. As Figure 9 shown, the electronic device may include: a processor 91, a memory 92, and computer program instructions stored in the memory 92 and executable on the processor 91. When the processor 91 executes the computer program instructions, it implements the method provided in any of the foregoing embodiments.

[0221] Optionally, the above-mentioned various components of the electronic device can be connected through a system bus.

[0222] The memory 92 can be a separate storage unit or a storage unit integrated in the processor 91. The number of processors 91 is one or more.

[0223] It should be understood that the processor 91 can be a Central Processing Unit (CPU), or other general-purpose processors 91, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), etc. The general-purpose processor 91 can be a microprocessor 91 or the processor 91 can also be any conventional processor 91, etc. The steps of the method disclosed in conjunction with the present application can be directly embodied as being completed by the execution of the hardware processor 91, or by a combination of hardware and software modules in the processor 91.

[0224] The system bus can be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, etc. The system bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of simplicity of representation, only a thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The memory 92 may include a Random Access Memory (RAM), and may also include a Non-Volatile Memory (NVM), such as at least one disk memory 92.

[0225] All or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions. The foregoing program can be stored in a readable memory 92. When the program is executed, it executes the steps including the above method embodiments; and the foregoing memory 92 (storage medium) includes: Read-Only Memory (ROM), RAM, flash memory 92, hard disk, solid-state drive, magnetic tape, floppy disk, optical disc, and any combination thereof.

[0226] The electronic device provided in the embodiments of the present application can be used to execute the method provided in any of the above method embodiments, and its implementation principle and technical effects are similar, and will not be elaborated here.

[0227] An embodiment of the present application provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions run on a computer, the computer is caused to execute the above method.

[0228] The above computer-readable storage medium may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory, electrically erasable programmable read-only memory, erasable programmable read-only memory, programmable read-only memory, read-only memory, magnetic memory, flash memory, a magnetic disk or an optical disk. The readable storage medium may be any available medium accessible by a general-purpose or special-purpose computer.

[0229] Optionally, the readable storage medium is coupled to the processor, so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium may also be a component of the processor. The processor and the readable storage medium may be located in an application specific integrated circuit (ASIC). Of course, the processor and the readable storage medium may also exist as discrete components in a device.

[0230] An embodiment of the present application further provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium, and when the at least one processor executes the computer program, the above method can be implemented.

[0231] It should be understood that the present application is not limited to the exact structures that have been described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present application is only limited by the appended claims.

Claims

1. A method for detecting abnormality of a transmission tower, characterized in that: The method comprises: Acquire an image to be detected corresponding to the transmission tower to be detected; The image to be detected is input into a pre-trained detection model to obtain a target semantic segmentation result, wherein the target semantic segmentation result is damage information of each component in the transmission tower to be detected. The detection model is obtained by training a double-branch semantic segmentation network model based on training data, wherein the training data includes: a plurality of first images corresponding to the transmission tower and annotation information corresponding to the plurality of first images; the double-branch semantic segmentation network model includes: a plurality of first basic modules of semantic branches and a plurality of second basic modules of spatial branches, and a bidirectional attention fusion module is arranged between each two adjacent first basic modules and each two adjacent second basic modules.

2. The method according to claim 1, characterized in that Before inputting the image to be detected into a pre-trained detection model to obtain a target semantic segmentation result, the method further includes: Acquire the plurality of first images corresponding to the transmission tower and annotation information corresponding to the plurality of first images; The dual-branch semantic segmentation network model is trained according to the multiple first images and the annotation information corresponding to the multiple first images to obtain the detection model.

3. The method according to claim 2, characterized in that The step of training the dual-branch semantic segmentation network model according to the plurality of first images and the annotation information corresponding to the plurality of first images to obtain the detection model includes: For each first image, input the first image into the backbone network in the dual-branch semantic segmentation network model to obtain a second image after downsampling; Inputting the second image into the semantic branch and the spatial branch respectively, and fusing the semantic features and the spatial features through the bidirectional attention fusion module, to obtain a third image corresponding to the semantic branch and a fourth image corresponding to the spatial branch respectively; Inputting the third image into the pyramid pooling module in the dual-branch semantic segmentation network model to obtain a first semantic feature containing multi-scale information; Inputting the fourth image and the first semantic feature into a feature fusion module in the dual-branch semantic segmentation network model to obtain first fusion information; According to the semantic segmentation results corresponding to the first fusion information corresponding to each first image and the annotation information corresponding to each first image, the parameters in the dual-branch semantic segmentation network model are adjusted, and the adjusted dual-branch semantic segmentation network model is used as the detection model.

4. The method according to claim 3, characterized in that The step of inputting the second image into the semantic branch and the spatial branch respectively, and fusing the semantic features and the spatial features through the bidirectional attention fusion module to obtain a third image corresponding to the semantic branch and a fourth image corresponding to the spatial branch respectively, comprises: Determine a second semantic feature and a first spatial feature corresponding to the second image according to the second image, the first first basic module of the semantic branch, the first second basic module of the spatial branch, and the first bidirectional attention fusion module; Determine a third semantic feature and a second spatial feature according to the second semantic feature, the second first basic module of the semantic branch, the first spatial feature, the second second basic module of the spatial branch, and the second bidirectional attention fusion module; According to the third semantic feature, the third first basic module, the second spatial feature, and the third second basic module, a third image corresponding to the semantic branch and a fourth image corresponding to the spatial branch are determined.

5. The method according to claim 4, characterized in that The determining, according to the second image, the first first basic module of the semantic branch, the first second basic module of the spatial branch, and the first bidirectional attention fusion module, the second semantic feature and the first spatial feature corresponding to the second image comprises: Inputting the second image into the first basic module of the semantic branch to obtain a fifth image after convolution and downsampling processing; Inputting the second image into the first second basic module of the spatial branch to obtain a sixth image after convolution processing; The fifth image and the sixth image are input into a first bidirectional attention fusion module to obtain a second semantic feature corresponding to the fused fifth image and a first spatial feature corresponding to the sixth image.

6. The method according to claim 5, characterized in that The determining of the third semantic feature and the second spatial feature according to the second semantic feature, the second first basic module of the semantic branch, the first spatial feature, the second second basic module of the spatial branch, and the second bidirectional attention fusion module includes: Inputting the second semantic feature and the fifth image into the second first basic module of the semantic branch to obtain a seventh image after convolution and downsampling processing; Inputting the first spatial feature and the sixth image into the second second basic module of the spatial branch to obtain an eighth image after convolution processing; The seventh image and the eighth image are input into a second bidirectional attention fusion module to obtain a third semantic feature corresponding to the fused seventh image and a second spatial feature corresponding to the eighth image.

7. The method according to claim 6, characterized in that The step of determining, according to the third semantic feature, the third first basic module, the second spatial feature, and the third second basic module, the third image corresponding to the semantic branch and the fourth image corresponding to the spatial branch comprises: Inputting the third semantic feature and the seventh image into the third first basic module of the semantic branch to obtain a third image after convolution and downsampling processing; The second spatial feature and the eighth image are input into the third second basic module of the spatial branch to obtain a fourth image after convolution processing.

8. An abnormality detection device for a transmission tower, characterized in that: The device comprises: An acquisition module is used to acquire the image to be detected corresponding to the transmission tower to be detected; A determination module is used to input the image to be detected into a pre-trained detection model to obtain a target semantic segmentation result, wherein the target semantic segmentation result is damage information of each component in the transmission tower to be detected. The detection model is obtained by training a double-branch semantic segmentation network model based on training data, wherein the training data includes: multiple first images corresponding to the transmission tower and annotation information corresponding to the multiple first images, and the double-branch semantic segmentation network model includes: multiple first basic modules of semantic branches and multiple second basic modules of spatial branches, and a bidirectional attention fusion module is arranged between each two adjacent first basic modules and each two adjacent second basic modules.

9. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the method according to any one of claims 1 to 7.

Images