A visual inseparable bolt defect detection method based on bolt attribute and position

Abstract

The application discloses a visual indistinguishable bolt defect detection method based on bolt attributes and positions, selects a transformer architecture DETR as a basic model, and uses Resnet50 as a backbone network; an extensible encoder module is introduced, multi-scale features of a target are acquired under the condition of only using C5 layer features to improve the accuracy of target detection, and the problem of slow training speed caused by using multi-scale features in the DETR is solved; while the absolute position coding in the DETR is reserved, iRPE relative position coding is added to the multi-head self-attention module of the encoder to consider the direction, distance and interaction between the query and the relative position coding in the encoder self-attention mechanism, and the relative position relationship between each element is more accurately acquired; a bolt attribute classifier and a bolt defect classifier are designed, the attribute information of the bolt is combined with the bolt features with the position information, and the accuracy of bolt defect classification is further improved.

Description

Technical Field

[0001] This invention relates to the field of image analysis technology, and in particular to a visual method for detecting defects in inseparable bolts based on bolt properties and location. Background Technology

[0002] Transmission line faults are a major cause of large-scale power outages, and line operation and maintenance are crucial for ensuring the safety of the power system. Transmission lines contain numerous bolts, widely found in towers, insulators, and fittings, playing a key role in fixing components and maintaining the structural stability of the transmission line. However, bolts are small in size, have complex defect types, and are subjected to a large number of stresses. Furthermore, the operating environment of outdoor transmission lines is directly or indirectly affected by factors such as variable weather, severe natural disasters, vegetation growth around the corridor, and biological activity in the vicinity. Bolts can easily transition from a normal state to a defective state. Common defects such as loose nuts or pins can lead to component loosening and affect structural stability; critical defects such as missing pins or nuts can affect power transmission safety and potentially cause large-scale line faults, seriously threatening the safe operation of the power grid. Therefore, defect detection of bolts is essential to ensure the normal operation of transmission lines.

[0003] In recent years, with the development of artificial intelligence technology, using drones to inspect power transmission lines and then using computer vision and image processing technology to automate the aerial images of the power transmission lines has become the main method.

[0004] However, using deep learning methods to detect defects in bolts in power transmission lines presents two main challenges:

[0005] 1. Compared to other hardware in power transmission lines, bolts are very small, making them difficult to inspect. This is a key factor affecting the accuracy of bolt defect detection. Therefore, in bolt defect detection, the model must first be able to detect small-scale targets.

[0006] 2. Most existing bolt defect identification and detection methods assume that defects are visually separable. A bolt is considered defective if a part of it is missing. However, bolt defects in transmission lines actually fall into the category of visually inseparable defects. For example, bolts on a bag-type suspension clamp require both a nut and a washer; a bolt is considered defective only when either the nut or washer is missing. However, bolts on bag-type suspension clamps do not require a pin, therefore, the defect of a missing pin does not exist on these clamps. Conversely, bolts on U-shaped rings require both a nut and a pin, but not a missing washer. Therefore, bolt defect detection cannot rely solely on target detection algorithms; it also requires the introduction of relevant knowledge from the power industry to assist in defect identification models.

[0007] Therefore, under the above background, it is of paramount importance to introduce relevant knowledge from the power industry into deep learning models to solve the current problems in the detection of bolt defects in transmission lines, and to further improve the detection accuracy while meeting industrial needs. Summary of the Invention

[0008] The purpose of this invention is to provide a visually inseparable bolt defect detection method based on bolt attributes and location, which solves the problems of small bolt targets being difficult to detect and the visual inseparability of bolt defects in transmission lines. A new model is designed that can simultaneously detect bolt targets and their defects, and the location and attribute knowledge of bolts in transmission lines is introduced to further improve the accuracy of bolt defect detection.

[0009] To achieve the above objectives, the present invention provides the following solution:

[0010] A visual method for detecting defects in inseparable bolts based on bolt attributes and location includes the following steps:

[0011] S1, construct a hardware-bolt dataset to obtain images of hardware and bolts of different categories and scales;

[0012] S2, the DETR transformer model is selected as the basic architecture, and the ResNet50 model is used as the backbone network;

[0013] S3 incorporates an expandable encoder module to improve the model's adaptability to hardware and bolts of different scales.

[0014] S4 retains the original absolute position encoding in DETR while adding iRPE relative position encoding to the encoder's self-attention mechanism. It comprehensively considers the influence of direction, position, and queries on relative position encoding to accurately obtain the relative positional relationship between features, thereby further improving the accuracy of target detection and bolt defect identification.

[0015] S5, after the detection head, a bolt attribute classifier and a bolt defect classifier are designed. When the model detects that the target is a bolt, it first classifies the bolt attributes, and then combines the bolt attribute information with the bolt features with position information to further identify bolt defects.

[0016] Furthermore, in step S1, a hardware-bolt dataset is constructed to obtain images of hardware and bolts of different categories and scales, specifically including:

[0017] The areas containing hardware and bolts on power transmission lines and towers were cropped from aerial images, cleaned, and selected from those that were clear and contained a large variety and number of hardware and bolts. The dataset was labeled according to the COCO dataset format, and attributes and defect categories were assigned to each target bolt.

[0018] Furthermore, in step S2, the transformer model DETR is selected as the basic architecture, and the ResNet50 model is used as the backbone network, specifically including:

[0019] First, a ResNet50 model is used as the backbone network to extract features from the hardware-bolt images. Then, the image features are added to the absolute position encoding and fed into the DETR encoder. The encoder learns the relationships between features through a multi-head self-attention mechanism and then feeds the optimized features into the DETR decoder. The decoder initializes 100 learnable queries. First, the queries are fed into the decoder's self-attention module for optimization. Then, the optimized queries are combined with the image features output from the encoder and fed into the DETR's cross-attention module to obtain queries with image features. Finally, bipartite matching is used to apply the queries to object detection classification and regression to obtain the category information and bounding box position information of the hardware and bolts.

[0020] Furthermore, in step S3, an expandable encoder module is added to improve the model's adaptability to fittings and bolts of different scales, specifically including:

[0021] First, an extensible encoder module is introduced to enhance feature extraction capabilities and robustness. Since using a Feature Pyramid Network (FPN) in DETR would lead to excessive computation, DETR performs poorly in detecting small targets. To address this issue, standard convolutions and dilated convolutions are stacked to expand the receptive field of the C5 layer features. The formula for calculating the receptive field of dilated convolution is:

[0022] K = (rate-1) × (k-1) + k (1)

[0023] In the formula, K is the dilated convolution receptive field, rate is the dilated convolution rate, and k is the convolution kernel size.

[0024] The scalable encoder module consists of two parts: a projection layer and residual blocks. The projection layer uses 1×1 convolutions to reduce the channel dimension, and then adds a 3×3 convolutional layer to refine the semantic information of the context. Subsequently, the residual blocks generate output features containing multiple receptive fields covering all target scales by stacking four consecutively dilated convolutional residual blocks with different dilation rates.

[0025] Finally, the original scale range and the magnified scale range are combined by adding corresponding features to obtain an output feature containing multiple receptive fields. The C5 layer features alone can cover all object scales, obtain feature maps with richer semantic information, and greatly reduce the amount of computation.

[0026] Furthermore, in step S4, a two-dimensional image relative position encoding (iRPE) is added. By considering the interaction between direction, distance, and the self-attention query and relative position embedding in the encoder, the model can obtain the relative positional relationships of different targets in the image, further improving the accuracy of target detection and bolt defect identification. Specifically, this includes:

[0027] First, the original absolute positional encoding in DETR is retained. After the image features are divided into element blocks, in order for the model to utilize the sequence order, the positional information of each element block is labeled, and an absolute positional encoding P = (P1, P2, ..., P...) is added. n ) to the element block embedded as X.

[0028] X i =X i +P i (2)

[0029] In the formula, X i For the i-th element block, P i Fixed encoding using sine functions of different frequencies.

[0030] Secondly, addressing the limitation of DETR in fully utilizing the pairwise relationships between different input elements, a two-dimensional image relative position encoding method, iRPE, is introduced. A segmented indexing function g(x) maps the relative distance between elements to an integer in a finite set. The relative position encoding can then be indexed by this integer, and this condition is shared across different positional relationships. The segmented indexing function significantly reduces the computational cost and number of parameters for the sequence. The formula for the segmented indexing function is:

[0031]

[0032] In the formula, [·] represents integer operations, sign(x) determines the sign of x, returns 1 for positive input, -1 for negative input, and 0 otherwise. α is the segmentation point, β controls the output within the range of [-β, β], and γ adjusts the curvature of the logarithmic part.

[0033] Then, to fully consider the importance of the orientation between elements, the horizontal and vertical codes are calculated separately and then combined, as shown in the formula:

[0034]

[0035] I x(i,j)=g(x i -x j (5)

[0036] I y (i,j)=g(y i -y j (6)

[0037] In the formula, (x i ,y i ) and (x j ,y j ) represent the positions of the i-th and j-th element blocks, respectively. x (i,j) and I y (i,j) are integers indexed by g(x) in the horizontal and vertical directions, respectively. and These are relative position codes in the horizontal and vertical directions, respectively, and are learnable vectors. ij This is the encoding of the relative positions after combination.

[0038] Finally, r ij The self-attention module of the DETR encoder is introduced to help the model obtain the relative positional relationships of different targets in the image, further improving the accuracy of target detection and bolt defect identification. The formula is as follows:

[0039]

[0040]

[0041]

[0042]

[0043] In the formula, W Q W K and W V Both are learnable parameter matrices. ij T b is the transpose matrix of the relative position encoding. ij This is the relative position encoding after combining it with the query in the encoder. z These are the dimensions of Q and K, which here serve as a form of normalization. ij This represents the attention value for the key value in the encoder. For e ij The output after softmax operation. i This is the output of the i-th element after passing through the self-attention module.

[0044] Furthermore, in step S5, a bolt attribute classifier and a bolt defect classifier are designed. When the model detects a bolt as the target, it first classifies the bolt's attributes, and then combines the bolt attribute information with bolt features containing location information to further identify bolt defects. Specifically, this includes:

[0045] First, the DETR decoder outputs 100 optimized queries with image features, where each query contains the features and location information of each target. Then, the DETR target detection head uses these queries for classification and bounding box regression to obtain the category information and bounding box location information of the hardware-bolt image.

[0046] Then, the model determines whether the current image contains the category of bolts. If bolts are present, the corresponding queries are obtained and used as the classification of bolt attributes. Both the bolt attribute classifier and the bolt defect classifier are composed of MLPs. By feeding the queries with bolt features into the bolt attribute classifier, scores for each type of bolt attribute can be obtained. Here, the bolt attributes are categorized into three types: nuts, pins, and washers.

[0047] Finally, the obtained bolt attribute scores are used as vector f. attribute Multiply by a learnable weight matrix W attribute After obtaining the bolt's attribute information, it is combined with Queries containing bolt features and bolt position information, using the following formula:

[0048] f attribute =[S nut ,S pin ,S shim (11)

[0049] F = f bolt +f attribute W attribute (12)

[0050] In the formula, S nut S pin and S shim These are the scores for nuts, pins, and washers, respectively, as classified by the bolt attribute classifier. attribute W is a fractional vector of the bolt. attribute f is a learnable weight matrix. bolt F represents the bolt features with location information in the Queries. F is the output feature that combines the bolt attribute information and the bolt features with location information.

[0051] The value F is then fed into the bolt defect classifier to classify whether the bolt has a defect and the type of defect. The defect types are divided into four categories: missing bolt, missing pin, missing washer, and normal bolt.

[0052] According to specific embodiments provided by the present invention, the following technical effects are disclosed: The present invention provides a visually inseparable bolt defect detection method based on bolt attributes and position, comprising the steps of: constructing a hardware-bolt dataset to obtain images of hardware and bolts of different types and scales; selecting the transformer architecture DETR as the basic model, using ResNet50 as the backbone network, and adding an extensible encoder module to obtain multi-scale features of the target using only C5 layer features to improve the accuracy of target detection, while solving the problem of slow training speed when using multi-scale features in DETR; in addition to the absolute position encoding of DETR itself, iRPE relative position encoding is added to the multi-head self-attention module of the encoder to consider the interaction between direction, distance, and self-attention query and relative position encoding in the encoder, to more accurately obtain the relative position relationship of each element block; a bolt attribute classifier and a bolt defect classifier are designed and added after the detection head to combine bolt features with position information and bolt attribute information to further improve the accuracy of bolt defect classification. This invention applies DETR to bolt defect detection, effectively improving the accuracy of visually inseparable bolt defect detection by combining bolt properties and location, thus meeting the needs of end-to-end bolt defect detection. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 This is a flowchart of a visually inseparable bolt defect detection method based on bolt attributes and location according to an embodiment of the present invention;

[0055] Figure 2 This is a schematic diagram of the network structure of an expandable encoder module according to an embodiment of the present invention;

[0056] Figure 3 This is a schematic diagram of the overall structure of an embodiment of the present invention;

[0057] Figure 4 This is a diagram illustrating the effect of target detection for hardware-bolts according to an embodiment of the present invention.

[0058] Figure 5 This is a diagram illustrating the effect of bolt defect detection in an embodiment of the present invention. Detailed Implementation

[0059] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0060] The purpose of this invention is to provide a visually inseparable bolt defect detection method based on bolt attributes and location, which solves the problems of bolt targets in transmission lines being relatively small and difficult to detect, the low performance of DETR for small-scale targets, and the visual inseparability of bolt defects in transmission lines. This method aims to improve the accuracy of bolt defect detection while meeting industrial requirements.

[0061] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0062] like Figure 1 As shown, the present invention provides a visually inseparable bolt defect detection method based on bolt attributes and location, comprising the following steps:

[0063] S1, construct a hardware-bolt dataset to obtain images of hardware and bolts of different categories and scales;

[0064] S2, the DETR transformer model is selected as the basic architecture, and the ResNet50 model is used as the backbone network;

[0065] S3 incorporates an expandable encoder module to improve the model's adaptability to hardware and bolts of different scales.

[0066] S4 retains the original absolute position encoding in DETR while adding iRPE relative position encoding to the encoder's self-attention mechanism. It comprehensively considers the influence of orientation, position, and self-attention queries in the encoder on relative position encoding, so as to accurately obtain the relative positional relationship between features and further improve the accuracy of target detection and bolt defect identification.

[0067] S5, after the detection head, a bolt attribute classifier and a bolt defect classifier are designed. When the model detects that the target is a bolt, it first classifies the bolt attributes, and then combines the bolt attribute information with the bolt features with position information to further identify bolt defects.

[0068] The basic flowchart of this invention is as follows:Figure 1 As shown.

[0069] Deep learning models require a large number of dataset image samples for training. Since most drone aerial images are global images of power transmission lines, they need to be cropped according to the model's input image size. Furthermore, to visually compare the improved accuracy of the model before and after the upgrade in detecting visually inseparable bolt defects, it is necessary to further filter samples containing bolt types. Therefore, in step S1, a hardware-bolt dataset is constructed to obtain hardware-bolt images of different categories and scales, specifically including:

[0070] The areas containing hardware on power transmission lines and towers in aerial images were cropped, cleaned, and selected from those that were clear and contained a large variety and number of hardware and bolts. The dataset was labeled according to the COCO dataset format, and attributes and defect categories were labeled for each target bolt.

[0071] In this method, considering the visual inseparability problem in bolt defect detection in transmission lines, it is necessary to incorporate relevant knowledge from the power industry into the deep learning model, while also requiring higher accuracy in target detection of fittings and bolts. This method introduces the transformer model DETR to improve the accuracy of the bolt defect detector. Specifically, in step S2, the transformer model DETR is selected as the basic architecture, and the ResNet50 model is used as the backbone network, including:

[0072] First, a ResNet50 model is used as the backbone network to extract features from the hardware-bolt images. Then, the image features are added to the absolute position encoding and fed into the DETR encoder. The encoder learns the relationships between features through a multi-head self-attention mechanism and then feeds the optimized features into the DETR decoder. The decoder initializes 100 learnable queries. First, the queries are fed into the decoder's self-attention module for optimization. Then, the optimized queries are combined with the image features output from the encoder and fed into the DETR's cross-attention module to obtain queries with image features. Finally, bipartite matching is used to apply the queries to object detection classification and regression to obtain the category information and bounding box position information of the hardware and bolts.

[0073] In step S3, an expandable encoder module is added to improve the model's adaptability to fittings and bolts of different scales. Specifically, this includes:

[0074] First, an extensible encoder module is introduced to enhance feature extraction capabilities and robustness. Since using a Feature Pyramid Network (FPN) in DETR would lead to excessive computation, DETR performs poorly in detecting small targets. To address this issue, standard convolutions and dilated convolutions are stacked to expand the receptive field of the C5 layer features. The formula for calculating the receptive field of dilated convolution is:

[0075] K = (rate-1) × (k-1) + k (1)

[0076] In the formula, K is the dilated convolution receptive field, rate is the dilation rate, and k is the kernel size.

[0077] The scalable encoder module consists of two parts: a projection layer and residual blocks. The projection layer uses 1×1 convolutions to reduce the channel dimension, and then adds a 3×3 convolutional layer to refine the semantic information of the context. Subsequently, the residual blocks generate output features containing multiple receptive fields covering all target scales by stacking four consecutively dilated convolutional residual blocks with different dilation rates.

[0078] Finally, the original scale range and the magnified scale range are combined by adding corresponding features to obtain an output feature containing multiple receptive fields. The C5 layer features alone can cover all object scales, obtain feature maps with richer semantic information, and greatly reduce the amount of computation.

[0079] A schematic diagram of the network structure of the expandable encoder module in this invention is shown below. Figure 2 As shown.

[0080] In step S4, a two-dimensional image relative position encoding (iRPE) is added. By considering the interaction between direction, distance, queries, and relative position embedding, the model can obtain the relative positional relationships of different targets in the image, further improving the accuracy of target detection and bolt defect identification. Specifically, this includes:

[0081] First, the original absolute positional encoding in DETR is retained. After the image features are divided into element blocks, in order for the model to utilize the sequence order, the positional information of each element block is labeled, and an absolute positional encoding P = (P1, P2, ..., P...) is added. n ) to the element block embedded as X.

[0082] X i =X i +P i (2)

[0083] In the formula, X i For the i-th element block, P i Fixed encoding using sine functions of different frequencies.

[0084] Secondly, addressing the limitation of DETR in fully utilizing the pairwise relationships between different input elements, a two-dimensional image relative position encoding method, iRPE, is introduced. A segmented indexing function g(x) maps the relative distance between elements to an integer in a finite set. The relative position encoding can then be indexed by this integer, and this condition is shared across different positional relationships. The segmented indexing function significantly reduces the computational cost and number of parameters for the sequence. The formula for the segmented indexing function is:

[0085]

[0086] In the formula, [·] represents integer operations, sign(x) determines the sign of x, returns 1 for positive input, -1 for negative input, and 0 otherwise. α is the segmentation point, β controls the output within the range of [-β, β], and γ adjusts the curvature of the logarithmic part.

[0087] Then, to fully consider the importance of the orientation between elements, the horizontal and vertical codes are calculated separately and then combined, as shown in the formula:

[0088]

[0089] I x (i,j)=g(x i -x j (5)

[0090] I y (i,j)=g(y i -y j (6)

[0091] In the formula, (x i ,y i ) and (x j ,y j ) represent the positions of the i-th and j-th element blocks, respectively. x (i,j) and I y (i,j) are integers indexed by g(x) in the horizontal and vertical directions, respectively. and These are relative position codes in the horizontal and vertical directions, respectively, and are learnable vectors. ij This is the encoding of the relative positions after combination.

[0092] Finally, r ij The self-attention module of the DETR encoder is introduced to help the model obtain the relative positional relationships of different targets in the image, further improving the accuracy of target detection and bolt defect identification. The formula is as follows:

[0093]

[0094]

[0095]

[0096]

[0097] In the formula, W Q W K and W V Both are learnable parameter matrices. ij T b is the transpose matrix of the relative position encoding. ij This is the relative position encoding after combining it with the query in the encoder. z These are the dimensions of Q and K, which here serve as a form of normalization. ij This represents the attention value for the key value in the encoder. For e ij The output after softmax operation. i This is the output of the i-th element after passing through the self-attention module.

[0098] In step S5, a bolt attribute classifier and a bolt defect classifier are designed. When the model detects a bolt as the target, it first classifies the bolt's attributes, and then combines the bolt attribute information with bolt features containing location information to further identify bolt defects. Specifically, this includes:

[0099] First, the DETR decoder outputs 100 optimized queries with image features, where each query contains the features and location information of each target. Then, the DETR target detection head uses these queries for classification and bounding box regression to obtain the category information and bounding box location information of the hardware-bolt image.

[0100] Then, the model determines whether the current image contains the category of bolts. If bolts are present, the corresponding queries are obtained and used as the classification of bolt attributes. Both the bolt attribute classifier and the bolt defect classifier are composed of MLPs. By feeding the queries with bolt features into the bolt attribute classifier, scores for each type of bolt attribute can be obtained. Here, the bolt attributes are categorized into three types: nuts, pins, and washers.

[0101] Finally, the obtained bolt attribute scores are used as vector f. attribute Multiply by a learnable weight matrix W attribute After obtaining the bolt's attribute information, it is combined with Queries containing bolt features and bolt position information, using the following formula:

[0102] f attribute =[S nut,S pin ,S shim (11)

[0103] F = f bolt +f attribute W attribute (12)

[0104] In the formula, S nut S pin and S shim These are the scores for nuts, pins, and washers, respectively, as classified by the bolt attribute classifier. attribute W is a fractional vector of the bolt. attribute f is a learnable weight matrix. bolt F represents the bolt features with location information in the Queries. F is the output feature that combines the bolt attribute information and the bolt features with location information.

[0105] The value F is then fed into the bolt defect classifier to classify whether the bolt has a defect and the type of defect. The defect types are divided into four categories: missing bolt, missing pin, missing washer, and normal bolt.

[0106] This invention integrates the two improved methods proposed in steps S3, S4, and S5 into the basic network proposed in step S2, ultimately forming a complete visual method for detecting inseparable bolt defects based on bolt attributes and location. The network structure is shown in Figure 3.

[0107] The target detection effect of the method of the present invention on hardware and bolts is as follows: Figure 4 As shown.

[0108] The bolt defect detection effect of the method of the present invention is as follows: Figure 5 As shown, this invention, based on the DETR model, uses ResNet50 as the backbone network and adds an extensible encoder module. It obtains multi-scale features of the target using only the C5 layer features to improve the accuracy of target detection, while also solving the problem of slow training speed when using multi-scale features in DETR. In addition to retaining the absolute position encoding of DETR itself, this invention adds iRPE relative position encoding to the multi-head self-attention module of the encoder to consider the interaction between direction, distance, and the query and relative position encoding in the encoder's self-attention, thus more accurately obtaining the relative positional relationship of each element block. Furthermore, this invention adds a bolt attribute classifier and a bolt defect classifier after the detection head, combining bolt features with positional information and bolt attribute information to further improve the accuracy of bolt defect classification. Therefore, the method described in this invention effectively solves the visual inseparability problem in bolt defect detection in transmission lines while ensuring small-scale bolt target detection, meeting the needs of practical industry.

[0109] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for visually inseparable bolt defect detection based on bolt attributes and location, characterized in that, Includes the following steps: S1, construct a hardware-bolt dataset to obtain images of hardware and bolts of different categories and scales; S2, the DETR transformer model is selected as the basic architecture, and the ResNet50 model is used as the backbone network; S3 introduces an expandable encoder module to improve the model's adaptability to hardware and bolts of different scales. S4 retains the original absolute position encoding in DETR while adding iRPE relative position encoding to the encoder's self-attention mechanism. It comprehensively considers the influence of direction, position, and queries on relative position encoding to accurately obtain the relative position relationship between features, thereby further improving the accuracy of target detection and bolt defect identification. S5. Design a bolt attribute classifier and a bolt defect classifier. When the model detects that the target is a bolt, it first classifies the bolt attributes, and then combines the bolt attribute information with the bolt features with location information to further identify bolt defects. Specifically, in S5, a bolt attribute classifier and a bolt defect classifier are designed. When the model determines that the target in the hardware-bolt image is a bolt, the features of the target are first input into the bolt attribute classifier to classify the attribute information of the current bolt. Then, the attribute information of the bolt is combined with the bolt features with location information to further identify bolt defects, specifically including: First, the DETR decoder outputs 100 optimized queries with image features, where each query contains the features and location information of each target. Then, the DETR target detection head uses the queries for classification and bounding box regression to obtain the category information and bounding box location information of the hardware-bolt image. Then, the model determines whether the current image contains the category of bolt. If it contains bolt, it obtains the corresponding queries and uses these queries as the classification of bolt attributes. Both the bolt attribute classifier and the bolt defect classifier are composed of MLP. By inputting the queries with bolt features into the bolt attribute classifier, the scores of various bolt attributes can be obtained. Here, the bolt attributes are divided into three categories: nut, pin and washer. Finally, the obtained bolt attribute scores are used as a vector. Multiply by a learnable weight matrix After obtaining the bolt's attribute information, it is combined with Queries containing bolt features and bolt position information, using the following formula: （11） （12） In the formula, , and These are the scores for nuts, pins, and washers, respectively, as classified by the bolt attribute classifier. Let be the fractional vector of the bolt. The weight matrix is ​​a learnable matrix. F represents the bolt features with location information in the Queries, and F is the output feature that combines bolt attribute information and bolt features with location information. Then, F is further fed into the bolt defect classifier to classify whether the current bolt has a defect and the defect category. The defect categories are divided into four types: missing bolt, missing pin, missing washer, and normal bolt.

2. The visual non-separable bolt defect detection method based on bolt attributes and location according to claim 1, characterized in that, The construction of the hardware-bolt dataset, which yields images of hardware and bolts of different categories and scales, specifically includes: The areas containing fittings and bolts on power transmission lines and towers in aerial images were cropped, cleaned, and selected from those images that were clear and contained a large variety and number of fittings and bolts. The dataset was labeled according to the COCO dataset format, and on this basis, the attributes and defect categories of each target bolt were labeled.

3. The visual non-separable bolt defect detection method based on bolt attributes and location according to claim 1, characterized in that, The selection of the DETR transformer model as the basic architecture and the use of the ResNet50 model as the backbone network specifically includes: First, a ResNet50 model is used as the backbone network to extract features from the hardware-bolt images. Then, the image features are added to the absolute position encoding and fed into the DETR encoder. The encoder learns the relationship between features through a multi-head self-attention mechanism and then feeds the optimized features into the DETR decoder. The decoder initializes 100 learnable queries. First, the queries are fed into the decoder's self-attention module for optimization. Then, the optimized queries are combined with the image features output by the encoder and fed into the DETR's cross-attention module to obtain queries with image features. Finally, the queries are used for object detection classification and regression through bipartite matching to obtain the category information and bounding box position information of the hardware and bolts.

4. The method for visually inseparable bolt defect detection based on bolt attributes and location according to claim 1, characterized in that, The introduction of an extensible encoder module improves the model's adaptability to hardware and bolts of different scales, specifically including: First, an extensible encoder module is introduced to enhance feature extraction capabilities and robustness. Since using a Feature Pyramid Network (FPN) in DETR would result in excessive computation, DETR performs poorly in detecting small targets. To address this issue, standard convolution and dilated convolution are superimposed to expand the receptive field of the C5 layer features. The formula for calculating the receptive field of dilated convolution is as follows: （1） In the formula, To dilate the receptive field of the convolution, The dilatational convolution ratio is... The kernel size; The scalable encoder module consists of two parts: a projection layer and residual blocks. The projection layer uses 1×1 convolutions to reduce the channel dimension, and then adds a 3×3 convolutional layer to refine the semantic information of the context. After that, the residual blocks generate output features containing multiple receptive fields of all target scales by stacking four consecutive dilated convolutional residual blocks with different dilation rates. Finally, the original scale range and the magnified scale range are combined by adding corresponding features to obtain an output feature containing multiple receptive fields. The C5 layer features alone can cover all object scales, obtain feature maps with richer semantic information, and greatly reduce the amount of computation.

5. The method for visually inseparable bolt defect detection based on bolt attributes and location according to claim 1, characterized in that, By incorporating 2D image relative position encoding (iRPE), and considering the interaction between orientation, distance, and the self-attention query and relative position embedding in the encoder, the model can acquire the relative positional relationships of different targets in the image, further improving the accuracy of target detection and bolt defect identification. Specifically, this includes: First, the original absolute positional encoding in DETR is retained. After the image features are divided into element blocks, in order for the model to utilize the sequence order, the positional information of each element block is labeled and absolute positional encoding is added. to embed as The element block, the formula is: （2） In the formula, For the i-th element block, A fixed code for using sine functions of different frequencies; Secondly, to address the limitation of DETR in fully utilizing the pairwise relationships between different input elements, a two-dimensional image relative position encoding method, iRPE, is introduced, which utilizes a segmented indexing function. By mapping the relative distances between elements to an integer in a finite set, the relative positional encoding can be indexed by this integer, and this condition is shared across different positional relationships. The segmented indexing function can significantly reduce the computational cost and number of parameters for sequences. The formula for the segmented indexing function is: （3） In the formula, Integer operations, Decide The symbol indicates whether the input is positive or negative; a positive input returns 1, a negative input returns -1, and otherwise returns 0. As the segmentation point, control Output within range Adjust the curvature of the logarithmic part; Then, to fully consider the importance of the orientation between elements, the horizontal and vertical codes are calculated separately and then combined, as shown in the formula: （4） （5） （6） In the formula, and The first The and the first The position of each element block and respectively in the horizontal and vertical directions The integer after the index; and These are relative position codes in the horizontal and vertical directions, respectively, and are learnable vectors. Encoding the relative positions after combination; Finally The self-attention module of the DETR encoder is introduced to help the model obtain the relative positional relationships of different targets in the image, further improving the accuracy of target detection and bolt defect identification. The formula is as follows: （7） （8） （9） （10） In the formula, , and All of these are learnable parameter matrices. The transpose matrix of relative position encoding. This is the relative position encoding after combining with the query in the encoder; These are the dimensions of Q and K, which here serve as a form of normalization. This represents the attention value of the query to the key value in the encoder. for The output after softmax operation For the first The output of each element after passing through the self-attention module.

Images