A fitting classification detection method based on an improved attention mechanism

Abstract

The application discloses a kind of based on improved attention mechanism's fitting classification detection method, improved Swin Transformer architecture is selected as main network;Introduce the framework module based on the convolution of self-attention, utilize shallow convolution network and mobile window multi-head attention mechanism, the region containing rich fitting feature information is extracted to improve the accuracy of target classification;Among them, the improved Swin Transformer network includes two independent regions, i.e. the convolution based on self-attention and Swin Transformer.The convolution based on self-attention utilizes shallow convolution network and mobile window multi-head attention mechanism, and extracts the region containing rich fitting specific information.Subsequently, the features obtained based on the convolution of self-attention are spliced with the original image processed by linear embedding in Swin Transformer, and then input into Swin Transformer to identify the type of fitting, to more accurately identify fitting.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image analysis, and in particular to a fitting classification detection method based on an improved attention mechanism. BACKGROUND

[0002] With the rapid development of China's new generation of power system construction, as an important part of the power system, the intelligent detection and analysis of power transmission and transformation equipment becomes increasingly important. The power transmission line is an important lifeline of the power system. Due to its direct exposure to the outdoor natural environment, it is easily affected by external factors such as wind, rain, snow and thunder, and thus prone to failure. Therefore, regular inspection of the reliability and operation of the power transmission line plays a crucial role in maintaining the safety of the power system. The traditional manual inspection method cannot meet the needs of the construction of the smart grid. Through the use of unmanned aerial vehicles to inspect the power transmission line and obtain pictures, the use of artificial intelligence and computer vision technology has gradually become the mainstream. In the power transmission link, the main research objects of power vision technology include insulators, conductors, fittings, towers, stay wires and grounding devices.

[0003] The inspection mode of visual processing and analysis of the inspection pictures has been closely related to the power transmission line fault, which is one of the important reasons for large-scale power grid outage accidents. Line operation and maintenance is an important link to ensure the safety of the power system. Fittings are widely used iron or aluminum metal accessories on the power transmission line, which are of various types and mainly used for supporting, fixing, connecting bare conductors, conductors and insulators, etc. Due to the complex contact environment, the power transmission line may have electrical flashover, material aging, mechanical tension and other situations. Once discovered in time, it will cause a major circuit damage accident. At present, the use of aircraft for power transmission line inspection has become a conventional inspection method. The use of computer vision and image processing technology for automatic processing of aerial images and aerial videos realizes automatic positioning and identification of power transmission line fitting faults, which greatly improves the efficiency of power maintenance and repair. Accurate identification of fitting targets is the premise of fitting fault judgment. Due to the particularity of the scene and target of the power transmission line aerial image fitting detection, there are certain differences from general target detection. Therefore, accurate identification of fitting targets is necessary for ensuring the normal operation of the power transmission line.

[0004] In recent years, with the development of artificial intelligence technology, the use of unmanned aerial vehicles for power transmission line inspection and the use of computer vision and image processing technology for automatic processing of power transmission line aerial images have become the main means at present.

[0005] However, there are two difficulties in using deep learning methods to classify and identify fittings in the power transmission line:

[0006] 1. The size and variety of fittings are numerous, so it is difficult for humans to quickly and accurately classify and detect targets in the drawings. This is a key factor affecting the accuracy of fitting defect detection. Therefore, in the fitting detection and maintenance, the model first has the ability to accurately identify the type of target.

[0007] 2. Most existing fitting classification detection methods still have the problem of insufficient recognition ability. There may be problems of not being able to accurately identify the fittings. However, each type of fitting in the transmission line has a corresponding detection procedure. If the target fitting is misidentified, it may lead to the problem of not finding potential defects in the subsequent process. Therefore, the classification and identification of fittings also need to introduce a model with stronger recognition ability to distinguish the type of fittings.

[0008] Therefore, in the above background, the knowledge related to the power field is introduced into the deep learning model to solve the problems existing in the current transmission line fitting recognition and classification, and to further improve the fitting type detection accuracy while meeting the industrialization demand becomes the top priority. SUMMARY

[0009] The purpose of the present application is to provide a fitting classification detection method based on an improved attention mechanism, to solve the problem of insufficient recognition ability of most existing fitting classification detection methods, and to design a new model to further improve the accuracy of fitting classification and identification.

[0010] To achieve the above purpose, the present application provides the following scheme:

[0011] A fitting classification detection method based on an improved attention mechanism, comprising the following steps:

[0012] A fitting dataset is constructed, which contains fitting images of different categories and different scales.

[0013] The constructed metal dataset is input into the improved attention mechanism fitting classification detection model to obtain the classification result of the fitting.

[0014] The improved attention mechanism fitting classification detection model includes an Attention-based Convolution (ABC) framework module, a Swin Transformer model and a second softmax function.

[0015] The ABC framework module includes two regions, a shallow convolution region and a moving window multi-head self-attention region. The shallow image convolution region is used to preliminarily extract and refine the features of the fitting image, and the moving window multi-head self-attention region is used to extract some regions containing rich fitting feature information.

[0016] The features obtained by inputting the original fitting image into the ABC framework module are spliced with the features output after inputting the original fitting image into the linear embedding layer of the SwinTransformer model in the channel dimension, and then the spliced result is input into the remaining part of the Swin Transformer model. The output end result of the Swin Transformer is input into the second softmax function to obtain the probability of each fitting, and the fitting type corresponding to the maximum probability is selected as the classification result of the fitting.

[0017] The ABC framework module is proposed to improve the performance of the fitting classification and recognition task, and the ABC framework is combined with the Swin Transformer to obtain better prediction performance.

[0018] The shallow convolution region of the ABC framework module contains eight convolution layers, uses a convolution kernel with a 3x3 receptive field, and the convolution step is fixed at 1 pixel; the spatial padding of the convolution layer input is to retain the same dimension size after convolution, and the padding of the 3x3 convolution layer is 1 pixel; spatial pooling is added after the 2nd and 5th convolution layers, and the spatial pooling is performed by two max pooling layers, and the max pooling layer is performed on a pixel window of 2x2 with a step of 2; all hidden layers are equipped with nonlinear correction function, and the moving window multi-head self-attention region contains multi-head self-attention mechanism and moving window mechanism.

[0019] First, the original fitting image is input into the convolution region in the ABC framework module, and then the output result of the convolution region in the ABC framework module is input into the moving window multi-head self-attention region; specifically, the original fitting image is taken as an input tensor X, and the dimension of the input tensor X is h x w x c, where h represents the height, w represents the width, and c represents the number of channels. The input of the ABC framework module is a fixed RGB image of 224x224 size, and the output X1 of the convolution layer is used as the input of the moving window multi-head self-attention mechanism. The input tensor X becomes X1 after passing through the convolution region, and the dimension is 56x56x16. The tensor X1 is input into the self-attention mechanism region to obtain tensor X2. In the self-attention mechanism region, an input tensor X1 is taken, and the dimension of X1 is h x w x c, where h represents the height, w represents the width, and c represents the number of channels. X1 is convolved into three independent tensors: Q with a dimension of h x w x c Q , K with a dimension of h x w x c K , and V with a dimension of h x w x c V , where c Q , c K , and c Vand represent the number of channels in the corresponding tensor; the purpose of self-attention is to calculate the weighted sum of these values ​​multiplied by V, where the weight is calculated as the similarity between the query Q and the corresponding key K. Therefore, in order to calculate the similarity, Q and K usually have the same shape, that is, c Q =c V , the output of a single self-attention mechanism is calculated as:

[0020]

[0021] where Q′, K′, and are the flattened tensors used for the dot product.

[0022] After scaling operation, the similarity matrix Q′·K′ T Divide by a coefficient And apply the first softmax function, perform a dot product between the normalized similarity matrix and V, and generate a matrix of dimensions h×w×c K Self-attention map S n , n is the number of attention probes in the multi-head attention mechanism, n i is the i-th multi-head self-attention probe.

[0023] Since the two-dimensional feature map is flattened into a one-dimensional vector in formula (1), the original structure of the original image tensor is destroyed. In order to improve the efficiency when processing structured data such as images and multi-dimensional features, relative position encoding is used. The relative position encoding is composed of l i,j It represents the relative position of Q' relative to K', where i, j represent the coordinates in the tensor Q'. The attention logic is calculated as follows:

[0024]

[0025] Among them, q i is the feature vector representing pixel Q′ i:=(i x ,i y )'s i-th row, k j is the feature vector j representing pixel K′:=(j x ,j y ), and They are the relative width of the code j x -i x and relative height j y -i y With the relative position encoding, the output of a single multi-head self-attention mechanism can be reformulated as

[0026]

[0027] in and is the matrix of relative positions, the number of attention probes in the multi-head attention mechanism can be adjusted according to actual conditions, for example, it can be set to 4.

[0028] A key design element of the ABC framework module is the transfer of window partitioning between consecutive self-attention layers. The shifted window connects with the window of the previous layer, providing connection information between them, greatly enhancing the modeling capability. The first module adopts the conventional window partitioning strategy starting from the top-left pixel. The 56x56 feature maps are evenly divided into 8x8 windows with a size of 7x7 (M=7). Then, the window is moved out from the window of the conventional partitioning strategy to offset with the window of the previous layer, and then the moved window is rearranged into 7x7 windows for self-attention mechanism calculation.

[0029] The calculation method of the entire multi-head self-attention region using the moving window partitioning method is as follows:

[0030]

[0031] wherein represents the output feature of the ABC framework module.

[0032] The Swin Transformer network is selected as the basic architecture, and the ABC framework module + SwinTransformer model is used as the backbone network, specifically including:

[0033] After inputting the tensor X of the original image of the fitting into the ABC framework module, the tensor X of the original image of the fitting is used again as the input of the Swin Transformer. First, the tensor X of the input sample image is processed by the patch partition layer of the Swin Transformer, and the dimension becomes 56x56x48. Then, it is mapped to the specified dimension X3 by the linear embedding layer of the Swin Transformer again. The dimension of X3 is 56x56x128. The role of the patch partition layer is to crop the input original image into 16 image blocks through the conv2d.

[0034] In the output of the self-attention mechanism in formula (4), that is, the output of the ABC framework module, the dimension of the output tensor X2 is 56x56x16, and the dimension of the output X3 of the linear embedding layer of the Swin Transformer is also 56x56x128, so the two tensors are spliced along the channel dimension, the feature tensor output by the ABC framework module is spliced with the feature tensor output by the linear embedding layer of the Swin Transformer in the channel dimension, to obtain Y with a dimension of 56x56x144, and then the spliced tensor Y is used to replace X3, which continues to be input to the network layer after the linear embedding layer in the Swin Transformer, and finally the probability of each fitting is obtained at the output end of the Swin Transformer through a second softmax function, and the fitting type corresponding to the maximum probability is selected as the final fitting classification result.

[0035] The application adds a new loss function to prevent redundancy caused by overlapping of the areas of attention of the moving window multi-head attention mechanism and improve the performance of the network.

[0036] In order to alleviate the overlapping problem, a diversity loss is introduced to learn different and non-overlapping areas, by minimizing the product of the corresponding outputs of the two attention probes, and The diversity loss is expressed as:

[0037]

[0038] Each probe in the moving window multi-head attention mechanism produces a S n (h', w'), where S n (h', w') represents the area of attention of the corresponding probe, (h', w') represents the position of the corresponding probe in the self-attention, n i is the i-th multi-head self-attention probe, S n (h', w') can be regarded as a weight matrix with a dimension of 56x56x16, in S n (h', w'), the richer the area with fitting feature information is, the larger the weight matrix corresponding to the area is, when the product of the S n (h', w') represented by two attention probes is 0, the areas of attention of the two attention probes do not overlap, when the overlap loss obtained by multiplying different probes with each other is 0, the areas of attention of different probes do not overlap, which prevents the redundancy of learning caused by multiple attention probes paying attention to the same area at the same time.

[0039] For the classification task, the cross entropy loss function (CE) is used, and its expression is as follows:

[0040] L CE = -∑ n q(n)logp(n) (6)

[0041] Wherein q(n) is the real probability distribution, p(n) is the predicted probability distribution, and n is the type of the tuning fork;

[0042] The total loss of training the network is the sum of the two loss functions, that is, the sum of the cross entropy loss function and the diversity learning loss function;

[0043] L = L CE + 10L Overlap (7)

[0044] During the network training process, the training of the network is guided by the loss function, and the loss function is mainly used in the training stage of the model. After each batch of training data is input into the model, the predicted value is output through forward propagation, then the loss function calculates the difference value between the predicted value and the true value, that is, the loss value. After obtaining the loss value, the model updates each parameter through back propagation to reduce the loss between the true value and the predicted value, so that the predicted value generated by the model approaches the true value, thereby achieving the purpose of learning. When the loss function is 0, at this time, it indicates that the output of the network is the ideal output.

[0045] The present application discloses the following technical effects:

[0046] The application provides a hardware fitting classification detection method based on an improved attention mechanism, selects a SwinTransformer model as a backbone network, and introduces an Attention-based Convolution (ABC) framework module which uses a shallow convolution and a moving window multi-head self-attention mechanism. The shallow image convolution is used for preliminary extraction and refinement of the image, and the multi-head self-attention mechanism is used for extracting some regions containing rich hardware fitting feature information; the features obtained by the ABC are spliced with the output of the linear embedding layer of the SwinTransformer in the channel dimension, and then input to the remaining part of the SwinTransformer, so that the model can better capture the global and local information of the image, and achieve better results by learning the relationship between different features of the image; a new loss function is introduced, so that each probe of the multi-head self-attention mechanism shows different hardware fitting feature regions, and prevents overlapping feature extraction of the multi-head feature extractor. The hardware fitting classification detection method based on the improved attention mechanism is applied to the detection of the types of hardware fittings, and the accuracy of the hardware fitting classification detection is improved by combining the SwinTransformer and the Attention-based Convolution (ABC) framework module. BRIEF DESCRIPTION OF DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0048] Figure 1 is a flowchart of a hardware fitting classification detection method based on an improved attention mechanism. DETAILED DESCRIPTION

[0049] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0050] The purpose of the present application is to provide a hardware fitting classification detection method based on an improved attention mechanism. The hardware fitting classification detection method based on the improved attention mechanism is applied to the detection of the types of hardware fittings, solves the problem of insufficient accuracy of hardware fitting classification, and achieves the purpose of improving the accuracy of hardware fitting classification detection while meeting the industrialization demand.

[0051] Specifically, the application discloses a fitting classification detection method based on an improved attention mechanism, an improved Swin Transformer architecture is selected as a backbone network; an Attention-based Convolution (ABC) framework module is introduced, a shallow convolutional network and a moving window multi-head attention mechanism are used to extract a region containing rich fitting feature information to improve the accuracy of target classification; wherein the improved Swin Transformer realizes two independent regions, namely the Attention-based Convolution and the Swin Transformer. The Attention-based Convolution uses a shallow convolutional network and a moving window multi-head attention mechanism to extract a region containing rich fitting specific information. Subsequently, the features obtained by the Attention-based Convolution are spliced with the original image processed by the linear embedding in the Swin Transformer, and then input into the Swin Transformer to identify the type of the fitting and more accurately identify the fitting. The Attention-based Convolution also introduces a diversity loss to guide the training of the self-attention mechanism, reduce the overlap between patches, and thus discover important patches of diversity, further improving the accuracy of fitting classification.

[0052] In order to make the above-mentioned purpose, features and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below in combination with the drawings and specific embodiments.

[0053] As shown in Figure 1 The application provides a fitting classification detection method based on an improved attention mechanism, which comprises the following steps:

[0054] A fitting dataset is constructed, and the fitting dataset contains fitting images of different categories and different scales.

[0055] The constructed metal dataset is input into the fitting classification detection model based on the improved attention mechanism to obtain the classification result of the fitting.

[0056] The fitting dataset is constructed, and the fitting dataset contains fitting images of different categories and different scales.

[0057] Deep learning models require a large number of dataset image samples as support during training. Since most of the images collected by unmanned aerial vehicles are global images of power transmission lines, they need to be cropped according to the input image size of the model. The region containing the fitting in the aerial image of the power transmission line and the tower is cropped, cleaned, and the images with clear images, more types and numbers of fittings are selected. The dataset is labeled according to the COCO dataset format, and the category of each target fitting is labeled on this basis.

[0058] In the present application, in order to solve the problem of low accuracy in the classification identification of fittings in power transmission lines, it is necessary to introduce relevant knowledge in the power field into the deep learning model, and at the same time, it is necessary to have higher precision in the classification and detection of fittings. The present application selects the Swin Transformer network as the basic architecture, and uses the ABC+Swin Transformer model as the backbone network to improve the accuracy of the fitting classification detector. Among them, the Swin Transformer model is selected as the backbone network, a new module named ABC is introduced, and the fitting classification detection model improved by the attention mechanism includes an Attention-based Convolution (ABC) framework module, a Swin Transformer model and a second softmax function.

[0059] Specifically, the metal data set constructed by the present application is input into the untrained fitting classification detection model improved by the attention mechanism, the output end result of the model is input into the softmax function to obtain the probability of each fitting, the fitting type corresponding to the maximum probability is selected as the classification result of the fitting to train the model, and a trained fitting classification detection model, i.e. the fitting classification detection model improved by the attention mechanism, is obtained. The original fitting picture to be detected is input into the trained fitting classification detection model, and the fitting type corresponding to the maximum probability in the output result of the model is selected as the classification result of the fitting.

[0060] The ABC framework module includes two regions, a shallow convolution region and a moving window multi-head self-attention region. The shallow image convolution region is used to preliminarily extract and refine the features of the fitting image, and the moving window multi-head self-attention region is used to extract some regions containing rich fitting feature information.

[0061] The features obtained by inputting the original fitting image into the ABC framework module are spliced with the features output after inputting the original fitting image into the linear embedding layer of the Swin Transformer model in the channel dimension, and then the spliced result is input into the remaining part of the Swin Transformer model. The output end result of the Swin Transformer is input into the second softmax function to obtain the probability of each fitting, and the fitting type corresponding to the maximum probability is selected as the classification result of the fitting.

[0062] The present application proposes an ABC framework module to improve the performance based on the fitting classification identification task, and combines the ABC framework with the Swin Transformer to obtain better prediction performance, as follows:

[0063] The shallow convolution region of the ABC framework module includes eight convolution layers, uses a convolution kernel with a very small 3*3 receptive field, and the convolution step is fixed to 1 pixel; the spatial padding of the convolution layer input is to keep the same dimension size after convolution, and the padding of the 3*3 convolution layer is 1 pixel; spatial pooling is added after the 2nd and 5th convolution layers, and the spatial pooling is performed by two maximum pooling layers, the maximum pooling layer is performed on a 2*2 pixel window with a step of 2; all hidden layers are equipped with nonlinear correction function, and the moving window multi-head self-attention region includes multi-head self-attention mechanism and moving window mechanism.

[0064] First, the original fitting image is input into the convolution region in the ABC framework module, and then the output result of the convolution region in the ABC framework module is input into the moving window multi-head self-attention region; specifically, the fitting original image is taken as an input tensor X, and the dimension of the input tensor X is h*w*c, wherein h represents height, w represents width, and c represents the number of channels; the input of the ABC framework module is a fixed 224*224 size RGB image, the output X1 of the convolution layer is used as the input of the moving window multi-head self-attention mechanism, the input tensor X becomes X1 after passing through the convolution region, and the dimension is 56*56*16; the tensor X1 is input into the region of the self-attention mechanism to obtain the tensor X2, in the region of the self-attention mechanism, an input tensor X1 is taken, the dimension of which is h*w*c, wherein h represents height, w represents width, and c represents the number of channels; X1 is convolved into three independent tensors: Q with the dimension of h*w*c Q , K with the dimension of h*w*c K , and V with the dimension of h*w*c V , wherein c Q , c K , and c V represent the number of channels in the corresponding tensor; the purpose of the self-attention is to calculate the weighted sum of these values multiplied by V, wherein the weight is calculated as the similarity between the query Q and the corresponding key K, therefore, in order to calculate the similarity, Q and K usually have the same shape, i.e. c Q =c V , the output of the single self-attention mechanism is calculated as:

[0065]

[0066] wherein Q', K' and are flattened tensors used for dot multiplication.

[0067] After the scaling operation, i.e. dividing the similarity matrix Q'·K' T by a coefficient and applying the first softmax function, the dot product between the normalized similarity matrix and V is performed to generate the tensor with the dimension of h*w*cK Self-attention map S n , n is the number of attention probes in the multi-head attention mechanism, n i is the i-th multi-head self-attention probe.

[0068] Since the two-dimensional feature map is flattened into a one-dimensional vector in formula (1), the original structure of the original image tensor is destroyed. In order to improve the efficiency when processing structured data such as images and multi-dimensional features, relative position encoding is used. The relative position encoding is composed of l i,j It represents the relative position of Q' relative to K', where i, j represent the coordinates in the tensor Q'. The attention logic is calculated as follows:

[0069]

[0070] Among them, q i is the feature vector i representing pixel Q′: x ,i y )'s i-th row, k j is the feature vector j representing pixel K′:=(j x ,j y ), and They are the relative width of the code j x -i x and relative height j y -i y With the relative position encoding, the output of a single multi-head self-attention mechanism can be reformulated as

[0071]

[0072] in and It is a matrix of relative positions. The number of attention probes in the multi-head attention mechanism can be adjusted according to actual conditions, for example, it can be set to 4.

[0073] A key design element of the ABC framework module is its transfer of window partitions between consecutive self-attention layers. The shifted window is connected to the window of the previous layer, providing the connection information between them and greatly enhancing the modeling ability. The first module adopts a conventional window partitioning strategy starting from the upper left corner pixel. The 56×56 feature map is evenly divided into 8×8 windows of size 7×7 (M=7). Then, the window is shifted from the window of the conventional partition to the upper left corner. This is done to offset the window of the previous layer, and then rearrange the moved windows into 7×7 windows before performing the calculation of the self-attention mechanism.

[0074] The calculation method of the whole multi-head self-attention region is as follows by using the mobile window partition method:

[0075]

[0076] wherein represents the output feature of the ABC framework module.

[0077] The Swin Transformer network is selected as the basic architecture, and the ABC framework module + SwinTransformer model is used as the backbone network, and specifically includes:

[0078] The features obtained by the ABC are spliced with the output of the linear embedding layer of the Swin Transformer in the channel dimension, and then input to the remaining part of the Swin Transformer, so that the model can better capture the global and local information of the image, and achieve better results by learning the relationship between different features of the image.

[0079] After inputting the tensor X of the original image of the fitting to the ABC framework module, the tensor X of the original image of the fitting is used again as the input of the Swin Transformer. First, the tensor X of the input sample image is processed by the patch partition layer of the Swin Transformer, and the dimension becomes 56x56x48. Then, the tensor X is mapped to the specified dimension X3 by the linear embedding layer of the Swin Transformer again, and the dimension of X3 is 56x56x128. The function of the patch partition layer is to crop the input original image into 16 image blocks by using the conv2d.

[0080] In the output of the self-attention mechanism in formula (4), that is, the output of the ABC framework module, the dimension of the output tensor X2 is 56x56x16, and the dimension of the output X3 of the linear embedding layer of the Swin Transformer is also 56x56x128, so the two tensors are spliced along the channel dimension, the feature tensor output by the ABC framework module is spliced with the feature tensor output by the linear embedding layer of the Swin Transformer in the channel dimension, to obtain Y with a dimension of 56x56x144, and then the spliced tensor Y is used to replace X3, which continues to be input to the network layer after the linear embedding layer in the Swin Transformer, and finally the probability of each fitting is obtained at the output end of the Swin Transformer through a second softmax function, and the fitting type corresponding to the maximum probability is selected as the final fitting classification result.

[0081] The application adds a new loss function to prevent redundancy caused by overlapping of the areas of attention of the moving window multi-head attention mechanism and improve the performance of the network. Since the number of areas containing fitting information in the ABC framework module is determined by the number of probes in the moving window multi-head self-attention in the ABC framework module, it is found in the implementation process that the areas of attention of the multi-head probes have a tendency to overlap, especially in areas rich in feature information. Such overlapping of the areas of attention may cause redundant learning sources, so that other areas containing specific fitting information cannot be discovered. In order to alleviate this overlapping problem, a diversity loss is introduced to learn different and non-overlapping areas, by minimizing the product of the corresponding outputs of two attention probes, and The diversity loss is expressed as:

[0082]

[0083] Each probe in the moving window multi-head attention mechanism produces a S n (h′,w′), where S n (h′,w′) represents the area of attention of the corresponding probe, (h′,w′) represents the position of the corresponding probe in the self-attention, n i is the i-th multi-head self-attention probe, S n (h′,w′) can be regarded as a weight matrix with a dimension of 56x56x16, in S n (h′,w′), the richer the area with fitting feature information, the larger the weight matrix corresponding to the area, and when the S nWhen the result of the multiplication of (h', w') is 0, the areas of interest of the two attention probes do not overlap, and when the overlap loss obtained by multiplying different probes with each other is 0, the areas of interest of different probes do not overlap, which prevents the learning redundancy caused by multiple attention probes paying attention to the same area at the same time.

[0084] For the classification task, the cross entropy loss function (CE) is used, and its expression is as follows:

[0085] L CE = -∑ n q(n)logp(n) (6)

[0086] where q(n) is the real probability distribution, p(n) is the predicted probability distribution, and n is the type of the inductor;

[0087] The total loss of training the network is the sum of the two loss functions, that is, the sum of the cross entropy loss function and the diversity learning loss function;

[0088] L = L CE + 10L Overlap (7)

[0089] During the network training process, the training of the network is guided by the loss function, and the loss function is mainly used in the training stage of the model. After each batch of training data is input into the model, the predicted value is output through forward propagation, and then the loss function calculates the difference value between the predicted value and the real value, that is, the loss value. After obtaining the loss value, the model updates the parameters through back propagation to reduce the loss between the real value and the predicted value, so that the predicted value generated by the model approaches the real value, thereby achieving the purpose of learning. When the loss function is 0, at this time, the output of the network is the ideal output.

[0090] The application proposes an improved attention mechanism hardware classification detection method, that is, an improved SwinTransformer model. The ABC framework is used to improve the performance of the hardware classification task, and the ABC is combined with the Swin Transformer to obtain better prediction performance. Our framework combines shallow convolution and multi-head attention mechanism, and uses a moving window method. The shallow convolution uses a few convolution kernels of several convolution networks to condense information, enhances the features of the image, and processes the input information into the same size, so as to splice with the information of the subsequent attention network calculation and Swin transformer. The multi-head attention mechanism enables the network to learn and find the area containing rich hardware feature information, and displays these areas. Finally, the area rich in hardware information features obtained by the ABC framework is spliced along the channel dimension with the image preliminarily processed by the linearembedding layer in the Swin Transformer, and the spliced image tensor is input to the subsequent network of the SwinTransformer, and the important hardware feature area obtained by the ABC is spliced with the original input image, and then the Swin Transformer is used to perform the hardware classification task, which makes good use of the ability of the Swin Transformer to mine long-distance dependencies and parallel computing to learn more dependencies between important hardware features. The addition of the ABC framework well compensates for the local sensitivity and translation invariance of the Swin Transformer. The ABC framework splices the important area containing rich hardware feature information into the original image, which can fully mobilize the long dependence of the Swin Transformer, that is, to extract stronger features by learning the dependencies between different features. Therefore, the entire network can not only extract important hardware feature information areas, but also use the ability of the Swin Transformer to learn the mutual relationship between features to further improve the prediction accuracy. It can be seen that the method described in the application effectively improves the recognition rate of hardware classification recognition and meets the needs of actual industry.

[0091] The principles and implementation modes of the application are described by using specific examples in the present application, and the above examples are only used to help understand the method of the application and its core idea; at the same time, for those skilled in the art, according to the idea of the application, the specific implementation mode and application range will be changed. In view of the above, the content of the specification should not be understood as a limitation of the application.

Claims

1. A method for detecting classification of hardware based on improved attention mechanism, characterized in that, The method comprises: A metal fitting dataset is constructed, and the metal fitting dataset contains metal fitting images of different categories and different scales; The constructed metal fitting dataset is input into a metal fitting classification detection model with an improved attention mechanism to obtain a classification result of the metal fitting; The metal fitting classification detection model with the improved attention mechanism comprises a convolutional ABC framework module based on self-attention, a Swin Transformer model, and a second softmax function; The ABC framework module comprises two regions, a shallow convolution region and a moving window multi-head self-attention region, the shallow image convolution region is used to preliminarily extract and refine features of the metal fitting image, and the moving window multi-head self-attention region is used to extract a region containing rich metal fitting feature information; The features obtained by inputting the original metal fitting image into the ABC framework module are spliced with the features output after inputting the original metal fitting image into a linear embedding layer of the Swin Transformer model in a channel dimension, and then the spliced result is input into the remaining part of the Swin Transformer model, the output end result of the Swin Transformer is input into the second softmax function to obtain a probability of each metal fitting, and a metal fitting type corresponding to the maximum probability is selected as the classification result of the metal fitting.

2. The method of claim 1, wherein the method is characterized by, The ABC framework module is proposed to improve the performance of the metal fitting classification and recognition task, and the ABC framework is combined with the Swin Transformer to obtain better prediction performance; The shallow convolution region of the ABC framework module comprises eight convolution layers, a convolution kernel with a 3*3 receptive field is used, and a convolution step is fixed as 1 pixel; spatial padding of the convolution layer input is used to keep the same dimension size after convolution, the padding of the 3*3 convolution layer is 1 pixel; spatial pooling is added after the 2nd and 5th convolution layers, the spatial pooling is performed by two maximum pooling layers, the maximum pooling layer is performed on a 2*2 pixel window with a step of 2; all the hidden layers are equipped with a nonlinear correction function, and the moving window multi-head self-attention region comprises a multi-head self-attention mechanism and a moving window mechanism; First, the original fitting image is input to the convolution region in the ABC framework module, and then the output result of the convolution region in the ABC framework module is input to the moving window multi-head self-attention region; specifically, taking the fitting original image as an input tensor X, assuming that the dimension of the input tensor X is h x w x c, wherein h represents height, w represents width, and c represents the number of channels, the input of the ABC framework module is a fixed 224 x 224 size RGB image, the output X1 of the convolution layer is used as the input of the moving window multi-head self-attention mechanism, the input tensor X becomes X1 after passing through the convolution region, and the dimension is 56 x 56 x 16, and then the tensor X1 is input to the region of the self-attention mechanism to obtain the tensor X2, in the region of the self-attention mechanism, assuming that an input tensor X1 has a dimension of h x w x c, wherein h represents height, w represents width, and c represents the number of channels, X1 is convolved into three independent tensors: Q with a dimension of h x w x c Q , K with a dimension of h x w x c K , and V with a dimension of h x w x c V , wherein c Q , c K , and c V represent the number of channels in the corresponding tensor; the purpose of self-attention is to calculate the weighted sum of these values multiplied by V, wherein the weight is calculated as the similarity between the query Q and the corresponding key K, therefore, in order to calculate the similarity, Q and K usually have the same shape, i.e., c Q =c V , and the output of a single self-attention mechanism is calculated as: Q', K' and V' are flattened tensors used for point multiplication; After the scaling operation, i.e. dividing the similarity matrix Q' · K' by a coefficient T and applying the first softmax function, the dot product between the normalized similarity matrix and V is generated, resulting in a self-attention map S of dimension h x w x c K n n is the number of attention heads in the multi-head attention mechanism, n i is the i-th multi-head self-attention head;​​ Since the two-dimensional feature map is flattened into a one-dimensional vector in formula (1), the original structure of the original image tensor is destroyed, in order to improve the efficiency when processing images and multi-dimensional feature structured data, relative position encoding is adopted, which is represented by l i,j , which represents the relative position of Q' with respect to K', where i, j represents the coordinates in the tensor Q', and the attention logic is calculated as follows: where q i is the i-th row of the feature vector i:=(i x ,i y ) of pixel Q', k j is the j-th row of the feature vector j:=(j x ,j y ) of pixel K', and and are learnable parameters that encode the position information within the relative width j x -i x and relative height j y -i y respectively. With relative position encoding, the output of a single multi-head self-attention mechanism can be reformulated as wherein and is a matrix of relative positions, the number of attention probes in the multi-head attention mechanism can be adjusted according to actual conditions; A key design element of the ABC framework module is the shift of window partition between successive self-attention layers, the shifted window is concatenated with the window of the previous layer to provide the connection information between them, greatly enhancing the modeling capability, the first module adopts the conventional window partition strategy starting from the top-left pixel, 56x56 feature maps are evenly divided into 8x8 windows with a size of 7x7 The window is shifted from the window of the conventional partition to the top, and then the shifted window is rearranged into 7x7 windows for self-attention mechanism calculation; The calculation method of the whole multi-head self-attention region is as follows by using the moving window partition method: wherein represents the output features of the ABC framework module.

3. The method of claim 2, wherein the method comprises: The Swin Transformer network is selected as the basic architecture, the ABC framework module + Swin Transformer model is used as the backbone network, and the specific process comprises: After inputting the tensor X of the original image of the fitting into the ABC framework module, the tensor X of the original image of the fitting is used as the input of the Swin Transformer again. First, the input sample image tensor X is input into the patch partition layer of the Swin Transformer, and the dimension becomes 56x56x48. Then, the tensor X is mapped to the specified dimension X3 by the linear embedding layer of the Swin Transformer again, and the dimension of X3 is 56x56x128. The patch partition layer is used to crop the input original image into 16 image blocks by the conv2d. In the output of the self-attention mechanism in formula (4), that is, the output of the ABC framework module, the dimension of the output tensor X2 is 56x56x1, and the dimension of the output X3 of the linear embedding layer of the Swin Transformer is also 56x56x128. Therefore, the two tensors are spliced along the channel dimension, and the feature tensors output by the ABC framework module and the linear embedding layer of the Swin Transformer are spliced in the channel dimension to obtain Y with a dimension of 56x56x144. Then, the spliced tensor Y is used to replace X3, and the network layer after the linear embedding layer of the Swin Transformer is continued to be input. Finally, the probability of each fitting is obtained at the output end of the Swin Transformer by the second softmax function, and the fitting type corresponding to the maximum probability is selected as the final fitting classification result.

4. The method of claim 1, wherein the method is characterized by, A new loss function is added to prevent redundancy caused by the overlapping of the areas focused by the moving window multi-head attention mechanism and to improve the performance of the network. To alleviate the overlap problem, a diversity loss is introduced to learn different and non-overlapping regions by minimizing the product of the corresponding outputs in two attention probes, and The diversity loss is formulated as: Each probe in the mobile window multi-head attention mechanism will generate an S n (h', w'), where S n (h', w') represents the region of interest of the corresponding probe, (h', w') represents the position of the corresponding probe in the self-attention, n i is the i-th multi-head self-attention probe, S n (h', w') can be regarded as a weight matrix with a dimension of 56x56x16, and in S n (h', w'), the richer the region with the information of the feature of the gold-plated object, the larger the weight matrix corresponding to the region, and when the product of S n (h', w') of two attention probes is 0, the regions of interest of the two attention probes do not overlap, and when the overlap loss obtained by multiplying different probes with each other is 0, the regions of interest of different probes do not overlap, which prevents the redundancy of learning caused by multiple attention probes paying attention to the same region at the same time. The cross-entropy loss function is used for classification tasks, and its expression is as follows: L CE = -∑ n q(n) log p(n) (6) where q(n) is the real probability distribution, p(n) is the predicted probability distribution, and n is the type of fitting. The total loss of training the network is the sum of the two loss functions, that is, the sum of the cross-entropy loss function and the diversity learning loss function. L = L CE +10L Overlap (7) During the network training process, the loss function is used to guide the training of the network. The loss function is mainly used in the training stage of the model. After each batch of training data is input into the model, the predicted value is output through forward propagation. Then the loss function calculates the difference between the predicted value and the true value, that is, the loss value. After obtaining the loss value, the model updates the parameters through back propagation to reduce the loss between the true value and the predicted value, so that the predicted value generated by the model approaches the true value, thereby achieving the purpose of learning. When the loss function is 0, the output of the network is the ideal output at this time.

Images