Belt damage classification method based on multi-window fusion self-attention model

By building a multi-window fusion self-attention network, the local window information and semantic features of the belt laser image are extracted, the noise interference false alarm problem is solved, efficient belt damage detection is achieved, and detection accuracy and efficiency are improved.

CN118918357AActive Publication Date: 2024-11-08SHENYANG AEROSPACE UNIVERSITY
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202410918335.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-10
Publication Date
2024-11-08
Estimated Expiration
2044-07-10

AI Technical Summary

Technical Problem

There are noise interference and false alarm problems in the existing belt damage detection technology, resulting in frequent shutdown and inspection, affecting the normal operation of material transmission.

Method used

Using the belt injury classification method based on the multi-window fusion self-attention model, the local window information and semantic features of the belt laser image are extracted, the information is fused and feature maps are generated. The probability of damage or interference between the softmax activation layer and the fully connected layer output.

Benefits of technology

It effectively reduces the amount of calculation, improves the performance of belt damage detection, can accurately distinguish between real damage and interference, and improves the accuracy and efficiency of detection.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918357A_ABST
    Figure CN118918357A_ABST
Patent Text Reader

Abstract

The invention discloses a belt damage classification method based on a multi-window fusion self-attention model, and the method comprises the following steps: constructing a network structure based on multi-window fusion self-attention, and the network structure based on multi-window fusion self-attention is composed of a main channel at the upper side and a branch channel at the lower side; inputting a feature map of a belt laser image into the network structure, respectively extracting local window information of the image and cross-window information of different levels of semantic features through a main path and a branch path, and fusing the extracted information in the main path to generate a feature map with a specific size; according to the belt damage classification method based on the multi-window fusion self-attention model provided by the invention, local window information and cross-window information of different levels of semantic features are respectively extracted through two paths based on a multi-window fusion self-attention network, and the extracted information is fused at a trunk; therefore, the network can effectively utilize the image information and can greatly reduce the calculation amount.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a belt damage classification method based on a multi-window fusion self-attention model. Background Art

[0002] In the metallurgical, mining and other industrial fields, belt conveyors are a common method of transporting materials. Damage to the belt, such as scratches and tears, will seriously affect the normal transmission of the belt. If the belt damage is not handled in time, in serious cases, the entire belt may be scrapped. The cost of a transmission belt ranges from hundreds of thousands to millions or even tens of millions. Scrapping will bring huge economic losses to the company. Therefore, it is increasingly important to detect belt damage during operation.

[0003] With the development of artificial intelligence technology, image-based belt damage detection technology has developed rapidly. During belt damage detection, the detector emits laser stripes to continuously scan the bottom of the transmission belt to generate a belt profile image, and then uses the image to determine whether the belt is damaged. However, in order to obtain higher detection sensitivity, some noise is often detected as damage. Common noise may be caused by dust, uneven lighting, etc. If these noises are regarded as real belt damage, frequent shutdown inspection operations will seriously affect the normal transmission of materials.

[0004] Therefore, proposing a method to effectively remove noise interference and false alarms in the inspection results is a problem that needs to be solved in belt damage detection. Summary of the invention

[0005] In view of this, the present invention provides a belt damage classification method based on a multi-window fusion self-attention model, and applies the self-attention network based on multi-window fusion to the damage classification of laser belt images.

[0006] The technical solution provided by the present invention is as follows: a belt damage classification method based on a multi-window fusion self-attention model, comprising the following steps:

[0007] S1: Extract feature map of belt laser image;

[0008] S2: construct a network structure based on multi-window fusion self-attention, wherein the network structure based on multi-window fusion self-attention consists of a trunk path on the upper side and a branch path on the lower side; the trunk path and the branch path each include four states, and each state of the branch path consists of an image splitting module, a self-attention module, and a window combination module; each state of the upper path consists of an image splitting module, a self-attention module, a window combination module, and a window fusion module;

[0009] S3: Input the feature map of the belt laser image into the network structure, extract the local window information of the image and the cross-window information of semantic features of different levels through the trunk path and the branch path respectively, and fuse the extracted information in the trunk path, and generate a feature map of a specific size after four feature information extractions;

[0010] S4: Use the average pooling operation to convert the feature map into a feature vector, and the feature vector is input into a fully connected layer with two hidden nodes;

[0011] S5: The softmax activation layer is connected to the fully connected layer to output the feature map of the belt laser image as the probability of real damage or interference.

[0012] Preferably, in S3, the feature map of the belt laser image is input into the network structure, including:

[0013] S31: The image splitting modules in the trunk pathway and the branch pathway respectively split the specific size into continuous non-overlapping window images;

[0014] S32: The window image of the branch pathway is input into the self-attention module connected thereto to calculate the multi-head self-attention based on the window, and is also input into the trunk pathway, and is input into the fusion module together with the window image extracted by the trunk pathway to realize the fusion of the local window information and the cross-window feature information of different levels of semantics, and the fused features are then input into the self-attention module of the trunk pathway;

[0015] S33: The outputs of the self-attention modules in the trunk and branch pathways are input into the window combination modules connected thereto, and the feature maps f1 and f2 of different states of the trunk and branch networks are adaptively fused;

[0016] As the network deepens, the scale of the feature map input to each state continues to decrease, and the receptive field represented by the window of the same size continues to increase. The receptive field of the window image in the i+1th state is 4 times that of the window image in the i-th state.

[0017] Preferably, the self-attention module is composed of a window attention module and a multi-layer perceptron with a residual structure, and the image is first regularized before being input into the self-attention and multi-layer perceptron modules;

[0018] Use Relu activation to regularize it: where SoftPlus(x) = log e (1+e x ), which is the difference between Q and K T Normalize the multiplication of ;

[0019] In the self-attention module, the feature map is evenly divided into multiple windows, each window is represented by a column vector token, and multi-head attention calculation is performed within the window;

[0020] In the self-attention module of the branch pathway, only the multi-head self-attention of the feature map from the branch pathway is calculated; in the self-attention module of the trunk pathway, the multi-head self-attention of the fused feature map of the trunk and branch pathways needs to be calculated.

[0021] Preferably, the self-attention module in S32 calculates the window-based multi-head self-attention, including:

[0022] S321: Perform linear transformation on the input image to generate Q, K and V vectors, where Q represents query, K represents key, and V represents value;

[0023] S322: Divide the result after linear transformation into h heads, each head learns different attention weights; for four different states, the number of multi-head attentions is 3, 6, 12 and 24 respectively;

[0024] S323: Calculate the self-attention of each head, as shown in formula (1).

[0025]

[0026] d is the scale factor;

[0027] S324: The output of multiple heads after self-attention calculation, through the learnable linear transformation weight matrix W O Merge into the final multi-head attention output, as expressed in formula (2).

[0028] MultiHead(Q,K,V)=Concat(head1,...,head h )W O (2).

[0029] Preferably, in S33, the feature graphs f1 and f2 of different states of the trunk and branch networks are adaptively fused through a window combination module; including:

[0030] S331: input the feature maps f1 and f2 to the grouped convolution module respectively, in which all feature maps at the same position are grouped into the same group, and the feature maps f1 and f2 are grouped and convolved according to the grouping results;

[0031] S332: After the group convolution, the feature map of the branch path is point-convolved so that the number of its channels is consistent with the number of channels of the trunk;

[0032] S333: Use global average pooling operation to extract global spatial information S1 and S2 of the trunk and branch pathway feature maps;

[0033] S334: Use the softmax activation function to extract the attention coefficient of the feature map. The fused feature map F is expressed as:

[0034] F=Soft(Ave(Conv 分组 (f1)))*Conv 分组 (f1)+Soft(Ave(Conv 分组 (f1)))*Conv 点 (Conv 分组 (f2))(3).

[0037] The present invention provides a belt damage classification method based on a multi-window fusion self-attention model. The method constructs a self-attention network based on multi-window fusion and applies it to damage classification of laser belt images. The self-attention network based on multi-window fusion extracts local window information and cross-window information of semantic features of different levels through two pathways, and fuses the extracted information in the backbone, so that the network can effectively utilize image information and greatly reduce the amount of calculation. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the description of the embodiments of the present invention or the prior art. Obviously, the drawings described below are only part of the embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor.

[0039] Figure 1 The overall structure diagram of the belt damage classification model based on the multi-window fusion self-attention model provided by the present invention;

[0040] Figure 2 A structural diagram of the Transformer module provided by the present invention;

[0041] Figure 3 This is a structural diagram of the fusion module provided by the present invention. DETAILED DESCRIPTION

[0042] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present invention. Instead, they are merely examples of systems consistent with some aspects of the present invention as detailed in the appended claims.

[0043] The Transformer model is a deep learning model based entirely on the self-attention mechanism and has been widely used in the field of image processing. Among them, the ViT (Vision transformer) model has significantly improved performance in image recognition tasks. ViT divides the image into small blocks of 16×16 pixels, each block is regarded as a "word", represented by a 1×1 vector, and then the processed image blocks are input into an encoder composed of a multi-layer Transformer model. However, the laser contour image has a high resolution and a large image size, which greatly increases the computational complexity of the ViT model. Therefore, some scholars have proposed a window-based ViT method. This method divides a large-scale image into multiple windows and calculates self-attention within each window. However, this method ignores the semantic relationship between different windows.

[0044] To this end, the present invention provides a belt damage classification method based on a multi-window fusion self-attention model, comprising the following steps:

[0045] S1: Extract feature map of belt laser image;

[0046] Specifically, before the input image enters the network path, a convolution operation with a kernel size of 3×3 and a channel number of 48 is used to extract the feature map of the belt laser image with a size of 1280×400×3. The size of the feature map is 640×200×48, and then the feature map is input into the state of the network structure;

[0047] S2: Construct a network structure based on multi-window fusion self-attention. The network structure is as follows Figure 1 As shown in the figure. The network consists of two pathways: the upper side (trunk) and the lower side (branch). Both pathways include 4 states. Each state of the lower pathway consists of three modules: image splitting, self-attention module, and window combination. In addition to the above three modules, each state of the upper pathway also includes a window fusion module.

[0048] Specifically, the sizes of the input images of the four states are 640×200×48, 320×100×96, 160×50×192 and 80×25×384, respectively, and the self-attention module is repeated 2, 2, 6 and 2 times in the four states to extract more useful image information.

[0049] S3: In both the upper and lower paths, the image splitting module divides the image of size m×n×k into continuous non-overlapping (m / 8)×(n / 8)×k window images of size 8×8 pixels. Afterwards, the window image of the lower path is input into the self-attention module connected to it to calculate the window-based multi-head self-attention, and is also input into the upper path. Together with the window image extracted by the upper path, it is input into the fusion module to achieve the fusion of local window information and different levels of semantic cross-window feature information. The fused features are input into the self-attention module of the upper path. In both the upper and lower paths, the output of the self-attention module is input into the window combination module connected to it. This module first combines the 8×8 pixel windows into a feature map of size m×n×k, and then generates a feature map of size (m / 2)×(n / 2)×(2×k) through a convolution operation with a stride of 2, a number of channels of 2k, and a kernel size of 3×3.

[0050] As the network deepens, the scale of the feature map input to each state continues to decrease, but the receptive field represented by the window of the same size continues to increase. The receptive field of the window image in the i+1th state is 4 times that of the window image in the i-th state. Therefore, fusing the window feature maps of the i-th state of the upper network and the i+1th state of the lower network in the upper network enables the network to learn the local information of the current window in the i-th state, and can also learn the high-semantic cross-window information of the window where the current window of the i-th state is located and the three adjacent windows through the i+1th state window image. Accordingly, the feature fusion module fuses feature images of different states, allowing the network to learn local information within the window and high-semantic cross-window information more quickly and effectively.

[0051] After four times of feature information extraction, a feature map of size 8×8×768 can be generated based on the multi-window fusion self-attention network.

[0052] S4: The average pooling operation is used to transform the feature map into a 768×1 feature vector, which is input into a fully connected layer with two hidden nodes. Finally, the softmax activation layer is connected to the fully connected layer to output the probability that the image input to the network is real damage or interference.

[0053] S5: The softmax activation layer is connected to the fully connected layer to output the feature map of the belt laser image as the probability of real damage or interference.

[0054] Regarding the above self-attention module, Figure 2This is the structural diagram of the self-attention module. The module is mainly composed of a window attention module with a residual structure and a multi-layer perceptron. The image is first regularized before being input into the self-attention and multi-layer perceptron modules to avoid overfitting. In the self-attention module, the feature map is evenly divided into multiple 8×8 windows, each window is represented by a 1×64 column vector token, and multi-head attention calculation is performed within the window.

[0055] In the self-attention module of the lower path, only the multi-head self-attention of the feature map from the lower path is calculated. In the self-attention module of the upper path, the multi-head self-attention of the fused feature map from the upper and lower paths needs to be calculated. Although the window-based multi-head attention can effectively reduce the computational complexity of feature extraction, it introduces the problem of insufficient attention to cross-window information. However, the present invention solves this problem by fusing features with different receptive fields and different semantic levels to achieve learning of cross-window information.

[0056] When calculating window-based self-attention, the input image is first linearly transformed to generate query (Query, Q), key (Key, K) and value (Value, V) vectors. The result of the linear transformation is then divided into h heads, each of which learns different attention weights. For the four different stages, the number of multi-head attentions is 3, 6, 12 and 24 respectively. Finally, the self-attention of each head is calculated, as shown in formula (1).

[0057]

[0058] d is a scaling factor to prevent the inner product of the transposed Q and K from being too large. Since Q, K, and V come from feature images at different semantic levels when calculating the self-attention of the upper path, Relu activation is used to regularize them first. e (1+e x ), which is the difference between Q and K T Normalize the product of . DWC is a depth-wise convolution, which separates the position encoding from the self-attention calculation to achieve stronger local inductive bias. The output of multiple heads after self-attention calculation is transformed through a learnable linear transformation weight matrix W O Merge into the final multi-head attention output, as expressed in formula (2).

[0059] MultiHead(Q,K,V)=Concat(head1,...,head h )W O (2).

[0060] In addition, regarding the above-mentioned window combination module, Table 1 shows the relationship between the window receptive field size and its number at different stages. It can be seen that since the window receptive fields are different at different stages, learning the window self-attention of the feature images at different stages can not only obtain the local information inside the window, but also obtain the cross-window information of the window and its adjacent windows by learning the feature images of higher stages and larger receptive fields. Therefore, the method of the present invention fuses the feature images of different stages and effectively utilizes global and local information to improve the performance of belt damage detection in laser images.

[0061] Table 1 Relationship between the size of window receptive field and the number of windows at different stages

[0062] Stage Receptive field size Number of windows 1 (m,n) k 2 (2m,2n) 2k 3 (4m,4n) 4k 4 (8m,8n) 8k

[0063] The present invention adaptively fuses the feature graphs f1 and f2 of different states of the trunk and branch networks. The fusion module structure is as follows: Figure 3 As shown. The feature maps f1 and f2 have the same size, different number of channels, and the feature maps are composed of all window images at different positions. The feature maps f1 and f2 are first input into the grouped convolution module respectively, in which all feature maps at the same position are divided into the same group, and the feature maps f1 and f2 are grouped and convolved according to the grouping results. After the grouped convolution, the feature map of the branch path is point-convolved to make its number of channels consistent with the number of channels through the trunk. Then, the global average pooling operation is used to extract the global spatial information S1 and S2 of the trunk and branch feature maps. Next, the softmax activation function is used to extract the attention coefficient of the feature map. Finally, the fused feature map F can be expressed as:

[0064] F=Soft(Ave(Conv 分组 (f1)))*Conv 分组 (f1)+Soft(Ave(Conv 分组 (f1)))*Conv 点 (Conv 分组 (f2))(3).

[0066] The above method can be used to classify it into two categories: real damage and interference. In order to verify the effectiveness of the method of the present invention, the embodiment of the present invention classifies 5336 belt defect laser images into two categories: defects and non-defects. The 5-fold method is used to verify the performance of the algorithm, that is, 4 sets of data are used for training and 1 set of testing each time. Repeat the experiment 5 times until all the data have been tested. In the embodiment of the present invention, the average accuracy of all 5336 test data is 0.910. It can be seen that the method proposed in the present invention can effectively classify belt defect images and remove the influence of non-real defects.

[0067] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. A belt damage classification method based on multi-window fusion self-attention model, It is characterized in that The steps include: S1: Extract feature map of belt laser image; S2: construct a network structure based on multi-window fusion self-attention, wherein the network structure based on multi-window fusion self-attention consists of a trunk path on the upper side and a branch path on the lower side; the trunk path and the branch path each include four states, and each state of the branch path consists of an image splitting module, a self-attention module, and a window combination module; each state of the upper path consists of an image splitting module, a self-attention module, a window combination module, and a window fusion module; S3: Input the feature map of the belt laser image into the network structure, extract the local window information of the image and the cross-window information of semantic features of different levels through the trunk path and the branch path respectively, and fuse the extracted information in the trunk path, and generate a feature map of a specific size after four feature information extractions; S4: Use the average pooling operation to convert the feature map into a feature vector, and the feature vector is input into a fully connected layer with two hidden nodes; S5: The softmax activation layer is connected to the fully connected layer to output the feature map of the belt laser image as the probability of real damage or interference.

2. According to claim 1, a belt damage classification method based on a multi-window fusion self-attention model, It is characterized in that In S3, the feature map of the belt laser image is input into the network structure, including: S31: The image splitting modules in the trunk pathway and the branch pathway respectively split the specific size into continuous non-overlapping window images; S32: The window image of the branch pathway is input into the self-attention module connected thereto to calculate the multi-head self-attention based on the window, and is also input into the trunk pathway, and is input into the fusion module together with the window image extracted by the trunk pathway to realize the fusion of the local window information and the cross-window feature information of different levels of semantics, and the fused features are then input into the self-attention module of the trunk pathway; S33: The outputs of the self-attention modules in the trunk and branch pathways are input into the window combination module connected to them, and the feature maps of different states of the trunk and branch networks are adaptively fused. 1 and f 2 ; As the network deepens, the scale of the feature map input to each state continues to decrease, and the receptive field represented by the window of the same size continues to increase. The receptive field of the window image in the i+1th state is 4 times that of the window image in the i-th state.

3. A belt damage classification method based on a multi-window fusion self-attention model according to claim 1, It is characterized in that The self-attention module is composed of a window attention module with a residual structure and a multi-layer perceptron, and the image is first regularized before being input into the self-attention and multi-layer perceptron modules; Use Relu activation to regularize it: where SoftPlus(x) = log e (1+e x ), which is the difference between Q and K T Normalize the multiplication of ; In the self-attention module, the feature map is evenly divided into multiple windows, each window is represented by a column vector token, and multi-head attention calculation is performed within the window; In the self-attention module of the branch pathway, only the multi-head self-attention of the feature map from the branch pathway is calculated; in the self-attention module of the trunk pathway, the multi-head self-attention of the fused feature map of the trunk and branch pathways needs to be calculated.

4. According to claim 2, a belt damage classification method based on a multi-window fusion self-attention model, It is characterized in that The self-attention module in S32 calculates the window-based multi-head self-attention, including: S321: Perform linear transformation on the input image to generate Q, K and V vectors, where Q represents query, K represents key, and V represents value; S322: Divide the result after linear transformation into h heads, each head learns different attention weights; for four different states, the number of multi-head attentions is 3, 6, 12 and 24 respectively; S323: Calculate the self-attention of each head, as shown in formula (1). d is the scale factor; S324: The output of multiple heads after self-attention calculation, through the learnable linear transformation weight matrix W O Merge into the final multi-head attention output, as expressed in formula (2). MultiHead(Q,K,V)=Concat(head 1 ,...,head h )W O (2)。 5. The belt damage classification method based on multi-window fusion self-attention model according to claim 2, It is characterized in that In S33, the window combination module is used to adaptively fuse the feature maps of different states of the trunk and branch networks. 1 and f 2 ; include: S331: The feature map f 1 and f 2 They are input into the grouped convolution module respectively, in which all feature maps with the same position are grouped into the same group, and the feature maps f are processed according to the grouping results. 1 and f 2 , perform group convolution; S332: After the group convolution, the feature map of the branch path is point-convolved so that the number of its channels is consistent with the number of channels of the trunk; S333: Use global average pooling operation to extract global spatial information S1 and S2 of the trunk and branch pathway feature maps; S334: Use the softmax activation function to extract the attention coefficient of the feature map. The fused feature map F is expressed as: F=Soft(Ave(Conv 分组 (f 1 )))*Conv 分组 (f 1 )+soft(Ave(Conv 分组 (f 1 )))*Conv 点 (Conv 分组 (f 2 ))(3)。

Citation Information

Patent Citations

  • Remote sensing image classification method and device based on space and channel feature extraction

    CN116188836A

  • Fine granularity classification method based on multi-granularity interaction and feature recombination network

    CN116883748A

  • Remote sensing image classification method, equipment and device based on small samples

    CN118015322A