A thread rotation classification method based on convolutional neural network

Abstract

The invention relates to the technical field of convolutional neural networks, and specifically discloses a thread hand direction classification method based on a neural network. In the process of feature extraction, a channel attention module adopts channel attention with receptive field fusion, so that the network pays more attention to the local context information of the feature; the spatial attention adopts a triple spatial attention module, and the spatial information is integrated through a cross-dimensional interaction method, which is more conducive to the learning of the feature space position; a Res2Net backbone network that pays more attention to fine-grained features is adopted, and the channel attention with receptive field fusion and triple spatial attention are embedded in the block of the backbone network, so as to correct the attention of the features learned by the original backbone network; feature extraction is performed by adopting the method of first channel attention and then spatial attention, so as to strengthen the network's extraction and learning of fine-grained features of thread hand direction at the channel and space levels.

Description

Technical Field

[0001] The present invention relates to the technical field of convolutional neural networks, and in particular to a thread hand direction classification method based on convolutional neural networks. Background Art

[0002] In the study of automatic recognition and classification of thread direction, the traditional algorithm based on the relative inclination angle of the helix has certain limitations. It can only capture macro features such as grayscale and straight lines, but lacks deep feature descriptions. It fails to achieve satisfactory results under larger image zoom, variable backgrounds, and more demanding lighting conditions. In recent years, the emergence of deep learning networks has brought new solutions to image classification, avoiding the complexity and non-universality of manual feature selection, and has an accuracy that traditional visual algorithms cannot achieve for macro features such as object size, color, and shape.

[0003] The targets of common industrial defect detection tasks are relatively macroscopic and have obvious differences. The characteristics of various defects are significant. Therefore, deep learning-based recognition tasks can use convolutional neural networks to extract sufficient features. These key features can be used to better realize thread type recognition and defect detection tasks. However, due to the same target thread type, thread direction recognition is only distinguishable in the subtle features of the left and right directions. If the background and irrelevant areas cannot be avoided as much as possible, it will be difficult to extract the distinctive features of the direction, resulting in poor distinguishability of the direction of rotation of the network model. At the same time, in actual recognition, such as the extraction of the thread direction feature of bicycle pedals, the thread area occupies a small proportion of the overall background of the image, and the left and right thread areas are distinguishable only in the spiral part. Therefore, ordinary recognition networks can no longer achieve both speed and quality for thread classification, which is undoubtedly unable to meet production needs.

[0004] Therefore, in the design of the pedal thread direction recognition network, it is necessary to further ignore the thread background features, focus on the fine-grained level features of the direction, and capture the features of the thread helix to the greatest extent, laying the foundation for the final thread direction recognition. How to increase the attention to the fine-grained features of the thread direction is the key and difficulty of this classification task. Summary of the invention

[0005] In order to solve the difficulties of subtle target differences and easy interference of background features in the process of thread hand direction recognition, the present invention starts from modeling spatial attention in a cross-dimensional manner and fusion of local correlations of different receptive fields, and proposes a thread hand direction classification method based on a convolutional neural network, including the following steps:

[0006] S1: extracting the thread rotation direction features of the thread in the thread area image through the feature extraction network and outputting them as the initial thread rotation direction feature map;

[0007] S2: Determine whether the output of the current layer feature extraction network is mapped to the corresponding thread area image within a preset residual value. If so, perform hand direction classification on the target thread area image through the current feature extraction network. If not, proceed to the next step.

[0008] S3: Through the channel attention module in the CBAM module, the initial receptive field and the optimized receptive field are acquired and fused according to the thread rotation feature map, and the fused feature map after the receptive field fusion is extracted;

[0009] S4: Acquire the channel attention feature map based on the fusion feature map;

[0010] S5: Through the spatial attention module in the CBAM module, the interaction information of the cross-dimensional interaction of each dimension combination of the channel is obtained according to the channel attention feature map;

[0011] S6: perform element addition and average pooling processing in sequence according to the mutual information of each dimension combination, and the output after pooling is used as the input of the next layer of feature extraction network;

[0012] S7: Enter the next layer of feature extraction network and output the thread direction feature map according to the input, and return to step S2;

[0013] The channel dimensions include the channel C and the height H and width W in the channel space dimensions.

[0014] Furthermore, in step S3, the initial receptive field is obtained by the channel attention module after convolving the thread rotation feature map with a convolution kernel of a preset size, and the optimized receptive field is obtained by the channel attention module after performing a hole convolution on the thread rotation feature map based on a convolution kernel of a preset size.

[0015] Furthermore, in step S4, the acquisition of the channel attention feature map can be expressed as the following formula group:

[0016]

[0017] z′=δ(Mz)

[0018] X′=z′Y

[0019] Where Y is the fusion feature map, z is the feature of the channel dimension, h is the size of the height H in the channel space dimension, w is the size of the width W in the channel space dimension, and F gap (Y) is the global average pooling process of Y, T(i,j) is the pixel with coordinates (i,j) on the fusion feature map Y; z′ is the weight vector, δ is the ReLU function, M is the fully connected weight; X′ is the channel attention feature map.

[0020] Furthermore, the dimension combination includes:

[0021] The first branch consists of dimension C and dimension H, the second branch consists of dimension C and dimension W, and the third branch consists of dimension H and dimension W.

[0022] Furthermore, the interactive information needs to go through the following steps in sequence before being obtained:

[0023] Selectively rotate the channel attention feature map according to the branch category;

[0024] The selectively rotated channel attention feature map is pooled through the maximum pooling layer channel and the average pooling layer channel respectively;

[0025] Concatenate the outputs of the maximum pooling layer channel and the average pooling layer channel;

[0026] The concatenated channel output is processed by batch normalization and activation to obtain weight information, and the weight information is multiplied with the selectively rotated channel attention feature map;

[0027] According to the branch category, the multiplied channel attention feature map is selectively reversely rotated accordingly;

[0028] The selective rotation and the corresponding selective reverse rotation have the same rotation angle and opposite rotation directions.

[0029] Further, the selective rotation includes:

[0030] When the branch category is the first branch, the channel attention feature map is rotated 90° counterclockwise along the axis in the height H direction;

[0031] When the branch category is the second branch, the channel attention feature map is rotated 90° counterclockwise along the axis in the width W direction;

[0032] When the branch category is the third branch, the channel attention feature map is not rotated.

[0033] Furthermore, in step S6, the acquisition of the thread direction characteristic diagram can be expressed as the following formula:

[0034]

[0035] Where y is the thread direction characteristic diagram, is the rotation tensor of the first branch, is the rotation tensor of the second branch; for The concatenated output after passing through the maximum pooling layer channel and the average pooling layer channel, for The concatenated output after passing through the maximum pooling layer channel and the average pooling layer channel, is the concatenation output of the dimension combination of dimension H and dimension W in X′ after passing through the maximum pooling layer channel and the average pooling layer channel respectively; σ is the activation function; ψ1 is The two-dimensional convolutional layer of The two-dimensional convolution layer of A two-dimensional convolutional layer.

[0036] Furthermore, the feature extraction network is a multi-scale feature network.

[0037] Compared with the prior art, the present invention has at least the following beneficial effects:

[0038] (1) The thread hand direction classification method based on convolutional neural network described in the present invention introduces hole convolution instead of ordinary convolution, reduces the amount of calculation and further improves the learning effect of channel attention on each receptive field by fusing different receptive fields;

[0039] (2) Different receptive fields have different local correlations. Fusion of features from different receptive fields can link different local correlations. Thus, channel weight allocation is used to allow the network to distinguish the weights of the focus positions on the feature map, further enhancing the ability to learn the relationship between local features and channels.

[0040] (3) Through the acquisition and fusion of cross-dimensional interactive information, the features of different dimensions are interactively learned without increasing the computational overhead, which preserves the channel and space dependencies, enhances the network's learning ability for spatial features, and can more fully extract spatial information;

[0041] (4) A multi-scale feature network (Res2Net) is selected as the feature extraction network to integrate multi-scale features and perform fine-grained representation extraction to further improve the classification accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A method step diagram of a thread rotation classification method based on a convolutional neural network;

[0043] Figure 2 Schematic diagram of average pooling and maximum pooling;

[0044] Figure 3 Schematic diagram of residual block;

[0045] Figure 4 It is a structural diagram of the SAM module;

[0046] Figure 5 It is a structural diagram of the CAM module;

[0047] Figure 6 It is a structural diagram of the SENet module;

[0048] Figure 7 Schematic diagram of ordinary convolution and hole convolution;

[0049] Figure 8 Schematic diagram of the structure of the receptive field fusion channel attention module combined with dilated convolution;

[0050] Fig. 9 Schematic diagram of the triple attention mechanism structure;

[0051] Fig.10 This is a schematic diagram of the overall architecture of the RTNet algorithm; DETAILED DESCRIPTION

[0052] The following are specific embodiments of the present invention and the accompanying drawings to further describe the technical solution of the present invention, but the present invention is not limited to these embodiments.

[0053] Embodiment 1

[0054] Before explaining the technical contents of the present invention in detail, this embodiment first explains the corresponding structure and functions of the network optimization learning basis on which the present application relies, that is, the convolutional neural network, to facilitate the understanding of the technical contents of the present application.

[0055] Convolution operation is a common computing method in traditional image processing and one of the foundations of convolutional neural networks in the field of images. Convolution operation can extract local and multi-dimensional feature information of the input image. Compared with artificial neural networks, it has two main advantages: local perception mechanism and weight sharing mechanism. It greatly reduces the amount of calculation and improves the training speed. Its network structure mainly includes: convolution layer, pooling layer and fully connected layer.

[0056] The convolution layer operates by sliding the convolution kernel on the input to obtain the corresponding feature map, and then selectively extracts the local feature information of the input. The formula is as follows:

[0057]

[0058] In the formula, l represents the number of layers in the network, M represents the receptive field of the input layer, δ represents the activation function, represents the jth feature map of the lth layer, w represents the convolution kernel, b represents the bias top, * represents the convolution operation, and the result of the calculation is used as the input of the next layer of the network.

[0059] Among them, the activation function in the convolution operation performs nonlinear mapping on the features. Commonly used activation functions include Sigmoid function, Tanh function, ReLU (Rectified Linear Unit) function, etc.

[0060] The function of the ReLU function is expressed as: Relu(x) = max(0, x), which is a piecewise function. When x≤0, the function value is 0 and the derivative is 0. When x>0, the function value is equal to x and the derivative is 1, so it converges quickly. Therefore, the ReLU function can effectively suppress the vanishing gradient, making it have a wider range of application prospects.

[0061] There are two main types of pooling layers: average-pooling and max-pooling. Figure 2 As shown in the figure, average pooling is to average the local feature pixels in the feature map with a certain step size, which is often used to extract image background information; maximum pooling is to extract the highest value of the local feature points in the feature map, which can retain more texture edge information of the image.

[0062] The fully connected layer connects all output features to a layer of neurons respectively, and outputs the category probability through softmax to achieve classification. To prevent overfitting, a Dropout layer can be added to the fully connected layer.

[0063] At the same time, the targets of common industrial defect detection tasks are relatively macroscopic, and there are obvious differences, and the characteristics of various defects are significant. Therefore, deep learning-based recognition tasks can use convolutional neural networks to extract sufficient features, and these key features can be used to better realize thread type recognition and defect detection tasks.

[0064] However, since the target threads are of the same type, only the subtle features of the left and right hand directions are distinguishable in thread direction recognition. If the background and irrelevant areas cannot be avoided as much as possible, it will be difficult to extract the distinctive features of the hand direction, resulting in poor hand direction distinguishability of the network model. In actual recognition, the thread area occupies a small part of the overall background of the image, and the left and right hand thread areas are distinguishable only in the spiral part. Therefore, only by extracting appropriate spiral features can the thread direction be effectively identified, which is challenging for ordinary recognition networks.

[0065] Therefore, in the design of the thread direction recognition network, it is necessary to further ignore the thread background features, focus on the fine-grained level features of the direction, and capture the features of the helical part of the thread to the greatest extent, laying the foundation for the final thread direction recognition. How to increase the attention to the fine-grained features of the thread direction is the key and difficulty of this classification task.

[0066] After understanding the basic composition of the convolutional neural network and the functions of its various functional layers, the present invention takes into account the small overall area occupied by the threads in the pedals and the subtle differences, such as Figure 1 As shown, a thread hand direction classification method based on convolutional neural network is proposed, including the following steps:

[0067] S1: extracting the thread rotation direction features of the thread in the thread area image through the feature extraction network and outputting them as the initial thread rotation direction feature map;

[0068] S2: Determine whether the output of the current layer feature extraction network is mapped to the corresponding thread area image within a preset residual value. If so, perform hand direction classification on the target thread area image through the current feature extraction network. If not, proceed to the next step.

[0069] S3: Through the channel attention module in the CBAM module, the initial receptive field and the optimized receptive field are acquired and fused according to the thread rotation feature map, and the fused feature map after the receptive field fusion is extracted;

[0070] S4: Acquire the channel attention feature map based on the fusion feature map;

[0071] S5: Through the spatial attention module in the CBAM module, the interaction information of the cross-dimensional interaction of each dimension combination of the channel is obtained according to the channel attention feature map;

[0072] S6: perform element addition and average pooling processing in sequence according to the mutual information of each dimension combination, and the output after pooling is used as the input of the next layer of feature extraction network;

[0073] S7: Enter the next layer of feature extraction network and output the thread direction feature map according to the input, and return to step S2;

[0074] The channel dimensions include the channel C and the height H and width W in the channel space dimensions.

[0075] First, the feature extraction network of the present invention uses a multi-scale feature network (that is, Rse2Net). The reason for choosing this feature extraction network is that the thread rotation direction recognition and classification problem to be solved by the present invention has subtle feature differences between the rotation directions, and the thread part occupies a small proportion of the entire image. Therefore, if the thread feature is to be extracted, it is necessary to achieve fine-grained characterization extraction, which is impossible for ordinary feature extraction networks. Among them, the residual modules of all feature extraction networks can be expressed by the following formula:

[0076] H(x)=F(x)+x

[0077] In the formula, x is the input of the residual module, F(x) is the output of the convolution branch, and H(x) is the total output of the residual module. If the residual is 0, it means that the learned features are good enough. In this case, the above formula only makes an identity transformation and does not increase the amount of calculation. Of course, you can also set a preset residual value according to the accuracy requirements.

[0078] From the structural explanation, Res2Net (such as Figure 3 .b) Compared with ordinary ResNet (such as Figure 3 .a), the 3×3 convolution kernel in the residual module of ResNet is removed and replaced by Figure 3 The variable structure shown in .b. This structure groups the input features (such as Figure 3 .b) in layers (multiple 3×3 layers, Figure 3 The output of the convolution kernel group (such as Figure 3 .b) are connected, and the increased computational and memory overhead is relatively small. The output of each convolution kernel group forward propagated by this structure can be expressed as follows:

[0079]

[0080] In the formula, s represents the number of feature channels after the 1×1 convolution layer, x i K represents a feature subset, and the number of channels in each feature subset is equal. i For each subset x i Corresponding to different 3×3 convolutional layers, y i It represents the output. This structural design makes the output contain different numbers and combinations of receptive field scales. Finally, it enters the 1×1 convolution to better integrate the global information with the local information. At the same time, it can be seen that s can change the receptive field to a certain extent, that is, the larger s is, the larger the receptive field is. By increasing the receptive field, the extraction of fine-grained level representation is achieved.

[0081] To further explain, the extraction of image features by convolutional neural networks relies on feature location and acquisition by the spatial attention mechanism. The traditional spatial attention mechanism is mainly based on the CBAM (Convolutional Block Attention Module) module, which contains two sub-modules: SAM (Spatial Attention Module) and CAM (Channel Attention Module).

[0082] SAM is the spatial attention module, which focuses on "where" the key information is. Figure 4The figure shows the structure of the SAM module. SAM includes a series of average pooling layers (avgpool) and maximum pooling layers (maxpool), followed by a 7×7 convolutional layer. The calculation process can be expressed by the following formula:

[0083]

[0084] In the formula, F represents the feature map of the middle layer, M s (F) represents the extraction of spatial attention features (X″) on the feature map (X′), δ is the ReLU activation function, and f 7×7 represents the 7×7 convolution function, They represent the average pooling layer and the maximum pooling layer respectively.

[0085] However, SAM only extracts spatial features through two pooling methods. Due to the diversity of spatial position features, this method is not comprehensive for feature extraction of the entire dimensional space.

[0086] CAM is the channel attention module, which focuses on "how to go" of key information. Figure 5 The figure shows the structure of the CAM module. The CAM includes an average pooling layer and a maximum pooling layer in parallel, followed by a perceptron. The calculation process can be expressed as follows:

[0087]

[0088] In the formula, F (Input feature F) represents the intermediate layer feature map, M c (F) represents the channel attention feature extraction on the feature map, σ represents the activation function, and the ReLU activation function is also used here. They represent the average pooling layer and the maximum pooling layer respectively. W0 and W1 are the weights of the Multi-Layer Perceptron (MLP) layer, and their dimensions are r is the scaling factor.

[0089] The maximum pooling feature extraction in CAM is mainly used to filter the salient features of the current target, while the purpose of average pooling is to filter the global background information of the current target. The two single-channel feature maps after pooling enter the multi-layer perceptron respectively, in order to increase the nonlinearity of the features. Compared with the fully connected layer, the MLP has an intermediate hidden layer, so the overall number of parameters is greatly reduced. After the output features are added and activated, the channel attention features are obtained.

[0090] In addition to the CAM module, the SENet module can also achieve this function, which mainly consists of two parts: Squeeze part and Excitation part. The structure diagram is as follows Figure 6 shown.

[0091] where F tr is a convolution module without bias, consisting of multiple two-dimensional convolution kernels. For any given input X of size H′×W′×C′, F tr Map X into a feature map U of shape H×W×C through convolution.

[0092] The Squeeze module is an average pooling layer that converts the feature map U passed through the Ftr module into a 1×1×C output Z through pooling. c , the calculation process can be expressed by the following formula:

[0093]

[0094] In the formula, u c It represents the spatial feature information of each channel in U, that is, U can be expressed as U=[u1,u2,…,u c ] represents, h and w represent the height and width of the feature map respectively.

[0095] The Excitation module follows closely to further capture the features between channels, which can be expressed as follows:

[0096] s=F ex (z,W)=σ(g(z,W))=σ(W1δ(W0z))

[0097] Where δ represents the ReLU function, σ represents the Sigmoid function, W0 and W1 are the fully connected weights, r is the scaling factor, and s is the output of the Excitation module, with a dimension of 1×1×C.

[0098] F scale The module multiplies the feature information obtained by the Squeeze module with the weight obtained by the Excitation module to obtain the final feature information The formula expression is:

[0099]

[0100] In the formula, s c Represents the weight corresponding to the c-th channel information in the feature matrix U.

[0101] Both CAM and SE (SENet) modules have the ability to capture the inter-channel relationship of features, which can ensure the integrity and relevance of features. However, since the above channel attention modules directly process the feature map of the forward input, there is a lack of mining for local relevance, and thus the contextual information of fine-grained features cannot be learned more fully.

[0102] Based on the above, it can be seen that SAM is not comprehensive in extracting spatial features, and SENet and CAM cannot fully exploit the different receptive field representations of features. Due to the difficulty of rotation direction recognition targets with subtle differences and easy interference from background features, the present invention further studies the cross-dimensional modeling of spatial attention and the fusion of local correlations of different receptive fields, improves the channel attention weight allocation, and proposes RTNet based on channel attention and triple spatial attention based on receptive field (Receptive Field) fusion, which strengthens the Res2Net network's learning of fine-grained features of thread rotation direction, and further improves the accuracy of thread rotation direction recognition. Specifically as follows,

[0103] 1) Channel attention mechanism based on receptive field fusion:

[0104] The weight extraction between feature map channels in SENet and CBAM is based on the input feature map, that is, the receptive field of the extracted features is fixed. For example, in SENet, the attention module directly averages the input feature map and then performs subsequent channel weight allocation; in the channel attention module of CBAM, the input is pooled and weights are allocated through the MLP perceptron, and then the two one-dimensional vectors after weight allocation are superimposed to obtain the final channel weight allocation result. The above direct pooling operation on the input feature map can ensure the integrity and relevance of the features, but it fails to fully tap the receptive field representation capabilities of the features at all levels.

[0105] The present invention improves channel attention through receptive field fusion and introduces hole convolution to replace ordinary convolution, thereby reducing the amount of calculation and improving the network's learning effect on the contextual relationship of multiple receptive fields.

[0106] Based on existing research (such as InceptionNet), we know that by fusing multiple receptive fields in the feature map through convolution operations, and then adding the feature map element by element, the feature map through information fusion is made more locally relevant, so that the acquired weights add local relevant information, which is more conducive to the learning of subtle features.

[0107] Therefore, the present invention uses two different sizes of convolution kernels, 3×3 (the size of the initial receptive field) and 5×5, to extract and fuse the local correlation features of different receptive fields, thereby more effectively allocating channel weights. By fusing different local correlation features, this dataset can be learned at a fine-grained level to obtain a more distinctive representation, thereby further improving the accuracy of thread direction recognition.

[0108] According to research, the use of 5×5 convolution kernels helps to expand the receptive field of feature maps, but at the same time, it introduces too much calculation, which affects the network training speed and the entire reasoning speed. Therefore, without affecting the test accuracy, the present invention introduces a dilated convolution to replace the 5×5 convolution kernel. As shown in Figure 7, the same size 3×3 ordinary convolution ( Figure 7 .a, dilation rate is 1) and dilated convolution ( Figure 7 .b, the dilation rate is 2), the receptive field of the atrous convolution is equivalent to that of a 5×5 standard convolution.

[0109] like Figure 8 As shown in FIG. 1 , it is a schematic diagram of the structure of the receptive field fusion channel attention module combined with the dilated convolution designed in the present invention. First, the input feature Figure X ∈R H’ × W’ ×C performs convolution operations through 3×3 ordinary convolution and hole convolution to obtain the feature map A∈R of the same size. H×W×C and B∈R H×W×C After element-wise addition, they are fused into a feature map Y, which has the fusion features of two receptive fields.

[0110] In order to capture the relationship between channels, it is necessary to retain the characteristics of the channel dimension, that is, to use global average pooling to aggregate the information of each channel. The calculation process of the obtained feature map can be expressed as:

[0111]

[0112] The calculated feature map is distributed by channel, so the fully connected layer is used to learn and distribute the channel weights, and the ReLU activation function is used for nonlinear activation. The formula is as follows:

[0113] z′=δ(Mz)

[0114] By multiplying the input feature map and the weight vector z′ element by element, we get the feature of fused channel attention Figure X ′, the formula is as follows:

[0115] X′=z′Y

[0116] In the above formulas, Y is the fusion feature map, z is the feature of the channel dimension, h is the size of the height H in the channel space dimension, w is the size of the width W in the channel space dimension, and F gap (Y) is the global average pooling process of Y, Y(i,j) is the pixel with coordinates (i,j) on the fusion feature map Y; z′ is the weight vector, δ is the ReLU function, M is the fully connected weight; X′ is the channel attention feature map.

[0117] By introducing dilated convolution instead of ordinary convolution, the amount of calculation is reduced while the learning effect of channel attention on the contextual relationship of each receptive field is further improved by fusing different receptive fields. Different receptive fields have different local correlations. Fusion of features from different receptive fields can link different local correlations, thereby using channel weight allocation to allow the network to distinguish the weights of the focus position of the feature map, further enhancing the ability to learn the local features and the relationship between channels.

[0118] 2) Triplet Spatial Attention:

[0119] SAM fuses the information of each channel through two pooling methods to extract spatial attention features. However, compressing channel information alone cannot fully extract spatial attention features.

[0120] In order to improve the comprehensiveness of spatial attention feature extraction from input feature maps, this paper uses triplet-attention (TA) mechanism as the spatial attention mechanism. TA is a new module that can obtain interactive feature information by using a three-branch structure to interactively calculate cross-dimensional attention weights.

[0121] The structure of the triple attention mechanism module is as follows Fig. 9 As shown, for a given input χ of shape C×H×W, the triple attention mechanism creates three branches, capturing the interactive information of the first branch (C, H), the second branch (C, W), and the third branch (H, W) dimensions respectively.

[0122] The first branch is responsible for capturing the interactive information of the two dimensions of channel C and height dimension H in the spatial dimension. After inputting χ (that is, the channel attention feature map output by the channel attention mechanism based on receptive field fusion), this branch rotates χ 90° counterclockwise along the height H direction. At this time, the rotation tensor is defined as The shape is W×H×C, then Enter the maximum pooling layer and the average pooling layer respectively, and concatenate the outputs of the two single channels to obtain a shape of 2×H×C The features of the two channel feature maps are fused through fused convolution. After batch normalization and activation, the obtained weights are multiplied with the input feature map, and then rotated 90° clockwise along the H axis to restore it to the input shape. At this point, the cross-dimensional interactive attention feature learning of the first branch is completed.

[0123] The second branch learns the (C, W) dimension mutual information in the same way (90° counterclockwise rotation along the axis in the width W direction). The third branch does not perform the rotation operation.

[0124] The outputs of the three branches are added element by element and then enter the average pooling layer to obtain the final attention feature containing cross-dimensional interaction information. The whole process can be expressed by the following formula:

[0125]

[0126] Where y is the thread direction characteristic diagram, is the rotation tensor of the first branch, is the rotation tensor of the second branch; for The concatenated output after passing through the maximum pooling layer channel and the average pooling layer channel, for The concatenated output after passing through the maximum pooling layer channel and the average pooling layer channel, is the concatenation output of the dimension combination of dimension H and dimension W in X′ after passing through the maximum pooling layer channel and the average pooling layer channel respectively; σ is the activation function; ψ1 is The two-dimensional convolutional layer of The two-dimensional convolution layer of A two-dimensional convolutional layer.

[0127] like Fig.10 The figure shows the overall architecture diagram of the RTNet algorithm designed by the present invention. Combining the explanation of the technical content of the present invention in this embodiment, it can be seen that the thread rotation classification method based on convolutional neural network described in the present invention, in the process of feature extraction, the channel attention module adopts the channel attention of receptive field fusion, so that the network pays more attention to the local context information of the feature; the spatial attention adopts a triple spatial attention module, which fuses the spatial information through cross-dimensional interaction, which is more conducive to the learning of the feature space position.

[0128] The Res2Net backbone network that pays more attention to fine-grained features is adopted, and the channel attention and triple spatial attention of receptive field fusion are embedded into the blocks of the backbone network, so as to correct the attention of the features learned by the original backbone network and obtain a more effective representation.

[0129] The feature extraction is performed by first using channel attention and then using spatial attention. First, the output feature map of the 1×1 convolution layer is passed through a 3×3 convolution kernel and a 3×3 dilated convolution with a 5×5 convolution kernel receptive field (with a dilation rate of 2), and the two feature maps with different receptive fields are fused to more effectively learn the channel weights; the obtained channel attention features are input into the triple spatial attention, and then the spatial information of the features is captured through cross-dimensional interaction. The outputs of the three spatial attention features are added and averaged, and then a residual connection is formed with the block input as the output. Through the combined use of the above two attention modules, the network is strengthened in the extraction and learning of fine-grained features of thread rotation at the channel and spatial levels.

[0130] It should be noted that all directional indications in the embodiments of the present invention (such as up, down, left, right, front, back, etc.) are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0131] In addition, in the present invention, descriptions such as "first", "second", "one", etc. are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0132] In the present invention, unless otherwise clearly specified and limited, the terms "connection", "fixation", etc. should be understood in a broad sense. For example, "fixation" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements, unless otherwise clearly defined. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0133] In addition, the technical solutions between the various embodiments of the present invention can be combined with each other, but it must be based on the fact that ordinary technicians in the field can implement it. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

Claims

1. A thread hand direction classification method based on convolutional neural network, characterized in that: Includes steps: S1: extracting the thread rotation direction features of the thread in the thread area image through the feature extraction network and outputting them as the initial thread rotation direction feature map; S2: Determine whether the output of the current layer feature extraction network is mapped to the corresponding thread area image within a preset residual value. If so, perform hand direction classification on the target thread area image through the current feature extraction network. If not, proceed to the next step. S3: Through the channel attention module in the CBAM module, the initial receptive field and the optimized receptive field are acquired and fused according to the thread rotation feature map, and the fused feature map after the receptive field fusion is extracted; S4: Acquire the channel attention feature map based on the fusion feature map; S5: Through the spatial attention module in the CBAM module, the interaction information of the cross-dimensional interaction of each dimension combination of the channel is obtained according to the channel attention feature map; S6: perform element addition and average pooling processing in sequence according to the mutual information of each dimension combination, and the output after pooling is used as the input of the next layer of feature extraction network; S7: Enter the next layer of feature extraction network and output the thread direction feature map according to the input, and return to step S2; The channel dimensions include the channel C and the height H and width W in the channel space dimensions.

2. A thread hand direction classification method based on convolutional neural network as claimed in claim 1, characterized in that: In step S3, the initial receptive field is obtained by performing convolution of a preset size convolution kernel on the thread hand direction feature map by the channel attention module, and the optimized receptive field is obtained by performing a hole convolution on the thread hand direction feature map by the channel attention module based on a preset size convolution kernel.

3. A thread hand direction classification method based on convolutional neural network as claimed in claim 1, characterized in that: In step S4, the acquisition of the channel attention feature map can be expressed as the following formula group: z′=δ(Mz) X′=z′Y Where Y is the fusion feature map, z is the feature of the channel dimension, h is the size of the height H in the channel space dimension, w is the size of the width W in the channel space dimension, and F gap (Y) is the global average pooling process of Y, T(i,j) is the pixel with coordinates (i,j) on the fusion feature map Y; z′ is the weight vector, δ is the ReLU function, M is the fully connected weight; X′ is the channel attention feature map.

4. A thread hand direction classification method based on convolutional neural network as claimed in claim 1, characterized in that: The dimension combinations include: The first branch consists of dimension C and dimension H, the second branch consists of dimension C and dimension W, and the third branch consists of dimension H and dimension W.

5. A thread hand direction classification method based on convolutional neural network as claimed in claim 4, characterized in that: The interactive information needs to go through the following steps in sequence before being obtained: Selectively rotate the channel attention feature map according to the branch category; The selectively rotated channel attention feature map is pooled through the maximum pooling layer channel and the average pooling layer channel respectively; Concatenate the outputs of the maximum pooling layer channel and the average pooling layer channel; The concatenated channel output is processed by batch normalization and activation to obtain weight information, and the weight information is multiplied with the selectively rotated channel attention feature map; According to the branch category, the multiplied channel attention feature map is selectively reversely rotated accordingly; The selective rotation and the corresponding selective reverse rotation have the same rotation angle and opposite rotation directions.

6. A thread hand direction classification method based on convolutional neural network as claimed in claim 5, characterized in that: The selective rotation includes: When the branch category is the first branch, the channel attention feature map is rotated 90° counterclockwise along the axis in the height H direction; When the branch category is the second branch, the channel attention feature map is rotated 90° counterclockwise along the axis in the width W direction; When the branch category is the third branch, the channel attention feature map is not rotated.

7. A thread hand direction classification method based on convolutional neural network as claimed in claim 5, characterized in that: In step S6, the acquisition of the thread direction characteristic diagram can be expressed as the following formula: Where y is the thread direction characteristic diagram, is the rotation tensor of the first branch, is the rotation tensor of the second branch; for The concatenated output after passing through the maximum pooling layer channel and the average pooling layer channel, for The concatenated output after passing through the maximum pooling layer channel and the average pooling layer channel, is the concatenation output of the dimension combination of dimension H and dimension W in X′ after passing through the maximum pooling layer channel and the average pooling layer channel respectively; σ is the activation function; ψ1 is The two-dimensional convolutional layer of The two-dimensional convolutional layer of A two-dimensional convolutional layer.

8. A thread hand direction classification method based on convolutional neural network as claimed in claim 1, characterized in that: The feature extraction network is a multi-scale feature network.

Images