Discrete wavelet-based visual converter network classification method

Abstract

The invention discloses a network classification method based on a discrete wavelet visual converter, and belongs to the technical field of image processing. According to the method, a multi-head attention layer in a visual converter is analyzed, a multi-head attention module with discrete wavelet bottlenecks is designed and built to improve an existing multi-head attention layer module, and the method comprises the following steps that S1, a data set is evenly divided into a training set and a test set; s2, building a multi-head attention module of a discrete wavelet bottleneck; and S3, training the established discrete wavelet-based visual converter network on a natural image classification data set, and testing to verify the effectiveness of the discrete wavelet bottleneck-based multi-head attention module. According to the method, a traditional multi-head attention module is improved by constructing a multi-head attention module with discrete wavelet bottlenecks, so that feature information is re-integrated in a channel domain, a space domain and a frequency domain, and the calculation amount is reduced on the premise of ensuring information flow integrity.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a classification method based on a discrete wavelet visual transformer network. Background Art

[0002] Visual Transformer Networks (VTNs) have become an important tool for computer vision and image processing tasks, such as image classification, object detection, and image restoration. Multi-head attention layers are a crucial component of VTNs, easily capturing long-range dependencies between different parts of an image. However, traditional multi-head attention mechanisms still face two challenges.

[0003] First, a major challenge with multi-head attention mechanisms is their tendency to overlook critical local details, such as edges, textures, and fine-grained features. While they excel at capturing global context, they often struggle to extract these important local features. One popular approach is to integrate convolutional neural networks with multi-head attention mechanisms, offering a solution. Obviously, this approach also comes with a certain computational burden.

[0004] Another challenge with multi-head attention is computational cost. This computational complexity becomes particularly pronounced when processing large-scale or high-resolution images, resulting in significant computational overhead and memory requirements. Therefore, it is crucial to improve the multi-head attention module to reduce computational overhead while ensuring a complete information flow. Summary of the Invention

[0005] In response to the shortcomings of the existing technology, the present invention provides a classification method based on a discrete wavelet visual transformer network, constructs a multi-head attention module with a discrete wavelet bottleneck to improve the traditional multi-head attention module, so as to reduce the amount of calculation while ensuring the integrity of the information flow, and for image classification tasks, uses the multi-head attention module with a discrete wavelet bottleneck to replace the multi-head attention module of the visual transformer network, and builds a network based on a discrete wavelet visual transformer, which can improve the accuracy of image classification while reducing the computational complexity, so as to solve the problems raised in the above background technology.

[0006] To achieve the above objectives, the present invention provides the following technical solutions: a classification method based on a discrete wavelet visual transformer network, which analyzes the multi-head attention layer in the visual transformer network, designs a multi-head attention module with a discrete wavelet bottleneck, replaces the multi-head attention layer with the multi-head attention module with the discrete wavelet bottleneck, builds a discrete wavelet visual transformer network, and classifies images based on the multi-head attention module with the discrete wavelet bottleneck, comprising the following steps:

[0007] S1. Construct training set and test set;

[0008] Prepare a natural image classification dataset and divide the dataset into training set and test set;

[0009] S2, build a multi-head attention module with scattered wavelet bottleneck;

[0010] Design a multi-head attention module with discrete wavelet bottleneck, embed the module into the visual transformer network, and use the multi-head attention module with discrete wavelet bottleneck to reconstruct feature information in spatial, channel and frequency domains.

[0011] Among them, the method of reconstructing spatial, channel and frequency domain feature information using a multi-head attention module with discrete wavelet bottleneck includes:

[0012] S21. Build a channel domain fusion layer of one-dimensional discrete wavelet transform for discrete wavelet bottleneck for dimensionality reduction;

[0013] S22. Build a spatial feature fusion layer of the two-dimensional discrete wavelet transform with a discrete wavelet bottleneck, and fuse the high-frequency and low-frequency feature information. The fused feature information is used as the key and value input of the discrete multi-head attention.

[0014] S23, embed and build a multi-head attention module with discrete wavelet bottleneck;

[0015] S24, building a channel domain fusion layer of the one-dimensional discrete wavelet transform for the discrete wavelet bottleneck of dimensionality upgrading, and splicing the obtained result with the result of the module output in step S23 as the final output;

[0016] S3, train and verify the effectiveness of the module;

[0017] The constructed discrete wavelet visual transformer network was trained on natural image classification datasets and tested to verify the effectiveness of the multi-head attention module of the discrete wavelet bottleneck.

[0018] To further optimize the technical solution, in step S21, the method for constructing a channel domain fusion layer of a one-dimensional discrete wavelet transform for a discrete wavelet bottleneck for dimensionality reduction includes the following specific steps:

[0019] First, the input features Decompose using one-dimensional discrete wavelet transform to obtain and

[0020] Then, the dimensionality is reduced by group convolution and reconstructed using one-dimensional inverse discrete wavelet transform to obtain

[0021] The two-step process is shown below:

[0022] Xlow ,X high =1D DWT(X

[0023]

[0024] Among them, 1D DWT represents one-dimensional discrete wavelet transform, 1D IDWT represents one-dimensional inverse discrete wavelet transform, Indicates "Conv 1x1-BN-ReLU".

[0025] To further optimize the technical solution, in step S22, the method for using the spatial feature fusion layer of the two-dimensional discrete wavelet transform of the discrete wavelet bottleneck includes the following specific steps:

[0026] First, the features Decompose using two-dimensional discrete wavelet transform to obtain low-frequency information High-frequency information and The absolute value of the high-frequency component is added to the low-frequency component as a weight to obtain and X ll , and Splicing along the channel dimension, the process is as follows:

[0027] X ll ,X lh ,X hl ,X hh =2D DWT(X0)

[0028]

[0029] Among them, 2D DWT represents two-dimensional discrete wavelet transform, As the input of the Key and Value of discrete multi-head attention;

[0030] Then, X ll ,X lh ,X hl , and X hh Feature extraction is performed by group convolution and reconstruction is performed using two-dimensional inverse discrete wavelet transform. The process is as follows:

[0031]

[0032] Among them, 2D IDWT represents two-dimensional inverse discrete wavelet transform, Indicates “Conv 3x3-BN-ReLU”.

[0033] To further optimize this technical solution, in step S23, building a multi-head attention module with a discrete wavelet bottleneck includes the following specific steps:

[0034] The linear projection of X is taken as As and Input;

[0035] Attention uses the dot product of Query and Key, and then performs The scaling of , through softmax to obtain the attention weight matrix, the weight matrix is ​​used to calculate the weighted sum of Value, so as to obtain the output of each head. The process is shown in the following formula:

[0036]

[0037] To further optimize the technical solution, in step S24, the method for constructing a channel domain fusion layer of a one-dimensional discrete wavelet transform for a discrete wavelet bottleneck for dimensionality increase includes the following specific steps:

[0038] First, the input features Decompose using one-dimensional discrete wavelet transform to obtain and The dimensionality is increased by group convolution and reconstructed using one-dimensional inverse discrete wavelet transform. Perform residual connection on X and X2 to get The process is as follows:

[0039] X′ low ,X′ high =1D DWT(X1)

[0040]

[0041] X WBS =X+X2

[0042] Then, each head in step S23 is independently calculated, and the final result is each head and X WBS Connect through linear layers to produce the final attention output. The process is shown in the following formula:

[0043]

[0044] Among them, WMSA-WBS represents the multi-head attention module with discrete wavelet bottleneck.

[0045] To further optimize this technical solution, in step S2, the designed multi-head attention module with discrete wavelet bottleneck is encapsulated as an independent module, embedded in various types of visual transformer networks, and trained and tested on classic image classification datasets.

[0046] Compared with the prior art, the present invention provides a classification method based on discrete wavelet visual transformer network, which has the following beneficial effects:

[0047] This classification method based on the discrete wavelet visual transformer network constructs a multi-head attention module with discrete wavelet bottleneck to improve the traditional multi-head attention module, thereby reducing the amount of computation while ensuring the integrity of the information flow. For image classification tasks, the multi-head attention module with discrete wavelet bottleneck is used to replace the multi-head attention module of the visual transformer network, and a discrete wavelet visual transformer network is built, which can improve the accuracy of image classification while reducing computational complexity. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 Schematic diagram of the flow of the discrete wavelet visual transformer network classification method of the present invention;

[0049] Figure 2 Schematic diagram of the process of the multi-head attention module based on discrete wavelet bottleneck of the present invention;

[0050] Figure 3 It is a flow chart of the present invention based on the discrete wavelet visual transformer network. DETAILED DESCRIPTION

[0051] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0052] Example 1:

[0053] See also Figure 1-Figure 3 A classification method based on discrete wavelet visual transformer network is proposed. By analyzing the multi-head attention layer in the visual transformer network, a multi-head attention module with discrete wavelet bottleneck is constructed to improve the traditional multi-head attention module, so that the amount of calculation is reduced while ensuring the integrity of the information flow. For image classification tasks, the multi-head attention module with discrete wavelet bottleneck is used to replace the multi-head attention module of the visual transformer network, and a visual transformer network based on discrete wavelet is constructed, which can improve the accuracy of image classification while reducing the computational complexity.

[0054] The method comprises the following steps:

[0055] S1. Prepare a natural image classification dataset and divide the above datasets into training sets and test sets.

[0056] S2. Design and build a multi-head attention module with discrete wavelet bottleneck, embed it into the visual transformer network, and use the multi-head attention module with discrete wavelet bottleneck to reconstruct spatial, channel and frequency domain feature information.

[0057] S3. The constructed discrete wavelet transform visual transformer network is trained on natural image classification datasets and tested to verify the effectiveness of the multi-head attention module of the discrete wavelet bottleneck.

[0058] Specifically, in step S2, the method of reconstructing spatial, channel, and frequency domain feature information using a multi-head attention module with a discrete wavelet bottleneck includes:

[0059] S21. Build a channel domain fusion layer of one-dimensional discrete wavelet transform for discrete wavelet bottleneck for dimensionality reduction.

[0060] S22. Build a spatial feature fusion layer of the two-dimensional discrete wavelet transform with a discrete wavelet bottleneck, and fuse the high-frequency and low-frequency feature information. The fused feature information is used as the key and value input of the discrete multi-head attention.

[0061] S23. Embed and build a multi-head attention module with discrete wavelet bottleneck.

[0062] S24, building a channel domain fusion layer of the one-dimensional discrete wavelet transform of the discrete wavelet bottleneck for dimensionality upgrading, and splicing the obtained result with the result of the module output in step S23 as the final output.

[0063] Specifically, in step S21, the method for constructing a channel domain fusion layer of a one-dimensional discrete wavelet transform for a discrete wavelet bottleneck for dimensionality reduction includes the following specific steps:

[0064] First, the input features Decompose using one-dimensional discrete wavelet transform to obtain and

[0065] Then, the dimensionality is reduced by group convolution and reconstructed using one-dimensional inverse discrete wavelet transform to obtain

[0066] The two-step process is shown below:

[0067] X low ,X high =1D DWT(X)

[0068]

[0069] Among them, 1D DWT represents one-dimensional discrete wavelet transform, 1D IDWT represents one-dimensional inverse discrete wavelet transform, Indicates "Conv 1x1-BN-ReLU".

[0070] Specifically, in step S22, the method for using the spatial feature fusion layer of the two-dimensional discrete wavelet transform of the discrete wavelet bottleneck includes the following specific steps:

[0071] First, the features Decompose using two-dimensional discrete wavelet transform to obtain low-frequency information High-frequency information and The absolute value of the high-frequency component is added to the low-frequency component as a weight to obtain and X ll , and Splicing along the channel dimension, the process is as follows:

[0072] X ll ,X lh ,X hl ,X hh =2D DWT(X0)

[0073]

[0074] Among them, 2D DWT represents two-dimensional discrete wavelet transform, As the input of the Key and Value of discrete multi-head attention;

[0075] Then, X ll ,X lh ,X hl , and X hh Feature extraction is performed by group convolution and reconstruction is performed using two-dimensional inverse discrete wavelet transform. The process is as follows:

[0076]

[0077] Among them, 2D IDWT represents two-dimensional inverse discrete wavelet transform, Indicates “Conv 3x3-BN-ReLU”.

[0078] Specifically, in step S23, building a multi-head attention module with a discrete wavelet bottleneck includes the following specific steps:

[0079] The linear projection of X is taken as As and Input;

[0080] Attention uses the dot product of Query and Key, and then performs The scaling of , through softmax to obtain the attention weight matrix, the weight matrix is ​​used to calculate the weighted sum of Value, so as to obtain the output of each head. The process is shown in the following formula:

[0081]

[0082] Specifically, in step S24, the method for constructing a channel domain fusion layer of a one-dimensional discrete wavelet transform for dimensionality-upgrading discrete wavelet bottleneck includes the following specific steps:

[0083] First, the input features Decompose using one-dimensional discrete wavelet transform to obtain and The dimensionality is increased by group convolution and reconstructed using one-dimensional inverse discrete wavelet transform. X and X2 are connected by residual, and we get The process is as follows:

[0084] X′ low ,X′ high =1D DWT(X1)

[0085]

[0086] X WBS =X+X2

[0087] Then, each head in step S23 is independently calculated, and the final result is each head and X WBS Connect through linear layers to produce the final attention output. The process is shown in the following formula:

[0088]

[0089] Among them, WMSA-WBS represents the multi-head attention module with discrete wavelet bottleneck.

[0090] Furthermore, in the discrete wavelet method of the multi-head attention module with discrete wavelet bottleneck, only one wavelet function is used. In future practical applications, different wavelet functions will be selected for different scenarios.

[0091] Example 2:

[0092] Based on the discrete wavelet-based visual transformer network classification method described in Example 1, the effectiveness of the image classification method using a visual transformer network with a multi-head attention module using a discrete wavelet bottleneck was trained and verified on the classic image classification dataset ImageNet-1K. The classic network architectures involved in the experiment included the classic deep networks ResNet, PVT, PVTV2, and Swin-ViT. The ImageNet-1K dataset contains 1,280,000 training images and 50,000 test images, representing 1,000 different categories. Each image is 224×224 pixels in size.

[0093] Table 1 Classification results of discrete wavelet visual transformer network (%)

[0094]

[0095] This embodiment conducted a large number of experiments on the ImageNet-1K dataset, involving various types of visual transformer networks. As shown in Table 1, the experimental results all demonstrated the excellence of the classification method of the discrete wavelet visual transformer network of the present invention.

[0096] In the description of this specification, the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0097] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A classification method based on a discrete wavelet visual transformer network, by analyzing the multi-head attention layer in the visual transformer network, designing a discrete wavelet bottleneck multi-head attention module, using the discrete wavelet bottleneck multi-head attention module to replace the multi-head attention layer, building a discrete wavelet visual transformer network, and classifying images based on the discrete wavelet bottleneck multi-head attention module, characterized in that: The following steps are involved: S1, build training set and test set; Prepare a natural image classification dataset and divide the dataset into a training set and a test set; S2, build a multi-head attention module with scattered wavelet bottleneck; Design a multi-head attention module with discrete wavelet bottleneck, embed the module into the visual transformer network, and use the multi-head attention module with discrete wavelet bottleneck to reconstruct feature information in spatial, channel and frequency domains. Among them, the method of reconstructing spatial, channel and frequency domain feature information using a multi-head attention module of a discrete wavelet bottleneck includes: S21, build a channel domain fusion layer of one-dimensional discrete wavelet transform for discrete wavelet bottleneck for dimensionality reduction; S22, build a spatial feature fusion layer of the two-dimensional discrete wavelet transform of the discrete wavelet bottleneck, and fuse the high-frequency and low-frequency feature information. The fused feature information is used as the input of the Key and Value of the discrete multi-head attention; S23, embed and build a multi-head attention module with discrete wavelet bottleneck; S24, constructing a channel domain fusion layer of a one-dimensional discrete wavelet transform for the discrete wavelet bottleneck of dimensionality upgrading, and splicing the obtained result with the result of the module output in step S23 as the final output; S3, train and verify the effectiveness of the module; The constructed discrete wavelet visual transformer network is trained on natural image classification datasets and tested to verify the effectiveness of the multi-head attention module of the discrete wavelet bottleneck.

2. According to claim 1, the classification method based on discrete wavelet visual transformer network is characterized in that: In the step S21, the method for constructing a channel domain fusion layer of a one-dimensional discrete wavelet transform for a discrete wavelet bottleneck for dimensionality reduction comprises the following specific steps: First, the input features Using one-dimensional discrete wavelet transform to decompose, we get and Then, the dimension is reduced by group convolution and reconstructed using one-dimensional inverse discrete wavelet transform to obtain The two-step process is shown below: X low ,X hig h=1D DWT(X) Among them, 1D DWT represents one-dimensional discrete wavelet transform, 1D IDWT represents one-dimensional inverse discrete wavelet transform, Stands for "Conv1x1-BN-ReLU".

3. According to claim 1, the classification method based on discrete wavelet visual transformer network is characterized in that: In step S22, the method for using the spatial feature fusion layer of the two-dimensional discrete wavelet transform of the discrete wavelet bottleneck includes the following specific steps: First, the features Decompose using two-dimensional discrete wavelet transform to obtain low-frequency information High frequency information and The absolute value of the high-frequency component is added as a weight to the low-frequency component to obtain and X ll , and Splicing along the channel dimension, the process is as follows: X ll ,X lh ,X hl ,X hh =2D DWT(X0) Among them, 2D DWT represents two-dimensional discrete wavelet transform, As the input of Key and Value of discrete multi-head attention; Then, X ll ,X lh ,X hl , and X hh The feature extraction is performed by group convolution and the reconstruction is performed by two-dimensional inverse discrete wavelet transform. The process is as follows: Among them, 2D IDWT represents two-dimensional inverse discrete wavelet transform, Stands for "Conv 3x3-BN-ReLU".

4. According to claim 1, the classification method based on discrete wavelet visual transformer network is characterized in that: In step S23, building a multi-head attention module with discrete wavelet bottleneck includes the following specific steps: The linear projection of X is taken as As and Input; Attention uses the dot product of Query and Key, and then performs The scale of is obtained through softmax to obtain the attention weight matrix, which is used to calculate the weighted sum of Value to obtain the output of each head. The process is shown in the following formula:

5. According to claim 1, the classification method based on discrete wavelet visual transformer network is characterized in that: In step S24, the method for constructing a channel domain fusion layer of a one-dimensional discrete wavelet transform for a discrete wavelet bottleneck of dimensionality increase comprises the following specific steps: First, the input features Using one-dimensional discrete wavelet transform to decompose, we get and The dimensionality is increased by group convolution and reconstructed using one-dimensional inverse discrete wavelet transform. X and X2 are connected by residual, and we get The process is as follows: X′ low ,X′ high =1D DWT(X1) X WBS =X+X2 Then, each head in step S23 is independently calculated, and the final result is each head and X WBS Connect through linear layers to produce the final attention output. The process is shown in the following formula: Among them, WMSA-WBS represents the multi-head attention module with discrete wavelet bottleneck.

6. The discrete wavelet visual transformer network classification method according to claim 1 is characterized in that: In step S2, the designed discrete wavelet bottleneck multi-head attention module is encapsulated as an independent module, embedded into various types of visual transformer networks, and trained and tested on classic image classification datasets.

Images