Transformer-based image feature extraction method and system, and computer equipment

Abstract

The present invention discloses a transformer-based image feature extraction method, system, and computer device. This method first divides a feature map into local windows of equal size, each representing information about a region of the image. Local window shortcuts (LWS) enable interaction between local windows, allowing global image information to be shared. This adds diverse global image information to the features extracted by high-resolution transformers. The image features extracted by this method are suitable for high-precision image classification, resulting in more accurate classification results.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a transformer-based image feature extraction method and system, and computer equipment. Background Art

[0002] Image feature extraction can reduce the dimensionality of the original input data or recombine the original features for subsequent use. In recent years, as transformers have migrated from natural language processing to computer vision, using transformers to extract image features has become a popular trend. Compared to vision transformers, high-resolution transformers contain richer spatial information and can extract high-resolution image features for image classification tasks. However, the basic transformer block of a high-resolution transformer, consisting of a local window self-attention submodule and a feedforward subnetwork, cannot achieve sufficient interaction between local windows, and the extracted high-resolution features lack global information. Summary of the Invention

[0003] The present invention provides an image feature extraction method and system based on an enhanced high-resolution transformer, as well as a computer device, for overcoming the defects of existing high-resolution transformers, such as the inability to achieve sufficient interaction between local windows and the lack of global information in the extracted high-resolution features.

[0004] To achieve the above objectives, the present invention proposes an image feature extraction method based on an enhanced high-resolution transformer, comprising the following steps:

[0005] Acquire a target image set; the target image set includes camera photos, mobile phone photos, microscopic images and infrared images;

[0006] Preprocessing the images in the target image set to obtain an extracted image set;

[0007] Inputting the extracted image set into the trained feature extraction model, using a convolutional neural network to perform coarse feature extraction on the images in the extracted image set, and then using an enhanced high-resolution transformer network to perform feature interaction on the coarse extracted features to obtain global fusion features;

[0008] The enhanced high-resolution transformer network includes a local window self-attention submodule and a feedforward subnetwork, and both the local window self-attention submodule and the feedforward subnetwork are added with local window shortcuts to realize feature vector interaction between local windows.

[0009] To achieve the above objectives, the present invention further proposes an image feature extraction system based on an enhanced high-resolution transformer, comprising:

[0010] An image acquisition module is used to acquire a target image set; the target image set includes camera photos, mobile phone photos, microscopic images and infrared imaging images;

[0011] A preprocessing module, configured to preprocess the images in the target image set to obtain an extracted image set;

[0012] A feature extraction module is used to input the extracted image set into a trained feature extraction model, use a convolutional neural network to perform coarse feature extraction on the images in the extracted image set, and then use an enhanced high-resolution transformer network to perform feature interaction on the coarse extracted features to obtain global fusion features;

[0013] The enhanced high-resolution transformer network includes a local window self-attention submodule and a feedforward subnetwork, and both the local window self-attention submodule and the feedforward subnetwork are added with local window shortcuts to realize feature vector interaction between local windows.

[0014] To achieve the above object, the present invention further proposes a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0015] To achieve the above object, the present invention further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above method when executed by a processor.

[0016] Compared with the prior art, the present invention has the following beneficial effects:

[0017] The image feature extraction method based on an enhanced high-resolution transformer, provided by this invention, first divides the feature map into local windows of equal size. Each local window represents information about a region of the image. Local window shortcuts (LWS) enable interaction between local windows, allowing global image information to be shared. This adds diverse global image information to the features extracted by the high-resolution transformer. The image features extracted by this invention are suitable for high-precision image classification, resulting in more accurate classification results. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying any creative work.

[0019] Figure 1 Flowchart of the image feature extraction method based on enhanced high-resolution transformer provided by the present invention;

[0020] Figure 2 This is a diagram of the basic transformer block structure of the enhanced high-resolution transformer network in the present invention;

[0021] Figure 3 This is a structural diagram of the local window shortcut in the present invention;

[0022] Figure 4 Schematic diagram of using a fully connected layer to implement interaction between local windows in the present invention;

[0023] Figure 5 It is a visualization comparison diagram of the features of the present invention and the existing method; wherein a is a visualization diagram of the features of the existing method, and b is a visualization diagram of the features of the present invention.

[0024] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0025] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0026] In addition, the technical solutions between the various embodiments of the present invention can be combined with each other, but it must be based on the fact that ordinary technicians in this field can implement it. When the combination of technical solutions is mutually contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0027] The present invention proposes an image feature extraction method based on enhanced high-resolution transformer, such as Figure 1 As shown, the following steps are included:

[0028] 101: Acquire a target image set; the target image set includes camera photos, mobile phone photos, microscopic images, and infrared imaging images.

[0029] 102: Preprocess the images in the target image set to obtain an extracted image set.

[0030] 103: Inputting the extracted image set into the trained feature extraction model, using a convolutional neural network to perform coarse feature extraction on the images in the extracted image set, and then using an enhanced high-resolution transformer network to perform feature interaction on the coarse extracted features to obtain global fusion features;

[0031] The enhanced high-resolution transformer network includes a local window self-attention submodule and a feedforward subnetwork, and both the local window self-attention submodule and the feedforward subnetwork are added with local window shortcuts to realize feature vector interaction between local windows.

[0032] In one embodiment, for step 101, a large number of publicly available datasets are available for image classification. The present invention uses the mini-ImageNet dataset (Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al., "Matching networks for one-shot learning," Advances in neural information processing systems, vol. 29, 2016.), which contains 100 image categories, with 600 images per category. 500 images from each category are randomly selected as the training set, and the remaining images are used as the test set. The category to which each image belongs is its corresponding true label.

[0033] In the next embodiment, for step 102, preprocessing the images in the target image set to obtain the input image set includes:

[0034] The resolutions of the images in the target image set are scaled and normalized to obtain the input image set.

[0035] To better utilize the performance of the feature extraction model, this embodiment adopts the following image set preprocessing method for image classification tasks:

[0036] The input image resolution is uniformly scaled to 224×224;

[0037] Half of the training images are horizontally flipped to increase the number of training images;

[0038] Normalize the image so that the mean is 0 and the standard deviation is 1.

[0039] In another embodiment, for step 103, the feature extraction model is first trained using an NVIDIA GeForce RTX 3090 graphics card with 24GB of video memory and 86GB of RAM. The operating system is Ubuntu 18.04, and the network model is implemented in the PyTorch deep learning framework.

[0040] The main training strategies (or parameters) are set as follows:

[0041] Network parameter initialization: random;

[0042] Number of epochs: 300

[0043] Batch size: 112;

[0044] Optimizer: AdamW optimizer;

[0045] weight decay size: 0.05;

[0046] Loss function: cross entropy loss function;

[0047] Learning rate adjustment strategy: The initial value is 0.000000109375. The learning rate warm-up method is used to reach the maximum learning rate of 0.000109375 at the 20th epoch, and then the cosine learning rate decay strategy is used to drop to 0.00000109375 at the 300th epoch.

[0048] Image enhancement method: random enhancement, mixing, and cropping hybrid strategies are adopted.

[0049] In this embodiment, the high-resolution transformer network is enhanced as Figure 2 As shown, it includes a local window self-attention submodule (LwsLWSA) and a feedforward subnetwork (LwsFFN), and both the local window self-attention submodule and the feedforward subnetwork are added with a local window shortcut (LWS). The local window shortcut of this embodiment is as follows Figure 3 shown.

[0050] Image Matrix Rearrange as input feature map before inputting transformer network Feature Map X LW After inputting the transformer network, three parts of features are extracted at the same time, including LwsLWSA without shortcut (i.e., local window self-attention without LWS), original shortcut (i.e., the original shortcut of the high-resolution transformer network) and LWS, such as Figure 2 shown.

[0051] In LwsLWSA without shortcut, X LW First, it is divided into non-overlapping local windows: The size of the local window is K × K. Then, MHSA (Multi-Head Self-Attention) operation is performed separately in each local window. The MHSA formula for the mth window is as follows:

[0052]

[0053]

[0054]

[0055] Where, a∈{1,…,A} and m∈{1,2,…,M}. A refers to the number of “heads” in the multi-head attention, D refers to the number of channels, N refers to the input pixel value, and M refers to the number of local windows. Refers to the output of MHSA performed in the mth window. The output of LwsLWSA without shortcut is obtained by fusing the outputs of MHSA of different local windows together:

[0056]

[0057] Input feature map X LW When you enter LWS, the Local Window Shortcut (LWS) performs the following operations:

[0058] Divide the input feature map into M non-overlapping local windows;

[0059] Simplifying the expression of each of the local windows;

[0060] Use structured efficient linear layers to achieve interaction between local windows;

[0061] Restore the dimensional information of the local window to obtain the interacted global image features.

[0062] In one embodiment, the mth local window Represents a pixel value of size K 2 In order to make the LWS of the present invention applicable to various transformer network structures without making the network too complicated and increasing the computational cost, each local window is expressed in the simplest way, that is, a value is used to replace the K 2 ×D different local windows of data.

[0063] Simplify the expression of each local window, including:

[0064] The average pooling operation is used to simplify the size of the local window, using a single pixel Represents the overall information of the mth window, P m The calculation method for the i-th element in is as follows:

[0065]

[0066]

[0067]

[0068] Where, is the mth local window; and K is the size of the local window, which is a square with K pixel values ​​in both length and width;

[0069] The average pooling operation is used to aggregate the information contained in all pixels in the local window.

[0070] Connect the local windows after the average pooling operation to realize the input feature X LW To local window single pixel multi-channel features Conversion:

[0071] X LS =Concat[P1,P2,…,P M ]

[0072] Where, (m=1,2,…,M) represents the pixel representation of the mth local window after simplification;

[0073] For single pixel multi-channel feature X LS Simplify and get single channel features

[0074]

[0075] Where σ is the ReLU activation function; is the weight superimposed on the channel.

[0076] Since different channels represent different features of the image, it is not possible to simply average the channel information. This process uses a fully connected layer to weight each channel, and uses an activation function after the fully connected layer to enhance the nonlinearity of the feature. In subsequent network training, the channel weights will be continuously iteratively optimized. The multi-channel feature X LS After processing, it becomes a single-channel feature

[0077] In the next embodiment, to achieve the most adequate interaction between local windows, a fully connected layer can be used between the simplified local windows, such as Figure 4 As shown. At this point, each local window can fully interact with other local windows, and there are many possibilities for weight optimization. However, the number of parameters required for the fully connected layer is M 2 , the computational complexity is O(M 2 ), directly using the fully connected layer will greatly increase the network complexity and computational cost.

[0078] Therefore, the present invention adopts a structured efficient linear layer (ACDC layer) to replace the fully connected layer.

[0079] The interaction between local windows is achieved by using a structured efficient linear layer, including:

[0080] Use structured efficient linear layers to achieve interaction between local windows,

[0081] X LP′ =F acdc (X LP ,W)=σ(ACD′C -1 X LP )

[0082] Where, is the single-pixel single-channel feature of the local window after interaction; σ is the ReLU activation function; and A and D′ are both diagonal matrices with M parameters. C is the second-order discrete cosine transform (DCT) matrix. All elements in C are real numbers and C is an orthogonal matrix. -1 =C T . DCT can be calculated by the Fast Fourier Transform (FFT) algorithm. σ is also a ReLU activation function to enhance the nonlinearity of the interaction. The stacked structure of multi-layer ACDC has been proven to be used to approximately replace the fully connected layer. In order to minimize the network overhead, the present invention only uses one layer of ACDC+ReLU function to realize the interaction between simplified local windows. At this time, the required number of parameters is 2M and the computational complexity is O(MlogM), which is greatly reduced compared to the fully connected layer.

[0083] In another embodiment, the dimensions of the simplified local window remain unchanged after the ACDC layer + ReLU interaction, and each local window is still represented by a single value. In order to enable the output of the LWS to be integrated with the output of the parallel branch in the transformer network, it is necessary to restore the dimensional information of each local window.

[0084] Restore the dimensional information of the local window to obtain the interactive global image features, including:

[0085] The combination of fully connected layer + ReLU activation function is used to restore the channel information of the local window and obtain the single-pixel multi-channel feature X LS ′,

[0086]

[0087] Where σ is the ReLU activation function; is the weight superimposed on the channel;

[0088] Use the identity mapping to restore the size information of the local window. This step will not introduce new parameters.

[0089] X LS′ =Concat[V1,V2,…V M ]

[0090]

[0091] X LW′ =Concat[V1′,V2′,…,V′ M ]

[0092] Where, X LW′ It is the image feature containing global information obtained after the input feature passes through the local window shortcut; W ones is a matrix whose elements are all 1.

[0093] Performance comparison:

[0094] The performance of the image feature extraction method based on the enhanced high-resolution transformer provided by the present invention is compared with the existing image feature extraction method based on the high-resolution transformer. The main difference between the present invention and the existing method is that both the local window self-attention submodule and the feedforward subnetwork are equipped with a local window shortcut.

[0095] The classification accuracy, required parameters, and FLOPs (floating-point operations) of the models trained by LWS-HRFormer (model of the present invention) and HRFormer (model of the existing method) on all test images are shown in Table 1. As can be seen from Table 1, the image feature extraction method LWS proposed in the present invention can improve the classification accuracy of the HRFormer model by 0.70%, and the model complexity and FLOPs overhead are almost unchanged. Figure 5The feature maps extracted by the HRFormer and LWS-HRFormer models trained on the mini-ImageNet dataset are visualized. The feature maps come from the branch with the highest input pixel of the model, with a size of 56×56. The feature maps on the three different channels are output from the 2nd, 18th, and 32nd layer transformer blocks. Figure 5 It is not difficult to find from the feature maps in the transformer block of the same layer that the LWS structure can make the features output by the transformer network more diverse and clearer, indicating that the extracted features increase the diversity of global image information, thereby making the transformer network using LWS perform better.

[0096] Table 1 Performance of different models on the mini-ImageNet dataset

[0097]

[0098] The present invention also proposes an image feature extraction system based on an enhanced high-resolution transformer, comprising:

[0099] An image acquisition module is used to acquire a target image set; the target image set includes camera photos, mobile phone photos, microscopic images and infrared imaging images;

[0100] A preprocessing module, configured to preprocess the images in the target image set to obtain an extracted image set;

[0101] A feature extraction module is used to input the extracted image set into a trained feature extraction model, use a convolutional neural network to perform coarse feature extraction on the images in the extracted image set, and then use an enhanced high-resolution transformer network to perform feature interaction on the coarse extracted features to obtain global fusion features;

[0102] The enhanced high-resolution transformer network includes a local window self-attention submodule and a feedforward subnetwork, and both the local window self-attention submodule and the feedforward subnetwork are added with local window shortcuts to realize feature vector interaction between local windows.

[0103] The present invention further provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0104] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above-mentioned method when executed by a processor.

[0105] The above description is only a preferred embodiment of the present invention and does not limit the patent scope of the present invention. All equivalent structural transformations made by using the contents of the present invention description and drawings under the inventive concept of the present invention, or direct / indirect application in other related technical fields are included in the patent protection scope of the present invention.

Claims

1. A transformer-based image feature extraction method, characterized in that: The following steps are involved: Acquire a target image set; the target image set includes camera photos, mobile phone photos, microscopic images and infrared images; Preprocessing the images in the target image set to obtain an extracted image set; Inputting the extracted image set into the trained feature extraction model, using a convolutional neural network to perform coarse feature extraction on the images in the extracted image set, and then using an enhanced high-resolution transformer network to perform feature interaction on the coarse extracted features to obtain global fusion features; The enhanced high-resolution transformer network includes a local window self-attention submodule and a feedforward subnetwork, each of which is added with a local window shortcut to enable feature vector interaction between local windows; The local window shortcut performs the following operations: Divide the input feature map into Non-overlapping local windows; Simplifying the expression of each of the local windows; Use structured efficient linear layers to achieve interaction between local windows; Restore the dimensional information of the local window to obtain the interacted global image features; Simplify the expression of each local window, including: The average pooling operation is used to simplify the size of the local window, using a single pixel Represents the overall information of the mth window, Middle The elements are calculated as follows: ； ； ； Where, For the A local window; and ; is the size of the local window; Connect the local windows after the average pooling operation to realize the input feature To local window single pixel multi-channel features Conversion: ； Where, ( ) represents the pixel expression of the mth local window after simplification; For the single pixel multi-channel feature Simplify and get single channel features : ； Where, is the ReLU activation function; is the weight superimposed on the channel.

2. The image feature extraction method according to claim 1, wherein: Preprocessing the images in the target image set to obtain an input image set includes: The resolutions of the images in the target image set are scaled and normalized to obtain an input image set.

3. The image feature extraction method according to claim 1, wherein: The interaction between local windows is achieved by using a structured efficient linear layer, including: Use structured efficient linear layers to achieve interaction between local windows, ； Where, It is the single-pixel single-channel feature of the local window after interaction; is the ReLU activation function; and ; and All contain The diagonal matrix of parameters, is the discrete cosine transform matrix of the second kind, The elements in are all real numbers and is an orthogonal matrix, .

4. The image feature extraction method according to claim 1, wherein: Restore the dimensional information of the local window to obtain the interactive global image features, including: The combination of fully connected layer + ReLU activation function is used to restore the channel information of the local window to obtain single-pixel multi-channel features. , ； Where, is the ReLU activation function; is the weight superimposed on the channel; Use the identity mapping to restore the size information of the local window, ； ； ； Where, It is the image feature containing global information obtained after the input feature passes through the local window shortcut; , , It is a matrix whose element values ​​are all 1; M is the number of local windows.

5. A transformer-based image feature extraction system, characterized in that: include: An image acquisition module is used to acquire a target image set; the target image set includes camera photos, mobile phone photos, microscopic images and infrared imaging images; A preprocessing module, configured to preprocess the images in the target image set to obtain an extracted image set; A feature extraction module is used to input the extracted image set into a trained feature extraction model, use a convolutional neural network to perform coarse feature extraction on the images in the extracted image set, and then use an enhanced high-resolution transformer network to perform feature interaction on the coarse extracted features to obtain global fusion features; The enhanced high-resolution transformer network includes a local window self-attention submodule and a feedforward subnetwork, each of which is added with a local window shortcut to enable feature vector interaction between local windows; The local window shortcut performs the following operations: Divide the input feature map into Non-overlapping local windows; Simplifying the expression of each of the local windows; Use structured efficient linear layers to achieve interaction between local windows; Restore the dimensional information of the local window to obtain the interacted global image features; Simplify the expression of each local window, including: The average pooling operation is used to simplify the size of the local window, using a single pixel Represents the overall information of the mth window, Middle The elements are calculated as follows: ； ； ； Where, For the A local window; and ; is the size of the local window; Connect the local windows after the average pooling operation to realize the input feature To local window single pixel multi-channel features Conversion: ； Where, ( ) represents the pixel expression of the mth local window after simplification; For the single pixel multi-channel feature Simplify and get single channel features : ； Where, is the ReLU activation function; is the weight superimposed on the channel.

6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 4 are implemented.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images