Image Classification Method Based on Adaptive Region of Interest Transformer

Abstract

The present invention discloses an image classification method based on adaptive attention region Transformer: Step 1, select n images as training sample set P = {p1, p2, ..., p n}, then the i-th = {1, 2, ..., n} image p in the training sample set P i Use a scale-invariant feature transform to obtain N feature points. Step 2: Obtain the center point M(x, y) of the densest region among the N feature points in step 1. Step 3: Determine the region of interest and the non-region of interest. Step 4: Extract the features F1 and F2 of the region of interest and non-region of interest, respectively, and fuse them to form the image feature F. Step 5: Use the MART network to fuse the image feature F obtained in step 4 with the new classification label and the new location label to form the image feature label FT. Step 6: Perform image classification. This method solves the problem of reduced classification accuracy and robustness in existing methods that indiscriminately divide the image into blocks of the same size for image classification.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing methods, and in particular relates to an image classification method based on an adaptive region of interest transformer. Background Art

[0002] Image classification, which distinguishes categories based on feature differences between them, is an important fundamental problem in the field of computer vision. It is also an important foundation for high-level visual tasks such as image detection and behavior analysis.

[0003] Currently, the main image classification methods at home and abroad include convolutional neural network-based image classification methods and Transformer-based image classification methods. Among them, the Transformer-based image classification method divides the image into image blocks of equal size and then learns the attention features between the image blocks. However, the attention areas in different images vary, and existing methods do not take into account the different levels of attention paid to the entire image by different image regions. This method of indiscriminately dividing the image into image blocks of equal size for image classification results in the inability to more effectively learn the features of the attention areas, resulting in defects such as reduced classification accuracy and robustness. Summary of the Invention

[0004] The purpose of the present invention is to provide an image classification method based on an adaptive region of interest transformer, which solves the problem of reduced classification accuracy and robustness caused by the existing method of indiscriminately dividing the image into image blocks of the same size for image classification.

[0005] The technical solution adopted by the present invention is an image classification method based on Adaptive Attention Region Transformer (AART), specifically: Step 1, select n images as a training sample set P = {p1, p2, ..., p n}, then the i-th = {1, 2, ..., n} image p in the training sample set P i Use scale-invariant feature transformation to obtain N feature points;

[0006] Step 2: Obtain the center point M(x,y) of the densest area among the N feature points in step 1;

[0007] Step 3: Determine the focus area and non-focus area based on the center point M(x, y) of the most dense area obtained in step 2;

[0008] Step 4: extract the features of the focus area and non-focus area obtained in step 3 as F1 and F2 respectively, and fuse them to form the image feature F;

[0009] Step 5: The Multi-scale Attention Region Transformer (MART) network creates a new classification label and a new location label, and then fuses the image feature F obtained in step 4 with the new classification label and the new location label to form an image feature label FT.

[0010] In step 6, the image feature label FT obtained in step 5 is used to learn the attention between image blocks to obtain the attention feature AT, thereby performing image classification.

[0011] The present invention is also characterized in that

[0012] The specific process of step 1 is:

[0013] Step 1.1, select n images as the training sample set P = {p1, p2, ..., p n};

[0014] Step 1.2: The i-th image p in the training sample set P obtained in step 1.1 is i Use the cv2.normalize() method to shrink the image value to between 0 and 255, thereby obtaining the image information quantized into int8;

[0015] Step 1.3: After step 1.2, the quantized image information is obtained and input into the cv2.xfeatures2d.SIFT_create().detectAndCompute() method to obtain the image p i N feature points.

[0016] The specific process of step 2 is:

[0017] Step 2.1: Input the N feature points of the image obtained in step 1 into the numpy.mean() method to obtain the mean coordinate position Mean(x1,y1) of the N feature point coordinates.

[0018] In step 2.2, the mean coordinate position Mean(x1,y1) obtained in step 2.1 is input into the seeds parameter of the sklearn.cluster.MeanShift() method, and the bin_seeding parameter of the method is set to True; the fit() method of the method is called, and the N feature points obtained in step 1.2 are input into the fit() method to obtain the clustering information L of the N feature points;

[0019] Step 2.3, call the cluster_centers_ attribute of the cluster information L of the N feature points obtained in step 2.2 to obtain the cluster center coordinates among the N feature points, that is, the center point M(x, y) of the most dense area.

[0020] The specific process of step 3 is:

[0021] Step 3.1: Use the center point M(x,y) of the densest area obtained in step 2.3 as the center point M'(x',y') of the area of interest;

[0022] Step 3.2, set the width W and height H of the focus area, and calculate the coordinates A(x) of the upper left corner of the focus area according to the center point M'(x',y') of the focus area obtained in step 3.1. a ,y a ) and the lower right corner coordinate B(x b ,y b );

[0023] Step 3.3, according to the coordinates of the upper left corner vertex A(x a ,y a ) and the lower right corner vertex coordinate B(x b ,y b ) to determine whether the coordinates are beyond the image range; create and As the upper left vertex A(x a ,y a ) on the X-axis and Y-axis with the lower right corner vertex B (x b ,y b ) Values on the X and Y axes that exceed the image range; the original width and height of the image are OW and OH;

[0024] Among them, the value of the upper left corner vertex A of the focus area on the X axis exceeds the image range The calculation method is shown in formula (1):

[0025]

[0026] In formula (1), x′ is the X-axis coordinate of the center point M′ of the region of interest, and W is the width of the region of interest;

[0027] The value of the upper left corner vertex A of the focus area on the Y axis exceeds the image range The calculation method is shown in formula (2):

[0028]

[0029] In formula (2), y′ is the Y-axis coordinate of the center point M′ of the focus area, and H is the height of the focus area.

[0030] The value of point B, the lower right corner of the focus area, on the X axis exceeds the image range. The calculation method is shown in formula (3):

[0031]

[0032] In formula (3), x′ is the X-axis coordinate of the center point M′ of the focus region, W is the width of the focus region, and OW is the original width of the image.

[0033] The value of point B, the lower right corner of the focus area, on the Y axis exceeds the image range. The calculation method is shown in formula (4):

[0034]

[0035] In formula (4), y′ is the Y-axis coordinate of the center point M′ of the focus area, H is the height of the focus area, and OH is the original height of the image.

[0036] Step 3.4: The upper left corner vertex A(x a ,y a ) on the X-axis and Y-axis with the lower right corner vertex B (x b ,y b ) values on the X and Y axes that are outside the image range and Then we need to calculate the upper left vertex A'(x' a ,y′ a ) and the lower right vertex B'(x′ b ,y′ b )coordinate;

[0037] Among them, the X-axis coordinate x′ of point A' a The calculation method of is shown in formula (5):

[0038]

[0039] In formula (5), x′ is the X-axis coordinate of the center point M′ of the focus area, W is the width of the focus area, It is the value of the upper left corner vertex A of the focus area on the X axis that exceeds the image range. The value of point B, the lower right corner of the focus area, on the X-axis exceeds the image range;

[0040] The Y-axis coordinate y′ of point A' a The calculation method of is shown in formula (6):

[0041]

[0042] In formula (6), y′ is the Y-axis coordinate of the center point M′ of the focus area, H is the height of the focus area, It is the value of the upper left corner vertex A of the focus area on the Y axis that exceeds the image range. The value of point B, the lower right corner of the focus area, on the Y axis exceeds the image range;

[0043] The X-axis coordinate x′ of point B' b The calculation method of is shown in formula (7):

[0044]

[0045] In formula (7), x′ is the X-axis coordinate of the center point M′, W is the width of the area of interest, It is the value of the upper left corner vertex A of the focus area on the X axis that exceeds the image range. The value of point B, the lower right corner of the focus area, on the X-axis exceeds the image range;

[0046] The Y-axis coordinate y′ of point B' b The calculation method of is shown in formula (8):

[0047]

[0048] In formula (8), y′ is the Y-axis coordinate of the center point M′, H is the height of the area of interest, It is the value of the upper left corner vertex A of the focus area on the Y axis that exceeds the image range. The value of point B, the lower right corner of the focus area, on the Y axis exceeds the image range;

[0049] Step 3.5: The upper left corner vertex A'(x') after the focus area obtained in step 3.4 is out of range a , y′ a ) and the lower right vertex B'(x′ b , y′ b ) coordinates, the image of the focus area can be obtained, and the remaining image area is the image of the non-focus area.

[0050] The specific process of step 4 is:

[0051] Step 4.1, use the torch.as_tensor() method to convert the image information of the region of interest and the non-region of interest obtained in step 3.5 into the image information of the region of interest and the non-region of interest of float32 type;

[0052] Step 4.2: Convert the image information of the region of interest obtained in step 4.1 into float32 type using a smaller convolution kernel size of The step size and convolution kernel size are the same as S, the number of channels is 2S×2S×3, and the convolution operation is performed to extract image features to obtain the small-scale fine-grained features F1 of the focus area. The dimension of the fine-grained feature F1 is The non-interest area image information converted into float32 type in step 4.1 is convolved with a larger convolution kernel size of 2S×2S, a step size of 2S, and a channel number of 2S×2S×3 to extract image features, thereby dividing the non-interest area image into larger image blocks, and then obtaining the large-scale coarse-grained feature F2 of the non-interest area. The dimension of the coarse-grained feature F2 is

[0053] Step 4.3, merge the second and third dimensions of the small-scale fine-grained features F1 of the focus area obtained in step 4.2. The dimension of the merged fine-grained features F1 is Merge the second and third dimensions of the large-scale coarse-grained feature F2 of the non-focused area obtained in step 4.2. The dimension of the merged coarse-grained feature F2 is The fine-grained feature F1 and the coarse-grained feature F2 are spliced in the second dimension. The dimension of the spliced image feature map F is Then swap the first and second dimensions of the image feature map F. The dimension of the image feature map F after swapping is

[0054] The specific process of step 5 is:

[0055] Step 5.1: Input the image feature map obtained in step 4 into the MART network; the network structure of the MART network is as follows: Figure 2 As shown in the figure, it consists of a position label position, a classification label cls_token, a Transformer Block and an MLP Head. First, the input image feature map is concatenated with the position label position and the classification label cls_token to form an image feature label; then the image feature label is input into the Transformer Block to learn the attention between image blocks to obtain the attention feature; the MART network creates a row vector tensor named cls_token as the classification label, and the dimension of cls_token is [1, 12S 2 ]; Concatenate the classification label cls_token and the feature map F in the first dimension. The dimension of the new feature map F' after concatenation is

[0056] Step 5.2, create a dimension The vector tensor is named position as the position label, and the new feature map F' obtained in step 5.1 and the position label position are added to form the image feature label FT. The dimension of the image feature label FT is

[0057] The specific process of step 6 is:

[0058] In step 6.1, the image feature label FT obtained in step 5.2 is input into the Transformer Block to learn the inter-block attention to obtain the image attention feature AT.

[0059] Step 6.2: Input the attention feature AT of the image obtained in step 6.1 into the classifier MLP Head. The attention feature AT is calculated through the linear connection layer to obtain the image p. i The probability of belonging to each category, and then find the maximum value from these probabilities, output the category with the largest probability, that is, the output image p i The classification results;

[0060] Step 6.3, repeat steps 1.2 to 6.2 to obtain image p i+1 The classification results are calculated until i+1>n is reached and the execution ends.

[0061] The beneficial effects of the present invention are:

[0062] Compared to some existing Transformer-based image classification methods, the proposed method uses an adaptive region-of-attention Transformer. This method obtains image feature points and then calculates the center point of the densest region of these feature points. This center point is used to define the region of interest, with the remaining image area as the non-region of interest. The region of interest is then divided into small blocks, where fine-grained features are extracted using a smaller convolution kernel; the non-region of interest is divided into larger blocks, where coarse-grained features are extracted using a larger convolution kernel. This allows for the extraction of image features of varying scales from different image regions, resulting in more accurate image classification. Existing Transformer-based image classification methods indiscriminately divide images into blocks of equal size. However, since different image regions have varying degrees of attention to the overall image, dividing the image into blocks of equal size for classification results in an inability to effectively learn features from the regions of interest, resulting in reduced classification accuracy and robustness. The proposed method, based on adaptive region-of-attention, divides different image regions into blocks of varying sizes, then learns attention features between the blocks. Classification is then performed using these learned attention features, resulting in higher classification accuracy and greater robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 This is a framework diagram of the image classification method based on the adaptive region of interest Transformer of the present invention;

[0064] Figure 2 This is a multi-scale attention region transformer (MART) network structure diagram of the image classification method based on the adaptive attention region transformer of the present invention. DETAILED DESCRIPTION

[0065] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0066] The present invention provides an image classification method based on adaptive attention region Transformer, the method framework is shown in the figure Figure 1 As shown, the specific steps include:

[0067] Step 1: Select n images as the training sample set P = {p1, p2, ..., p n}, then the i-th = {1, 2, ..., n} image p in the training sample set P i Use Scale-Invariant Feature Transform (SIFT) to obtain N feature points;

[0068] The specific process of step 1 is:

[0069] Step 1.1, select n images as the training sample set P = {p1, p2, ..., p n},

[0070] Step 1.2: The i-th image p in the training sample set P obtained in step 1.1 is i Use the cv2.normalize() method to shrink the image value to between 0 and 255, thereby obtaining the image information quantized into int8;

[0071] Step 1.3: After step 1.2, the quantized image information is obtained and input into the cv2.xfeatures2d.SIFT_create().detectAndCompute() method to obtain N feature points of image i.

[0072] Step 2: Use the Meanshift clustering algorithm to obtain the center point M(x,y) of the most dense area among the N feature points obtained in step 1;

[0073] The specific process of step 2 is as follows:

[0074] Step 2.1: Input the N feature points of the image obtained in step 1 into the numpy.mean() method to obtain the mean coordinate position Mean(x1,y1) of the N feature point coordinates.

[0075] In step 2.2, input the mean coordinate position Mean(x1,y1) obtained in step 2.1 into the seeds parameter of the sklearn.cluster.MeanShift() method, and set the bin_seeding parameter of this method to True. Call the fit() method of this method and input the N feature points obtained in step 1.2 into the fit() method to obtain the clustering information L of the N feature points.

[0076] Step 2.3, call the cluster_centers_ attribute of the cluster information L of the N feature points obtained in step 2.2 to obtain the cluster center coordinates among the N feature points, that is, the center point M(x, y) of the most dense area.

[0077] Step 3: Based on the center point M(x, y) of the densest area obtained in step 2, cut an image area with a width w and a height H as the focus area, and the remaining image area as the non-focus area, and determine the focus area and the non-focus area;

[0078] The specific process of step 3 is:

[0079] Step 3.1: Use the center point M(x,y) of the densest area obtained in step 2.3 as the center point M'(x',y') of the area of interest;

[0080] Step 3.2, set the width W and height H of the focus area, and calculate the coordinates A (x) of the upper left corner of the focus area according to the center point M'(x', y') of the focus area obtained in step 3.1. a ,y a ) and the lower right corner coordinate B(x b ,y b );

[0081] Step 3.3, according to the coordinates of the upper left corner vertex A(x a ,y a ) and the lower right corner vertex coordinate B(x b ,y b ) to determine whether the coordinates are beyond the image range; create and As the upper left vertex A(x a ,y a ) on the X-axis and Y-axis with the lower right corner vertex B (x b ,yb ) Values on the X and Y axes that exceed the image range; the original width and height of the image are OW and OH;

[0082] Among them, the value of the upper left corner vertex A of the focus area on the X axis exceeds the image range The calculation method is shown in formula (1):

[0083]

[0084] In formula (1), x′ is the X-axis coordinate of the center point M′ of the region of interest, and W is the width of the region of interest;

[0085] The value of the upper left corner vertex A of the focus area on the Y axis exceeds the image range The calculation method is shown in formula (2):

[0086]

[0087] In formula (2), y′ is the Y-axis coordinate of the center point M′ of the focus area, and H is the height of the focus area.

[0088] The value of point B, the lower right corner of the focus area, on the X axis exceeds the image range. The calculation method is shown in formula (3):

[0089]

[0090] In formula (3), x′ is the X-axis coordinate of the center point M′ of the focus region, W is the width of the focus region, and OW is the original width of the image.

[0091] The value of point B, the lower right corner of the focus area, on the Y axis exceeds the image range. The calculation method is shown in formula (4):

[0092]

[0093] In formula (4), y′ is the Y-axis coordinate of the center point M′ of the focus area, H is the height of the focus area, and OH is the original height of the image.

[0094] Step 3.4: The upper left corner vertex A(x a ,y a ) on the X-axis and Y-axis with the lower right corner vertex B (x b ,y b ) values on the X and Y axes that are outside the image range and Then we need to calculate the upper left vertex A'(x' a , y′ a) and the lower right vertex B'(x′ b , y′ b )coordinate;

[0095] Among them, the X-axis coordinate x′ of point A' a The calculation method of is shown in formula (5):

[0096]

[0097] In formula (5), x′ is the X-axis coordinate of the center point M′ of the focus area, W is the width of the focus area, It is the value of the upper left corner vertex A of the focus area on the X axis that exceeds the image range. The value of point B, the lower right corner of the focus area, on the X-axis exceeds the image range;

[0098] The Y-axis coordinate y′ of point A' a The calculation method of is shown in formula (6):

[0099]

[0100] In formula (6), y′ is the Y-axis coordinate of the center point M′ of the focus area, H is the height of the focus area, It is the value of the upper left corner vertex A of the focus area on the Y axis that exceeds the image range. The value of point B, the lower right corner of the focus area, on the Y axis exceeds the image range;

[0101] The X-axis coordinate x′ of point B' b The calculation method of is shown in formula (7):

[0102]

[0103] In formula (7), x′ is the X-axis coordinate of the center point M′, W is the width of the area of interest, It is the value of the upper left corner vertex A of the focus area on the X axis that exceeds the image range. The value of point B, the lower right corner of the focus area, on the X-axis exceeds the image range;

[0104] The Y-axis coordinate y′ of point B' b The calculation method of is shown in formula (8):

[0105]

[0106] In formula (8), y′ is the Y-axis coordinate of the center point M′, H is the height of the area of interest, It is the value of the upper left corner vertex A of the focus area on the Y axis that exceeds the image range. The value of point B, the lower right corner of the focus area, on the Y axis exceeds the image range;

[0107] Step 3.5: The upper left corner vertex A'(x') after the focus area obtained in step 3.4 is out of range a , y′ a ) and the lower right vertex B'(x′ b , y′ b ) coordinates, the image of the focus area can be obtained, and the remaining image area is the image of the non-focus area.

[0108] Step 4: extract the features of the focus area and non-focus area obtained in step 3 as F1 and F2 respectively, and then fuse them to form the image feature F;

[0109] The specific process of step 4 is:

[0110] Step 4.1, use the torch.as_tensor() method to convert the image information of the region of interest and the non-region of interest obtained in step 3.5 into the image information of the region of interest and the non-region of interest of float32 type;

[0111] Step 4.2: Convert the image information of the region of interest obtained in step 4.1 into float32 type using a smaller convolution kernel size of (For example, the convolution kernel is 8×8), the stride is the same as the convolution kernel size S, the number of channels is 2S×2S×3, and the convolution operation is performed to extract image features. Thus, the image of the focus area is divided into smaller image blocks, and the small-scale fine-grained features F1 of the focus area are obtained. The dimension of the fine-grained feature F1 is The non-focus area image information converted into float32 type obtained in step 4.1 is convolved with a larger convolution kernel size of 2S×2S (for example, the convolution kernel is 16×16), the step size is the same as the convolution kernel size of 2S, and the number of channels is 2S×2S×3. The convolution operation is performed to extract image features, thereby dividing the non-focus area image into larger image blocks, and then obtaining the large-scale coarse-grained feature F2 of the non-focus area. The dimension of the coarse-grained feature F2 is

[0112] Step 4.3, merge the second and third dimensions of the small-scale fine-grained features F1 of the focus area obtained in step 4.2. The dimension of the merged fine-grained features F1 is Merge the second and third dimensions of the large-scale coarse-grained feature F2 of the non-focused area obtained in step 4.2. The dimension of the merged coarse-grained feature F2 is The fine-grained feature F1 and the coarse-grained feature F2 are spliced in the second dimension. The dimension of the spliced image feature map F is Then swap the first and second dimensions of the image feature map F. The dimension of the image feature map F after swapping is

[0113] Step 5: Input the image feature map obtained in step 4 into the Multi-Scale Attention Region Transformer (MART) network. The MART network creates a new tensor named cls_token as the classification label and a new tensor named position as the position label. It then fuses the image feature F obtained in step 4 with the labels cls_token and position to form the image feature label FT.

[0114] The specific process of step 5 is:

[0115] Step 5.1: Input the image feature map obtained in step 4 into the MART network; the network structure of the MART network is as follows: Figure 2 As shown in the figure, it consists of a position label position, a classification label cls_token, a Transformer Block and an MLP Head. First, the input image feature map is concatenated with the position label position and the classification label cls_token to form an image feature label; then the image feature label is input into the Transformer Block to learn the attention between image blocks to obtain the attention feature. The structure of the Transformer Block is exactly the same as that of the Transformer Encoder of the Vision Transformer (ViT); finally, the learned attention feature is input into the MLPHead to obtain the classification result. The structure of the MLPHead is exactly the same as that of the MLPHead of ViT; the MART network creates a row vector tensor named cls_token as the classification label. The dimension of cls_token is [1, 12S 2 ]; Concatenate the classification label cls_token and the feature map F in the first dimension. The dimension of the new feature map F' after concatenation is

[0116] Step 5.2, create a dimension The vector tensor is named position as the position label, and the new feature map F' obtained in step 5.1 and the position label position are added to form the image feature label FT. The dimension of the image feature label FT is

[0117] Step 6: Input the image feature label FT obtained in step 5 into the Transformer Block to learn the attention between image blocks to obtain the attention feature AT, and then input AT into the classifier MLP Head to obtain the image p i The classification result is obtained by repeating steps 1 to 6 to finally obtain the image p i+1 The classification results are calculated until i+1>n is reached and the execution ends.

[0118] The specific process of step 6 is:

[0119] In step 6.1, the image feature label FT obtained in step 5.2 is input into the Transformer Block to learn the inter-block attention to obtain the image attention feature AT.

[0120] Step 6.2: Input the attention feature AT of the image obtained in step 6.1 into the classifier MLPHead. The attention feature AT is calculated through the linear connection layer to obtain the image p. i The probability of belonging to each category, and then find the maximum value from these probabilities, output the category with the largest probability, that is, the output image p i The classification results.

[0121] Step 6.3: Repeat steps 1.2 to 6.2 to obtain image p i+1 The classification results are calculated until i+1>n is reached and the execution ends.

[0122] The method of the present invention obtains the feature points of the image, and then calculates the center point of the most densely populated area among these feature points, and cuts out an area with the center point as the focus area, and the remaining area as the non-focus area. The focus area is then divided into small-sized image blocks, and fine-grained features are extracted using a smaller convolution kernel; the non-focus area is divided into large-sized image blocks, and coarse-grained features are extracted using a larger convolution kernel. In this way, image features of different scales are extracted from different areas of the image, small-scale fine-grained features are extracted from the focus area, and large-scale coarse-grained features are extracted from the non-focus area. The coarse and fine-grained features of different scales are then fused into a feature map, and the feature map is input into the multi-scale attention region Transformer (MART) network to learn the attention features between image blocks, and finally the attention features are used for classification.

[0123] Experimental part:

[0124] Example 1

[0125] Dataset:

[0126] CIFAR-10 has 10 common categories, and the dataset includes 50,000 training images and 10,000 test images. The image size in the dataset is 32×32, but for ease of input into the method, the images are resized to 224×224 during training.

[0127] Experimental environment:

[0128] A 3090 graphics card was used for training, and the batch size was set to 128. The models and methods used in the experiment, including our method, were implemented using the Pytorch library and the Pytorch Visual Model Library (timm). The OW in the method is 224, the OH is 224, the W is 64, the H is 64, and the S is 8. The initial learning rate is 0.01, the minimum learning rate is set to 2e-4, and the weight decay is set to 5e-4. The SGD optimizer is used with a momentum of 0.9. The learning rate is adjusted using cosine annealing, and Tmax is set to 60. A total of 100 epochs are trained. In order to increase the diversity of data forms, appropriate data augmentation is performed on the data. The data augmentation methods include: random cropping, random horizontal flipping, and normalization.

[0129] Example 2

[0130] Dataset:

[0131] CIFAR 100 has 100 common categories, and the dataset includes 50,000 images for training and 10,000 for testing. The images in the dataset are 32×32 color images. To facilitate input into the method, the images are resized to 224×224 during training.

[0132] Experimental environment:

[0133] A 3090 graphics card was used for training, and the batch size was set to 128. The models and methods used in the experiment, including our method, were implemented using the Pytorch library and the Pytorch Visual Model Library (timm). The OW in the method is 224, the OH is 224, the W is 64, the H is 64, and the S is 8. The initial learning rate is 0.01, the minimum learning rate is set to 2e-4, and the weight decay is set to 5e-4. The SGD optimizer is used with a momentum of 0.9. The learning rate is adjusted using cosine annealing, and Tmax is set to 60. A total of 100 epochs are trained. In order to increase the diversity of data forms, appropriate data augmentation is performed on the data. The data augmentation methods include: random cropping, random horizontal flipping, and normalization.

[0134] Example 3

[0135] To facilitate the comparison of different methods, this experiment uses the same parameter configuration for different methods. In order to verify the feasibility of the method proposed in this paper, some representative image classification methods with VIT as the backbone network were selected for comparison. Specifically, they include ViT-B, DeiT-B, CrossViT-B, CrossViT-15 and the AART method proposed in this paper. We sorted out the top 1 and top 5 accuracy of different methods in the evaluation of CIFAR 10 and CIFAR 100 validation sets. The parameters compared between different methods include: parameter number (Params (M)), computational complexity (FLOPs (G)) and evaluation accuracy (Top-1 Acc. (%)).

[0136] Experimental results:

[0137] Table 1, experimental results of various methods on the CIFAR 10 dataset:

[0138]

[0139] Table 2, experimental results of various methods on the CIFAR 100 dataset:

[0140]

[0141] Experimental summary:

[0142] By comparing the Top-1 Acc. in Table 1 with the Top-1 Acc. and Top-5 Acc. in Table 2, we find that our AART method consistently achieves higher classification accuracy and stronger robustness than other methods.

Claims

1. Image classification method based on adaptive attention region Transformer, characterized by: Specifically: Step 1, select n images as the training sample set P = {p1, p2, ..., p n }, then the i={1,2,...,n}th image p in the training sample set P i Use scale-invariant feature transformation to obtain N feature points; Step 2: Obtain the center point M(x,y) of the densest area among the N feature points in step 1; Step 3: Determine the focus area and non-focus area based on the center point M(x, y) of the densest area obtained in step 2; Step 4: extract the features of the focus area and non-focus area obtained in step 3 as F1 and F2 respectively, and fuse them to form the image feature F; Step 5: The MART network creates a new classification label and a new location label, and then fuses the image feature F obtained in step 4 with the new classification label and the new location label to form an image feature label FT. The specific process of step 5 is: Step 5.1: Input the image feature map obtained in step 4 into the MART network. The network structure of the MART network consists of a position label, a classification label, a Transformer Block, and an MLP Head. First, the input image feature map is concatenated with the position label and the classification label, cls_token, to form an image feature label. Then, the image feature label is input into the Transformer Block to learn the attention between image blocks and obtain the attention feature. The MART network creates a row vector tensor named cls_token as the classification label. The dimension of cls_token is [1, 12S 2 ]; Concatenate the classification label cls_token and the feature map F in the first dimension. The dimension of the new feature map F' after concatenation is Step 5.2, create a dimension The vector tensor is named position as the position label, and the new feature map F' obtained in step 5.1 and the position label position are added to form the image feature label FT. The dimension of the image feature label FT is In step 6, the image feature label FT obtained in step 5 is used to learn the attention between image blocks to obtain the attention feature AT, thereby performing image classification.

2. The image classification method based on adaptive region of interest transformer according to claim 1, characterized in that: The specific process of step 1 is: Step 1.1, select n images as the training sample set P = {p1, p2, ..., p n }; Step 1.2: The i-th image p in the training sample set P obtained in step 1.1 is i Use the cv2.normalize() method to shrink the image value to between 0 and 255, thereby obtaining the image information quantized into int8; Step 1.3: After step 1.2, the quantized image information is obtained and input into the cv2.xfeatures2d.SIFT_create().detectAndCompute() method to obtain the image p i N feature points.

3. The image classification method based on adaptive region of interest transformer according to claim 2, characterized in that: The specific process of step 2 is: Step 2.1: Input the N feature points of the image obtained in step 1 into the numpy.mean() method to obtain the mean coordinate position Mean(x1,y1) of the N feature point coordinates. In step 2.2, the mean coordinate position Mean(x1,y1) obtained in step 2.1 is input into the seeds parameter of the sklearn.cluster.MeanShift() method, and the bin_seeding parameter of the method is set to True; the fit() method of the method is called, and the N feature points obtained in step 1.2 are input into the fit() method to obtain the clustering information L of the N feature points; Step 2.3, call the cluster_centers_ attribute of the cluster information L of the N feature points obtained in step 2.2 to obtain the cluster center coordinates among the N feature points, that is, the center point M(x, y) of the most dense area.

4. The image classification method based on adaptive region of interest transformer according to claim 3, characterized in that: The specific process of step 3 is: Step 3.1: Use the center point M(x,y) of the densest area obtained in step 2.3 as the center point M'(x',y') of the area of interest; Step 3.2, set the width W and height H of the focus area, and calculate the coordinates A(x) of the upper left corner of the focus area according to the center point M'(x',y') of the focus area obtained in step 3.

1. a ,y a ) and the lower right corner coordinate B(x b ,y b ); Step 3.3, according to the coordinates of the upper left corner vertex A(x a ,y a ) and the lower right corner vertex coordinate B(x b ,y b ) to determine whether the coordinates are beyond the image range; create and As the upper left vertex A(x a ,y a ) on the X-axis and Y-axis with the lower right corner vertex B (x b ,y b ) Values on the X and Y axes that exceed the image range; the original width and height of the image are OW and OH; Among them, the value of the upper left corner vertex A of the focus area on the X axis exceeds the image range The calculation method is shown in formula (1): In formula (1), x′ is the X-axis coordinate of the center point M′ of the region of interest, and W is the width of the region of interest; The value of the upper left corner vertex A of the focus area on the Y axis exceeds the image range The calculation method is shown in formula (2): In formula (2), y′ is the Y-axis coordinate of the center point M′ of the area of interest, and H is the height of the area of interest; The value of point B, the lower right corner of the focus area, on the X axis exceeds the image range. The calculation method is shown in formula (3): In formula (3), x′ is the X-axis coordinate of the center point M′ of the focus area, W is the width of the focus area, and OW is the original width of the image; The value of point B, the lower right corner of the focus area, on the Y axis exceeds the image range. The calculation method is shown in formula (4): In formula (4), y′ is the Y-axis coordinate of the center point M′ of the region of interest, H is the height of the region of interest, and OH is the original height of the image; Step 3.4, according to the upper left corner vertex A(x a ,y a ) on the X-axis and Y-axis with the lower right corner vertex B (x b ,y b ) values on the X and Y axes that are outside the image range and Then we need to calculate the upper left vertex A'(x' a ,y′ a ) and the lower right vertex B'(x′ b ,y′ b )coordinate; Among them, the X-axis coordinate x′ of point A' a The calculation method of is shown in formula (5): In formula (5), x′ is the X-axis coordinate of the center point M′ of the focus area, W is the width of the focus area, It is the value of point A, the upper left corner of the focus area, that exceeds the image range on the X axis. The value of point B, the lower right corner of the focus area, on the X-axis exceeds the image range; The Y-axis coordinate y′ of point A' a The calculation method of is shown in formula (6): In formula (6), y′ is the Y-axis coordinate of the center point M′ of the focus area, H is the height of the focus area, It is the value of the upper left corner vertex A of the focus area on the Y axis that exceeds the image range. The value of point B, the lower right corner of the focus area, on the Y axis exceeds the image range; The X-axis coordinate x′ of point B' b The calculation method of is shown in formula (7): In formula (7), x′ is the X-axis coordinate of the center point M′, W is the width of the area of interest, It is the value of point A, the upper left corner of the focus area, that exceeds the image range on the X axis. The value of point B, the lower right corner of the focus area, on the X-axis exceeds the image range; The Y-axis coordinate y′ of point B' b The calculation method of is shown in formula (8): In formula (8), y′ is the Y-axis coordinate of the center point M′, H is the height of the area of interest, It is the value of the upper left corner vertex A of the focus area on the Y axis that exceeds the image range. The value of point B, the lower right corner of the focus area, on the Y axis exceeds the image range; Step 3.5: The upper left corner vertex A'(x') after the focus area obtained in step 3.4 is out of range a ,y′ a ) and the lower right vertex B'(x′ b ,y′ b ) coordinates, the image of the focus area can be obtained, and the remaining image area is the image of the non-focus area.

5. The image classification method based on adaptive region of interest transformer according to claim 4, characterized in that: The specific process of step 4 is: Step 4.1, use the torch.as_tensor() method to convert the image information of the region of interest and the non-region of interest obtained in step 3.5 into the image information of the region of interest and the non-region of interest of float32 type; Step 4.2: Convert the image information of the region of interest obtained in step 4.1 into float32 type using a smaller convolution kernel size of The step size and convolution kernel size are the same as S, the number of channels is 2S×2S×3, and the convolution operation is performed to extract image features to obtain the small-scale fine-grained features F1 of the focus area. The dimension of the fine-grained feature F1 is The non-interest area image information converted into float32 type in step 4.1 is convolved with a larger convolution kernel size of 2S×2S, a step size of 2S, and a channel number of 2S×2S×3 to extract image features, thereby dividing the non-interest area image into larger image blocks, and then obtaining the large-scale coarse-grained feature F2 of the non-interest area. The dimension of the coarse-grained feature F2 is Step 4.3, merge the second and third dimensions of the small-scale fine-grained features F1 of the focus area obtained in step 4.

2. The dimension of the merged fine-grained features F1 is Merge the second and third dimensions of the large-scale coarse-grained feature F2 of the non-focused area obtained in step 4.

2. The dimension of the merged coarse-grained feature F2 is The fine-grained feature F1 and the coarse-grained feature F2 are spliced in the second dimension. The dimension of the spliced image feature map F is Then swap the first and second dimensions of the image feature map F. The dimension of the image feature map F after swapping is 6. The image classification method based on adaptive region of interest transformer according to claim 1, characterized in that: The specific process of step 6 is: Step 6.1: Input the image feature label FT obtained in step 5.2 into the Transformer Block to learn the inter-block attention to obtain the image attention feature AT; Step 6.2: Input the attention feature AT of the image obtained in step 6.1 into the classifier MLP Head. The attention feature AT is calculated through the linear connection layer to obtain the image p. i The probability of belonging to each category, and then find the maximum value from these probabilities, output the category with the largest probability, that is, the output image p i The classification results; Step 6.3, repeat steps 1.2 to 6.2 to obtain image p i+1 The classification results are calculated until i+1>n is reached and the execution ends.

Images