An image direction recognition method fusing convolution and ViT

Abstract

The application belongs to the technical field of image classification and computer vision, and discloses an image direction recognition method fusing convolution and ViT. For the image direction recognition method, most of which consider semantic information of the image, the space and direction information are not considered enough. A direction pyramid ViT model is proposed, which fuses a convolutional neural network and a ViT framework to realize automatic recognition of the image direction. MOAB captures relative position information between a center subgraph and a surrounding neighborhood subgraph under different scales; a conditional position encoding and a deformable convolution are combined, namely DCPE. The deformable convolution is added in the conditional position encoding, which can adapt to input images of different sizes and express the rotation characteristics of the image; a direction convolution method is applied to Self-Attentation in the TransformerEncoder Block, namely DTEB. The DTEB compresses the resolution of the shallow features and improves the feature extraction capability. Through experiments, it is verified that the method more comprehensively expresses the direction semantics of the image, and is superior to the current most advanced model in accuracy and performance.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image classification and computer vision, and particularly relates to an image direction recognition method fusing convolution and ViT. BACKGROUND

[0002] With the development of digital imaging, photography and image understanding technology, the wide use of digital cameras, smart phones and other electronic products, people's demand for digital image storage, retrieval and processing tools is growing. These tools all need the direction information of the image, so that the image can be correctly processed and displayed. At present, detecting the direction of the image is very necessary in many fields, such as natural photography, medical diagnosis, robot-assisted automatic intervention system (RAIS), fingerprint verification system and face detection. Smart phones and digital cameras have a built-in orientation sensor that can track the direction of the camera when taking a picture and store it in the EXIF metadata of the image. In magnetic resonance imaging (MRI), the position and direction of the slice group are crucial to achieving high-quality image diagnosis and meeting various clinical work. In the bronchoscopy of the automatic robot-assisted intervention system (RAIS), there is a technology to detect the direction of the bronchial image branch, which can prevent the surgeon from being tired and minimize errors. The system using fingerprints for personal authentication needs the size, type and direction of the measured fingerprint matching pattern image. This method for detecting the direction of the fingerprint image significantly improves the accuracy of the system. In the face detection system, the image displayed upside down will reduce the visual saliency. Therefore, correcting the face image before sending it to the face recognition system can improve the detection performance and prevent dishonest behavior. Generally, the direction of the photo when taking a photo is determined by the rotation of the camera, and any angle is possible, but rotating 90° is the most common. Once the image processing tool detects that the image direction is a multiple of 90, the image direction is easy to correct. Therefore, it is usually assumed that the image rotation depends on one of the four directions (0°, 90°, 180° and 270°). Since the picture type and content are diverse, it is a challenging task to form an automatic direction detection system that can be widely used for different types of images.

[0003] In the current research, the image direction recognition method mostly uses image processing and deep learning algorithm. However, these methods have some problems: (1) mainly rely on low-level features of the image, such as texture, color and shape, etc. But due to the continuous increase of the number of image samples at present, and the semantic gap between low-level features and high-level image semantics, the manually constructed features are often limited in performance, and the influence of human visual mechanism on image direction perception is not fully considered. (2) For special images, such as abstract images, the content and semantics are relatively implicit and not obvious, which leads to unsatisfactory recognition results when these methods are applied to abstract paintings. (3) The structure of the network model used is relatively single, mostly directly fine-tuning the existing neural network, such as (VGG, AlexNet or ResNet, etc.). The model features mainly express the high-level semantic features of the image, but do not consider the influence of the spatial position information of the image content on the direction judgment. (4) The size requirement of the model for the input picture is consistent, if not, scaling or cropping is needed. However, the length and width of the image have a great influence on the direction judgment. SUMMARY

[0004] In view of the problems of the current image direction recognition, the application provides an image direction recognition method fusing convolution and ViT.

[0005] In order to achieve the above purpose, the application adopts the following technical scheme:

[0006] An image direction recognition method fusing convolution and ViT, comprising the following steps:

[0007] Step 1, rotate each image by four angles of 0 degrees, 90 degrees, 180 degrees and 270 degrees clockwise respectively, and finally get four images with different directions (0°, 90°, 180° and 270°) for each image;

[0008] Step 2, input an image, use a multi-scale Outlook Attentation module (MOAB) to calculate the attention of k×k pixel points in the neighborhood of each center point, k represents the neighborhood scale corresponding to the current position. MOAB can capture the relative position information of a certain center subgraph and its surrounding neighborhood subgraph, the specific steps are as follows:

[0009] Step 2.1, linearly transform the input image with size H×W×C to get a feature map with dimension H×W×k 4 , wherein H represents the height of the feature map, W represents the width of the feature map, and C represents the number of channels of the feature map;

[0010] Step 2.2, the H×W×k 4The dimension of the feature map is transformed into an attention feature map (HxW, kxk, kxk), and then a SoftMax is used to obtain an attention map AM (Attentation Map);

[0011] Step 2.3, the AM is mapped to a new feature space V (HxW, C, kxk) using the "Linear+Unfold" operation;

[0012] Step 2.4, the attention map AM is multiplied by the feature space V, and then the feature map is restored to the size of the original input image through the "Fold" operation;

[0013] Step 2.5, steps 2.1-2.4 are performed using different scales k (k=3, 5) to obtain two different scales of Outlook Attentation (OA), denoted as OA_3 and OA_5 respectively;

[0014] Step 2.6, OA_3, OA_5 and the input feature are linearly added, denoted as OA_Fusion=OA_3⊕OA_5⊕input feature. Wherein, ⊕ represents the addition of corresponding elements;

[0015] Step 2.7, finally, OA_Fusion is used with Layer Normalization (LayerNorm, LN) and Multilayer Perceptron (MLP) residual connection, i.e. OA_Fusion+LN&MLP(OA_Fusion), to obtain a new feature map;

[0016] Step 2.8, the feature map obtained in step 2.7 is subjected to Patch Embedding, including a convolution operation, Layer Normalization Layer Norm and flattening Flatten operation. Finally, a token sequence (Nxd) is obtained, where N represents the number of sequences and d represents the dimension of the sequence;

[0017] Step 3, combine the conditional position encoding (CPE) and the deformable convolution (DCN), i.e. replace the F function in CPE with DCN, denoted as DCPE. DCPE can encode different sizes of input and contain spatial arrangement information of each patch in the encoding, reflecting the rotation characteristics of the image. The specific steps are as follows:

[0018] Step 3.1, the token sequence obtained in step 2.8 is taken as input, and the dimension is transformed into HxWxC in the two-dimensional image space;

[0019] Step 3.2, convolution operation is performed with kernel_size = 3 and stride = 1, and the result of the convolution represents the position offset offset (H x W x 2M) of each pixel. Where M = 3 x 3, and the channel number 2M represents the offset in the "horizontal coordinate" and "vertical coordinate" directions. The output offset has the same spatial resolution as the input feature map;

[0020] Step 3.3, according to the offset obtained in step 3.2, the value of the new M points on the feature map is calculated. Since the offset can be a decimal number, the new coordinates are also decimal numbers, and the feature values of the four pixels around the coordinates are calculated by bilinear interpolation;

[0021] Step 3.4, the feature map obtained in step 3.3 is subjected to convolution operation kernel_size = 3 and stride = 1 to obtain a new feature map with dimensions H x W x C;

[0022] Step 3.5, finally, the dimensions of the feature map are transformed to obtain a new token sequence, and the dimensions of the new token sequence are the same as those of the token sequence in step 3.1;

[0023] Step 4, the Directional Convolution (DC) method is used to perform convolution fusion from different directions and scales (horizontal, vertical, and depth separable with dilation), and it is applied to the Multi Head Attention mechanism of the Transformer Encoder Block module in ViT, denoted as DTEB. DTEB compresses the shallow feature scale to the lowest resolution, increases the network feature extraction capability, and maintains high computational efficiency. The specific steps are as follows:

[0024] Step 4.1, the dimensions of the input feature map (H x W, C) are transformed to (H x W x C).

[0025] Step 4.2, the feature map is subjected to linear transformation W K to obtain the key value (K), the spatial resolution of the feature map is compressed by directional convolution (DC), and then W Q and W V are used to obtain the query (Q) and value (V) respectively, where W Q , W K and W V use 1 x 1 convolution;

[0026] Step 4.3, the multi-head attention mechanism is used to generate the attention weight of Q and K, and it is applied to V, i.e. where d Kdenotes the dimension of the key value (K), QK T denotes the dot product of Q and K.

[0027] Step 5, a framework is constructed that fuses convolutional neural networks and ViT, namely Directional Pyramidal Vision Transformer (DPVT). DPVT is divided into four stages: Stage_i (i = 1, 2, 3, 4), each of which uses a combination of MOAB, DCPE, and DTEB. The input to each stage is a 3D feature map, and the specific steps are as follows:

[0028] Step 5.1, in Stage_i (i = 1, 2, 3, 4), the feature map H i-1 ×W i-1 ×C i-1 A 3x3 convolution is used to downsample the resolution and increase the number of output channels. The downsample rate for Stage_1 is 4, and the downsample rate for Stage_2, Stage_3, and Stage_4 is 2. The size of the output patch is P1 = 4, P2 = P3 = P4 = 2, P i denotes the downsample rate of Stage_i, H i denotes the height of the feature map, W i denotes the width of the feature map, C i denotes the number of channels of the feature map. In this way, the size of the feature map can be flexibly adjusted at each stage to build a feature pyramid model.

[0029] Step 5.2, the feature map obtained in step 5.1 is input into the MOAB module to produce the correlation attention weight between the patch at each spatial position and its surrounding local neighborhood patch;

[0030] Step 5.3, after directional convolution and layer normalization (LayerNorm) operations, the feature map is further flattened (Flatten) to obtain the token sequence, with a dimension of

[0031] Step 5.4, DCPE is used to encode the position of the token sequence, and the original token sequence is added to obtain the token sequence with position encoding. DCPE can extract the position relationship of each patch and express the rotation features of the image, and it can also adapt to different sizes of input images;

[0032] Step 5.5, input the token sequence with position encoding into the DTEB, reduce the transform computation complexity, and improve the accuracy of the network in image direction detection;

[0033] Step 5.6, transform the token sequence in step 5.5 into a 3-D feature map, and the output feature map sizes of Stage_1, Stage_2, Stage_3 and Stage_4 stages are respectively and That is, the number of channels is increased to 4 times, 2 times, 2 times and 2 times of the input image; therefore, the feature maps of the four stages are respectively and

[0034] Step 5.7, finally, a global average pooling layer (GAP) is applied to the output feature map of Stage_4 by using a structure similar to a convolutional neural network. A full connection (FC) layer is used for the final classification mapping;

[0035] Step 5.8, in order to adapt to different scale application scenarios, three DPTV models with different capacities are constructed by selecting different values of parameters in each Stage_i.

[0036] Further, the "Linear" operation in the "Linear+Unfold" operation in step 2.3 is a linear transformation on the input feature X n×in A linear transformation is performed, that is:

[0037] Y n×out =X n×in W in×out +b n×out

[0038] Wherein, X n×in is the input feature, Y n×out is the output feature, W is the parameter to be learned by the model, b is the vector bias, n is the number of rows of the input vector, in is the number of input neurons, out is the number of output neurons, and in is the same as out;

[0039] The "Unfold" operation is to extract a sliding local region block from a batch of input samples to realize a sliding window operation of local connection. In the present application, the input size is (C, H, W), the convolution kernel size is 3x3, the step is 1, the padding is 1, and the output size is (Cx3x3, HxW).

[0040] Further, the "Fold" operation and the "Unfold" operation in step 2.4 are opposite, that is, a series of sliding blocks are spliced into a tensor. In the present application, the input size is (C x 3 x 3, H x W), the convolution kernel size is 3 x 3, the step is 1, the padding is 1, and the output size is (C, H, W).

[0041] Further, the value of the new M points on the feature map in step 3.3 is calculated using the following formula,

[0042]

[0043] wherein Coord = {(0, 0), (1, 1), (0, 1), (-1, 0), (0, -1), (-1, 1), (1, -1), (-1, -1)}, pos_i represents the coordinates of the i-th point, x(pos_i) represents the pixel value of the i-th point, co is the coordinates of each element in the convolution kernel, w is the convolution kernel, offset represents the offset, and co+offset represents the offset position.

[0044] Further, step 4 uses a directional convolution (DC) method, which is composed of five convolution kernels in parallel, which are: (1) a 3 x 3 convolution kernel; (2) a 1 x 3 convolution kernel; (3) a 3 x 1 convolution kernel; (4) a depth separable dilated convolution with a 3 x 3 convolution kernel and a dilated rate of 3; and (5) a depth separable dilated convolution with a 3 x 3 convolution kernel and a dilated rate of 5. Then, the corresponding elements of the outputs of the five branches are added.

[0045] Further, the DPTV models with three different capacities in step 5.8 are DPTV-Tiny, DPTV-Small and DPTV-Base.

[0046] Further, the depth separable convolution with a dilated rate is divided into two steps: the first step is a layer-by-layer convolution with a dilated rate, that is, the convolution kernel is filled with "dilated rate-1" 0s, and the receptive field of the convolution is different due to different dilated rates; the input feature is H x W x C in , and a convolution kernel of [3+2x(rate-1)]x[3+2x(rate-1)]xC in is used for layer-by-layer convolution of the input feature, wherein C in is the input channel number, and rate is the dilated rate; after convolution, the output feature dimension is H x W x C in ; the second step is point-by-point convolution, which uses a 1 x 1 x C in xC outThe convolution kernel is used for convolution operation on the output feature of the first step, the feature map is weighted combined in the depth direction, a new feature map is generated, and the dimension is HxWxC out , wherein C out is the output channel number.

[0047] The original convolution kernel size adopted by the application is 3, the expansion rate is 3 and 5 respectively, so the expansion convolution kernel size obtained is 3+(3-1)x(3-1)=7 and 3+(3-1)x(5-1)=11. If the input feature is HxWxC in , 7x7xC in and 11x11xC in convolution kernels are used for layer-by-layer convolution on the input feature, wherein C in is the input channel number. After convolution, the output feature dimension is HxWxC in . The second step is point-by-point convolution, and the 1x1xC in xC out convolution kernel is used for convolution operation on the output feature of the first step, the feature map is weighted combined in the depth direction, a new feature map is generated, and the dimension is HxWxC out , wherein C out is the output channel number.

[0048] The structure of the five branches can fit more information, wherein the 1x3 and 3x1 convolution kernels are used for capturing horizontal and vertical direction information in the image, the depth separable convolution has a larger receptive field while capturing direction information, and can also learn the rotating spatial information of different scales in the image, thereby reducing the amount of calculation.

[0049] Compared with the prior art, the application has the following advantages:

[0050] (1) The multi-scale Outlook Attentation module (MOAB) is used to calculate the attention of each center point with k x k positions in the surrounding neighborhood of different scales. The relative position information of a certain center subgraph and its surrounding neighborhood subgraph can be better captured.(2) In order to adapt the position encoding to different sizes of input and better express the rotation characteristics, the conditional position encoding (CPE) and the variable convolution DCN are combined, that is, the F function (original text uses deep separable convolution) in CPE is replaced by DCN, which is called DCPE. DCPE can encode different sizes of input, and the spatial arrangement information of each patch can be included in the encoding, which reflects the rotation characteristics of the image.(3) In order to express the direction attribute while improving the lightweight degree of the model, a direction convolution is used to perform convolution fusion from different directions and scales (horizontal, vertical, and deep separable with expansion rate), and it is applied to K and V of the Self-Atterntation of the Transformer module. The shallow feature scale is compressed to the lowest resolution, which increases the network feature extraction capability and maintains high efficient calculation efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 is the schematic diagram of the image of the present application;

[0052] Figure 2 is the schematic diagram of the MOAB structure of the present application;

[0053] Figure 3 is the schematic diagram of the DCPE structure of the present application;

[0054] Figure 4 is the schematic diagram of the DTEB structure of the present application;

[0055] Figure 5 is the DPVT network model framework of the present application. DETAILED DESCRIPTION

[0056] In order to make the technical problems, technical solutions and beneficial effects of the present application clearer, the present application is further described in detail in combination with the embodiments and drawings. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application. The technical solutions of the present application are described in detail below in combination with the embodiments and drawings, but the protection scope is not limited thereto.

[0057] Embodiment 1

[0058] As shown in Figures 1-5 , the present application selects a public data set for experiment, and the specific implementation steps are as follows:

[0059] Step 1: Select the public data set SUN-397 and INRIA Holidays (INRIA) data set. SUN-397 contains 397 classes of scenes, each containing at least 100 pictures, a total of 108754 images. INRIA includes 1491 images from various scene types (natural, man-made, water and fire effects, etc.), a total of 500 image groups, each representing a different scene or object. Rotate each image clockwise by four angles of 0 degrees, 90 degrees, 180 degrees and 270 degrees, and each image will get 4 categories of images, respectively (0°, 90°, 180° and 270°).

[0060] Step 2: Input an image, use the multi-scale Outlook Attentation module (MOAB) to calculate the attention of each center point and the k*k pixel points in the surrounding different scale neighborhood, and k represents the neighborhood scale corresponding to the current position. MOAB can better capture the relative position information of a certain center subgraph and its surrounding neighborhood subgraph, and the specific steps are as follows:

[0061] Step 2.1, linearly transform the input image with size HxWxC to get a feature map with dimension HxWxk 4 , where H represents the height of the feature map, W represents the width of the feature map, and C represents the number of channels of the feature map.

[0062] Step 2.2, transform the dimension of the HxWxk 4 feature map into an attention feature map (HxW, kxk, kxk), and then use SoftMax to get the attention map AM (Attentation Map).

[0063] Step 2.3, use the "Linear+Unfold" operation to map AM to a new feature space V(HxW, C, kxk). "Linear" is a linear transformation performed on the input feature X n×in , that is:

[0064] Y n×out =X n×in W in×out +b n×out

[0065] Where X n×in is the input feature, Y n×out is the output feature, W is the parameter to be learned by the model, b is the vector bias, n is the number of rows of the input vector, in is the number of input neurons, and out is the number of output neurons. In this invention, in and out are the same.

[0066] The "Unfold" operation is to extract the sliding local region block from the input sample of a batch, and realize the sliding window operation of local connection. In the present application, the input size is (C, H, W), the convolution kernel size is 3x3, the step is 1, the padding is 1, and the output size is (C*3*3, H*W).

[0067] Step 2.4, after matrix multiplication of AM and V, the feature map can be restored to the size of the original input image through the "Fold" operation. The "Fold" operation is the opposite of the "Unfold" operation, which is to splice a series of sliding blocks into a tensor. In the present application, the input size is (C*3*3, H*W), the convolution kernel size is 3x3, the step is 1, the padding is 1, and the output size is (C, H, W).

[0068] Step 2.5, steps 2.1-2.4 are performed respectively with different scales (k=3, 5) to obtain two different scales of Outlook Attentation (OA), which are denoted as OA_3 and OA_5.

[0069] Step 2.6, linearly add OA_3, OA_5 and the input feature, denoted as OA_Fusion=OA_3⊕OA_5⊕input feature. Wherein, ⊕ represents the addition of corresponding elements.

[0070] Step 2.7, finally, OA_Fusion is used with Layer Normalization (LayerNorm, LN) and Multilayer Perceptron (MLP) residual connection, i.e. OA_Fusion+LN&MLP(OA_Fusion), to obtain a new feature map.

[0071] Step 2.8, the feature map obtained in step 2.7 is subjected to Patch Embedding, including a convolution operation, Layer Normalization LayerNorm and flattening Flatten operation. Finally, a token sequence (Nxd) is obtained, where N represents the number of sequences, and d represents the dimension of the sequence.

[0072] Step 3: combine Conditional Position Encoding (CPE) and Dynamic Convolution (DCN), i.e. replace the F function in CPE with DCN, denoted as DCPE. DCPE can encode different sizes of input and contain spatial arrangement information of each patch in the encoding, which reflects the rotation characteristics of the image. The specific steps are as follows.

[0073] Step 3.1, the token sequence obtained in step 2.8 is taken as input, and the dimension is transformed into HxWxC in the two-dimensional image space.

[0074] Step 3.2, convolution operation is performed with kernel_size = 3, stride = 1, and the result of the convolution represents the position offset offset (H x W x 2M) of each pixel. Where M = 3 x 3, and the channel number 2M represents the offset in the "horizontal coordinate" and "vertical coordinate" directions. The output offset has the same spatial resolution as the input feature map.

[0075] Step 3.3, according to the offset obtained in step 3.2, calculate the value of the new M points on the feature map, using the following formula,

[0076]

[0077] Where Coord = {(0, 0), (1, 1), (0, 1), (1, 0), (-1, 0), (0, -1), (-1, 1), (1, -1), (-1, -1)}, pos_i represents the coordinates of the i-th point, co+offset represents the pixel value of the i-th point, co is the coordinates of each element in the convolution kernel, w is the convolution kernel, offset represents the offset, and co+offset represents the offset position.

[0078] Since the offset can be a decimal number, the new coordinates are also decimal numbers, and the feature values of the four pixels around the coordinates are calculated by bilinear interpolation.

[0079] Step 3.4, the feature map obtained in step 3.3 is convolved with kernel_size = 3, stride = 1 to obtain a new feature map with dimensions H x W x C.

[0080] Step 3.5, finally, the dimensions of the feature map are transformed to obtain a new token sequence, and the dimensions of the new token sequence are the same as those of the token sequence in step 3.1.

[0081] Step 4: A Directional Convolution (DC) method is used to perform convolution fusion from different directions and scales (horizontal, vertical, and depth separable with dilation), and it is applied to the Multi Head Atterntation mechanism of the Tranformer Encoder Block module in ViT, denoted as DTEB. DTEB compresses the shallow feature scale to the lowest resolution, increases the network feature extraction capability, and maintains efficient computing efficiency. The specific steps are as follows:

[0082] Step 4.1, transform the dimensions of the input feature map (H x W, C) to (H x W x C).

[0083] Step 4.2, feature map through linear transformation W K Obtain key value (K) by direction convolution (DC) to compress the spatial resolution of the feature map, and then use W Q and W V Obtain query (Q) and value (V) respectively, where W Q , W K and W V All use 1x1 convolution.

[0084] The direction convolution method is composed of five convolution kernels in parallel, which are: (1) 3x3 convolution kernel; (2) 1x3 convolution kernel; (3) 3x1 convolution kernel; (4) depth separable dilated convolution with 3x3 convolution kernel and dilated rate of 3; (5) depth separable dilated convolution with 3x3 convolution kernel and dilated rate of 5. Then the outputs of the five branches are added element by element.

[0085] The depth separable convolution with dilated rate is divided into two steps: the first step is layer-by-layer convolution with dilated rate, that is, the convolution kernel is filled with "dilated rate-1" 0, and the receptive field of the convolution is different with different dilated rates. The original convolution kernel size used in the application is 3, and the expansion rates are 3 and 5 respectively, so the dilated convolution kernel sizes obtained are 3+(3-1)x(3-1)=7 and 3+(3-1)x(5-1)=11 respectively. If the input feature is HxWxC in , the 7x7xC in and 11x11xC in convolution kernels are used for layer-by-layer convolution of the input feature, where C in is the number of input channels. After convolution, the output feature dimension is HxWxC in . The second step is point-by-point convolution, which uses a 1x1xC in xC out convolution kernel to convolve the output feature of the first step, which combines the features in the depth direction to generate a new feature map with a dimension of HxWxC out , where C out is the number of output channels.

[0086] The structure of the five branches can fit more information, where the 1x3 and 3x1 convolution kernels are used to capture the horizontal and vertical direction information in the image, and the depth separable convolution has a larger receptive field while capturing direction information, which can also learn the rotated spatial information of different scales in the image and reduce the amount of calculation.

[0087] Step 4.3, use multi-head attention mechanism to generate attention weight of Q and K, and apply it to V, that is where d K means the dimension of key value (K) and QKT denotes the dot product of Q and K.

[0088] Step 5: A framework that combines convolutional neural networks and ViT is constructed, namely the Directional Pyramidal Vision Transformer (DPVT) model. DPVT is divided into four stages, Stage_i (i = 1, 2, 3, 4), and each stage uses a combination of MOAB, DCPE, and DTEB. The input of each stage is a 3D feature map. The specific steps are as follows:

[0089] Step 5.1, in Stage_i stage, the feature map H i-1 ×W i-1 ×C i-1 A 3x3 convolution is used to downsample the resolution and increase the number of output channels. The downsample rate of the first stage (Stage_1) is 4, and the downsample rate of the other three stages (Stage_2, Stage_3, and Stage_4) is 2. The size of the output patch is P1 = 4, P2 = P3 = P4 = 2, where P i denotes the downsample rate of Stage_i, H i denotes the height of the feature map, W i denotes the width of the feature map, and C i denotes the number of channels of the feature map. In this way, the size of the feature map can be flexibly adjusted at each stage to build a feature pyramid model.

[0090] Step 5.2, the feature map obtained from step 5.1 is input into the MOAB module to produce the correlation attention weight between the patch at each spatial position and its surrounding local neighborhood patch.

[0091] Step 5.3, after a directional convolution and a LayerNorm operation, the feature map is further flattened to obtain a token sequence with a dimension of

[0092] Step 5.4, the token sequence is positionally encoded using DCPE and added to the original token sequence to obtain a token sequence with positional encoding. DCPE can extract the positional relationship of each patch and express the rotation features of the image, and it can also adapt to different sizes of input images.

[0093] Step 5.5, the token sequence with positional encoding is input into DFEB to reduce the computational complexity of the Transformer and improve the accuracy of the network in image direction detection.

[0094] Step 5.6, the dimension of the token sequence in step 5.5 is transformed into a 3-D feature map, and the feature map size of different stages is (Stage_1), (Stage_2, Stage_3 and Stage_4), that is, the number of channels increases to 4 times (Stage_1), 2 times (Stage_2, Stage_3 and Stage_4) of the input image. Therefore, the feature maps of the four stages are and

[0095] Step 5.7, finally, a global average pooling layer (GAP) is applied to the output feature map of Stage_4 using a structure similar to a convolutional neural network. A fully connected layer (Full Connection, FC) is used for the final classification mapping;

[0096] Step 5.8, in order to adapt to different scale application scenarios, three different capacity DPTV models are constructed by selecting different values of parameters in each Stage_i, which are DPTV-Tiny, DPTV-Small and DPTV-Base. The detailed architecture of the three models is shown in Table 1, where k, C, P, H and L represent the neighborhood scale corresponding to the current position, the number of output channels, the downsampling rate, the number of attention heads and the number of DTEB, respectively. kernel_size represents the size of the convolution kernel, stride represents the convolution step, and Patch embedding represents the embedding patch in ViT.

[0097] Table 1

[0098]

[0099]

[0100] Step 6: The experimental environment adopted is Pycharm, and the deep learning framework is pytorch (GPU). 70% of each data set is selected as the training set, and 30% is selected as the test set. The original image size remains unchanged. The 10-fold cross-validation method is adopted, so the final evaluation index is the average accuracy after 10-fold cross-validation.

[0101] The experimental related parameters are set as follows: an AdamW optimizer with a momentum of 0.9, an initial learning rate of 0.002, a weight decay of 0.05, a total of 150 epochs, and a training batch size of 128. L2 regularization is added to prevent overfitting. The evaluation methods used in the method of the application include model parameters (M), floating point operations per second (FLOPs (G)), and accuracy (Accuracy (%)).

[0102] To fully verify the effectiveness and applicability of the method of the application and detect the effect of the fusion of the DCPE, MOAB and DTEB modules, we compared the influence of different module combinations (models 1-7) on the classification results. The experiment was carried out under the same experimental environment, data set, image preprocessing and network hyperparameter settings. As shown in Table 2, the backbone is PVT-v2-Base. Model 1 changes the PEG of the backbone to DCPE, which can process input images of different sizes. Model 2 is added to MOAB after embedding patches in Backbone. Model 3 replaces the Transformer Block in the backbone with DTEB. Model 4 uses DCPE and MOAB, retaining the original Transformer module. Model 5 changes the PEG to DCPE, and the Transformer module is replaced by DTEB. Model 6 uses MOAB and DTEB. Model 7 is the model we proposed, which also uses DCPE, MOAB and DTEB. These models use the "X-Base" structure.

[0103] The experimental results are shown in Table 2. The performance of models 1-7 is better than that of Backbone. The accuracy of models using only one structure in DCPE, MOAB and DTEB (models 1-3) is 0.9%-4.4% lower than that of combined models (models 4-6). Compared with model 4, model 7 uses DTEB, and the accuracy is improved by 1.5%-3.1%. Compared with model 5, model 5 uses MOAB, and the accuracy is improved by 0.5%-2.3%. Compared with model 6, model 6 uses DCPE, and the accuracy is improved by 0.8%-2.2%. The reasons for analyzing these results are as follows: (1) DCPE combines CPE and DCN, which can encode images of different sizes. The encoding can include the spatial arrangement information of each patch and reflect the rotation characteristics of the image. (2) MOAB encodes spatial information by measuring the similarity between each pair of tokens, so its feature learning ability is stronger than that of convolution. In addition, it also uses different attention scales to expand the perception field of orientation judgment. (3) In DTEB, the directional convolution of self-attention in the transformer block compresses the resolution of the input features, captures the directional information, and has a larger perception field, increasing the network feature extraction capability while maintaining high computational efficiency.

[0104] Table 2

[0105]

[0106] To verify the impact of input image size on image orientation detection, we conducted experiments on two different datasets using two types of input. One is to crop the image size to 224x224, and the other is to keep the original image size unchanged. Three different capacity DPVT models were used in the experiment. As shown in Table 3, when using the original image as input, the accuracy is 1.3%~6.1% higher than other images. The results show that the aspect ratio of the image is one of the factors affecting the orientation recognition, and this influence is particularly evident in painted images (such as abstract images).

[0107] Table 3

[0108]

[0109]

[0110] To verify the overall effectiveness of the model, we compared the performance of DPVT with the most advanced models, including CNN models, Transformer models, and the combination of CNN and Transformer models in Table 4. The parameters, FLOPs, and accuracy rates of the models were evaluated on two different datasets. The experimental results are shown in Table 4. First, we compared DPVT with two CNN models, ResNet and RegNetY. The proposed model DPVT has fewer parameters and computational costs than ResNet, but the accuracy is improved by 1.4%~7.8%. Compared with RegNetY-16G, DPVT Large has 15.6 fewer parameters and 2.2 less computational cost, but the accuracy is improved by 0.7%~6.3%.

[0111] Secondly, on smaller scale models, we compared DPVT Small with seven Transformer models with "X-Small", namely PoolFormer-S12, DeiT-S, TNT-S, T2T-ViT-19, PVT-v2-S, CPVT-S and Swin-S. On five different types of datasets, the accuracy of DPVT-Small is 0.1%~1.9%, 1.3%~3.0%, 0.2%~2.2%, 0.1%~2.4%, 2.6%~5.3%, 0.4%~3.3% and 0.1%~2.0% higher than the other seven models, respectively. However, compared with the other seven models, the parameter quantity of DPVT Small is reduced by 2.2~29.5, and the computational cost of DPVT Small is reduced by 1.4~4.

[0112] Thirdly, for the combined model of CNN and Transformer, DPVT also shows significant performance advantages. For the basic scale model, compare DVPTbase with five models with "X-base", namely Twins-SVT-B, Shu-sulu e-B, CMT-B, VOLO-D2, Next-ViT-B. The accuracy of DVPT-Base is 0.8%-4.9%, 0.5%-5.1%, 1.6%-3.4%, 2.3%-6.2% and 0.2%-2.8% higher than the other five models respectively. In addition, compared with Shuffle-B and CMT-B, the number of parameters of DVPT-Base is reduced by 41.4 and 14.4, and the computational cost of DVPT-Base is reduced by 6.2 and 4.7. For larger scale models, compare DVPT-Large with Twins-SVT-L, the accuracy of DPVT-Large is 0.6%-6.4% higher than other methods, and the number of parameters and FLOPs is reduced by 30.8 and 1.3 respectively.

[0113] Table 4

[0114]

[0115]

[0116]

[0117] Compared with the existing image direction perception method, the advantages of the present application are: (1) the multi-scale Outlookattentation module is adopted, so that the position of each center point needs to be calculated with kxk positions in the surrounding neighborhood of different scales. The relative position information of a certain center subgraph and its surrounding neighborhood subgraph can be better captured. (2) In order to make the position coding adapt to different sizes of input, and better express the rotation characteristics, we combine the conditional position coding (CPE) and the variable convolution DCN, that is, replace the F function (original text uses depth separable convolution) in CPE with DCN. After this replacement, DCPE can encode different sizes of input, and the encoding can contain the spatial arrangement information of each patch, and reflect the rotation characteristics of the image. (3) In order to express the direction attribute while improving the lightweight degree of the model, we use a direction convolution to perform convolution fusion from different directions and scales (horizontal, vertical, depth separable with expansion rate), and apply it to the K and V of the Self-Atterntation in the Transformer module. The shallow feature scale is compressed to the lowest resolution, which increases the network feature extraction capability and maintains high efficient computing efficiency.

[0118] The description herein of any aspect of the application using terms such as "processing" or "comparing" or "determining" or "displaying" or "transmitting" or "receiving" or "identifying" or "selecting" or "extracting" or "generating" or "calculating" or "associating" or "determining" or "recommending" or "providing" or "receiving" or "receiving" or "receiving" or "receiving" or "receiving" or "receiving" or "receiving" or "receiving" or "receiving" or "receiving" or "receiving" or "receiving" or "receiving" or "receiving" or "receiving" or "receiving" or "receiving" or "receiving" or "receiving" or "receiving" or "receiving" or "receiving" or "receiving" or "receiving" or "receiving" or "receiving" or "receiving" or "receiving" or "receiving" or "receiving" or "receiving" or "receiving" or "receiving" or "receiving" or "receiving" or "receiving

Claims

1. A method for image orientation recognition by fusing convolution and ViT, characterized in that, Comprising the following steps: Step 1, rotate each image by four angles of 0 degree, 90 degree, 180 degree and 270 degree clockwise respectively, and each image finally gets four images in different directions; Step 2, input an image, and calculate the attention of each center point and kxk pixel points in the surrounding different scale neighborhood by using a multi-scale Outlook Attentation module (MOAB), wherein k represents the neighborhood scale corresponding to the current position; the MOAB can capture the relative position information of a certain center subgraph and its surrounding neighborhood subgraph, and the specific steps are as follows: Step 2.1, linearly transforming an input image with a size of HxWxC to obtain a feature map with a dimension of HxWxk, wherein H represents a height of the feature map, W represents a width of the feature map, and C represents a channel number of the feature map 4 ; Step 2.2, the HxWxk obtained in step 2.1 is transformed into an attention feature map, and then a SoftMax is used to obtain an attention map AM. 4 The dimension of the feature map is transformed into an attention feature map, and then a SoftMax is used to obtain an attention map AM. Step 2.3, the attention map AM is mapped to a new feature space by using the "Linear+Unfold" operation; Step 2.4, the attention map AM is multiplied with the feature space, and then the feature map is restored to the size of the original input image by using the "Fold" operation; Step 2.5, steps 2.1-2.4 are performed respectively by using different scales k=3 or 5 to obtain OA in different scales; OA_3 and OA_5 are recorded respectively; Step 2.6, OA_3, OA_5 and the input feature are linearly added, and recorded as OA_Fusion=OA_3⊕OA_5⊕inputfeature; wherein, ⊕ represents corresponding element addition; Step 2.7, finally, OA_Fusion is used with layer normalization LN and multi-layer perceptron MLP residual connection, i.e. OA_Fusion+LN&MLP(OA_Fusion), to obtain a new feature map; Step 2.8, the feature map obtained in step 2.7 is subjected to Patch Embedding, including: a convolution operation, layer normalization Layer Norm and flattening Flatten operation, finally, a token sequence (Nxd) is obtained, wherein N represents the number of sequences, and d represents the dimension of the sequence; Step 3, combine the conditional position encoding and the variability convolution, i.e. replace the F function in the conditional position encoding with the variability convolution, and record it as DCPE, and the specific steps are as follows: Step 3.1, the token sequence obtained in step 2.8 is taken as input, and the dimension is transformed into HxWxC in the two-dimensional image space; Step 3.2, a convolution operation is performed by using kernel_size=3 and stride=1, and the result of the convolution represents the position offset offset (HxWx2M) of each pixel; wherein, M=3x3, and the channel number 2M represents the offset in the "horizontal coordinate" and "vertical coordinate" directions, and the output offset has the same spatial resolution as the input feature map; Step 3.3, according to the offset obtained in step 3.2, the values of the new M points on the feature map are calculated; Step 3.4, the feature map obtained in step 3.3 is subjected to a convolution operation kernel_size=3 and stride=1 to obtain a new feature map, and the dimension is HxWxC; Step 3.5, finally, the dimension of the feature map is transformed to obtain a new token sequence, and the dimension of the new token sequence is the same as that of the token sequence in step 3.1; Step 4, a directional convolution method is used to perform convolution fusion from different directions and scales, and is applied to the multi-head self-attention mechanism of the Tranformer Encoder Block module in ViT, denoted as DTEB; the DTEB compresses the shallow feature scale to the lowest resolution, increases the network feature extraction capability, and maintains high computational efficiency, and the specific steps are as follows: Step 4.1, the dimension of the token sequence obtained in step 3.5 is transformed into a two-dimensional feature map (HxWxC); Step 4.2, the feature map is compressed in spatial resolution by a linear transformation W K obtaining a key value K, compressing the spatial resolution of the feature map by directional convolution, then using W Q and W V obtaining a query Q and a value V, respectively, where W Q , W K and W V all use 1x1 convolution; Step 4.3, generate attention weights for Q and K using multi-head attention mechanism and apply them to V, i.e., where d K is the dimension of the key value K, QK T denotes the dot product of Q and K; Step 5, a framework combining convolutional neural network and ViT is constructed, namely Directional Pyramid ViT model, DPVT, DPVT is divided into four stages Stage_i (i=1, 2, 3, 4), each stage uses the combination of MOAB, DCPE and DTEB, and the input of each stage is a 3D feature map, and the specific steps are as follows: Step 5.1, at Stage_i stage, feature map H i−1 x W i−1 x C i−1 A 3x3 convolution is used to downsample the resolution and increase the number of output channels; the downsample rate of Stage_1 is 4, and the downsample rate of Stage_2, Stage_3 and Stage_4 stages is 2; the size of the output patch is , where, denotes the downsample rate of Stage_i, denotes the height of the feature map, denotes the width of the feature map, denotes the number of channels of the feature map; Step 5.2, the feature map obtained in step 5.1 is input into the MOAB module to generate the correlation attention weight between the patch at each spatial position and the patch in its local neighborhood; Step 5.

3. After the direction convolution and layer normalization operations, the feature map is further flattened to obtain a token sequence, and the dimension is ; Step 5.4, the token sequence is positionally encoded using DCPE, and is added to the original token sequence to obtain a token sequence with positional encoding; Step 5.5, the token sequence with positional encoding is input into DTEB to reduce the complexity of Transformer calculation and improve the accuracy of image direction detection of the network; Step 5.6, transform the token sequence in step 5.5 into a 3-D feature map, the output feature map size of Stage_1, Stage_2, Stage_3 and Stage_4 stage is , , and , i.e. the number of channels is increased by 4 times, 2 times, 2 times and 2 times; therefore, the feature map of the 4 stages is , , and , respectively, the size of the original image Step 5.7, the global average pooling layer is used to act on the output feature map of Stage_4 by using the structure of convolutional neural network; and a fully connected layer is used for final classification mapping; Step 5.8, in order to adapt to different application scenarios, three DPTV models with different capacities are constructed by selecting different values of parameters in each Stage_i.

2. The image orientation recognition method of claim 1, wherein, The "Linear" operation in the "Linear+Unfold" operation in step 2.3 is a linear transformation on the input feature X n×in A linear transformation is performed, namely: ； wherein, is an input feature, is an output feature, is a parameter to be learned by the model, is a vector bias, is the number of rows of the input vector, is the number of input neurons, is the number of output neurons, is the same as . The "Unfold" operation is to extract a sliding local region block from a batch of input samples to realize a local connection sliding window operation.

3. The image orientation recognition method of claim 1, wherein, The "Fold" operation in step 2.4 is the opposite of the "Unfold" operation, which splices a series of sliding blocks into a tensor.

4. The image orientation recognition method of claim 1, wherein, In step 3.3, the value of the new M points on the feature map is calculated using the following formula, ； wherein, , denotes the coordinates of the i-th point, denotes the pixel value of the i-th point, is the coordinates of each element within the convolution kernel, is the convolution kernel, offset denotes an offset, denotes the offset position.

5. The image orientation recognition method of claim 1, wherein, In step 4, the directional convolution method is composed of five convolution kernels in parallel, specifically: (1) 3x3 convolution kernel; (2) 1x3 convolution kernel; (3) 3x1 convolution kernel; (4) depth separable dilated convolution with 3x3 convolution kernel and expansion rate of 3; (5) depth separable dilated convolution with 3x3 convolution kernel and expansion rate of 5; then the corresponding element addition is performed on the outputs of the five branches.

6. The image orientation recognition method of claim 1, wherein, In step 5.8, three DPTV models with different capacities are constructed, specifically DPTV-Tiny, DPTV-Small and DPTV-Base.

7. The image orientation recognition method of claim 5, wherein, The depth separable dilated convolution is divided into two steps: The first step is layer-by-layer convolution, that is, filling the convolution kernel with "rate-1" 0, and the receptive field of the convolution is different due to different expansion rates; the input feature is HxWxC in , and a [3+2x(rate-1)]x[3+2x(rate-1)]x C in convolution kernel is used for layer-by-layer convolution of the input feature, wherein C in is the input channel number, and rate is the expansion rate; after convolution, the output feature dimension is HxWxC in ; The second step is pointwise convolution, using 1×1×C++. in ×C out The convolution kernel performs a convolution operation on the output features from the first step, weighting and combining the feature maps along the depth direction to generate a new feature map with dimensions H×W×C. out , where C out This is the number of output channels.

Images