An image direction prediction method based on multi-scale fusion and attention mechanism

Abstract

The application discloses an image direction prediction method based on multi-scale fusion and attention mechanism and belongs to the technical field of computer vision and image processing. For the automatic prediction problem of the image direction, the last residual structure output feature map of the last four parts of the ResNet50 network is obtained, the four spatial attention maps are obtained through an attention mechanism module respectively, and corresponding elements of the four spatial attention maps and corresponding original feature maps are added. Then, three smaller scale features are up-sampled to the same resolution as the maximum scale by using bilinear interpolation, and the final multi-scale attention fusion features, called local features, are obtained by splicing along the channel. Secondly, four different scale VR_LBP feature maps of the image are taken as network inputs, four feature maps are obtained by using ResNet50 to fuse residual dilated convolution, and global features are obtained by adding corresponding elements of the feature maps. Finally, the local features and the global features are spliced and fused, and then direction prediction is finally realized through GAP and a fully connected layer.

Description

Technical Field

[0001] This invention belongs to the field of image processing and computer vision perception technology, specifically relating to an image orientation prediction method based on multi-scale fusion and attention mechanism. Background Technology

[0002] Advances in digital imaging technology, along with the proliferation of digital cameras, smartphones, and other devices, have led to a significant increase in the number of photos taken. Since cameras are not always level during shooting, the resulting photos often require rotation correction to ensure they are displayed in the correct orientation—the direction in which the scene initially appeared. The correct orientation of an image is defined as the direction in which the scene initially appeared. Most digital cameras have a built-in orientation sensor that allows the camera's orientation to be recorded in the image's EXIF ​​metadata during shooting; however, this field is not uniformly managed and updated by several image processing applications and image formats. Therefore, automatic detection of standard image orientation is an important task for some applications, such as automatically creating digital albums, digitizing analog photos, and computer vision applications that require images to be input in a vertical orientation. In these cases, user intervention is required, and humans can utilize their image understanding abilities to identify the correct orientation of a photo. Generally, the orientation of a photo is determined by the camera's rotation at the time of shooting, although any angle is possible, with a 90-degree rotation being the most common. Therefore, it is generally assumed that an image was taken in one of four directions (0 degrees (top), 90 degrees (right), 180 degrees (bottom), and 270 degrees (left)). Due to the wide variability of the scene content, automating this task is a challenging endeavor.

[0003] Currently, most image orientation recognition methods employ image processing and machine learning algorithms. However, these methods have several problems: (1) Some orientation detection methods rely on low-level features and then use appropriate classifiers to achieve image orientation detection; however, low-level features cannot capture a large amount of semantic content in the image. (2) Some orientation detection methods using neural networks require scaling the original image; for example, using a VGG network will scale the image to 224×224. However, the aspect ratio of the image is one of the factors for judging the image, and scaling the image will result in the loss of some image information. (3) Most current neural network methods for image orientation detection are based on fine-tuning of existing backbone networks without considering whether the extracted features can express human visual perception, resulting in low generalization ability of the model. Summary of the Invention

[0004] To address the current problems in image orientation recognition, this invention provides an image orientation prediction method based on multi-scale fusion and attention mechanisms.

[0005] To achieve the above objectives, the present invention employs the following technical solutions:

[0006] An image orientation prediction method based on multi-scale fusion and attention mechanism includes the following steps:

[0007] Step 1: Rotate each image clockwise by three angles: 90 degrees, 180 degrees, and 270 degrees. Each image will eventually be displayed in four different directions: top, right, bottom, and left.

[0008] Step 2: Use ResNet50 with residual attention mechanism to extract local features from each image. The specific steps are as follows:

[0009] Step 2.1: The ResNet50 network consists of 6 parts: convolutional layers, C0, C1, C2, C3, and C4. C0 contains one 7×7 convolutional layer with a stride of 2 and one 3×3 max pooling layer with a stride of 2. C1, C2, C3, and C4 are 1 / 4, 1 / 8, 1 / 16, and 1 / 32 times the size of the original image, respectively, and contain 3, 4, 6, and 3 bottleneck layers (BTNK) respectively.

[0010] Step 2.2: Denote the output feature maps of the last residual structure of each part C1, C2, C3, and C4 as C1, C2, C3, and C4, respectively; pass each feature map at a different scale through an attention mechanism module (Convolutional Block Attention Module, CBAM) to obtain four spatial attention maps, denoted as A1, A2, A3, and A4.

[0011] Step 2.3: Add the spatial attention map to the corresponding original feature map element by element, denoted as . in, This indicates that corresponding elements are added together;

[0012] Step 2.4: Upsample the feature maps of the three small scales F2, F3, and F4 to the same scale as F1 using bilinear interpolation, concatenate them along the channels, and then perform a 1×1 convolution operation to obtain the final multi-scale attention fusion feature, the local feature, denoted as: Local_Feature = concat(F1, up_2x(F2), up_4x(F3), up_8x(F4)); where concat represents feature concatenation, and up_2x represents upsampling by 2 times;

[0013] Step 3: Use the VR_LBP (Variable rotation Local binary patterns) feature maps of the image at four different scales as network input, and use ResNet50 to extract global features of the image by fusing residual dilated convolution.

[0014] Step 3.1: In the standard RGB color model, calculate the VR_LBP feature map that expresses the "direction" characteristics of the image; four different VR_LBP scales are used in the calculation process. 1,8 VR_LBP 2,16 VR_LBP 3,24 and VR_LBP 4,32 Four VR_LBP feature maps are generated, denoted as P1, P2, P3, and P4 respectively, and used as inputs to the RestNet50 network.

[0015] Step 3.2: Input the four VR_LBP feature maps of different scales, P1, P2, P3, and P4, into RestNet50. Denote the output feature maps of the last convolutional block of the RestNet50 network as {RP1, RP2, RP3, RP4}. Input these four feature maps into residual dilated convolutional blocks with corresponding sampling rates. The four sampling rates correspond to the R values ​​in the VR_LBP feature maps. The residual dilated convolutional block is formed by adding a 1×1 convolution shortcut connection to a 3×3 dilated convolution. The shortcut connection matches the spatial dimension of the feature maps, and the residual block achieves identity mapping while extracting image features through convolution. After the residual dilated convolution block, four feature maps are obtained, denoted as RPD1, RPD2, RPD3, and RPD4. The corresponding elements of the four feature maps RPD1, RPD2, RPD3, and RPD4 are added together to obtain the global feature; Global_Feature = RPD1⊕RPD2⊕RPD3⊕RPD4;

[0016] Step 4: The local features obtained in Step 2.4 are concatenated and fused with the global features obtained in Step 3.2 to finally achieve direction prediction;

[0017] Step 4.1: Downsample the local feature Local_Feature to the same resolution as the global feature Global_Feature using bilinear interpolation, and then concatenate them to obtain the fused feature; LG_Feature = concat(down(Local_Feature), Global_Feature), where down represents downsampling;

[0018] Step 4.2: The LG_Feature is processed by Global Average Pooling (GAP) to obtain a one-dimensional vector; then it is processed by a 256-segment fully connected layer to predict the image orientation.

[0019] Step 4.3: The logistic regression maximum likelihood loss function is used as the loss function to achieve orientation classification and realize automatic prediction of image orientation. The loss function is defined as follows:

[0020]

[0021]

[0022] Among them, h θ (x) represents the probability that sample x belongs to a certain class; y i For the predicted direction category, x i Let represent the feature of the i-th sample, m be the number of samples, θ be the parameters obtained by the network model, and T be the transpose of the matrix.

[0023] Furthermore, in step 2.1, there are two types of bottleneck layers, BTNK1 and BTNK2. BTNK2 has three conv+BN+ReLU convolutional blocks on the left, which add the convolution result F(x) to the input x, i.e., F(x)+x, and then pass it through a ReLU activation function. This module has the same number of input and output channels. BTNK1 has three conv+BN+ReLU convolutional blocks F(x) on the left and one conv+BN convolutional block G(x) on the right, which plays a role in matching the difference between the input and output dimensions, i.e., F(x) and G(x) have the same number of channels, and then sum them up F(x)+G(x). This module has different numbers of input and output channels. The ResNet50 network is composed of multiple bottleneck layers of different types stacked together.

[0024] Furthermore, in step 2.2, the attention mechanism module CBAM combines both the channel attention module and the spatial attention module. The channel attention module processes the input feature map through global max pooling and global average pooling to obtain two C×1×1 features, where C represents the number of channels. These two C×1×1 features are then fed into a two-layer neural network MLP.

[0025] The first layer of the two-layer neural network MLP has C / r neurons, where r is the reduction rate and the activation function is RelU. The second layer has C neurons. This two-layer neural network is shared.

[0026] The two features output by the MLP are added element-wise, and then a sigmoid activation operation is performed to generate the final channel attention feature. Finally, the channel attention feature is multiplied element-wise with the input feature to generate the input feature required by the spatial attention module.

[0027] The spatial attention module first applies global max pooling and global average pooling to the output features of the channel attention module based on the channel dimension, respectively, to obtain two 1×H×W single-channel pooled features. Then, these two 1×H×W single-channel pooled features are concatenated based on the channel dimension, followed by a 7×7 convolution operation to reduce the number of channels to 1. Then, a sigmoid function is used to generate spatial attention features. Finally, the spatial attention features generated by the sigmoid function and the input features of the spatial attention module are multiplied element-wise to obtain a weighted feature that combines channel attention and spatial attention.

[0028] Furthermore, in step 3.1, the VR_LBP feature map uses a pixel in the image as its center point, based on... Interpolation yields a neighborhood point centered on a circular set of sampling points, where R is the radius and P is the number of sampling points. Then, the value of the center pixel is compared with the values ​​of its neighboring pixels. If a neighboring pixel's value is greater than the center pixel's value, the neighborhood pixel is set to 1; otherwise, it is set to 0. The circular sampling points are then read clockwise, ultimately forming a binary sequence. This sequence is then converted to decimal, resulting in VR_LBP. R,P The code is calculated as follows:

[0029]

[0030] Among them, gray c It is the gray level of the current pixel, gray i It is the gray level of its domain; when x is less than 0, t(x) is 0, otherwise it is 1.

[0031] Furthermore, step 3.2, dilated convolution, has a hyperparameter called dilation rate, which defines the spacing between values ​​when the convolution kernel processes data. It is achieved by filling the convolution kernel with dilation rate - 1 zeros. Therefore, different dilation rates result in different receptive fields, allowing for the acquisition of multi-scale information. The convolution kernel of dilated convolution is K = k + (k-1)(r-1), where k is the original convolution kernel size and r is the dilation rate parameter. Four different dilated convolution kernels are used for VR_LBP at different scales, with r being 1, 2, 3, and 4, respectively.

[0032] Furthermore, in step 4.2, the window size of global average pooling is the size of the entire feature map. For each channel of the output feature map, an average value is calculated for all pixels. After global average pooling, a feature vector of size 1×1×C is obtained, where C is the number of channels in the original feature map.

[0033] Compared with the prior art, the present invention has the following advantages:

[0034] (1) The feature map output from the last residual structure of the last four parts of the ResNet50 network is passed through an attention mechanism module (CBAM) to obtain four spatial attention maps, which are then added to the corresponding elements of the original feature maps. Then, the three smaller-scale features are upsampled to the same resolution as the largest scale using bilinear interpolation, and then concatenated along the channels to obtain the final multi-scale attention fusion features, called local features. This method uses an attention mechanism to achieve better consistency between machine and human vision in judging direction. (2) The four LBP feature maps of the image at different scales are used as network inputs. ResNet50 is used to fuse residual dilated convolutions to obtain four feature maps, and then the corresponding elements of the feature maps are added to obtain global features. Extracting the directional characteristics of the image from multiple scales can express the directional characteristics of the image from different perspectives, improving the accuracy of direction detection. (3) The local features and global features are concatenated and fused, and then passed through GAP and fully connected layers to finally achieve direction prediction. The GAP module converts images of arbitrary input size into feature vectors of fixed size, reducing overfitting and accelerating the convergence speed of the network. Attached Figure Description

[0035] Figure 1 This is a schematic diagram of the present invention;

[0036] Figure 2 This is a flowchart of the present invention;

[0037] Figure 3 These are the dilated convolution kernels of different scales in this invention;

[0038] Figure 4 These are VR_LBP feature maps at different scales according to the present invention;

[0039] Figure 5 This is the network model framework of the present invention;

[0040] Figure 6 This is a schematic diagram of the performance index data of the eight models LModel1 to LModel8 of this invention.

[0041] Figure 7 This is a schematic diagram of the performance index data of the 10 models Gmodel1 to model10 of this invention. Detailed Implementation

[0042] To make the technical problems, technical solutions, and beneficial effects of this invention clearer, the invention will be further described in detail with reference to the embodiments and accompanying drawings. It should be understood that the specific embodiments described herein are only for explaining the invention and are not intended to limit the invention. The technical solutions of this invention are described in detail below with reference to the embodiments and accompanying drawings, but the scope of protection is not limited thereto.

[0043] Example 1

[0044] This invention selects a publicly available dataset for experimentation, and the specific implementation steps are as follows:

[0045] Step 1: Select the publicly available SUN dataset for testing. This dataset contains 108,754 images and 397 scene categories. Rotate each image clockwise by three angles: 90 degrees, 180 degrees, and 270 degrees. Each image will then be displayed in four different directions: top, right, bottom, and left.

[0046] Step 2: Extract local features of the image using the ResNet50 fusion residual attention mechanism. The specific steps are as follows:

[0047] Step 2.1: The ResNet50 network consists of 6 parts: convolutional layers, C0, C1, C2, C3, and C4. C0 contains one 7×7 convolutional layer with a stride of 2 and one 3×3 max pooling layer with a stride of 2. C1, C2, C3, and C4 are 1 / 4, 1 / 8, 1 / 16, and 1 / 32 times the size of the original image, respectively, and contain 3, 4, 6, and 3 bottleneck layers (BTNK) respectively. The bottleneck layer BTNK has two types: BTNK1 and BTNK2. BTNK2 has three conv+BN+ReLU convolutional blocks on the left, which add the result F(x) after convolution to the input x, i.e., F(x)+x, and then pass it through a ReLU activation function. This module has the same number of input and output channels. BTNK1 has three conv+BN+ReLU convolutional blocks F(x) on the left and one conv+BN convolutional block G(x) on the right. It plays the role of matching the difference between the input and output dimensions, i.e., F(x) and G(x) have the same number of channels, and then sum them up F(x)+G(x). This module has different numbers of input and output channels.

[0048] Step 2.2: Denote the output feature maps of the last residual structure of each part C1, C2, C3, and C4 as C1, C2, C3, and C4, respectively. Pass each of these four feature maps at different scales through a Convolutional Block Attention Module (CBAM) to obtain four spatial attention maps, denoted as A1, A2, A3, and A4. The CBAM combines channel attention and spatial attention modules. The channel attention module processes the input feature map through global max pooling and global average pooling to obtain two C×1×1 features, where C represents the number of channels. These two C×1×1 features are then fed into a two-layer neural network MLP. The first layer of the MLP has C / r neurons, where r is the reduction rate and the activation function is RelU. The second layer has C neurons. This two-layer neural network is shared. The two features output by the MLP are element-wise summed, followed by a sigmoid activation operation to generate the final channel attention feature. Finally, the channel attention feature is... The channel attention features are multiplied element-wise with the input features to generate the input features required by the spatial attention module. The spatial attention module first applies global max pooling and global average pooling to the output features of the channel attention module based on the channel dimension, respectively, to obtain two 1×H×W single-channel pooled features. Then, these two 1×H×W single-channel pooled features are concatenated based on the channel dimension, and then subjected to a 7×7 convolution operation to reduce the number of channels to 1. After that, a sigmoid function is used to generate spatial attention features. Finally, the spatial attention features generated by the sigmoid function are multiplied element-wise with the input features of the spatial attention module to obtain a weighted feature that combines channel attention and spatial attention.

[0049] Step 2.3: Add the spatial attention map to the corresponding original feature map element by element, denoted as .

[0050] Step 2.4: Upsample the feature maps of the three small scales F2, F3, and F4 to the same scale as F1 using bilinear interpolation, concatenate them along the channels, and then perform a 1×1 convolution operation to obtain the final multi-scale attention fusion feature, the local feature, denoted as: Local_Feature = concat(F1, up_2x(F2), up_4x(F3), up_8x(F4)); where concat represents feature concatenation, and up_2x represents upsampling by 2 times;

[0051] Step 3: Use the four VR_LBP (Variable Rotation Local Binary Patterns) feature maps of the image at different scales as network input, and use ResNet50 to extract global features of the image through residual dilated convolution. The specific steps are as follows:

[0052] Step 3.1: In the standard RGB color model, calculate the VR_LBP feature map that expresses the "direction" characteristics of the image; four different VR_LBP scales are used in the calculation process. 1,8 VR_LBP 2,16 VR_LBP 3,24 and VR_LBP 4,32 Four VR_LBP feature maps are generated, denoted as P1, P2, P3, and P4 respectively, and used as input to the RestNet50 network; a VR_LBP feature map is generated by taking a certain pixel in the image as the center. Point, based on Interpolation yields a neighborhood point centered on a circular set of sampling points, where R is the radius and P is the number of sampling points. Then, the value of the center pixel is compared with the values ​​of its neighboring pixels. If a neighboring pixel's value is greater than the center pixel's value, the neighborhood pixel is set to 1; otherwise, it is set to 0. The circular sampling points are then read clockwise, ultimately forming a binary sequence. This sequence is then converted to decimal, resulting in VR_LBP. R,P The code is calculated as follows:

[0053]

[0054] Among them, gray c It is the gray level of the current pixel, gray i It is the gray level of its domain; when x is less than 0, t(x) is 0, otherwise it is 1.

[0055] Step 3.2: Input the four VR_LBP feature maps of different scales, P1, P2, P3 and P4, into RestNet50. Denote the output feature map of the last convolutional block of the RestNet50 network as {RP1, RP2, RP3, RP4}. Input these four feature maps into the residual dilated convolutional blocks of the corresponding sampling rates. The four sampling rates correspond to the R values ​​in the VR_LBP feature maps.

[0056] A residual dilated convolutional block is formed by adding a 1×1 convolutional shortcut connection to a 3×3 dilated convolution. The shortcut connection matches the spatial dimension of the feature maps, while the residual block simultaneously achieves identity mapping during image feature extraction. After passing through the residual dilated convolutional block, four feature maps are obtained, denoted as RPD1, RPD2, RPD3, and RPD4. The global feature is obtained by summing the corresponding elements of these four feature maps. This indicates that corresponding elements are added together.

[0057] Dilated convolution has a hyperparameter called the dilation rate, which defines the spacing between values ​​processed by the convolution kernel. This dilation rate is essentially filled with zeros equal to the dilation rate - 1. Therefore, different dilation rates result in different receptive fields, thus acquiring multi-scale information. The kernel size of dilated convolution is K = k + (k-1)(r-1), where k is the original kernel size and r is the dilation rate. This invention uses four different dilated convolution kernels for VR_LBP at different scales, with r values ​​of 1, 2, 3, and 4.

[0058] Step 4: Concatenate and fuse the local features obtained in Step 2.4 with the global features obtained in Step 3.2 to finally achieve direction prediction. The specific steps are as follows:

[0059] Step 4.1: Downsample the local feature (Local_Feature) to the same resolution as the global feature (Global_Feature) using bilinear interpolation, then concatenate them to obtain the fused feature. LG_Feature = concat(down(Local_Feature), Global_Feature), where down indicates downsampling.

[0060] Step 4.2: LG_Feature is processed by global average pooling to obtain a one-dimensional vector; then, it is processed by a 256-segment fully connected layer to predict the image orientation; the window size of GAP is the size of the entire feature map. For each channel of the output feature map, an average value is calculated for all pixels. After global average pooling, a feature vector of size 1×1×C is obtained, where C is the number of channels in the original feature map.

[0061] Step 4.3: The logistic regression maximum likelihood loss function is used as the loss function to achieve orientation classification and realize automatic prediction of image orientation. The loss function is defined as follows:

[0062]

[0063]

[0064] Among them, h θ (x) represents the probability that sample x belongs to a certain class; y i For the predicted direction category, x i Let represent the feature of the i-th sample, m be the number of samples, θ be the parameters obtained by the network model, and T represent the matrix transpose.

[0065] Step 5: The experimental environment used was Anaconda3, and the deep learning framework was TensorFlow (GPU). 70% of each dataset was selected as the training set, and 30% as the test set. The original image size remained unchanged. A 10-fold cross-validation method was used; therefore, the final evaluation metric was the average accuracy after 10-fold cross-validation.

[0066] ResNet50 was pre-trained using the ImageNet dataset, and the obtained convolutional layer parameters were applied to the method proposed in this invention. Based on this, other modules were fine-tuned.

[0067] Experimental parameter settings: batch size was set to 128, the network was trained end-to-end using the momentum SGD optimizer, momentum was set to 0.9, learning rate was 0.001, and the number of iterations was 30. L2 regularization was added to prevent overfitting. This invention addresses a multi-class classification problem; therefore, classification accuracy (ACC), macro-average precision (MAP), macro-average recall (MAR), and confusion matrix were used to evaluate the model's performance.

[0068] To fully verify the effectiveness and applicability of the method of this invention, this invention proposes the effectiveness of local features in image orientation detection tasks. Ablation experiments were conducted on the feature fusion methods of different layers of ResNet50 used in this part of the network structure and CBAM. The global features adopt the structure we proposed, generating 8 different models. As shown in Table 1, the feature map used by Lmodel1 and Lmodel5 is C4, the feature maps used by Lmodel2 and Lmodel6 are C3 and C4, the feature maps used by Lmodel3 and Lmodel7 are C2, C3 and C4, and the feature maps used by Lmodel4 and Lmodel8 are C1, C2, C3 and C4.

[0069] Furthermore, the integration of CBAM was considered. Lmodel1 through Lmodel4 did not incorporate CBAM, instead using direct upsampling fusion. Lmodel5 through Lmodel8 incorporated CBAM. Lmodel1 is ReanNet50, the backbone network. Lmodel8 is the model proposed in this invention. Figure 6 The performance metrics of the eight models show that the model proposed in this invention (LModel8) has an accuracy of 99.2%, a macro average precision of 97.1%, and a macro average recall of 95.5%, which are superior to other models.

[0070] Table 1

[0071]

[0072] To verify the effectiveness of using eye-tracking heatmaps as labels and the effectiveness of the global features proposed in this invention in image orientation detection tasks, ablation experiments were conducted on "VR_LBP" images of different scales used in this part of the network structure with residual dilated convolution. Local features adopted the structure proposed in this invention, generating 10 different models, as shown in Table 2. First, "VR_LBP" images of different scales of the original image were calculated. The original image or LBP images of different scales were then selected as network inputs to the models. As shown in the table, the inputs to Gmodel1 and Gmodel2 were the original images, while the inputs to Gmodel3 and Gmodel7 were VR_LBP images. 1,8 The input to Gmodel4 and Gmodel8 is VR_LBP. 1,8 and VR_LBP 2,16 The input for Gmodel5 and Gmodel9 is VR_LBP. 1,8 VR_LBP 2,16 and VR_LBP 3,24 The input for Gmodel5 and Gmodel10 is VR_LBP. 1,8 VR_LBP 2,16 VR_LBP 3,24 and VR_LBP 4,32 Furthermore, the inclusion of residual dilated convolutional layers was also considered. Gmodel1, Gmodel3 through Gmodel6 did not include residual dilated convolutional layers, while Gmodel2, Gmodel7 through Gmodel10 did. Figure 7 The performance metrics of the 10 models in this paper show that the model proposed in this invention (Gmodel10) has an accuracy of 99.2%, a macro-average precision of 97.5%, and a macro-average recall of 94.7%, which are better than other models.

[0073] Table 2

[0074]

[0075] To verify the effectiveness of fusing local and global features in this task, experiments were conducted on four models: a backbone network, a local feature network, a global feature network, and a network using both features. Table 3 shows that the fusion method proposed in this invention achieves a classification accuracy of 99.2% on the dataset, outperforming other single-feature models. Experimental results demonstrate that the accuracy of the model using local and global feature fusion is higher than that of the backbone network or networks using single features. This also verifies that when viewing an image, the judgment of image orientation considers both the specific content within the image and its overall layout, enabling our classification model to achieve good classification results across various image types.

[0076] Table 3

[0077]

[0078] This invention was tested on the SUN dataset and compared with current related research, showing significant classification performance. The VR_LBP image descriptor designed in this invention effectively represents the directional characteristics of images. The fusion of local and global features helps the model perceive image direction from different visual features. This allows the method of this invention to perform well on various datasets.

[0079] Compared with existing image orientation perception methods, the advantages of this invention are: (1) This invention does not scale the original images in the dataset, maintaining the initial size of the images and preserving more effective information of the images. (2) Attention mechanism features are extracted from feature maps of different scales of the neural network model and fused to obtain local features. This method is similar to the human visual attention mechanism, acquiring more detailed information related to the target while ignoring other irrelevant information. Through this mechanism, high-value information can be quickly filtered from a large amount of information using limited attention resources. (3) "VR_LBP" (rotationally variable local binary pattern) features of different scales are extracted from the image, and ResNet50 is used to fuse residual dilated convolution to obtain 4 feature maps. Then, the corresponding elements of the feature maps are added to obtain global features. "VR_LBP" can more accurately express the orientation characteristics of the image and improve the generalization ability of the model. (4) The fusion of global features and local features can more comprehensively express the orientation semantics of the image and improve the classification accuracy of the model.

[0080] Contents not described in detail in this specification are prior art known to those skilled in the art. Although illustrative specific embodiments of the invention have been described above to facilitate understanding by those skilled in the art, it should be understood that the invention is not limited to the scope of the specific embodiments. Various modifications are readily apparent to those skilled in the art as long as they fall within the spirit and scope of the invention as defined and determined by the appended claims, and all inventions utilizing the concept of this invention are protected.

Claims

1. An image orientation prediction method based on multi-scale fusion and attention mechanism, characterized in that: Includes the following steps: Step 1: Rotate each image clockwise by three angles: 90 degrees, 180 degrees, and 270 degrees. Each image will eventually be displayed in four different directions: top, right, bottom, and left. Step 2: Use ResNet50 with residual attention mechanism to extract local features from each image. The specific steps are as follows: Step 2.1: The ResNet50 network consists of 6 parts: convolutional layers, C0, C1, C2, C3, and C4; C0 contains one 7×7 convolutional layer with a stride of 2 and one 3×3 max pooling layer with a stride of 2; C1, C2, C3, and C4 are 1 / 4, 1 / 8, 1 / 16, and 1 / 32 times the size of the original image, respectively, and contain 3, 4, 6, and 3 bottleneck layers (BTNK), respectively. Step 2.2: The last residual structure output feature maps of each part C1, C2, C3, and C4 are denoted as C1, C2, C3, and C4, respectively; and the feature maps at four different scales are passed through an attention mechanism module CBAM to obtain four spatial attention maps denoted as A1, A2, A3, and A4. Step 2.3: Add the spatial attention map to the corresponding original feature map element by element, denoted as F. i =A i ⊕C i (i = 1, 2, 3, 4), where ⊕ represents the sum of corresponding elements; Step 2.4: Upsample the feature maps of the three small scales F2, F3, and F4 to the same scale as F1 using bilinear interpolation, concatenate them along the channels, and then perform a 1×1 convolution operation to obtain the final multi-scale attention fusion feature, the local feature, denoted as: Local_Feature = concat(F1, up_2x(F2), up_4x(F3), up_8x(F4)); where concat represents feature concatenation, and up_2x represents upsampling by 2 times; Step 3: Use the four rotationally variable local binary pattern feature maps VR_LBP of the image at different scales as network input, and use ResNet50 to fuse residual dilated convolution to extract the global features of the image. The specific steps are as follows; Step 3.1: In the standard RGB color model, calculate the VR_LBP feature map that expresses the "direction" characteristics of the image; four different VR_LBP scales are used in the calculation process. 1,8 VR_LBP 2,16 VR_LBP 3,24 and VR_LBP 4,32 Four VR_LBP feature maps are generated, denoted as P1, P2, P3, and P4 respectively, and used as inputs to the RestNet50 network. Step 3.2: Input the four VR_LBP feature maps of different scales, P1, P2, P3, and P4, into RestNet50. The output feature maps of the last convolutional block of the RestNet50 network are denoted as RP1, RP2, RP3, and RP4. Input these four feature maps into residual dilated convolutional blocks with corresponding sampling rates. The four sampling rates correspond to the R values ​​in the VR_LBP feature maps. After passing through the residual dilated convolutional blocks, four feature maps are obtained, denoted as RPD1, RPD2, RPD3, and RPD4. Add the corresponding elements of these four feature maps (RPD1, RPD2, RPD3, and RPD4) to obtain the global feature: Global_Feature = RPD1⊕RPD2⊕RPD3⊕RPD4. Step 4: The local features obtained in Step 2.4 are concatenated and fused with the global features obtained in Step 3.2 to finally achieve direction prediction; Step 4.1: Downsample the local feature Local_Feature to the same resolution as the global feature Global_Feature using bilinear interpolation, and then concatenate them to obtain the fused feature; LG_Feature = concat(down(Local_Feature), Global_Feature), where down represents downsampling; Step 4.2: LG_Feature is processed by a global average pooling to obtain a one-dimensional vector; then, it is processed by a 256-byte fully connected layer to predict the image orientation. Step 4.3: The logistic regression maximum likelihood loss function is used as the loss function to achieve orientation classification and realize automatic prediction of image orientation. The loss function is defined as follows: Among them, h θ (x) represents the probability that sample x belongs to a certain class; y i For the predicted direction category, x i Let represent the feature of the i-th sample, m be the number of samples, θ be the parameters obtained by the network model, and T be the transpose of the matrix.

2. The image orientation prediction method based on multi-scale fusion and attention mechanism according to claim 1, characterized in that: In step 2.1, there are two types of bottleneck layers BTNK: BTNK1 and BTNK2. BTNK2 has three conv+BN+ReLU convolutional blocks on the left, which add the result F(x) after convolution to the input x, i.e., F(x)+x, and then pass through a ReLU activation function. This module has the same number of input and output channels. BTNK1 has three conv+BN+ReLU convolutional blocks F(x) on the left and one conv+BN convolutional block G(x) on the right, which plays the role of matching the difference between the input and output dimensions. Since F(x) and G(x) have the same number of channels, they are summed F(x)+G(x). Therefore, this module has different numbers of input and output channels. The ResNet50 network is composed of multiple bottleneck layers BTNK stacked together.

3. The image orientation prediction method based on multi-scale fusion and attention mechanism according to claim 1, characterized in that: In step 2.2, the attention mechanism module CBAM combines the channel attention module and the spatial attention module. The channel attention module processes the input feature map through global max pooling and global average pooling to obtain two C×1×1 features, where C represents the number of channels. These two C×1×1 features are then fed into a two-layer neural network MLP. The first layer of the two-layer neural network MLP has C / r neurons, where r is the reduction rate and the activation function is RelU. The second layer has C neurons. This two-layer neural network is shared. The two features output by the MLP are added element-wise, and then a sigmoid activation operation is performed to generate the final channel attention feature. Finally, the channel attention feature is multiplied element-wise with the input feature to generate the input feature required by the spatial attention module. The spatial attention module first applies global max pooling and global average pooling to the output features of the channel attention module based on the channel dimension, respectively, to obtain two 1×H×W single-channel pooled features. Then, these two 1×H×W single-channel pooled features are concatenated based on the channel dimension, followed by a 7×7 convolution operation to reduce the number of channels to 1. Then, a sigmoid function is used to generate spatial attention features. Finally, the spatial attention features generated by the sigmoid function and the input features of the spatial attention module are multiplied element-wise to obtain a weighted feature that combines channel attention and spatial attention.

4. The image orientation prediction method based on multi-scale fusion and attention mechanism according to claim 1, characterized in that: The VR_LBP feature map in step 3.1 uses a pixel in the image as the center point, based on... Interpolation yields a neighborhood point centered on a circular set of sampling points, where R is the radius and P is the number of sampling points. Then, the value of the center pixel is compared with the values ​​of its neighboring pixels. If a neighboring pixel's value is greater than the center pixel's value, the neighborhood pixel is set to 1; otherwise, it is set to 0. The circular sampling points are then read clockwise, ultimately forming a binary sequence. This sequence is then converted to decimal, resulting in VR_LBP. R,P The code is calculated as follows: Among them, gray c It is the gray level of the current pixel, gray i It is the gray level of its domain; when x is less than 0, t(x) is 0, otherwise it is 1.

5. The image orientation prediction method based on multi-scale fusion and attention mechanism according to claim 1, characterized in that: Step 3.2, dilated convolution, has a hyperparameter called dilation rate, which defines the spacing between values ​​when the convolution kernel processes data. It is filled with dilation rate minus one zero in the convolution kernel. Therefore, when different dilation rates are set, the receptive field is different, and multi-scale information is obtained. The convolution kernel of dilated convolution is K = k + (k-1)(r-1), where k is the original convolution kernel size and r is the dilation rate parameter of dilated convolution. Four different dilated convolution kernels are used according to different scales of VR_LBP, with r being 1, 2, 3 and 4 respectively.

6. The image orientation prediction method based on multi-scale fusion and attention mechanism according to claim 1, characterized in that: The residual dilated convolution block is formed by adding a 1×1 convolution shortcut connection to a 3×3 dilated convolution. The shortcut connection matches the spatial dimension of the feature map, while the residual block achieves identity mapping while extracting image features through convolution.

7. The image orientation prediction method based on multi-scale fusion and attention mechanism according to claim 1, characterized in that: In step 4.2, the window size of global average pooling is the size of the entire feature map. For each channel of the output feature map, an average value is calculated for all pixels. After global average pooling, a feature vector of size 1×1×C is obtained, where C is the number of channels in the original feature map.

Images