A multi-view image classification method based on a global filtering module
By combining a global filtering module and deep convolution, the problem of insufficient feature and semantic correspondence in appearance patent image classification by traditional machine learning algorithms is solved, achieving higher classification accuracy and efficiency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- GUANGDONG POLYTECHNIC NORMAL UNIV
- Filing Date
- 2023-02-07
- Publication Date
- 2026-06-02
Smart Images
Figure CN116091830B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of deep learning image classification technology, and more specifically, to a multi-view image classification method based on a global filtering module. Background Technology
[0002] Intellectual property rights are legally protected rights, representing the patents that rights holders enjoy in accordance with the law over the fruits of their labor. Design patents, as a type of industrial intellectual property, refer to aesthetically pleasing new designs suitable for industrial application, relating to the shape, pattern, or combination thereof, as well as the combination of color with shape and pattern. Publicly available data shows that the number of design patent applications is increasing year by year. However, this rapid growth in design patent applications has also generated numerous problems. For example, design patents are easily imitated or duplicated by individuals other than the inventor, thus necessitating strengthened examination and protection of design patents. How patent administration departments can utilize AI technology to enhance the examination and retrieval of design patents is a pressing issue for intellectual property protection. Improving the accuracy and efficiency of online design patent searches can enhance the quality and efficiency of patent examiners' examinations and provide support for the protection and innovation of design patents. Patent classification is the foundation and core of applications such as patent retrieval; therefore, how to utilize deep learning algorithms to improve the accuracy of design patent classification is a key issue in improving the effectiveness of design patent searches.
[0003] Traditional machine learning algorithms for image classification tasks primarily rely on manual selection of features from design patent images. However, this approach suffers from a significant problem: a poor correspondence cannot be established between the underlying features of the design patent image and the semantics of the design patent. Therefore, traditional machine learning methods cannot meet the current needs for design patent image classification and retrieval. Summary of the Invention
[0004] The technical problem to be solved by the present invention is to address the above-mentioned shortcomings of the prior art. The purpose of the present invention is to provide a multi-view image classification method based on a global filtering module.
[0005] The technical feature of this invention is: a multi-view image classification method based on a global filtering module, comprising the following steps:
[0006] Step S1. Obtain a multi-view image data sample set and divide it into a training set and a test set;
[0007] Step S2. Obtain multi-view image data from the training set, and perform block embedding processing on the multi-view image data to obtain a multi-view image block set;
[0008] Step S3. Generate image blocks from the multi-view image block set and embed them into the input MGViT network, defining the forward propagation process of the MGViT network;
[0009] Step S4. Define the global filtering module and the Transformer module, and define the loss function for iterative optimization;
[0010] Step S5. Input the training set into the MGViT network for training; input the test set into the trained MGViT network, and adjust the parameters of the MGViT network to obtain the optimized MGViT model;
[0011] Step S6. Input the image to be classified into the MGViT model, and perform Softmax prediction on the output of the MGViT model to obtain the prediction result.
[0012] As a further improvement, the training set and the test set are divided according to the proportion of different types of data labels in the image data sample set.
[0013] Furthermore, step S3 includes the following steps:
[0014] Step S31. The set of multi-view image blocks is represented as {V} 1 ,…,V L}, each view V j (j=
[0015] The size of the input image (1,…,L) is W×H×C. This is fed into the MGViT network to preprocess the input image, flattening it into non-overlapping blocks. The shape changes are as follows:
[0016]
[0017] Where C is the number of channels, H and W are the height and width of the image respectively, and W is the block size;
[0018] Then flatten each image patch into a one-dimensional vector, each vector having a size of P0*P0*C, where N is defined as:
[0019]
[0020] N is the number of image blocks generated, which is the effective input sequence length of ViT;
[0021] Since the images in the same batch are divided into blocks, the total number of blocks obtained is:
[0022] S = L * N;
[0023] Step S32. After segmenting and serializing the image, a linear transformation is then performed on each vector to map the vectorized block S into a latent D-dimensional embedding space, as shown below:
[0024]
[0025] in To perform a linear transformation on each flattened image patch, by When the dimension is reduced to D, this portion of the projected output is called the image patch embedding, and the learnable location embedding E pos Added to image patch embedding:
[0026]
[0027] in To preserve spatial location information between image blocks and share it across different views, the generated embedded block sequence a0 is transmitted to the Transformer encoder. Unlike the Transformer's default fixed-position encoding, ViT uses standard learnable 1-D positional encoding embedding.
[0028] Furthermore, step S4 includes the following steps:
[0029] Step S41. Define the Discrete Fourier Transform, given a two-dimensional signal X. m,n For any given condition, 0 ≤ m ≤ M-1, 0 ≤ n ≤ N-1, the two-dimensional DFT formula is as follows:
[0030]
[0031] Step S42. Define the Transformer encoder module, which consists of a global filtering module and an MLP block. Apply Layer norm (LN) before each block.
[0032] Step S43. Define a loss function based on the forward propagation process.
[0033] Further, in step S41, a learnable global filter is defined for mixing tokens representing different spatial locations, and for combining with the spectral features after FFT transformation, given a token. First, perform a two-dimensional FFT along the spatial dimension to transition to the frequency domain:
[0034]
[0035] in This represents a two-dimensional FFT, where X is a complex tensor representing the spectrum of x;
[0036] Through a learnable filter Multiply by X to modulate the spectrum:
[0037]
[0038] in It is the Hadamard product, which represents element-wise multiplication; the filter T is called the global filter, which has the same dimension as X and can represent any filter in the frequency domain.
[0039] Finally, inverse FFT is used to convert the modulated spectrum X back to the spatial domain and update the token:
[0040]
[0041] The global filter module can be equivalent to a deep global circular convolution with a filter size of H×W. In the frequency domain, the dot product operation has no inductive bias, which can be used to reduce redundant computation. If x is a real tensor, then its DFTX is conjugate symmetric and can be represented as... Based on the conjugate property, we can take only half of the value of X while retaining all the information:
[0042]
[0043] in, If the actual input is represented by a two-dimensional FFT, then the global filter is implemented as follows: This can reduce the parameters by half and also ensure... It is a real tensor.
[0044] Further, step S42 includes the following steps:
[0045] Step S421. The Transformer encoder mainly consists of a global filtering module and a multilayer perceptron block. The output can be written as:
[0046] a′ l =GlobalFilter(LN(a l-1 ))
[0047] a l =MLP(LN(a′) l ))+a′ l l∈1,2,…,L
[0048] Where LN(·) is the normalization operation, a l Represented as an encoded image;
[0049] Step S422. Define the feedforward network part in the multilayer perceptron block. The feedforward network consists of two fully connected layers. The first FC transforms the features from dimension 4, and the second FC restores the features from dimension 4. The non-linear activation functions in between all use GeLU, and feature transformation is performed along the embedding dimension, expanding the hidden dimension between the two fully connected layers to obtain a richer feature representation, as expressed below:
[0050]
[0051] Where f(·) represents a nonlinear activation function, the bias term is omitted in the above equation, and the input of the feedforward network is an embedding vector sequence;
[0052] Step S423. Define a depthwise convolution module, applying a k×k (k>1) convolution kernel to each channel, aggregating the internal features of the k×k convolution kernels to calculate new features, and reintroducing the depthwise convolution into the feedforward network of the Transformer. The calculation can be represented as:
[0053] Y = f(Y0W) d W2
[0054] in, It is the kernel of depthwise convolution.
[0055] Furthermore, in step S43, the MGViT network adopts L... cross (y,y ′ ), is the cross-entropy loss function used to calculate the loss between the predicted label and the true label, where the predicted label y ′ The cross-entropy loss between the true label y and the ground truth label y is shown in the following equation:
[0056]
[0057] Beneficial effects
[0058] Compared with the prior art, the advantages of this invention are as follows:
[0059] 1. Unlike previous multi-view image classification methods that ignored the importance of global information between views and caused the loss of visual information in the process of fusing multi-view features in the view pooling layer, this invention utilizes position encoding and Fourier transform acting in the frequency domain to mine long-distance dependencies within the image, ensuring information interaction between views.
[0060] 2. This invention introduces deep convolution into the model by taking into account the ability to retain local information in the feedforward network.
[0061] 3. The network operation method proposed in this invention significantly improves the accuracy of multi-view image classification tasks, which is of great significance for image classification. Attached Figure Description
[0062] Figure 1 This is a flowchart of the present invention;
[0063] Figure 2 Images from the Patent-MNIST dataset;
[0064] Figure 3 A comparison chart of training times for different models;
[0065] Figure 4 This is a flowchart of the global filtering module.
[0066] Figure 5 A flowchart of the Transformer encoder's workflow;
[0067] Figure 6 A flowchart illustrating the workflow for introducing the depthwise convolution module. Detailed Implementation
[0068] The present invention will be further described below with reference to specific embodiments shown in the accompanying drawings.
[0069] See Figures 1-6 A multi-view image classification method based on a global filtering module includes the following steps S1 to S6:
[0070] Step S1. Obtain a multi-view image data sample set and divide it into a training set and a test set. Since the difficulty of obtaining different types of data labels varies, the training set and test set are divided according to the proportion of different types of data labels in the image data sample set. Enhancement processing, such as cropping and transformation (translation or rotation), can be applied to the images in the training set.
[0071] Step S2. Obtain multi-view image data from the training set, and perform block embedding processing on the multi-view image data to obtain a multi-view image block set.
[0072] Step S3. Generate image patches from the multi-view image patch set and embed them into the input MGViT network, defining the forward propagation process of the MGViT network.
[0073] Step S3 includes the following steps S31 to S32:
[0074] Step S31. The set of multi-view image blocks is represented as {V} 1 ,…,V L}, each view V jThe size of the input image (j=1,…,L) is W×H×C. This is fed into the MGViT network to preprocess the input image, flattening it into non-overlapping blocks. The shape changes are as follows:
[0075]
[0076] Where C is the number of channels, H and W are the height and width of the image respectively, and W is the block size.
[0077] Then flatten each image patch into a one-dimensional vector, each vector having a size of P0*P0*C, where N is defined as:
[0078]
[0079] N is the number of image blocks generated, and is the effective input sequence length of ViT.
[0080] Since the images in the same batch are divided into blocks, the total number of blocks obtained is:
[0081] S = L * N.
[0082] Step S32. After segmenting and serializing the image into blocks, a linear transformation is then performed on each vector, i.e., a fully connected layer, to map the vectorized blocks S into a latent D-dimensional embedding space, as shown below:
[0083]
[0084] in To perform a linear transformation on each flattened image patch, by When the dimension is reduced to D, this portion of the projected output is called the image patch embedding, and the learnable location embedding E pos Added to image patch embedding:
[0085]
[0086] in To preserve spatial location information between image patches and share it across different views, the generated embedded patch sequence a0 is transmitted to the Transformer encoder. Unlike the Transformer, which uses fixed-position encoding by default, ViT uses standard learnable 1-D positional encoding embedding.
[0087] Step S4. Define the global filtering module and the Transformer module, and define the loss function for iterative optimization.
[0088] Step S4 includes the following steps S41 to S43:
[0089] Step S41. Define the Discrete Fourier Transform, given a two-dimensional signal X.m,n For any given condition, 0 ≤ m ≤ M-1, 0 ≤ n ≤ N-1, the two-dimensional DFT formula is as follows:
[0090]
[0091] In step S41, a learnable global filter is defined to mix tokens representing different spatial locations, and to combine them with the spectral features after FFT transformation, given a token. First, perform a two-dimensional FFT along the spatial dimension to transition to the frequency domain:
[0092]
[0093] in Let X represent a two-dimensional FFT, where X is a complex tensor representing the spectrum of x.
[0094] Through a learnable filter Multiply by X to modulate the spectrum:
[0095]
[0096] in It is the Hadamard product, which represents element-wise multiplication; the filter T is called the global filter, which has the same dimension as X and can represent any filter in the frequency domain.
[0097] Finally, inverse FFT is used to convert the modulated spectrum X back to the spatial domain and update the token:
[0098]
[0099] The global filter module can be equivalent to a deep global circular convolution with a filter size of H×W. In the frequency domain, the dot product operation has no inductive bias. Due to the unique properties of the DFT, it can be used to reduce redundant computations. If x is a real tensor, then its DFTX is conjugate symmetric and can be represented as... Based on the conjugate property, we can take only half of the value of X while retaining all the information:
[0100]
[0101] in, If the actual input is represented by a two-dimensional FFT, then the global filter is implemented as follows: This can reduce the parameters by half and also ensure... It is a real tensor.
[0102] Step S42. Define the Transformer encoder module, which consists of a global filtering module and an MLP block. Apply Layer norm (LN) before each block.
[0103] Step S42 includes the following steps S421 to S423:
[0104] Step S421. The Transformer encoder mainly consists of a global filter module and a multilayer perceptron (MLP) block. Therefore, the output can be written as:
[0105] a′ l =GlobalFilter(LN(a l-1 ))
[0106] a l =MLP(LN(a′) l ))+a′ l l∈1,2,…,L
[0107] Where LN(·) is the normalization operation, a l It is represented as an encoded image.
[0108] Step S422. Define the feedforward network part in the Multilayer Perceptron (MLP) block. The feedforward network consists of two fully connected layers. The first FC transforms the features from dimension 4, and the second FC restores the features from dimension 4. The non-linear activation functions in between all use GeLU, and feature transformation is performed along the embedding dimension, expanding the hidden dimension between the two fully connected layers to obtain a richer feature representation, as expressed below:
[0109]
[0110] Where f(·) represents a nonlinear activation function, the bias term is omitted in the above equation, and the input of the feedforward network is an embedding vector sequence.
[0111] Step S423. Define a deep convolution module. Apply a k×k (k>1) convolution kernel to each channel and aggregate the internal features of the k×k convolution kernels to calculate new features. Therefore, deep convolution is an effective method to introduce locality into the network. Considering this, this embodiment reintroduces deep convolution into the feedforward network of the Transformer. The calculation can be expressed as:
[0112] Y = f(Y0W) d W2
[0113] in, It is the kernel of depthwise convolution.
[0114] Step S43. Define the loss function according to the forward propagation process. In step S43, the MGViT network uses L... cross(y, y′) is the cross-entropy loss function used to calculate the loss between the predicted label and the true label, where the cross-entropy loss is between the predicted label y′ and the true label y. The cross-entropy loss function is shown in the following equation:
[0115]
[0116] Step S5. Input the training set into the MGViT network for training; input the test set into the trained MGViT network, and adjust the parameters of the MGViT network to obtain the optimized MGViT model.
[0117] Step S6. Input the image to be classified into the MGViT model, and perform Softmax prediction on the output of the MGViT model to obtain the prediction result.
[0118] This embodiment evaluates the proposed model on public datasets (ModelNet40 / 10) and the Guangdong Provincial Key Laboratory of Intellectual Property Big Data Design Dataset (Patent-MNIST), respectively, and makes a horizontal comparison with current image classification methods that perform better in multi-view image classification applications.
[0119] This experiment implemented the model using a PyTorch environment, using the MVIT model without a global filtering module as a comparison. The MVIT-Large / MGVIT-Large models were trained. Aadm was used as the optimizer, with an initial learning rate of 0.0001, β1 = 0.9, β2 = 0.999, and image size set to 384×384 pixels. All experiments were conducted using an Intel(R) Core(TM) i7-10700 CPU @ 2.90GHz and an NVIDIA GeForce RTX 2080Ti GPU.
[0120] The image patch size was uniformly set to 16×16. The MVIT model without a global filtering module simply added a deep convolution to the feedforward neural network. One important parameter in MGVIT, MLP_Ratio, refers to the ratio of the hidden layer dimension to the embedding layer dimension. By adjusting this parameter, we found that setting it to 6 achieved the best balance of parameter count, training speed, and accuracy. See the experimental section for detailed comparisons. It can be seen that under the same parameter settings, the MGVIT model has fewer parameters than the MVIT model without a global filtering module. This is because it operates in the frequency domain, reducing its complexity from quadratic to logarithmic levels, significantly reducing redundancy. Therefore, optimizing the model by introducing a global filtering module is feasible.
[0121] Table 1 Characteristics of MGVIT / MVIT variants
[0122] Model Encoder layers Hidden layer dimension D MLP_Ratio MLP Dimensions Head count Parameters MVIT-Base 6 768 - 3072 12 127.45M MVIT-Large 12 1024 - 4096 16 361.18M MGVIT-Base 6 768 6 - - 62.00M MGVIT-Large 12 1024 6 - - 200.01M
[0123] Table 2 Comparison of Multi-View Image Classification Methods
[0124] method Input method ModelNet40 (%) ModelNet10 (%) Patent-MNIST (%) Volumetric CNN 89.9 - - 3D-A-Nets Volume 90.5 - - LP-3DCNN 92.1 94.4 PointNet++ 91.9 - - 3DmFV-Net Points 91.6 95.2 - DeepCCFV 92.5 - - MVCNN 91.6 92.8 94.2 RotationNet 91.0 94.0 - 3D2SeqViews 93.4 94.7 - GVCNN 93.8 94.8 - DAN 93.5 95.4 94.5 MLVCNN Views 94.1 - - CAR-Net 95.2 95.8 - MVIT-Base 93.5 94.4 93.2 MVIT-Large 94.6 96.1 94.8 MGVIT-Base 94.2 95.2 93.4 MGVIT-Large 95.5 96.8 94.8
[0125] The experiment mainly compares and analyzes the experimental results of different multi-view methods. As shown in Table 2, the experimental effects of multi-view classification methods based on voxels, point clouds and views are compared respectively. From the experimental results in Table 2, the experimental classification results of the multi-view method based on views are better than those of the voxel and point cloud methods. In the ModelNet40 dataset, the best experimental results of the three classification methods in Table 2 are 92.1%, 92.5% and 95.5% respectively. It can be seen that the multi-view classification method based on views is significantly better than the classification effect of other methods. Among them, the classification accuracy of the MGVIT classification method in this paper reaches the best of 95.5%, which is higher than the classification accuracy of other view-based methods. In the ModelNet10 dataset, the classification accuracy of the MGVIT model proposed in this paper is 96.8%. The point cloud method achieves the best classification accuracy of 95.2%. The second highest multi-view classification method based on views is CAR-Net
[49] at 95.8%. The experimental effects of these methods are all lower than the experimental effect of MGVIT in this paper. Since the Patent-MNIST dataset is an image dataset, this paper only compares the view-based methods. The multi-view classification models MVCNN, DAN, and our proposed MGVIT achieve classification accuracies of 94.2%, 94.5%, and 94.8% respectively on the Patent-MNIST dataset, making our proposed method the best in terms of classification performance.
[0126] Analysis of the experimental results revealed that the MGVIT model achieved good classification performance on the three datasets. However, analysis of the Patent-MNIST dataset showed that the MGVIT model did not demonstrate the significantly better classification performance compared to other methods on the ModelNet10 / 40 dataset. This is because the Patent-MNIST dataset consists of two views, and the spatial location information of the front and back views is only partially correlated, requiring more attention to local information. The proposed model, on the other hand, is more sensitive to the spatial relationships between multiple views and has more associated interactive information. Therefore, the model proposed in this invention has a greater advantage on datasets with a larger number of views.
[0127] This section compares different MLP_Ratio values. As shown in Table 3, as the MLP_Ratio value increases, the number of parameters and training speed increase proportionally and inversely, respectively. Finally, the experimental settings in this paper are based on MLP_Ratio equal to 6, so that the ratio of the hidden layer dimension to the embedding layer dimension is 6, at which point the number of parameters and training speed are moderate.
[0128] Table 3 shows the impact of setting different MLP Ratios.
[0129]
[0130] Table 3 shows that MLP_Ratio has a significant impact on the number of model parameters. As the ratio increases, the embedding layer dimension increases linearly, and the number of parameters increases proportionally, but the speed decreases proportionally, and the training time increases. Accuracy peaks when MLP_Ratio is equal to 6 and 8, and shows a certain downward trend when set to 10. It can be seen that both excessively large and small values of MLP_Ratio lead to poor results. The accuracy is lowest when the ratio is 4. Therefore, based on a comprehensive judgment of training time and accuracy, choosing MLP_Ratio of 6 is the most suitable.
[0131] Table 4 compares the MGVIT-1 model (without global filtering and depthwise convolution), the MGVIT-2 model (with only global filtering), the MGVIT-3 model (with only depthwise convolution), and the MGVIT-4 model (a combination of both used in this paper). It also compares the MGVIT-1 model with the traditional MVCNN model. Experiments were conducted on the ModelNet10 dataset. The results are shown in Table 4. MGVIT-1 essentially performs 12-view image classification on top of the original VIT model. It shows that the accuracy of multi-view classification is 1.8% higher than that of single-view classification. This comparison verifies that multi-view classification, due to the richer spatial relationships arising from multiple perspectives of an object, leads to more accurate object classification. The MGVIT-3 model with added depthwise convolution achieves a 1.6% higher accuracy than the MGVIT-1 model. The experimental results demonstrate that adding depthwise convolution effectively increases locality, improving the model's ability to capture local image features and thus optimizing classification performance. The MGVIT-2 model improved the final accuracy from 94.0% to 95.9% compared to the MGVIT-1 model, a 1.9% improvement, fully demonstrating the effectiveness of the global filtering module and its significant impact on improving the accuracy of the classification model. Comparing the MGVIT-4 model, it can be seen that global filtering and depthwise convolution achieve optimal performance when used together, resulting in an overall accuracy improvement of 2.8% compared to the MGVIT-1 model. Compared to the traditional MVCNN method, the MGVIT-4 model achieves an overall accuracy improvement of 4.0%. Therefore, the improved model of this invention has better advantages than traditional CNN-based methods and is better suited for multi-view datasets.
[0132] Table 4 Ablation Experiment
[0133]
[0134]
[0135] The above are merely preferred embodiments of the present invention. It should be noted that those skilled in the art can make several modifications and improvements without departing from the structure of the present invention, and these will not affect the effectiveness of the implementation of the present invention or the practicality of the patent.
Claims
1. A method for multi-view image classification based on a global filtering module, characterized in that, The steps include the following: Step S1. Obtain a multi-view image data sample set and divide it into a training set and a test set; Step S2. Obtain multi-view image data from the training set, and perform block embedding processing on the multi-view image data to obtain a multi-view image block set; Step S3. Generate image blocks from the multi-view image block set and embed them into the input MGViT network, defining the forward propagation process of the MGViT network; Step S4. Define the global filtering module and the Transformer module, and define the loss function for iterative optimization; Step S5. Input the training set into the MGViT network for training; input the test set into the trained MGViT network, and adjust the parameters of the MGViT network to obtain the optimized MGViT model; Step S6. Input the image to be classified into the MGViT model, and perform Softmax prediction on the output of the MGViT model to obtain the prediction result; Step S3 includes the following steps: Step S31. The set of multi-view image blocks is represented as follows: Each view The size is The image is then fed into the MGViT network for preprocessing, where it is flattened into non-overlapping blocks. The shape changes in these blocks are as follows: Where C is the number of channels, H and W are the height and width of the image respectively, and W is the block size; Then flatten each image patch into a one-dimensional vector, with each vector having a size of [missing value]. ,in The definition of is: It represents the number of image blocks generated, and is the effective input sequence length of ViT; Since the images in the same batch are divided into blocks, the total number of blocks obtained is: ； Step S32. After segmenting and serializing the image, a linear transformation is then performed on each vector to map the vectorized block S into a latent D-dimensional embedding space, as shown below: in To perform a linear transformation on each flattened image patch, by Dimensions dropped to This portion of the projected output is called image patch embedding, and the learnable location embedding. Added to image patch embedding: in The generated embedded block sequence is used to preserve spatial location information between image blocks and to share it across different views. The data is transmitted to the Transformer encoder. Unlike the Transformer, which uses fixed-position encoding by default, ViT uses standard learnable 1-D positional encoding embedding. Step S4 includes the following steps: Step S41. Define the Discrete Fourier Transform, given a two-dimensional signal. The two-dimensional DFT formula is as follows: ； Step S42. Define the Transformer encoder module, which consists of a global filtering module and an MLP block. Apply Layer norm before each block. Step S43. Define a loss function based on the forward propagation process; In step S41, a learnable global filter is defined to mix tokens representing different spatial locations, and to combine them with the spectral features after FFT transformation, given a token. First, perform a two-dimensional FFT along the spatial dimension to transform it into the frequency domain: in Represents a two-dimensional FFT. It is a complex tensor representation The spectrum; Through a learnable filter Multiply To modulate the spectrum: in It's the Hadamard product, representing element-wise multiplication; a filter. Called a global filter, it has and The same dimension represents any filter in the frequency domain; Finally, the modulated spectrum is processed using inverse FFT. Return to the spatial domain and update the token: The global filter module is equivalent to a depthwise global recurrent convolution, with a filter size of... In the frequency domain, dot product operations do not have an inductive bias, which is used to reduce redundant calculations. If it is a real tensor, then its DFT It is conjugate symmetric, represented as Based on the conjugate property, only take It can store half the value while retaining all the information: in, If the actual input is represented by a two-dimensional FFT, then the global filter is implemented as follows: Reduce parameters by half and ensure It is a real number tensor; Step S42 includes the following steps: Step S421. The Transformer encoder, which includes a global filtering module and a multilayer perceptron block, outputs as follows: in, For normalization operations, Represented as an encoded image; Step S422. Define the feedforward network part in the multilayer perceptron block. The feedforward network consists of two fully connected layers. The first FC transforms the features from dimension 4, and the second FC restores the features from dimension 4. The non-linear activation functions in between all use GeLU, and feature transformation is performed along the embedding dimension, expanding the hidden dimension between the two fully connected layers to obtain rich feature representations, as expressed below: in, This represents a non-linear activation function. The bias term is omitted in the above equation. The input to the feedforward network is a sequence of embedding vectors. Step S423. Define a depthwise convolutional module. Apply a k×k, k>1 convolutional kernel to each channel. Aggregate the internal features of the k×k convolutional kernels to calculate new features. Reintroduce the depthwise convolution into the feedforward network of the Transformer. The calculation is represented as: in, It is the kernel of depthwise convolution. ; The MGViT network model was implemented using the PyTorch environment. Aadm was used as the optimizer in the experiments, with an initial learning rate of 0.0001. =0.9, =0.999, image size is set to 384×384 pixels; image block size is uniformly set to 16×16, and the important parameter MLP_Ratio in MGViT is set to 6.
2. The multi-view image classification method based on a global filtering module according to claim 1, characterized in that, The training set and the test set are divided according to the proportion of different types of data labels in the image data sample set.
3. The multi-view image classification method based on a global filtering module according to claim 1, characterized in that, In step S43, the MGViT network adopts The cross-entropy loss function calculates the loss between the predicted label and the true label, where the predicted label is... and real labels The cross-entropy loss between them is shown in the following formula: 。