Image classification method based on convolutional neural network-transformer hybrid architecture

Abstract

The application discloses an image classification method based on a convolutional neural network-converter hybrid architecture. The method comprises: using a pre-trained convolutional neural network to extract features of an image and generate an attention mask indicating a target position contour, the attention mask representing the probability of a corresponding element belonging to a target category; and using the attention mask as prior information to guide a converter network to focus on a recognizable area to determine the category of the image, wherein the converter network is trained with a set loss function as an optimization target. The application can automatically find a target recognizable area and learn the features of the area for region classification, thereby improving the accuracy of fine-grained image classification.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image analysis, and more particularly, to an image classification method based on a convolutional neural network-transformer hybrid architecture. BACKGROUND

[0002] Fine-grained image classification aims to distinguish different sub-classes under a basic class, such as distinguishing breeds of birds, cats, dogs, etc., or distinguishing models of cars, airplanes, etc. These samples to be classified belong to the same large class, so there is great similarity between different classes. However, due to factors such as posture, lighting, angle, etc., there is great difference between different samples of the same class. These two factors make the fine-grained classification problem much more difficult than the traditional classification problem. One of the keys to solving this problem is to find small feature regions that can be distinguished, such as the beak, head, wings, or tail of a bird.

[0003] Fine-grained image classification is one of the important tasks in the field of computer vision, and the development of modern intelligent transportation, industry, and Internet applications also requires fine-grained classification, such as distinguishing vehicle models, real-time plant or bird species queries, etc. Current fine-grained classification methods can be roughly divided into the following two categories.

[0004] The first category is the localization-learning method based on local features. This method first obtains the location of the key region for identification, and then learns the deep features of these regions. Early work in this category usually uses additional manual annotation information to assist in locating the discriminative region, such as bounding boxes indicating the target position and part annotations, etc. However, since manual annotation information requires a lot of manpower, the current mainstream method is usually based on attention mechanism, which can automatically find the discriminative region. This method uses a weakly supervised method to complete the training of the network, only requiring the class label of the picture, saving manpower.

[0005] The second category is the method based on high-order feature encoding. This method aims to obtain better image feature description by mining higher-order information. For example, the classic bilinear model obtains higher-order image features by performing inner product on the output features of two CNN branches, achieving better classification accuracy.

[0006] The localization-learning method has better interpretability, which conforms to human intuitive cognition, but increases the complexity of the model. The high-order feature encoding method has poor interpretability, and to obtain higher high-order information capacity, the model complexity also increases. Most current fine-grained image classification methods use the localization-learning method based on local features.

[0007] Patent application CN1114611181A discloses a method and device for fine-grained classification of vehicles. The specific implementation scheme is: first, extract the basic feature map from the residual network, and segment and aggregate it using a predefined region size, thereby constructing a global structure graph; the global structure graph is guided by a graph convolution network to generate a local attention map and a one-dimensional global feature map; the local attention map and the global feature map are multiplied by the pixel and then input into the classifier for classification.

[0008] Patent application CN113392875A discloses a method, system and device for fine-grained image classification. The specific implementation scheme is: first, extract the basic feature map from the convolutional neural network, add weight to the channel information of the basic feature map through the SENet module, input the basic feature map into the pyramid network, output multiple attention regions and sort them to train the network.

[0009] In summary, the current fine-grained image classification method is based on convolutional neural network. However, due to the local sliding window characteristics of convolutional neural network, it cannot capture global information and long-range dependencies between features well. SUMMARY

[0010] The purpose of the present application is to overcome the defects of the prior art, and to provide an image classification method based on a convolutional neural network-transformer hybrid architecture, which can automatically discover the target recognition area and learn the features of the area for classification. It is especially suitable for fine-grained image classification.

[0011] The technical solution of the present application provides an image classification method based on a convolutional neural network-transformer hybrid architecture. The method comprises the following steps:

[0012] Using a pre-trained convolutional neural network to extract the features of the image and generate an attention mask indicating the target position contour, the attention mask representing the probability that the corresponding element belongs to the target class;

[0013] Using the attention mask as prior information to guide the transformer network to focus on the recognition area to determine the image class, wherein the transformer network is trained with a set loss function as the optimization target.

[0014] Compared with the prior art, the present application has the advantages that the visual Transformer structure can directly model the global information of the image, capture the mutual correlation information between any regions of the image, and improve the network performance. At the same time, the pre-trained convolutional neural network is introduced as an attention module to extract the target position information in the image as prior knowledge embedded in the original image, which assists the Transformer in positioning the recognition area and learning the features, significantly improving the image classification accuracy.

[0015] Other features of the present application, its nature and advantages will become more apparent from the detailed description of exemplary embodiments of the application which follows, in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0016] The accompanying drawings incorporated in and forming a part of the specification illustrate embodiments of the application and, together with the description, serve to explain the principles of the application.

[0017] Figure 1 is a flowchart of an image classification method based on a convolutional neural network- transformer hybrid architecture according to an embodiment of the present application;

[0018] Figure 2 is a process schematic of an image classification method based on a convolutional neural network- transformer hybrid architecture according to an embodiment of the present application;

[0019] Figure 3 is a schematic diagram of a convolutional neural network- transformer hybrid architecture model according to an embodiment of the present application;

[0020] Figure 4 is a schematic diagram of a transformer encoder according to an embodiment of the present application. DETAILED DESCRIPTION

[0021] Various exemplary embodiments of the present application will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangements, numerical expressions, and numerical values of components and steps set forth in these embodiments are not limiting to the scope of the present application unless otherwise specifically stated.

[0022] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way limiting to the scope of the application or its applications or uses.

[0023] Techniques, methods, and apparatus known to those of ordinary skill in the relevant art can not be discussed in detail herein, but should be considered as part of the specification, where appropriate.

[0024] In all of the examples shown and discussed herein, any specific values should be interpreted as merely illustrative and not as a limitation. Thus, other examples of exemplary embodiments can have different values.

[0025] It should be noted that like numbers and letters refer to like items throughout the drawings, and that, once an item is defined in one drawing, it should not require further discussion in subsequent drawings.

[0026] Reference Figure 1As shown, in brief, the image classification method based on the convolutional neural network-Transformer hybrid architecture provided includes: step S110, extracting features of an image by using a pre-trained convolutional neural network and generating an attention mask indicating a target position contour, the attention mask representing a probability that a corresponding element belongs to a target category; and step S120, using the attention mask as prior information to guide a converter network to focus on a recognizable area to determine a category of the image, wherein the converter network is trained with a set loss function as an optimization target.

[0027] Specifically, taking fine-grained image classification as an example, combined with Figure 2 As shown, the image classification process based on the convolutional neural network-Transformer hybrid architecture provided mainly includes five links: 1) an original image is input into a pre-trained convolutional neural network, and a feature map extracted therefrom is processed to obtain an attention mask indicating a target approximate position contour; 2) the attention mask is embedded into the original image as prior knowledge information; 3) the embedded image is input into a Transformer network for feature extraction; 4) the Transformer network outputs a global feature vector of the image and inputs a loss module based on an ArcFace angle loss function; and 5) the loss module outputs a classification result. The network model of the present application follows basic deep learning training techniques in the training phase, that is, the model is trained by completing parameter update through back propagation on a data set, and the process requires a category label of the picture; the test / inference phase does not need to update the parameters, and the final output is the picture classification result.

[0028] Figure 3 It is a model principle diagram based on a convolutional neural network-Transformer hybrid architecture, mainly including a convolutional neural network attention module, a Transformer feature learning module and an ArcFace angle loss module, and embodiments of each module will be specifically introduced below.

[0029] (1) Convolutional neural network attention module

[0030] The module extracts features of an image by using a pre-trained convolutional neural network and generates an attention mask indicating a target position contour, and the attention mask will guide the Transformer network to focus on the target area as prior knowledge information, so as to find a recognizable area.

[0031] The type and specific structure of the convolutional neural network can be arbitrary, for example, VGG or residual network, etc., which only needs to match the input image size. In an embodiment, a VGG-16 network is used, and the size of the input image is 448x448. The feature map output from the conv5_3 layer of the network is represented as The feature maps are directly added in the channel dimension to obtain a single-channel intermediate feature map with a size of 14x14, and after normalization and Sigmoid operation, an attention mask is obtained. The mask can be resized to be consistent with the original image by bicubic interpolation method (bicubic interpolation), and all elements of the mask have a value between 0 and 1, indicating the attention weight of the corresponding pixel. The higher the weight, the greater the probability of belonging to the target. For example, the process of obtaining the attention mask is represented as:

[0032]

[0033] wherein M i represents the feature map of the i-th channel, Norm represents the normalization operation, for example, the 512-channel feature map is directly added to obtain a single feature map, and Norm is to normalize the elements on the feature map.

[0034] (2) Transformer feature learning module.

[0035] The module receives an image embedded with an attention mask for feature learning. The original RGB three-channel image is expanded to four channels, and the fourth channel is the scaled attention mask. In an embodiment, the backbone network of the Transformer feature learning module adopts ViT (Vision Transformer), which is a stack of multi-layer Transformer encoders. First, the input image tensor will be blocked and projected into a fixed-size one-dimensional feature vector. Specifically, assuming that the input image size is HxW, and the block size to be divided is PxP, there will be non-overlapping small blocks, each of which will be projected into a one-dimensional feature vector with a size of D. In order to facilitate classification and consider spatial relationships, an additional learnable vector for classification (aiming to find class information for each small block) and a learnable vector for position encoding (aiming to position encode each small block to maintain the spatial position information between the small blocks of the input image) will also be added. The final input encoder tensor is represented as:

[0036] z0=[X0;X1,X2,...,x N ]+x pos (2)

[0037] wherein x0, x pos represent the added classification vector and position encoding respectively, both of which are initialized as vectors of Gaussian distribution and set to be learnable and updated during the training process; x1 to x N represent one-dimensional feature vectors corresponding to each small block.

[0038] Referring to Figure 4As shown, the Transformer Encoder mainly performs multi-head self-attention (MSA) operations, the details of which can be found in the references ("Attention is all you need", Advances in neural information processing systems, 2017:5998-6008; Vaswani A, Shazeer N, Parmar N, et al.). The input image feature tensor is first normalized and subjected to MSA operations, and then subjected to a residual connection, that is, the initial tensor is added to the tensor after the operation to obtain an intermediate tensor; the intermediate tensor is also normalized, and then subjected to a multilayer perception (MLP) and residual connection to obtain the final output tensor, the process of which is given by equations (3) and (4). This is an encoder. A Transformer encoder is composed of multiple encoders stacked together. In one embodiment, the ViT network used consists of 12 encoder layers.

[0039] z′ l =MSA(LN(z) l-1 ))+z l-1 (3)

[0040] z l =MLP(LN(z′) l ))+z′ l (4)

[0041] Among them, z l-1 This represents the output of layer (l-1). Each layer is an encoder, which mainly consists of two modules: MSA and MLP. l This represents the intermediate feature vector of the l-layer encoder (located between MSA and MLP), where l = 1…L, and L represents the number of encoders.

[0042] In summary, the main processes of ViT include image block processing, image block embedding and location encoding, Transformer encoder and MLP classification processing.

[0043] (3) ArcFace corner loss module.

[0044] To achieve better classification results, in one embodiment of this invention, the ViT classification head is removed and replaced with an ArcFace (radian) loss module. The classification vector output by the last encoder of ViT (see x0 in Equation 2) is used as the global feature description of the image and input into the ArcFace angle loss module. The ArcFace angle loss module first has an n×D weight matrix, where n is the number of classes and D is the model dimension, which is the same size as the classification vector. Each D-dimensional vector of the weight matrix is ​​used as a feature vector of a class. During the testing / inference phase, the class of the image is determined based on the cosine distance between the classification vector and the class feature vector. During the training phase, a boundary angle m is added between the classification vector and the corresponding class vector to construct the loss function, as shown in Equation (5).

[0045]

[0046] Where N represents the number of images in a batch during training, assuming the i-th image belongs to the y-th image. i Categories This indicates that the image classification vector is related to the y-th element. i The angle θ between the category vectors j The angle between the classification vector and other category vectors is represented by s; s represents a scale value used to scale the cosine value between 0 and 1 to different intervals.

[0047] It should be noted that the attention module of the convolutional neural network only uses publicly available pre-trained network parameters. When training on fine-grained datasets, no parameter updates are required, which can save a lot of computing resources and time.

[0048] In summary, this invention innovatively introduces the Transformer structure into the fine-grained image classification problem, proposing a hybrid convolutional neural network-Transformer architecture model. This model uses a deep neural network to find key feature regions that can distinguish fine-grained categories and learns their features, thereby differentiating categories. This hybrid architecture model is flexible and can serve as a relatively general knowledge transfer framework. The convolutional neural network in the hybrid architecture model can use only publicly available pre-trained parameters without additional training, thus saving computational resources and training time. Experimental results demonstrate that the proposed hybrid architecture model exhibits excellent performance on multiple fine-grained image datasets.

[0049] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.

[0050] A computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.

[0051] Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0052] Computer readable program instructions for carrying out operations of the present application can be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate array (FPGA), or programmable logic array (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present application.

[0053] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0054] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0055] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0056] The flow diagrams and the block diagrams in the drawings are presented to illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flow diagrams and the block diagrams can represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logic functions. In some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks can sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flow diagrams, and combinations thereof, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and

[0057] Embodiments of the application have been described above. The description is illustrative of the embodiments of the application and is not meant to be limiting. Numerous modifications and variations are possible in light of the above teachings without departing from the scope and spirit of the described embodiments of the application. No limitation is intended to the details of construction or design except as described in the claims.

Claims

1. An image classification method based on a convolutional neural network-transformer hybrid architecture, comprising the following steps: extracting features of an image using a pre-trained convolutional neural network and generating an attention mask indicating the contour of a target position, the attention mask representing the probability that the corresponding element belongs to the target class; using the attention mask as prior information to guide a transformer network to focus on the discriminative region to determine the image class, wherein the transformer network is trained with a set loss function as the optimization objective; wherein the attention mask is obtained using the following formula: wherein M i represents the feature map of the i-th channel, and Norm represents a normalization operation; wherein, for the transformer network, it contains multiple layers of encoders, the image tensor input at first will be blocked and projected into a fixed size one-dimensional feature vector, and an additional learnable vector for classification and a learnable vector for position encoding are added, and the tensor input to the last encoder is represented as: z0= [x0; x1, x2,..., xN] + x pos wherein x0, x pos respectively represent the added classification vector and position encoding, both of which are initialized as vectors of Gaussian distribution and set to be learnable; x1 to x N represent one-dimensional feature vectors corresponding to each small block; wherein, the classification head of the transformer network is an angle loss module based on radian loss, and the classification vector output by the last encoder of the transformer network is input into the angle loss module as the global feature description of the image. wherein, the angle loss module is first provided with an n x D weight matrix, where n is the number of classes and D is the model dimension, each D-dimensional vector of the weight matrix is used as a feature vector of a class, and the cosine distance between the classification vector and the class feature vector is used to determine the class to which the image belongs.

2. The method of claim 1, wherein, The loss function of the transformer network is set as: wherein, N denotes the number of images in a batch in training, and it is assumed that the ith image belongs to the y i th class, denotes the angle between the ith image classification vector and the y i th class vector, and j denotes the angle between the classification vector and other class vectors, s denotes a scale value for scaling the cosine value between 0-1 to different intervals, and m denotes a boundary angle.

3. The method of claim 1, wherein, The convolutional neural network uses VGG or residual network.

4. A computer readable storage medium having stored thereon a computer program, wherein, The program is executed by a processor to implement the steps of the method according to any one of claims 1 to 3.

5. A computer device comprising a memory and a processor, having stored on the memory a computer program capable of running on the processor, characterized in that, The processor executes the program to implement the steps of the method according to any one of claims 1 to 3.

Images