Image classification method and device, electronic equipment and storage medium
By combining image classification methods with global and local features, the problem that existing biological classification models cannot be applied to traditional and fine-grained biological classification tasks at the same time is solved, and a high-accurate biological classification is achieved, which is suitable for multiple biological classification tasks.
Patent Information
- Application Number
- CN202411043986.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-31
- Publication Date
- 2024-11-26
AI Technical Summary
The existing biological classification model cannot be applied to both traditional biological classification tasks and fine-grained biological classification tasks, and the classification accuracy is low.
By combining the image classification method of global features and local features, the feature representation is encoded using the position weight matrix, redundant features are removed, global and local features are merged to generate classified features, and input a preset classifier for classification. The method includes feature extraction, global feature generation, local feature generation and classified feature merging modules, and can extract multi-level and multi-scale feature information from the image.
It is achieved that without changing the network structure, it can not only improve the accuracy of traditional biological classification tasks, but also be applicable to fine-grained biological classification tasks, significantly improve the classification accuracy and maintain the efficiency of training time.
Smart Images

Figure CN119027724A_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of artificial intelligence technology, and specifically relates to an image classification method, device, electronic device and storage medium. Background Art
[0002] As people pay more and more attention to environmental protection and their awareness of nature conservation continues to improve, the importance of protecting species diversity and promoting the harmonious development of man and nature has become more and more prominent. Biological classification provides a basis and method for us to understand and protect the biodiversity on Earth, which has important practical significance and scientific value.
[0003] In the prior art, machine models are usually used to classify images. However, existing biological classification models usually have low accuracy and cannot be applied to both traditional classification tasks and fine-grained classification tasks. Summary of the invention
[0004] In view of this, the purpose of the present application is to provide an image classification method, device, electronic device and storage medium, which are applicable to both traditional biological classification tasks and fine-grained biological classification tasks.
[0005] In a first aspect, an embodiment of the present application provides an image classification method, the method comprising: extracting features from an image to obtain a feature representation of the image; encoding the feature representation according to a position weight matrix to obtain a global feature, the position weight matrix comprising position weights of all elements in the feature representation, the position weights being obtained according to the coordinates of the elements in the feature representation; removing redundant features in the feature representation to obtain local features; merging the global features and the local features to obtain classification features; and inputting the classification features into a preset classifier to obtain a preset classification result.
[0006] The above-mentioned image classification method combines global features and local features for biological classification. It is not only applicable to traditional biological classification tasks and fine-grained biological classification tasks, but also can improve the classification accuracy.
[0007] In an optional embodiment of the present application, the feature extraction of the image to obtain the feature representation of the image includes: dividing the image into multiple first sub-blocks according to a first layout; calculating the representation of each first sub-block according to a self-attention mechanism to obtain the first representation of the image; dividing the first representation into multiple second sub-blocks according to a second layout, the first layout and the second layout both include multiple windows, the windows in the second layout are obtained by translating the windows in the first layout, and the windows are used to indicate the division method of the sub-blocks; calculating the representation of each second sub-block according to the self-attention mechanism to obtain the feature representation.
[0008] In the above embodiment, multi-level and multi-scale features can be extracted from the input data through the two-window mechanism, and comprehensive and high-quality feature information can be extracted from the input data.
[0009] In an optional embodiment of the present application, the method also includes: using a preset pooling kernel to encode the feature representation along the horizontal direction and the vertical direction respectively to obtain horizontal direction perception features and vertical direction perception features; merging the horizontal direction perception features and the vertical direction perception to obtain merged features; according to the length and height of the feature representation, dividing the merged features into a horizontal tensor and a vertical tensor; according to the number of channels of the feature representation, reducing the dimensions of the channels of the horizontal tensor and the vertical tensor to generate the position weight matrix.
[0010] The above embodiment generates a position weight matrix by capturing the dependencies in the horizontal and vertical directions in the feature graph, and the position weight matrix effectively enhances the global representation capability of the feature.
[0011] In an optional embodiment of the present application, removing redundant features in the feature representation to obtain local features includes: dividing the feature representation into a first feature and a second feature, the first feature having more information than the second feature; cross-reconstructing the first feature and the second feature to obtain refined features; merging deep features and shallow features in the refined features to obtain the local features, the shallow features containing more pixel information than the deep features.
[0012] In the above embodiment, feature reconstruction is performed to optimize the expression and reorganization of features and solve the problem of feature redundancy, thereby improving the local and global information integration capabilities of features.
[0013] In an optional embodiment of the present application, dividing the feature representation into a first feature and a second feature includes: multiplying the feature representation with an information weight and a non-information weight, respectively, to generate the first feature and the second feature, the information weight is obtained based on a preset first hyperparameter and a weight distribution, the non-information weight is obtained based on a preset second hyperparameter and the weight distribution, and the weight distribution is obtained by performing a feature evaluation on the feature representation.
[0014] The above embodiment provides a method for extracting information-rich features, which can quickly extract information-rich feature representations and information-less feature representations from feature representations.
[0015] In an optional embodiment of the present application, the method further includes: dividing the refined features into upper features and lower features along the channel direction according to a third hyperparameter; convolving the upper features with a preset convolution kernel to generate the deep features; and performing feature fusion on the lower features to generate the shallow features.
[0016] The above embodiment provides a method for extracting deep features, which can quickly and easily extract deep features and shallow features from feature representation.
[0017] In an optional embodiment of the present application, the merging of the global feature and the local feature to obtain the classification feature includes: multiplying the global feature, the local feature and the feature representation with adjustable coefficients respectively to obtain corresponding adjustment features; and merging all adjustable features to obtain the classification feature.
[0018] In the above embodiment, the global feature, the local feature and the feature representation are combined using an adjustable coefficient, which enhances the representation flexibility of the output feature and can better capture and represent features at different levels.
[0019] In a second aspect, an embodiment of the present application provides an image classification device, comprising: a feature extraction module, used to extract features from an image to obtain a feature representation of the image; a global feature generation module, used to encode the feature representation according to a position weight matrix to obtain a global feature, wherein the position weight matrix includes the position weights of all elements in the feature representation, and the position weights are obtained according to the coordinates of the elements in the feature representation; a local feature generation module, used to remove redundant features in the feature representation to obtain local features; a merging module, used to merge the global features and the local features to obtain classification features; and a classification module, used to input the classification features into a preset classifier to obtain a preset classification result.
[0020] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a memory and a processor, the processor being connected to the memory; the memory being used to store programs; and the processor being used to call the programs stored in the memory to execute any of the methods described in the first aspect.
[0021] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method described in any one of the first aspects is executed.
[0022] Other features and advantages of the present application will be described in the following description. The purpose and other advantages of the present application can be realized and obtained through the structures specifically pointed out in the written description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. The above and other purposes, features and advantages of the present application will be more clear as shown in the drawings.
[0024] Figure 1 A schematic diagram of a process flow of an image classification method provided in an embodiment of the present application is shown;
[0025] Figure 2 A schematic block diagram of an image classification device provided in an embodiment of the present application is shown;
[0026] Figure 3 A schematic block diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION
[0027] Hereinafter, embodiments of the present application are described in detail with reference to the accompanying drawings. It should be noted that although the same elements are shown in different drawings, they will be represented by the same reference numerals. In the following description, specific details such as detailed configuration and components are provided to only help fully understand the embodiments of the present application. Therefore, it should be clear to those skilled in the art that various changes and modifications to the embodiments described herein can be made without departing from the scope of the present application. In addition, for clarity and conciseness, the description of well-known functions and configurations is omitted. The terms described below are defined in consideration of the functions in the present application, and may be different according to the user, the user's intention or custom. Therefore, the definition of the term should be determined based on the content throughout this specification.
[0028] The present application may have various modifications and various embodiments, in which the embodiments are described in detail below with reference to the accompanying drawings. However, it should be understood that the present application is not limited to the embodiments, but includes all modifications, equivalents and substitutes within the scope of the present application.
[0029] The terms used herein are only used to describe various embodiments of the present application and are not intended to limit the present application. Unless the context clearly indicates otherwise, the singular form is intended to include the plural form. In the present application, it should be understood that the term "including" or "having" indicates the presence of features, quantities, steps, operations, structural elements, parts or combinations thereof, and does not exclude the presence of one or more other features, quantities, steps, operations, structural elements, parts or combinations thereof, or the possibility of adding one or more other features, quantities, steps, operations, structural elements, parts or combinations thereof.
[0030] Unless defined differently, all terms used herein have the same meaning as understood by a person skilled in the art to which the present application belongs. Unless explicitly defined in the present application, terms (such as those defined in a general dictionary) should be interpreted as having the same meaning as the contextual meaning in the relevant art, and should not be interpreted as having an idealized or overly formal meaning.
[0031] The electronic device according to one embodiment may be one of various types of electronic devices. The electronic device may include, for example, a portable communication device (e.g., a smart phone), a computer, a portable multimedia device, a portable medical device, a camera, a wearable device, or a household appliance. According to one embodiment disclosed, the electronic device is not limited to those described above.
[0032] The terms used in this application are not intended to limit the application, but are intended to include various changes, equivalents or alternatives of corresponding embodiments. Regarding the description of the accompanying drawings, similar reference numerals may be used to represent similar elements or related elements. Unless the relevant context clearly indicates otherwise, the singular form of the noun corresponding to the item may include one or more things. As used herein, each of the phrases such as "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C" and "at least one of A, B or C" may include all possible combinations of the items listed together in the corresponding one in the phrase. As used herein, terms such as "the 1st", "the 2nd", "the first" and "the second" may be used to distinguish the corresponding component from another component, and are not intended to limit the component in other aspects (e.g., importance or order). It is intended that if an element (e.g., a first element) is referred to as being “coupled with,” “coupled to,” “connected to,” or “connected to” another element (e.g., a second element), with or without the term “operably” or “communicatively,” it indicates that the element may be coupled with the other element directly (e.g., wired), wirelessly, or via a third element.
[0033] As used herein, the term "module" may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with other terms (e.g., "logic," "logic block," "component," and "circuit"). A module may be a single integrated component or its smallest unit or component suitable for performing one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application specific integrated circuit (ASIC).
[0034] To facilitate understanding, the relevant terms and concepts involved in the embodiments of the present application are first introduced below.
[0035] In biological classification, traditional classification tasks refer to tasks for classifying organisms with highly distinguishable features, such as distinguishing between tigers and birds. Fine-grained tasks refer to tasks for classifying organisms with very similar features, such as classifying between birds and fish.
[0036] In the prior art, models applied to traditional tasks perform poorly on fine-grained tasks, and similarly, models applied to fine-grained tasks perform poorly on traditional tasks. The inventors found that in the existing classification models, the models for traditional tasks mainly focus on global features, while the models for fine-grained tasks mainly focus on local features. This difference makes it impossible for the model to perform two tasks at the same time. The model must change the network structure before it can perform another task.
[0037] Based on the above problems, the present application provides an image classification method, which combines global features and local features for biological classification, which can not only be applied to traditional biological classification tasks and fine-grained biological classification tasks, but also improve the classification accuracy. The above image classification method can be applied to electronic devices with computing functions, and the above electronic devices can be personal terminal devices such as mobile phones, tablet computers, wearable devices, smart screens, augmented reality (AR) and virtual reality (VR) devices, laptops, ultramobile personal computers (UMPC), netbooks, personal digital assistants (PDA), artificial intelligence (AI) terminals and other devices with display functions, and can also be large electronic devices such as servers, server clusters, cloud computing resources, etc. In the embodiments of the present application, there is no special restriction on the specific form of the electronic device.
[0038] like Figure 1 As shown, Figure 1 FIG. 1 is a flow chart of an image classification method provided by an embodiment of the present application. Figure 1 As shown, the method includes steps 110 to 150:
[0039] Step 110: extract features from the image to obtain feature representation of the image.
[0040] Extract features from the input image and establish a feature representation of the input image. The input image refers to an image including at least one creature, such as an image including a bird, an image including a fish, and an image including a tiger. The image can be processed using a convolutional neural network or a residual neural network to generate a feature representation corresponding to the image.
[0041] In one embodiment of the present application, the feature extraction of the image to obtain the feature representation of the image includes: dividing the image into multiple first sub-blocks according to a first layout; calculating the representation of each first sub-block according to a self-attention mechanism to obtain the first representation of the image; dividing the first representation into multiple second sub-blocks according to a second layout, the first layout and the second layout both include multiple windows, the windows in the second layout are obtained by translating the windows in the first layout, and the windows are used to indicate the division method of the sub-blocks; calculating the representation of each second sub-block according to the self-attention mechanism to obtain the feature representation.
[0042] First, the input feature map can be an image for biological classification or information such as tokens after processing. For example, before extracting features from the input image, a PatchEmbedding operation can be performed. Each H×W×3 image is processed into Each patch is flattened into a 48-dimensional token. These tokens can be used to further obtain feature representation.
[0043] The input feature map is divided according to a first layout, and the first layout includes a plurality of preset fixed-size windows, for example, the first layout is set to a 7x7 window. After the feature map is divided according to the first layout, each window includes an independent sub-block of the feature map, and the sub-block obtained by dividing the first layout is referred to as the first sub-block in this application.
[0044] Multi-head Self-Attention (MSA) is performed on the features in each first sub-block to obtain the feature representation in each window. All window features are the first representation of the feature map. The feature representations in all windows can be spliced and reorganized to form the first representation.
[0045] The computational complexity of self-attention in this step is:
[0046] Ω(W-MSA)=4hwC 2 +2(M) 2 HkDJ
[0047] Although the window-based self-attention module reduces the computational complexity from quadratic to linear, the lack of cross-window communication and connection will inevitably limit its modeling and representation capabilities, so cross-window communication needs to be increased. In order to increase cross-window connections and communication while maintaining the computational efficiency of non-overlapping windows, the shift window method can be used.
[0048] The first representation is divided into a plurality of second sub-blocks using a second layout, wherein the second layout is obtained by performing a translation operation on the window in the first layout, for example, the window in the first layout is translated rightward and downward by half of the window size to obtain the second layout. The window in the second layout partially overlaps with the original window in the first layout.
[0049] Multi-head Self-Attention (MSA) is performed on the features in each second sub-block to generate the feature representation in each second sub-block. The features in all second sub-blocks are concatenated and reorganized to form the second representation.
[0050] For ease of description, the steps of using the unshifted window to perform sub-block division and calculate the feature representation can be expressed as W-MSA. The steps of performing sub-block division based on window shift and calculating the feature representation of self-attention can be expressed as SW-MAS.
[0051] In some embodiments of the present application, the above W-MSA and SW-MAS can be used alternately:
[0052]
[0053] in, represents the output characteristics of the lth W-MSA or SW-MAS; z l Represents the output features of the MLP (Multilayer Perceptron) in the lth block.
[0054] If W-MSA and SW-MAS are performed only once, the second representation at this time is the characteristic representation.
[0055] In some embodiments of the present application, W-MSA and SW-MAS may be used alternately for multiple times, thereby performing multiple feature extractions. In this case, the second representation needs to be further processed to obtain a feature representation.
[0056] In the above embodiment, the self-attention calculation is performed only in a small window, rather than on the entire feature map, through window division, which effectively reduces the amount of calculation and effectively captures local features. At the same time, the window translation mechanism enables features to interact across windows and capture global feature relationships. Multi-level and multi-scale features can be extracted from the input data, and comprehensive and high-quality feature information can be extracted from the input data.
[0057] Step 120, encoding the feature representation according to a position weight matrix to obtain a global feature, wherein the position weight matrix includes position weights of all elements in the feature representation, and the position weights are obtained according to the coordinates of the elements in the feature representation.
[0058] The feature representation is in the form of [x 1 ,x 2 ,…,x C ]∈R C×H×W , where each element has corresponding coordinates in the three dimensions of C, H, and W. The coordinates of one or more of the three dimensions are the coordinates of the element in the feature representation. The position weight of the element can be obtained by calculating the weight based on the coordinate information of the element. The position weights of all elements in the feature representation can form a matrix, namely the position weight matrix.
[0059] The feature representation is encoded based on the position weight matrix, so that the feature representation is enhanced through spatial information such as channel relationships, long-term dependencies, etc. to obtain global features.
[0060] In one embodiment of the present application, the method also includes: using a preset pooling kernel to encode the feature representation along the horizontal direction and the vertical direction respectively to obtain horizontal direction perception features and vertical direction perception features; merging the horizontal direction perception features and the vertical direction perception to obtain merged features; according to the length and height of the feature representation, dividing the merged features into a horizontal tensor and a vertical tensor; according to the number of channels of the feature representation, reducing the dimensions of the channels of the horizontal tensor and the vertical tensor to generate the position weight matrix.
[0061] Channel relationships and long-range dependencies are encoded through precise location information, which is divided into two steps: coordinate information embedding and coordinate attention generation.
[0062] Assuming the feature is represented as x′, we first use pooling kernels of size (H, 1) and (1, 2) to encode each channel along the horizontal and vertical coordinate directions. Therefore, the output of the Cth channel with a height of h is expressed as follows.
[0063]
[0064] The output of the cth channel with width w is expressed as follows.
[0065]
[0066] It aggregates features along two spatial directions, horizontal and vertical, and returns a pair of direction-aware attention maps. It preserves long-range dependencies in the horizontal (X) and vertical (Y) directions for channel attention.
[0067] The two direction-aware attention maps generated previously are concatenated and then transformed using a shared 1×1 convolution F 1 ,as follows.
[0068] f=δ(F 1 ([z h ,z w ]))
[0069] The generated f∈R C / r×H It is an intermediate feature map obtained by extracting features of spatial information in the horizontal and vertical directions, where r represents the downsampling ratio.
[0070] Split f into f along the spatial dimension h ∈R C / r×H With f w ∈R C / r×W These two separate tensors are then used to h With F w Two 1×1 convolutions transform f h With f w The number of channels is transformed to be consistent with the input x′, and we get:
[0071] g h =σ(F h (f h ))
[0072] g w =σ(F w (f w ))
[0073] The above convolution achieves attention to the elements within the feature representation in both horizontal and vertical directions, and the resulting attention map can well reflect whether the object of interest exists in the corresponding rows and columns.
[0074] Finally, let g h With g w As the weight of attention, the final output is obtained:
[0075]
[0076] At this time, not only the horizontal and vertical attention is completed, but also the function of channel attention is played.
[0077] The above embodiment introduces a coordinate attention mechanism to enhance the coordinate attention of features, and generates a position weight matrix by capturing the long-distance dependencies in the horizontal and vertical directions in the feature map. The position weight matrix effectively enhances the global representation capability of the features.
[0078] Step 130, removing redundant features in the feature representation to obtain local features.
[0079] Some features in the feature representation have little effect on the final classification and may also affect the use of other features. These features are called redundant features. Therefore, some prominent local features can be selected from the feature representation to remove redundant features. For example, the weight of the convolution kernel is dynamically adjusted according to different areas of the input feature map, or multiple different convolution kernels are used to capture local features more accurately, thereby improving the flexibility and effect of feature extraction.
[0080] There is no order in which global features and local features are extracted in this application. Local features can be obtained using feature representation first, and then global features can be obtained based on feature representation. Global features and local features can also be extracted at the same time. This application does not limit the order of extracting global features and local features.
[0081] In one embodiment of the present application, removing redundant features in the feature representation to obtain local features includes: dividing the feature representation into a first feature and a second feature, the first feature having more information than the second feature; cross-reconstructing the first feature and the second feature to obtain refined features; merging deep features and shallow features in the refined features to obtain the local features, the shallow features containing more pixel information than the deep features.
[0082] Evaluate the information content of different parts in the feature representation, for example, using normalization, local entropy, local information interaction and other methods to evaluate the information content of different parts. Select the part with more information as the first feature, and the other parts as the second feature.
[0083] Furthermore, dividing the feature representation into a first feature and a second feature includes: multiplying the feature representation with an information weight and a non-information weight, respectively, to generate the first feature and the second feature, the information weight being obtained based on a preset first hyperparameter and a weight distribution, the non-information weight being obtained based on a preset second hyperparameter and the weight distribution, and the weight distribution being obtained by performing a feature evaluation on the feature representation.
[0084] The information of different feature maps is evaluated using the scaling factors in the group normalization (GN) layer:
[0085]
[0086] Given an intermediate feature map X∈R(N×C×H×W), to standardize the input feature representation x′, subtract the mean μ and divide by the standard deviation σ. Where ε is a small constant for division stability, and γ and β are trainable affine transformations. The trainable parameter γ in the GN layer is used as a method to measure the spatial pixel difference in each batch and channel. The larger γ is, the more changes there are in the spatial pixels, which represents the importance of each mapping.
[0087]
[0088] Will be W γ The weights of the re-weighted feature map are mapped to the range (0, 1) by the sigmoid function and gated by the threshold. In this application, the weights above the threshold are set to 1, and the information weight W is obtained. 1 , set it to 0, and get the non-information weight W 2 , where the threshold is set to 0.5:
[0089] W = Gate (Sigmoid (GN (x′)))
[0090] The input feature x′ is respectively 1 and w 2 Multiply. Produce two weighted features, which are information-rich and less information in Informative and expressive spatial content, while With little or no information, that is is considered redundant.
[0091] The above embodiment provides a method for extracting information-rich features, which can quickly extract information-rich feature representations and information-less feature representations from feature representations.
[0092] In order to further reduce spatial redundancy, a reconstruction operation is performed. Features with less information Add them together to generate more informative features and save space. However, the above operation does not simply add the two parts directly, but uses a cross-reconstruction operation to fully combine the two weighted different information features while enhancing the information flow between them. Then connect the cross-reconstructed features X W1 and X W2 To obtain the spatially refined feature map X W The whole process of reconstruction in the above process can be expressed as:
[0093]
[0094] So far, the above operations not only separate the features with rich information from the features with less information, but also reconstruct them to achieve the purpose of enhancing representative features and reducing redundancy in spatial dimensions. However, even so, the feature map X after spatial refinement W There is still redundancy in the channel dimension.
[0095] Obtain deep features and shallow features in the refined features. Shallow features refer to features extracted by shallow networks that are closer to the input and contain more pixel information. They are some fine-grained information, such as some color, texture, edge, and angular information of the image. Deep features refer to features extracted by deep networks that are closer to the output and contain some coarse-grained information, including more abstract information such as semantic information.
[0096] The deep features and shallow features are combined to obtain local features.
[0097] In one embodiment of the present application, the method also includes: based on a third hyperparameter, dividing the refined features into upper features and lower features along the channel direction; using a preset efficient convolution kernel to convolve the upper features to generate the deep features; performing feature fusion on the lower features to generate the shallow features; merging the deep features and the shallow features in the refined features to obtain the local features.
[0098] For a given spatial fine feature X W , first split its total C channels into two parts, αC channels and (1-α)C channels, where α is a hyperparameter and 0≤α≤1. Then the number of channels of the two sets of features is compressed using a 1×1 convolution kernel to improve computational efficiency. The compression ratio r is also introduced to control the feature channels to balance the computational cost of CRU. After the split operation, X W Divided into two parts, X up With X low .
[0099] X up The high-performance convolution operation GWC and PWC are used to replace the ordinary k×k convolution to extract high-level representative information while reducing the computational cost. Although GWC reduces the parameters and computational complexity, it also cuts off the information flow between channel groups. PWC helps information flow between functional channels while compensating for the information loss. up Perform k×k GWC and 1×1 PWC operations on the network, and then aggregate and merge the outputs to obtain a representative feature map Y 1 In summary, the up-conversion stage can be expressed as:
[0100]
[0101] in, are the learnable matrices of GWC and PWC respectively, With Y 1 ∈R c×h×w They are the feature maps of the upper input and output respectively.
[0102] X low is input to the lower or lower layer transformation stage. Here, a 1×1 PWC operation is used to generate a feature map with shallow hidden details. In addition, a residual is used, that is, X is reused low The features of can obtain more feature maps without paying extra cost. Finally, connect the generated and reused features to get Y 2 :
[0103]
[0104] in, is the learnable matrix of PWC, With Y 2 ∈R c×h×w They are the feature maps of the lower layer input and lower layer output respectively.
[0105] The simplified SKNet method is used to adaptively merge the output features Y from the upper and lower conversion stages. 1 and Y 2 First, the Avg pooling method is used to calculate the channel statistics S m ∈R c×1×1 :
[0106]
[0107] Will be Y 1 With Y 2 The obtained S 1 With S 2 Superimposed together, the channel soft attention operation generates the feature importance vector β 1 ,β 2 ∈R C ,as follows:
[0108] β 1 +β 2 =1
[0109] Upper feature Y 1 With the following feature Y 2 In the weight β 1 With β 2 Under the influence of , the local feature x is obtained by merging in the channel direction.2
[0110] x 2 =β 1 Y 1 +β 2 Y 2 #
[0111] The above embodiment provides a method for extracting deep features, which can quickly and easily extract deep features and shallow features from feature representation. In addition, in the above embodiment, feature reconstruction is performed to optimize the expression and reorganization of features and solve the problem of feature redundancy, thereby improving the local and global information integration capabilities of features.
[0112] Step 140: merge the global feature and the local feature to obtain a classification feature.
[0113] The generated global features and local features are combined to obtain classification features, which can be input into a classifier for biological classification. The classifier is pre-designed and trained. For example, a Softmax layer can be pre-designed to classify images, and the Softmax layer can be used as a classifier after training.
[0114] In one embodiment of the present application, the merging of the global feature and the local feature to obtain the classification feature includes: multiplying the global feature, the local feature and the feature representation with adjustable coefficients respectively to obtain corresponding adjustment features; and merging all adjustable features to obtain the classification feature.
[0115] The obtained global feature x 1 , local features x 2 It is combined with the feature representation x′ through a residual connection to obtain the complete feature Y:
[0116] Y=Ax 1 +Bx 2 +Cx′
[0117] Among them, A, B, and C are adjustable coefficients.
[0118] In the above embodiment, the global feature, the local feature and the feature representation are combined using an adjustable coefficient, which enhances the representation flexibility of the output feature and can better capture and represent features at different levels.
[0119] Step 150: input the classification feature into a preset classifier to obtain a preset classification result.
[0120] The processed complete feature Y is input into the subsequent network to obtain the classification result. The classification result represents the prediction of the organism in the image. For example, if the classification result of the input image is a bird, it means that the classifier predicts that the organism in the image is a bird.
[0121] For example, using a linear layer as a classifier can be expressed as:
[0122] Output=Linear(AvgPool(LN(Y)))
[0123] The above-mentioned image classification method combines global features and local features for biological classification. It is not only applicable to traditional biological classification tasks and fine-grained biological classification tasks, but also can improve the classification accuracy.
[0124] After a series of ablation experiments, the above method can achieve a wide range of biological image classification, whether it is for traditional classification networks with obvious differences such as cats, dogs, birds, etc., or for similar types of organisms such as distinguishing different butterflies, stable results have been obtained, and compared with related methods, this method has been significantly improved.
[0125] The deep neural network constructed by the above method can perform traditional biological classification tasks and fine-grained biological classification tasks without changing the network structure. At the same time, by comparing with related networks, this network shows significant performance improvement in both traditional tasks and fine-grained tasks. Compared with related networks, this network does not increase too much complexity, and the training time is almost the same as the baseline.
[0126] like Figure 2 As shown, Figure 2 An image classification device 200 provided in an embodiment of the present application is shown, wherein the image classification device 200 comprises:
[0127] A feature extraction module 210 is used to extract features from an image to obtain a feature representation of the image;
[0128] A global feature generation module 220, configured to encode the feature representation according to a position weight matrix to obtain a global feature, wherein the position weight matrix includes position weights of all elements in the feature representation, and the position weights are obtained according to coordinates of the elements in the feature representation;
[0129] A local feature generation module 230, used to remove redundant features in the feature representation to obtain local features;
[0130] A merging module 240, configured to merge the global feature and the local feature to obtain a classification feature;
[0131] The classification module 250 is used to input the classification features into a preset classifier to obtain a preset classification result.
[0132] The present application does not restrict the order of extracting global features and local features. Therefore, the relationship between the global feature generation module 220 and the local feature generation module 230 is diverse. For example, the global feature generation module 220 and the local feature generation module 230 can be in a parallel relationship, and the feature extraction module 210 sends the feature representation to the global feature generation module 220 and the local feature generation module 230 respectively. The global feature generation module 220 sends the global feature to the merging module 240, and the local feature generation module 230 sends the local feature to the merging module 240, and the merging module 240 merges the global feature and the local feature.
[0133] In one embodiment of the present application, the feature extraction module 210 is specifically used to divide the image into multiple first sub-blocks according to a first layout; calculate the representation of each first sub-block according to a self-attention mechanism to obtain a first representation of the image; divide the first representation into multiple second sub-blocks according to a second layout, the first layout and the second layout both include multiple windows, the windows in the second layout are obtained by translating the windows in the first layout, and the windows are used to indicate the division method of the sub-blocks; calculate the representation of each second sub-block according to the self-attention mechanism to obtain the feature representation.
[0134] In one embodiment of the present application, the image classification device 200 also includes: a position weight matrix generation module, which is used to use a preset pooling kernel to encode the feature representation along the horizontal direction and the vertical direction respectively to obtain horizontal direction perception features and vertical direction perception features; merge the horizontal direction perception features and the vertical direction perception to obtain merged features; according to the length and height of the feature representation, the merged features are divided into a horizontal tensor and a vertical tensor; according to the number of channels of the feature representation, the channels of the horizontal tensor and the vertical tensor are reduced in dimension to generate the position weight matrix.
[0135] In one embodiment of the present application, the local feature generation module 230 is specifically used to divide the feature representation into a first feature and a second feature, the first feature containing more information than the second feature; cross-reconstructing the first feature and the second feature to obtain a refined feature; merging the deep features and shallow features in the refined feature to obtain the local feature, the shallow feature containing more pixel information than the deep feature.
[0136] In one embodiment of the present application, the local feature generation module 230 is also used to multiply the feature representation with an information weight and a non-information weight, respectively, to generate the first feature and the second feature, the information weight is obtained based on a preset first hyperparameter and a weight distribution, the non-information weight is obtained based on a preset second hyperparameter and the weight distribution, and the weight distribution is obtained by performing a feature evaluation on the feature representation.
[0137] In one embodiment of the present application, the local feature generation module 230 is also used to divide the refined features into upper features and lower features along the channel direction according to a third hyperparameter; use a preset convolution kernel to convolve the upper features to generate the deep features; and perform feature fusion on the lower features to generate the shallow features.
[0138] In one embodiment of the present application, the merging module 240 is specifically used to multiply the global feature, the local feature and the feature representation by the adjustable coefficient respectively to obtain the corresponding adjustment feature; and merge all the adjustable features to obtain the classification feature.
[0139] The image classification device provided in the embodiment of the present application has the same implementation principle and technical effects as those of the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference may be made to the corresponding contents in the aforementioned method embodiment.
[0140] like Figure 3 As shown, Figure 3 A schematic block diagram of an electronic device 300 provided in an embodiment of the present application is shown. The electronic device 300 includes: a processor 310 and a memory 320;
[0141] The processor 310, the memory 320 and each component are directly or indirectly electrically connected to each other to achieve data transmission or interaction. For example, these components can be electrically connected to each other via one or more communication buses or signal lines. The memory 320 is used to store computer programs, such as storing Figure 2 The software function module shown in the figure is the image classification device 200. The image classification device 200 includes at least one software function module that can be stored in the memory 320 in the form of software or firmware or fixed in the operating system (OS) of the electronic device 300. The processor 310 is used to execute the executable module stored in the memory 320, such as the software function module or computer program included in the image classification device 200.
[0142] Among them, the memory 320 can be, but is not limited to, random access memory (Random Access Memory, RAM), read only memory (Read Only Memory, ROM), programmable read-only memory (Programmable Read-Only Memory, PROM), erasable programmable read-only memory (Erasable Programmable Read-Only Memory, EPROM), electrically erasable read-only memory (Electric Erasable Programmable Read-Only Memory, EEPROM), etc.
[0143] The processor 310 may be an integrated circuit chip with signal processing capabilities. The above-mentioned processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), a microprocessor, etc.; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The various methods, steps and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. Alternatively, the processor 310 may also be any conventional processor, etc.
[0144] The embodiment of the present application further provides a non-volatile computer-readable storage medium (hereinafter referred to as storage medium), on which a computer program is stored. When the computer program is run by a computer such as the above-mentioned electronic device 300, the image classification method shown above is executed.
[0145] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.
[0146] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.
[0147] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.
[0148] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a computer-readable storage medium, including several instructions for enabling a computer device (which can be a personal computer, a laptop, a server, or an electronic device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned computer-readable storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0149] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. An image classification method, characterized in that: The method comprises: Extracting features from the image to obtain feature representation of the image; Encoding the feature representation according to a position weight matrix to obtain a global feature, wherein the position weight matrix includes position weights of all elements in the feature representation, and the position weights are obtained according to the coordinates of the elements in the feature representation; Removing redundant features in the feature representation to obtain local features; Combining the global feature and the local feature to obtain a classification feature; The classification features are input into a preset classifier to obtain a preset classification result.
2. The method according to claim 1, characterized in that The step of extracting features from an image to obtain a feature representation of the image includes: Dividing the image into a plurality of first sub-blocks according to a first layout; Calculating a representation of each first sub-block according to a self-attention mechanism to obtain a first representation of the image; Dividing the first representation into a plurality of second sub-blocks according to a second layout, wherein both the first layout and the second layout include a plurality of windows, and the windows in the second layout are obtained by translating the windows in the first layout, and the windows are used to indicate a division method of the sub-blocks; The representation of each second sub-block is calculated according to the self-attention mechanism to obtain the feature representation.
3. The method according to claim 1, characterized in that: The method further comprises: Using a preset pooling kernel to encode the feature representation along the horizontal direction and the vertical direction respectively, to obtain horizontal direction perception features and vertical direction perception features; Merging the horizontal direction perception feature and the vertical direction perception feature to obtain a merged feature; According to the length and height of the feature representation, the merged feature is divided into a horizontal tensor and a vertical tensor; According to the number of channels represented by the feature, the channels of the horizontal tensor and the vertical tensor are reduced in dimension to generate the position weight matrix.
4. The method according to claim 1, characterized in that: The removing redundant features in the feature representation to obtain local features includes: dividing the feature representation into a first feature and a second feature, the first feature having more information than the second feature; Cross-reconstructing the first feature and the second feature to obtain a refined feature; The deep features and shallow features in the refined features are merged to obtain the local features, and the shallow features contain more pixel information than the deep features.
5. The method according to claim 4, characterized in that The dividing the feature representation into a first feature and a second feature comprises: The feature representation is multiplied by an information weight and a non-information weight respectively to generate the first feature and the second feature, wherein the information weight is obtained based on a preset first hyperparameter and a weight distribution, and the non-information weight is obtained based on a preset second hyperparameter and the weight distribution, and the weight distribution is obtained by performing a feature evaluation on the feature representation.
6. The method according to claim 4, characterized in that The method further comprises: According to a third hyperparameter, the refined features are divided into upper features and lower features along a channel direction; Convolving the upper features using a preset convolution kernel to generate the deep features; The lower features are subjected to feature fusion to generate the shallow features.
7. The method according to claim 1, characterized in that The step of combining the global feature and the local feature to obtain a classification feature includes: Multiplying the global feature, the local feature and the feature representation by the adjustable coefficient respectively to obtain corresponding adjustment features; All adjustable features are combined to obtain the classification feature.
8. An image classification device, characterized in that: The device comprises: A feature extraction module, used to extract features from an image to obtain a feature representation of the image; A global feature generation module, used for encoding the feature representation according to a position weight matrix to obtain a global feature, wherein the position weight matrix includes position weights of all elements in the feature representation, and the position weights are obtained according to the coordinates of the elements in the feature representation; A local feature generation module, used to remove redundant features in the feature representation to obtain local features; A merging module, used for merging the global feature and the local feature to obtain a classification feature; The classification module is used to input the classification features into a preset classifier to obtain a preset classification result.
9. An electronic device, characterized in that: include: A memory and a processor, wherein the processor is connected to the memory; The memory is used to store programs; The processor is used to call the program stored in the memory to execute the method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is executed.
Citation Information
Patent Citations
Image classification method and device, terminal and storage medium
CN114219044A
Small sample image classification system and method thereof
CN115100470A
