Image classification method, image classification device and electronic equipment

By using a two-stage classification algorithm and a Transformer network to perform binary and fine classification of image patches, the problem of requiring pre-detection in existing technologies is solved, and efficient and accurate image classification is achieved, which is particularly suitable for canine recognition.

CN119027720APending Publication Date: 2024-11-26CHINA MOBILE (SUZHOU) SOFTWARE TECH CO LTD +1
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202410995073.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-24
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

Existing image classification algorithms require a pre-detection algorithm to detect whether there is an object to be classified in the image, resulting in low efficiency and increased computational load.

Method used

A two-stage classification algorithm is adopted, which uses two Transformer-based network structures to perform binary classification and fine classification of image patches. First, the coarse classification module determines whether the image contains the target object. Then, the image patches containing the target object are finely segmented and feature extracted to achieve fast and accurate detection and classification.

Benefits of technology

It improves the accuracy and speed of image classification, reduces the amount of computation, and is particularly suitable for identifying complex objects such as canine species. It can quickly and accurately detect and classify target objects in images.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119027720A_ABST
    Figure CN119027720A_ABST
Patent Text Reader

Abstract

The invention discloses an image classification method, an image classification device and electronic equipment, relates to the technical field of image recognition, and aims to solve the problem of low image classification accuracy. The method comprises the steps of performing block cutting processing on a to-be-classified image to obtain N first image blocks; performing binary classification processing on the N first image blocks based on a first converter module to obtain a first vector and N second vectors; under the condition that the first vectors indicate that the to-be-classified image contains the target to-be-classified object, L second image blocks are determined according to the N second vectors, and the L second image blocks are image blocks containing the image content of the target to-be-classified object in the N first image blocks; performing block cutting processing on the L second image blocks to obtain X third image blocks; and performing fine classification processing on the X third image blocks based on the second converter module to obtain a classification result of the target object to be classified. According to the embodiment of the invention, the accuracy and efficiency of image classification can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of image recognition technology, and in particular to an image classification method, an image classification device and an electronic device. Background Art

[0002] When classifying an image, it is usually divided into two steps: detection + recognition. First, the position of the object to be classified in the image is detected through the detection network, and then the classification result of the object to be classified is judged through a multi-classification network. In the whole process, the classification network is the key to judging the classification result.

[0003] In the related technology, a fine-grained image recognition method based on transformer and component feature fusion is proposed. This method also uses the transformer network structure for feature extraction, and then fuses the relevant features of the attention module for classification.

[0004] However, the proposal does not mention detection based on the Transformer fine-grained image recognition method. Therefore, this method requires a pre-detection algorithm to detect whether there is an object to be classified in the image, and when the object to be classified is detected in the image, the object to be classified is classified based on the transformer.

[0005] As can be seen from the above, the image classification algorithm in the related art has the problem of requiring a pre-detection algorithm to realize the detection of the object to be classified. Summary of the invention

[0006] The embodiments of the present application provide an image classification method, an image classification device, and an electronic device, which can adopt a two-stage classification algorithm and use two Transformer-based network structures to perform coarse classification and fine classification on the objects to be classified in the image, respectively, and convert the traditional detection + recognition idea into a two-classification + multi-classification idea, with high classification accuracy and faster speed.

[0007] In a first aspect, an embodiment of the present application provides an image classification method, comprising:

[0008] The image to be classified is cut into blocks to obtain N first image blocks, where N is an integer greater than 1;

[0009] Performing binary classification processing on the N first image blocks based on the first converter module to obtain a first vector and N second vectors, wherein the first vector is used to indicate whether the image to be classified contains a target object to be classified, the N second vectors correspond to the N first image blocks one by one, and the second vector is used to indicate whether the corresponding first image block contains the image content of the target object to be classified;

[0010] In a case where the first vector indicates that the image to be classified contains the target object to be classified, determining L second image blocks according to the N second vectors, where L is an integer greater than 1 and less than or equal to N, and the L second image blocks are image blocks containing image contents of the target object to be classified in the N first image blocks;

[0011] Performing block processing on the L second image blocks to obtain X third image blocks, where X is an integer greater than L;

[0012] Based on the second converter module, fine classification processing is performed on the X third image blocks to obtain a classification result of the target object to be classified.

[0013] In a second aspect, an embodiment of the present application further provides an image classification device, the device comprising:

[0014] The first block cutting module is used to cut the image to be classified into blocks to obtain N first image blocks, where N is an integer greater than 1;

[0015] A first classification module, configured to perform binary classification processing on the N first image blocks based on a first converter module to obtain a first vector and N second vectors, wherein the first vector is used to indicate whether the image to be classified contains a target object to be classified, the N second vectors correspond one-to-one to the N first image blocks, and the second vector is used to indicate whether the corresponding first image block contains the image content of the target object to be classified;

[0016] A first determining module, configured to determine L second image blocks according to the N second vectors when the first vector indicates that the image to be classified contains the target object to be classified, wherein L is an integer greater than 1 and less than or equal to N, and the L second image blocks are image blocks containing image contents of the target object to be classified in the N first image blocks;

[0017] A second slicing module, configured to slice the L second image blocks to obtain X third image blocks, where X is an integer greater than L;

[0018] The second classification module is used to perform fine classification processing on the X third image blocks based on the second converter module to obtain a classification result of the target object to be classified.

[0019] In a third aspect, an embodiment of the present application further provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps in the image classification method as described above when executing the computer program.

[0020] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps in the image classification method as described above are implemented.

[0021] In a fifth aspect, an embodiment of the present application further provides a computer program product, comprising computer instructions, which, when executed by a processor, implement the steps in the image classification method as described above.

[0022] In an embodiment of the present application, a block processing is performed on the image to be classified to obtain N first image blocks, where N is an integer greater than 1; a binary classification processing is performed on the N first image blocks based on a first converter module to obtain a first vector and N second vectors, wherein the first vector is used to indicate whether the image to be classified contains a target object to be classified, the N second vectors correspond to the N first image blocks one by one, and the second vector is used to indicate whether the corresponding first image block contains the image content of the target object to be classified; when the first vector indicates that the image to be classified contains the target object to be classified, L second image blocks are determined according to the N second vectors, wherein L is an integer greater than 1 and less than or equal to N, and the L second image blocks are image blocks containing the image content of the target object to be classified in the N first image blocks; the L second image blocks are block processed to obtain X third image blocks, where X is an integer greater than L; and a fine classification processing is performed on the X third image blocks based on the second converter module to obtain a classification result of the target object to be classified. By cutting the image to be classified into N first image blocks, and using the features extracted from the N first image blocks by the first converter module to perform binary classification on the image to be classified and the N first image blocks respectively, so as to determine whether the image to be classified includes a specified target object to be classified, and to determine whether each first image block contains at least part of the image content of the target object to be classified, the detection function of the target object to be classified can be realized based on the first converter module; then, for the first image block containing at least part of the image content of the target object to be classified, further cutting, feature extraction and other processes are performed to obtain feature information that is more accurate than that contained in the second vector, and then fine classification processing is performed according to the feature information to obtain the classification result of the target object to be classified. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0024] Figure 1 is a flow chart of an image classification method provided in an embodiment of the present application;

[0025] Figure 2 is a schematic diagram of a first converter module and a second converter module in an embodiment of the present application;

[0026] Figure 3 is a schematic diagram of the first converter module in an embodiment of the present application;

[0027] Figure 4 is a diagram of the second converter module architecture in an embodiment of the present application;

[0028] Figure 5 is a comparison diagram of the image classification method provided in the embodiment of the present application and the image classification method in the related art;

[0029] Figure 6 is a structural diagram of an image classification device provided in an embodiment of the present application;

[0030] Figure 7 It is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0032] In order to make the embodiments of the present application clearer, the relevant technical knowledge involved in the embodiments of the present application is first introduced as follows:

[0033] When classifying an image, it is usually divided into two steps: detection + recognition. First, the position of the object to be classified in the image is detected through the detection network, and then the classification result of the object to be classified is judged through a multi-classification network. In the whole process, the classification network is the key to judging the classification result.

[0034] In related technologies, commonly used image classification technology solutions are as follows:

[0035] 1) Build a fine-grained convolutional neural network, obtain a classifier through training, directly classify the object to be classified, and obtain the classification result of the object to be classified. This technical solution is a common method for processing classification tasks, mainly improving the network structure of convolutional neural networks (CNN).

[0036] Although this method has made some improvements in the network structure, it is limited by the characteristics of the CNN network. The receptive field of shallow features has certain limitations. Although deep features have stronger receptive fields, deeper networks require greater computational effort. In addition, convolutional neural networks extract features for the entire image and have limited ability to express fine-grained features.

[0037] 2) A fine-grained image recognition method based on transformer and component feature fusion. This method also uses the transformer network structure for feature extraction, and then fuses the relevant features of the attention module for classification.

[0038] However, the proposal does not mention detection based on the Transformer fine-grained image recognition method. Therefore, this method requires a pre-detection algorithm to detect whether there is an object to be classified in the image, and when the object to be classified is detected in the image, the object to be classified is classified based on the transformer.

[0039] In the image classification method provided in the embodiment of the present application, two converter modules with different precisions are set, and one converter module with lower precision is used to perform binary classification based on the image block. Since there are only two results in binary classification, one is the image content of the specified target object to be classified in the image block, and the other is the image content of the target object to be classified without the specified target object to be classified in the image block. Therefore, the converter module with lower precision can quickly and accurately obtain the binary classification result and determine the position of the target object to be classified in the image to be classified; the other converter module with higher precision is used to perform more accurate block cutting, feature extraction and classification processing on the image block containing the image content of the target object to be classified, so as to obtain the classification result of the target object to be classified.

[0040] For example, if the target object to be classified is a dog, then when the binary classification result of a converter module with lower accuracy indicates that the image to be classified contains image content of "dog", a converter module with higher accuracy can be further used to perform further block cutting, feature extraction and classification processing on the image block containing at least part of the image content of "dog" in the image to be classified, so as to determine which specific type of dog the dog in the image to be classified is.

[0041] Among them, with the development of artificial intelligence technology, artificial intelligence (AI) technology has been applied to all aspects of human life. Dog recognition technology is to detect dogs through cameras and determine the type of dogs. It can be applied to scenarios such as smart communities. By identifying the type of dogs, it can determine whether they are dangerous dogs to ensure the safety of community residents. The difficulty lies in the fact that there are many types of dogs, reaching more than 130, and some dogs have similar appearance characteristics, making it difficult to identify.

[0042] The image classification method provided in the embodiment of the present application can quickly and accurately detect and classify dogs in images.

[0043] It is worth noting that the embodiments of the present application can be used to classify any object in an image, such as classifying buildings according to architectural style, construction age, etc., or classifying vehicles according to model, manufacturer brand, etc., or classifying organisms according to breed, etc. For the sake of ease of explanation, the embodiments of the present application are usually explained by taking the example of determining the dog in the image according to the dog breed, which does not constitute a specific limitation here.

[0044] The image classification method, image classification device and electronic device provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.

[0045] See also Figure 1 , Figure 1 is a flow chart of the image classification method provided in the embodiment of the present application, such as Figure 1 As shown, the following steps are included:

[0046] Step 101: Slice the image to be classified to obtain N first image blocks, where N is an integer greater than 1.

[0047] In some implementations, the first image block may be an image block of a larger size, for example, a complete photo may be cut into 9 image blocks, or may be cut into a greater number of image blocks, which is not specifically limited herein.

[0048] It should be noted that the above-mentioned first image block is used for binary classification to determine whether the image to be classified contains the specified target object to be classified. When it is determined that the image to be classified contains the specified target object to be classified, the above-mentioned first image block is also used to determine the location of the target object to be classified.

[0049] Step 102: perform binary classification processing on the N first image blocks based on a first transformer module to obtain a first vector and N second vectors, wherein the first vector is used to indicate whether the image to be classified contains a target object to be classified, the N second vectors correspond one-to-one to the N first image blocks, and the second vector is used to indicate whether the corresponding first image block contains the image content of the target object to be classified.

[0050] In some embodiments, the first converter module may be referred to as a coarse classification module, which is used to perform binary classification on the image to be classified and each first image block in the image to be classified, and the first converter module includes a transformer network, which is used to perform feature extraction on the N first image blocks to obtain a first vector and N second vectors. Subsequently, based on the first vector, it can be determined whether the image to be classified contains the target object to be classified, and based on the processing of the N second vectors, it can be determined whether the first image blocks corresponding to each of the N second vectors contain at least part of the image content of the target object to be classified.

[0051] For example: by processing the first vector based on the specified excitation function (softmax, also called the normalized exponential function), a judgment result of whether the image to be classified contains the target object to be classified can be obtained. By processing the N second vectors respectively based on the specified softmax pair, a judgment result of whether the first image blocks corresponding to the N second vectors respectively contain at least part of the image content of the target object to be classified can be obtained.

[0052] Optionally, the above judgment result can be a probability value. For example, the judgment result is that the probability that the image to be classified contains the target object to be classified is 80%. At this time, it can be determined that the image to be classified contains the target object to be classified based on the probability value being greater than or equal to a preset threshold (such as 70%).

[0053] For example, taking the target object to be classified as a dog, the input image (i.e., the image to be classified) is divided into blocks and then input into a transformer-based coarse classification network. First, the entire image is classified to determine whether the entire image contains a dog. Then, the segmented image blocks are classified block by block to classify whether each image block contains partial regional features of a dog. Through this network, traditional target detection is converted into a simple image block binary classification problem.

[0054] Step 103: When the first vector indicates that the image to be classified contains the target object to be classified, determine L second image blocks based on the N second vectors, where L is an integer greater than 1 and less than or equal to N, and the L second image blocks are image blocks containing image content of the target object to be classified among the N first image blocks.

[0055] It should be noted that in the process of binary classification of image blocks based on the transformer network, it is possible to determine whether the image to be classified contains the target object to be classified based on the first vector. If the image to be classified contains the target object to be classified, it is possible to determine in which first image block or blocks the image content of the target object to be classified is distributed based on the N second vectors, that is, some first image blocks in the image to be classified may not contain the image content of the target object to be classified, and some first image blocks may contain the image content of the target object to be classified. At this time, the first image block containing the image content of the target object to be classified is determined as the second image block. At this time, the number of second image blocks is less than or equal to the number of first image blocks.

[0056] Step 104: Slice the L second image blocks to obtain X third image blocks, where X is an integer greater than L.

[0057] In some embodiments, the image size of the third image block is smaller than that of the second image block, so that the features extracted based on the third image block are more accurate than the features extracted based on the second image block.

[0058] It should be noted that in the embodiment of the present application, fine segmentation and feature extraction will not be performed on each first image block. Instead, fine segmentation and feature extraction will be performed on the second image block containing the image content of the target object to be classified. Compared with the method of extracting features from the entire image in the related art, this can increase the expressive power of the extracted fine-grained features while saving the amount of calculation.

[0059] Step 105: Perform fine classification processing on the X third image blocks based on the second converter module to obtain a classification result of the target object to be classified.

[0060] In some embodiments, the second converter module can be referred to as a fine classification module, which is used to perform fine classification on the target object to be classified in the image to be classified. The second converter module includes a transformer network, which is used to perform fine feature extraction on the X third image blocks, and analyze the classification result of the target object to be classified based on the obtained fine features.

[0061] In some embodiments, the coarse classification module is used to extract feature vectors of N first image blocks, and perform binary classification processing based on the feature vectors to determine whether the image to be classified contains the target object to be classified, and to determine whether each first image block contains partial features of the target object to be classified; then, the second image block containing partial features of the target object to be classified is subjected to refined feature extraction by the fine classification module, and the refined features extracted by the fine classification module are fused with the features extracted by the coarse classification module, so that the fine classification module analyzes the specific classification results of the target object to be classified based on the fused feature information.

[0062] For example: Figure 2 As shown, taking the classification of dogs in an image using the image recognition method provided in an embodiment of the present application as an example, the following two-stage dog recognition process can be implemented:

[0063] Phase 1: The input image is divided into blocks and then input into a transformer-based coarse classification network. First, the whole image is classified to determine whether the whole image contains dogs. Then, the segmented image blocks (i.e., the first image blocks) are classified block by block to classify whether each image block contains partial regional features of dogs. Through this coarse classification network, traditional object detection is converted into a simple image block binary classification problem.

[0064] Stage 2: The feature image block containing dogs (i.e., the second image block) is input into the transformer-based fine classification module through the branch selection module to further extract refined features. The fine classification module re-divides the image block containing dog features for a second time, dividing it into units of 4 pixels. Therefore, the extracted features contain more fine-grained features of dogs. The fine-grained features extracted by this network are fused with the coarse features of the corresponding image block extracted from the coarse classification network as the input of the final fine classification transformer network. Then, the first output vector P0 (i.e., the fifth vector) of the fine classification transformer network is passed through softmax for fine-grained classification to obtain the classification result of the dogs in the image.

[0065] In some embodiments, in this step, fine-grained features may be extracted from the X third image blocks based on a transformer network different from the transformer network in the first transformer module, and the type of the target object to be classified contained in the image to be classified is determined according to the extracted fine-grained features. For example, determining which dog type a dog in a photo belongs to.

[0066] In an embodiment of the present application, a block processing is performed on the image to be classified to obtain N first image blocks, where N is an integer greater than 1; a binary classification processing is performed on the N first image blocks based on a first converter module to obtain a first vector and N second vectors, wherein the first vector is used to indicate whether the image to be classified contains a target object to be classified, the N second vectors correspond to the N first image blocks one by one, and the second vector is used to indicate whether the corresponding first image block contains the image content of the target object to be classified; when the first vector indicates that the image to be classified contains the target object to be classified, L second image blocks are determined according to the N second vectors, wherein L is an integer greater than 1 and less than or equal to N, and the L second image blocks are image blocks containing the image content of the target object to be classified in the N first image blocks; the L second image blocks are block processed to obtain X third image blocks, where X is an integer greater than L; and a fine classification processing is performed on the X third image blocks based on the second converter module to obtain a classification result of the target object to be classified. By cutting the image to be classified into N first image blocks, and using the features extracted from the N first image blocks by the first converter module to perform binary classification on the image to be classified and the N first image blocks respectively, so as to determine whether the image to be classified includes a specified target object to be classified, and to determine whether each first image block contains at least part of the image content of the target object to be classified, the detection function of the target object to be classified can be realized based on the first converter module; then, for the first image block containing at least part of the image content of the target object to be classified, further cutting, feature extraction and other processes are performed to obtain feature information that is more accurate than that contained in the second vector, and then fine classification processing is performed according to the feature information to obtain the classification result of the target object to be classified.

[0067] As an optional implementation, the performing binary classification processing on the N first image blocks based on the first converter module to obtain the first vector and the N second vectors includes:

[0068] generating N fourth intermediate vectors based on the image contents of the N first image blocks;

[0069] Performing linear transformation on the N fourth intermediate vectors respectively to obtain N fifth intermediate vectors;

[0070] generating a sixth intermediate vector based on the classified characters, wherein the sixth intermediate vector has the same dimension as the fifth intermediate vector;

[0071] According to the arrangement positions of the N first image blocks in the image to be classified, adding position coding information to the N fifth intermediate vectors to obtain N seventh intermediate vectors;

[0072] Adding preset position coding information to the sixth intermediate vector to obtain an eighth intermediate vector;

[0073] Performing feature conversion processing on the eighth intermediate vector based on the first converter module to obtain a first vector;

[0074] Based on the first converter module, feature conversion processing is performed on the N seventh intermediate vectors to obtain N second vectors.

[0075] In some embodiments, the network structure of the rough classification module is mainly based on the encoder network structure of Transformer, which is mainly used to distinguish whether different image blocks contain dog regional features. By evenly dividing the input image (because it is a rough classification, the number of divided image blocks does not need to be too large, and the classification speed can be effectively improved, for example Figure 2 or Figure 3 As shown in the figure, the original image is divided into 9 image blocks, and more may be divided in actual use), and then each image block is judged whether it contains a dog area for binary classification. With the coarse classification module algorithm, the detection of dog target area can be converted into a transformer-based binary classification problem without the need for a pre-detection algorithm, and the binary classification problem is easier than the detection problem.

[0076] For example: Figure 3 As shown, taking the classification of dogs in a picture as an example, the process of binary classification based on the coarse classification module includes the following steps:

[0077] 1) Chop the input image into first image blocks of different sizes (such as 16*16). This is used for coarse classification, so the first image blocks do not need to be too small.

[0078] 2) Each first image block is converted into a one-dimensional fourth intermediate vector by vectorization. For example, N fourth intermediate vectors are represented as x1, x2, ..., x n , where n is the identifier of the first image block, which can be any integer between 1 and N.

[0079] 3) Obtain the content feature vectors of each first image block through linear transformation of the linear layer, for example: N fifth intermediate vectors are represented as z'1, z'2, ..., z' n .

[0080] 4) Convert the classification character (such as the character "cls") encoding into a one-dimensional sixth intermediate vector, for example: the sixth intermediate vector is represented as z'0, and z'0, z'1, z'2, ..., z' n Add the position encoding vectors corresponding to 0 to n to obtain the eighth intermediate vector z0 and N seventh intermediate vectors z1, z2, ..., zn .

[0081] Optionally, the position encoding formula can be used to implement the vector z' i Add the position information of the corresponding first image block to obtain the vector z i :

[0082]

[0083]

[0084]

[0085]

[0086] Wherein, t in the above formula represents the position, such as t=1 for the first image block, t=2 for the second image block, and so on; model represents the dimension, here we take 512; where k = 2i represents that the position is an even number such as 2, 4, and k = 2i+1 represents that the position is an odd number; the final output contains the vector z0, z1, z2, ..., z n .

[0087] 5) Substitute the above vectors z0, z1, z2, ..., z n Input to the first converter module to obtain N+1 vectors c'0, c'1, c'2, .... c' output by the first converter module n .

[0088] Among them, c'0 represents the first vector, which is a vector representation containing the entire image information; c'1, c'2, ....c' n represents N second vectors, which are vector representations of the first image blocks corresponding to each other, and vectors c'0, c'1, c'2, .... c' n The content features and position features of the corresponding images or image blocks are included respectively.

[0089] 6) First take the feature position vector c'0 and pass it through softmax Figure 2 Classification processing (determine whether the whole image contains dogs. If the result after softmax does not contain dogs, directly output the result. If so, the second vectors c'1, c'2, .... c' corresponding to each first image block are n After softmax, the image blocks are classified into two categories to determine whether each first image block contains local features of dogs.

[0090] It should be noted that in the related art vision transformer (VIT), vector c'0 is directly input into softmax for multi-classification. For vectors c'1, c'2, ....c' n For example, if there are 120 types of dogs, the VIT in the related art directly uses c'0 to output the classification results of 120 types through softmax to confirm the types of dogs, and its classification accuracy is often not high.

[0091] In the embodiment of the present application, c'0 is used for binary classification, which only confirms whether the whole image contains dogs without multi-classification to determine the type of dog. Compared with multi-classification, binary classification has a higher accuracy rate because of fewer categories. In addition, in the embodiment of the present application, vectors c'1, c'2, ....c' are not discarded. n , considering c'1, c'2, ....c' n Contains the position and content features of each first image block, which can be based on c'1, c'2, .... c' n Whether each first image block contains a local feature of a dog, for example, the N second vectors are input into softmax for binary classification to determine whether each first image block contains a partial feature of a dog.

[0092] As an optional implementation, the method further includes:

[0093] Performing feature extraction on the N first image blocks based on a convolutional feature extraction network to obtain N ninth intermediate vectors, wherein the dimension of the ninth intermediate vector is the same as the dimension of the second vector;

[0094] updating the second vector corresponding to a target image block according to the ninth intermediate vector, wherein the target image block is the first image block corresponding to the ninth intermediate vector;

[0095] The determining L second image blocks according to N second vectors when the first vector indicates that the image to be classified contains the target object to be classified includes:

[0096] When the first vector indicates that the image to be classified contains the target object to be classified, L second image blocks are determined according to the updated N second vectors.

[0097] In some embodiments, the N ninth intermediate vectors correspond one-to-one to the N first image blocks, and the N ninth intermediate vectors may represent shallow convolution features of the N first image blocks.

[0098] In some implementations, updating the second vector corresponding to the target image block according to the ninth intermediate vector may be performed by incorporating shallow convolution features of the corresponding first image block into the second vector.

[0099] For example: Assume that the N second vectors before updating are represented as: c'1, c'2, .... c' n , then Figure 3 As shown, the first image block is input into a 3-layer convolutional feature extraction network [conv1*1, conv3*3, conv1*1]x3) for image block feature extraction, and then a linear fully connected layer (FC) is used to output the extracted ninth intermediate vector, such as N ninth intermediate vectors represented by vectors H1, H2, ...H n Then, the vectors H1, H2, ...H can be transformed into n They are fused into their corresponding second vectors c'1, c'2, ....c' n In the above example, we get the updated second vectors c1, c2, ....c n :

[0100] c1=c'1+H1; c2=c'2+H2;…;c n =c' n +H n .

[0101] Thereafter, the updated second vectors c1, c2, ....c n The image is input into softmax for binary classification to determine whether the corresponding first image block contains some features of dogs.

[0102] It is worth mentioning that the above convolutional feature extraction network has a small size and a fast feature extraction speed.

[0103] In this embodiment, before using the second vector for binary classification to determine whether each first image block contains partial features of a dog, the convolution features of the corresponding first image block are also integrated into the second vector. This can improve the feature expression capability of the updated second vector, thereby improving the accuracy of the judgment result of whether each first image block contains partial features of a dog.

[0104] As an optional implementation manner, performing fine classification processing on the X third image blocks based on the second converter module to obtain the classification result of the target object to be classified includes:

[0105] Performing feature extraction on the X third image blocks based on the first transformer network to obtain X third vectors, wherein the X third vectors correspond to the X third image blocks one by one, and the third vectors include fine feature information of the target object to be classified in the corresponding third image blocks;

[0106] Performing feature fusion on the second vector and the third vector corresponding to the same second image block to obtain a fourth vector;

[0107] The fourth vector, the classification coding vector, and the second vector corresponding to the fourth image block are processed based on the second converter network to obtain a fifth vector and N sixth vectors, wherein the fourth image block is an image block that does not contain the image content of the target object to be classified among the N first image blocks; the classification coding vector is a random vector generated based on the classification character, and the classification coding vector contains position coding information of a preset position; the fifth vector corresponds to the classification coding vector; and the N sixth vectors correspond one-to-one to the N first image blocks;

[0108] Determining a classification result of the target object to be classified according to the fifth vector;

[0109] The second converter module includes the first converter network and the second converter network.

[0110] In some embodiments, the first converter network and the second converter network are two converter networks in a coarse classification module (i.e., the second converter module), wherein the first converter network is used to extract features from the X third image blocks, and the second converter network is used to further extract features from the fourth vector, the classification code vector, and the second vector corresponding to the fourth image block.

[0111] It should be noted that, by performing feature extraction on the X third image blocks through the first converter network, it is possible to more specifically extract refined feature features of the image containing the target object to be classified.

[0112] In some embodiments, the second vector and the third vector corresponding to the same second image block are feature fused to obtain a fourth vector. This can be done by fusing the second vector and the third vector corresponding to the same second image block to enhance the feature expression capability of the obtained fourth vector.

[0113] For example: based on the fact that the third vector contains more refined image features, and the second vector may contain other useful feature information, such as the position information of the corresponding second image block or the arrangement order of the corresponding second image block in the image to be classified, by performing feature fusion of the second vector and the third vector corresponding to the same second image block, the obtained fourth vector can reflect both the image content features of the corresponding second image block and the position features of the corresponding second image block.

[0114] In some implementations, the step of performing feature fusion on the second vector and the third vector corresponding to the same second image block to obtain a fourth vector includes:

[0115] splicing all the third vectors corresponding to the same second image to obtain a first intermediate vector;

[0116] Processing the first intermediate vector based on the first fully connected layer to obtain a second intermediate vector;

[0117] splicing the second intermediate vector and the second vector corresponding to the same second image to obtain a third intermediate vector;

[0118] The third intermediate vector is processed based on the second fully connected layer to obtain a fourth vector.

[0119] In some implementations, all the third vectors corresponding to each of the second images are concatenated to obtain L first intermediate vectors.

[0120] For example: Assume that the X third vectors are represented as x1, x2, ..., x X , all the third vectors corresponding to the same second image are concatenated, and the obtained first intermediate vectors are represented as c1, c2, ..., c L .

[0121] In some implementations, processing the L first intermediate vectors based on the first fully connected layer may be performing a linear transformation on the L first intermediate vectors using the first fully connected layer to obtain L second intermediate vectors.

[0122] In some implementations, the second intermediate vector and the second vector corresponding to the same second image are spliced ​​to obtain the third intermediate vector, which may be spliced ​​with the second intermediate vector and the second vector corresponding to each second image to obtain L third intermediate vectors in total. In this way, the third intermediate vector can reflect both the image content feature of the corresponding second image block and the position feature of the corresponding second image block.

[0123] In some embodiments, the third intermediate vectors are processed based on the second fully connected layer, and the second fully connected layer may be used to perform a linear transformation on L third intermediate vectors to obtain L fourth vectors, so as to facilitate the subsequent input of the fourth vectors into the second converter network for fine classification.

[0124] In some embodiments, the fourth vector, the classification coding vector, and the second vector corresponding to the fourth image block are processed based on the second converter network to obtain a fifth vector and N sixth vectors. The fourth vector, the classification coding vector, and the second vector corresponding to the fourth image block can be used as three input information of the second converter network, so as to use the second converter network to determine the fifth vector corresponding to the position information in the classification coding vector, and determine N sixth vectors corresponding one-to-one to the N first image blocks (that is, the N sixth vectors include L sixth vectors corresponding one-to-one to the L second image blocks, and NL sixth vectors corresponding one-to-one to the NL fourth image blocks).

[0125] It is worth mentioning that the fifth vector contains the feature information of the target object to be classified in the image to be classified, and the sixth vector contains the feature information of the partial content of the target object to be classified in the corresponding first image block. In this way, the specific classification result of the target object to be classified can be determined based on the fifth vector.

[0126] In some embodiments, the position information in the classification coding vector may be any preset position, and the sixth vector may include the position information of the first image block corresponding to each other, for example, based on the position information in the fourth vector, or the position information in the second vector corresponding to the fourth image block, the position information in each sixth vector is determined. In this way, the fifth vector and the sixth vector can be distinguished based on the position information in the classification coding vector.

[0127] For example: Figure 4 As shown, for each second image block, further feature extraction is performed through the transformer network in the fine classification module. Taking the feature extraction process of a second image block as an example, the following steps are specifically included:

[0128] 1a. Segment the second image block to obtain at least two third image blocks.

[0129] 2a. Extract the one-dimensional feature vector of each third image block, i.e., vector x i , i is the identifier of the third image block, which can be any integer between 1 and q, q represents the number of third image blocks obtained based on a second image block, if all second image blocks are divided into the same number of third image blocks, then X=q*L.

[0130] 3a. Vector xi Perform linear transformation to obtain the content feature vector of each third image block, such as vector z' i .

[0131] 4a. In the vector z' i Add the position information of the corresponding third image block to obtain a vector containing the content features and position features of each third image block, such as vector z i .

[0132] In some embodiments, the above position information may be position information in any form such as characters or codes, which is not specifically limited here. Taking the position information in the form of code as an example, the position coding formula in the above embodiment can be used to implement the vector z' i Add the position information of the corresponding third image block to obtain vectors z1, z2, ...z containing position features q .

[0133] 5a. Substitute the above vectors z1, z2, ...z q Input into the transformer network to obtain the third vector output by the transformer network, such as c1, c2, ....c q .

[0134] Another example: Figure 4 As shown, the process of performing feature fusion on the second vector and the third vector corresponding to the same second image block to obtain a fourth vector may include the following steps:

[0135] 1b. All the third vectors corresponding to the same second image are spliced ​​to obtain a first intermediate vector, such as the third vectors c1, c2, .... c corresponding to the same second image. q The vectors are concatenated to obtain a first intermediate vector, such as vector C.

[0136] 2b. Based on the first fully connected layer (FC), feature conversion processing is performed on vector C to obtain a second intermediate vector, such as vector M.

[0137] 3b. Concatenate the second intermediate vector corresponding to the same second image and the second vector to obtain a third intermediate vector.

[0138] In this step, useful feature information such as position features in the second vector corresponding to the same second image may be integrated into the second intermediate vector, so that repeated extraction of feature information in the second vector may be avoided.

[0139] 4b. Based on the second fully connected layer, the third intermediate vector is subjected to feature conversion processing to obtain a fourth vector, which can be used as a second converter network (such as Figure 4 The linear transformer group (liner transformer block) in the upper middle part) receives part of the input information so that the second transformer network outputs the classification result of the image content in the corresponding second image block according to the fourth vector.

[0140] It should be noted that the third intermediate vector and the fourth vector are in one-to-one correspondence with the second image block. For the remaining fourth image block in the N first image blocks, the corresponding second vector is directly used as part of the input information of the second converter network, so that the second converter network outputs the feature information of the image content in the corresponding fourth image block according to the second vector (the feature information of the fourth image block can indicate that the fourth image block does not contain the target object to be classified). In addition, the classification coding vector is also input to the second converter network, so that the second converter network outputs the feature information of the target object to be classified in the entire image to be classified.

[0141] In some embodiments, enabling the second transformer network to output feature information of the target object to be classified in the entire image to be classified based on the classification encoding vector includes:

[0142] Assuming that the preset position corresponding to the classification coding vector refers to the position numbered 0, the string code of classification (classification, cls) can be converted into a one-dimensional vector, and then the position code corresponding to 0 is added. After inputting into the transformer network, the fifth vector output by the transformer network is obtained, such as p0.

[0143] Thereafter, the fifth vector p0 may be input into the softmax layer to obtain a classification result of the image to be classified output by the softmax layer.

[0144] Optionally, the process of determining the classification coding vector may include the following steps:

[0145] 1c. Initialize cls randomly to a vector z'0, where the dimension of z'0 is the same as the second vector (z'1, z'2...z' n ) have the same dimensions.

[0146] 2c. Substitute t=0 into the position encoding formula to obtain the classification encoding vector z0 of position 0. The position encoding formula is the same as that in the above embodiment, and will not be described in detail here.

[0147] It is worth mentioning that the fine classification module in the embodiment of the present application mainly performs the second stage of refined classification based on the classification results of the coarse classification module, and the classification target is only for the image blocks output from the coarse classification network that contain partial features of the target object to be classified. The fine classification network performs refined feature extraction on the image blocks that contain partial features of the target object to be classified, and the extracted features are more targeted and have stronger expressive power. Before classification, the output features of the fine-grained classification network are integrated with the features of the corresponding image blocks in the coarse classification network, and also with the position coding features of the image blocks in the entire image, further enhancing the expressive power of the features.

[0148] For example: Taking dog image classification as an example, Figure 4 As shown in the figure, the workflow of the sub-classification module is as follows:

[0149] 1) According to the result of the coarse classification module, if the image block (i.e., the fourth image block) does not contain a dog, the second vector P used for classification in the coarse classification module is directly extracted as the input of the transformer layer of the fine-grained classification network ( Figure 4 left branch).

[0150] 2) If the image block contains dog features, the dog image block is further divided into 4*4 blocks and then input into the fine-grained feature extraction transformer network ( Figure 3 The right branch) is similar to the network structure of the coarse classification, with the following differences: the image block input to the fine-grained feature extraction transformer network is a third image block with a smaller size after secondary segmentation. The third image block is refined through the transformer network to obtain the third vectors c1, c2, .... c q (with the updated second vectors c1, c2, ....c in the coarse classification module n After that, the third vectors c1, c2, .... c corresponding to the same second image block are also q After concatenation, the output vector M (i.e., the second intermediate vector) is obtained after passing through the fully connected layer FC.

[0151] 3) The vector M is concatenated with the second vector P corresponding to the second image block in the coarse classification network, and the output vector (i.e., the third intermediate vector) after passing through a fully connected layer FC is used as the input of the transformer-based fine classification network.

[0152] 4) Convert the cls code into a one-dimensional vector, add the corresponding 0 position code to obtain the classification code vector z0 at the 0 position (different from the eighth intermediate vector z0 in the coarse classification module), and input the classification code vector z0 into the second transformer network, and obtain the fifth vector P0 output by the second transformer network (different from the first vector P0 in the coarse classification module).

[0153] 5) The fifth vector P0 is multi-classified through softmax (the specific number of categories is determined according to the dog category. For example, if there are 120 categories of dogs, a 120-dimensional vector is output after softmax) to obtain the specific dog category.

[0154] It is worth noting that the fine classification module in the embodiment of the present application is not like the VIT in the related art that directly vectorizes the image block and adds position coding as input. The fine classification module only adds position coding to CLS, and other image blocks are divided into two branches according to the results of the coarse classification module and input into the fine classification transformer. The second image block containing dog features will be divided twice and extracted through the transformer feature extraction of the right branch and the feature fusion with the feature of the second image block corresponding to the coarse classification will be input into the fine classification transformer. The fourth image block that does not contain dogs will directly extract the corresponding features in the coarse classification network and input it into the fine classification transformer, and finally the fine classification result is completed through CLS.

[0155] like Figure 5 As shown, in the related art, the dog classification method based on VIT mainly consists of the following two methods:

[0156] Method 1: Feature extraction and multi-classification are performed on the entire image based on VIT. For example, assuming that the entire image is divided into 16 image blocks, image features of the 16 image blocks will be extracted, and multi-classification will be performed based on the image features of the 16 image blocks.

[0157] In the above method 1, since VIT does not involve detection, most dog images are input into VIT for classification through full-image segmentation. At this time, many image blocks do not contain dog features, so many non-dog features are extracted, affecting the accuracy of the algorithm.

[0158] Method 2: First detect the dog image area in the picture, and then segment the image area. Assuming that the image area is segmented into 6 image blocks, the image features of the 6 image blocks will be extracted, and multiple classifications will be performed based on the image features of the 6 image blocks.

[0159] In the above method 2, a detection algorithm is used to determine the dog image area in the picture, and then VIT is used to classify the dog image area. The disadvantage is that a detection algorithm is required in advance, and the accuracy of the detection algorithm will affect the segmentation effect of VIT. In addition, detection and classification require feature extraction of the picture separately, which has the defect of repeated feature extraction.

[0160] In the embodiment of the present application, the rough classification module is first used to perform binary classification on the image blocks, and all the image blocks are divided into the second image blocks containing dogs and the fourth image blocks not containing dogs. This step can realize the function of the detection algorithm in the above-mentioned method 2. In addition, for the feature vector of the fourth image block not containing dogs (such as Figure 5 The eight image blocks that do not contain dog image features are directly input into the fine classification module. For the second image block containing dog features (such as Figure 5 The 8 image blocks containing dog image features in the image are input into the fine classification module for secondary refined feature extraction. Then they are fused with the features in the coarse classification module to enhance the feature expression ability, and then input into the fine classification module for species classification.

[0161] The image classification method provided in the embodiment of the present application can be executed by an image classification device. In the embodiment of the present application, an image classification device executing the image classification method is taken as an example to illustrate the image classification device provided in the embodiment of the present application.

[0162] Reference Figure 6 , the embodiment of the present application also provides an image classification device, such as Figure 6 As shown, the image classification device 600 includes:

[0163] The first block cutting module 601 is used to cut the image to be classified into blocks to obtain N first image blocks, where N is an integer greater than 1;

[0164] A first classification module 602 is used to perform binary classification processing on the N first image blocks based on the first converter module to obtain a first vector and N second vectors, wherein the first vector is used to indicate whether the image to be classified contains a target object to be classified, the N second vectors correspond to the N first image blocks one by one, and the second vector is used to indicate whether the corresponding first image block contains the image content of the target object to be classified;

[0165] A first determining module 603 is configured to determine L second image blocks according to the N second vectors when the first vector indicates that the image to be classified contains the target object to be classified, wherein L is an integer greater than 1 and less than or equal to N, and the L second image blocks are image blocks containing image contents of the target object to be classified in the N first image blocks;

[0166] A second slicing module 604 is configured to slice the L second image blocks to obtain X third image blocks, where X is an integer greater than L;

[0167] The second classification module 605 is used to perform fine classification processing on the X third image blocks based on the second converter module to obtain a classification result of the target object to be classified.

[0168] Optionally, the second classification module 605 includes:

[0169] a first feature extraction unit, configured to perform feature extraction on the X third image blocks based on a first transformer network to obtain X third vectors, wherein the X third vectors correspond one-to-one to the X third image blocks, and the third vectors include fine feature information of the target object to be classified in the corresponding third image blocks;

[0170] a feature fusion unit, configured to perform feature fusion on the second vector and the third vector corresponding to the same second image block to obtain a fourth vector;

[0171] A first processing unit is configured to process the fourth vector, the classification coding vector, and the second vector corresponding to the fourth image block based on a second converter network to obtain a fifth vector and N sixth vectors, wherein the fourth image block is an image block of the N first image blocks that does not contain the image content of the target object to be classified; the classification coding vector is a random vector generated based on classification characters, and the classification coding vector contains position coding information of a preset position; the fifth vector corresponds to the classification coding vector; and the N sixth vectors correspond one-to-one to the N first image blocks;

[0172] a determining unit, configured to determine a classification result of the target object to be classified according to the fifth vector;

[0173] The second converter module includes the first converter network and the second converter network.

[0174] Optionally, the feature fusion unit includes:

[0175] A first splicing subunit, configured to splice all the third vectors corresponding to the same second image to obtain a first intermediate vector;

[0176] A first processing subunit, configured to process the first intermediate vector based on a first fully connected layer to obtain a second intermediate vector;

[0177] A second splicing subunit is used to splice the second intermediate vector and the second vector corresponding to the same second image to obtain a third intermediate vector;

[0178] The second processing subunit is used to process the third intermediate vector based on the second fully connected layer to obtain a fourth vector.

[0179] Optionally, the first classification module 602 includes:

[0180] A first generating unit, configured to generate N fourth intermediate vectors based on image contents of the N first image blocks;

[0181] A transformation unit, configured to perform linear transformation on the N fourth intermediate vectors respectively to obtain N fifth intermediate vectors;

[0182] A second generating unit, configured to generate a sixth intermediate vector based on the classified characters, wherein the sixth intermediate vector has the same dimension as the fifth intermediate vector;

[0183] A second processing unit is configured to add position coding information to the N fifth intermediate vectors according to arrangement positions of the N first image blocks in the image to be classified, so as to obtain N seventh intermediate vectors;

[0184] a third processing unit, configured to add preset position coding information to the sixth intermediate vector to obtain an eighth intermediate vector;

[0185] a fourth processing unit, configured to perform feature conversion processing on the eighth intermediate vector based on the first converter module to obtain a first vector;

[0186] A fifth processing unit is used to perform feature conversion processing on the N seventh intermediate vectors based on the first converter module to obtain N second vectors.

[0187] Optionally, the image classification device 600 further includes:

[0188] a feature extraction module, configured to perform feature extraction on the N first image blocks based on a convolutional feature extraction network to obtain N ninth intermediate vectors, wherein the dimension of the ninth intermediate vector is the same as the dimension of the second vector;

[0189] an updating module, configured to update the second vector corresponding to a target image block according to the ninth intermediate vector, wherein the target image block is the first image block corresponding to the ninth intermediate vector;

[0190] The first determining module is specifically configured to:

[0191] When the first vector indicates that the image to be classified contains the target object to be classified, L second image blocks are determined according to the updated N second vectors.

[0192] The image classification device 600 provided in the embodiment of the present application can implement each process in the aforementioned image classification method embodiment and achieve the same technical effect. To avoid repetition, it will not be described here.

[0193] The present application also provides an electronic device. Figure 7 , Figure 7 is a structural diagram of an electronic device provided in an embodiment of the present application. Since the principle of solving the problem by the electronic device is similar to that of the aforementioned image classification method embodiment of the present application, the implementation of the electronic device can refer to the implementation of the method, and the repeated parts will not be repeated. Figure 7 As shown, the electronic device includes: a processor 700 and a memory 720, and the processor 700 is used to read the program in the memory 720 and perform the following process:

[0194] The image to be classified is cut into blocks to obtain N first image blocks, where N is an integer greater than 1;

[0195] Performing binary classification processing on the N first image blocks based on the first converter module to obtain a first vector and N second vectors, wherein the first vector is used to indicate whether the image to be classified contains a target object to be classified, the N second vectors correspond to the N first image blocks one by one, and the second vector is used to indicate whether the corresponding first image block contains the image content of the target object to be classified;

[0196] In a case where the first vector indicates that the image to be classified contains the target object to be classified, determining L second image blocks according to the N second vectors, where L is an integer greater than 1 and less than or equal to N, and the L second image blocks are image blocks containing image contents of the target object to be classified in the N first image blocks;

[0197] Performing block processing on the L second image blocks to obtain X third image blocks, where X is an integer greater than L;

[0198] Based on the second converter module, fine classification processing is performed on the X third image blocks to obtain a classification result of the target object to be classified.

[0199] Among them, Figure 7In the embodiment, the bus architecture may include any number of interconnected buses and bridges, specifically linking various circuits of one or more processors represented by processor 700 and memory represented by memory 720. The bus architecture may also link various other circuits such as peripheral devices (such as cameras), voltage regulators, and power management circuits, which are well known in the art and are not further described herein. The bus interface provides an interface. The processor 700 is responsible for managing the bus architecture and general processing, and the memory 720 may store data used by the processor 700 when performing operations.

[0200] Optionally, the processor 700 is further configured to read a program in the memory 720 and execute the following process:

[0201] Performing feature extraction on the X third image blocks based on the first transformer network to obtain X third vectors, wherein the X third vectors correspond to the X third image blocks one by one, and the third vectors include fine feature information of the target object to be classified in the corresponding third image blocks;

[0202] Performing feature fusion on the second vector and the third vector corresponding to the same second image block to obtain a fourth vector;

[0203] The fourth vector, the classification coding vector, and the second vector corresponding to the fourth image block are processed based on the second converter network to obtain a fifth vector and N sixth vectors, wherein the fourth image block is an image block that does not contain the image content of the target object to be classified among the N first image blocks; the classification coding vector is a random vector generated based on the classification character, and the classification coding vector contains position coding information of a preset position; the fifth vector corresponds to the classification coding vector; and the N sixth vectors correspond one-to-one to the N first image blocks;

[0204] Determining a classification result of the target object to be classified according to the fifth vector;

[0205] The second converter module includes the first converter network and the second converter network.

[0206] Optionally, the processor 700 is further configured to read a program in the memory 720 and execute the following process:

[0207] splicing all the third vectors corresponding to the same second image to obtain a first intermediate vector;

[0208] Processing the first intermediate vector based on the first fully connected layer to obtain a second intermediate vector;

[0209] splicing the second intermediate vector and the second vector corresponding to the same second image to obtain a third intermediate vector;

[0210] The third intermediate vector is processed based on the second fully connected layer to obtain a fourth vector.

[0211] Optionally, the processor 700 is further configured to read a program in the memory 720 and execute the following process:

[0212] generating N fourth intermediate vectors based on the image contents of the N first image blocks;

[0213] Performing linear transformation on the N fourth intermediate vectors respectively to obtain N fifth intermediate vectors;

[0214] generating a sixth intermediate vector based on the classified characters, wherein the sixth intermediate vector has the same dimension as the fifth intermediate vector;

[0215] According to the arrangement positions of the N first image blocks in the image to be classified, adding position coding information to the N fifth intermediate vectors to obtain N seventh intermediate vectors;

[0216] Adding preset position coding information to the sixth intermediate vector to obtain an eighth intermediate vector;

[0217] Performing feature conversion processing on the eighth intermediate vector based on the first converter module to obtain a first vector;

[0218] Based on the first converter module, feature conversion processing is performed on the N seventh intermediate vectors to obtain N second vectors.

[0219] Optionally, the processor 700 is further configured to read a program in the memory 720 and execute the following process:

[0220] Performing feature extraction on the N first image blocks based on a convolutional feature extraction network to obtain N ninth intermediate vectors, wherein the dimension of the ninth intermediate vector is the same as the dimension of the second vector;

[0221] updating the second vector corresponding to a target image block according to the ninth intermediate vector, wherein the target image block is the first image block corresponding to the ninth intermediate vector;

[0222] When the first vector indicates that the image to be classified contains the target object to be classified, L second image blocks are determined according to the updated N second vectors.

[0223] The electronic device provided in the embodiment of the present application can execute each process of the aforementioned image classification method embodiment, and its implementation principle and technical effect are similar, which will not be repeated in this embodiment.

[0224] The embodiment of the present application also provides a computer-readable storage medium for storing a computer program. The computer program can be executed by a processor, can implement the various steps of the aforementioned image classification method embodiment, and can achieve the same beneficial effects as the aforementioned image classification method embodiment. To avoid repetition, it will not be repeated here.

[0225] The embodiment of the present application also provides a computer program product, which is stored in a non-volatile storage medium. The computer program product is executed by at least one processor to implement the various steps of the aforementioned image classification method embodiment, and can achieve the same beneficial effects as the aforementioned image classification method embodiment. To avoid repetition, it will not be repeated here.

[0226] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0227] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may be physically included separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0228] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform some steps of the sending and receiving methods described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.

[0229] The above is a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles described in the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. An image classification method, characterized in that: include: The image to be classified is cut into blocks to obtain N first image blocks, where N is an integer greater than 1; Performing binary classification processing on the N first image blocks based on the first converter module to obtain a first vector and N second vectors, wherein the first vector is used to indicate whether the image to be classified contains a target object to be classified, the N second vectors correspond to the N first image blocks one by one, and the second vector is used to indicate whether the corresponding first image block contains the image content of the target object to be classified; In a case where the first vector indicates that the image to be classified contains the target object to be classified, determining L second image blocks according to the N second vectors, where L is an integer greater than 1 and less than or equal to N, and the L second image blocks are image blocks containing image contents of the target object to be classified in the N first image blocks; Performing block processing on the L second image blocks to obtain X third image blocks, where X is an integer greater than L; Based on the second converter module, fine classification processing is performed on the X third image blocks to obtain a classification result of the target object to be classified.

2. The method according to claim 1, characterized in that The performing fine classification processing on the X third image blocks based on the second converter module to obtain the classification result of the target object to be classified includes: Performing feature extraction on the X third image blocks based on the first transformer network to obtain X third vectors, wherein the X third vectors correspond to the X third image blocks one by one, and the third vectors include fine feature information of the target object to be classified in the corresponding third image blocks; Performing feature fusion on the second vector and the third vector corresponding to the same second image block to obtain a fourth vector; The fourth vector, the classification coding vector, and the second vector corresponding to the fourth image block are processed based on the second converter network to obtain a fifth vector and N sixth vectors, wherein the fourth image block is an image block that does not contain the image content of the target object to be classified among the N first image blocks; the classification coding vector is a random vector generated based on the classification character, and the classification coding vector contains position coding information of a preset position; the fifth vector corresponds to the classification coding vector; and the N sixth vectors correspond one-to-one to the N first image blocks; Determining a classification result of the target object to be classified according to the fifth vector; The second converter module includes the first converter network and the second converter network.

3. The method according to claim 2, characterized in that The step of fusing the second vector and the third vector corresponding to the same second image block to obtain a fourth vector includes: splicing all the third vectors corresponding to the same second image to obtain a first intermediate vector; Processing the first intermediate vector based on the first fully connected layer to obtain a second intermediate vector; splicing the second intermediate vector corresponding to the same second image and the second vector to obtain a third intermediate vector; The third intermediate vector is processed based on the second fully connected layer to obtain a fourth vector.

4. The method according to any one of claims 1 to 3, characterized in that The performing binary classification processing on the N first image blocks based on the first converter module to obtain a first vector and N second vectors includes: Generate N fourth intermediate vectors based on the image contents of the N first image blocks; Performing linear transformation on the N fourth intermediate vectors respectively to obtain N fifth intermediate vectors; generating a sixth intermediate vector based on the classified characters, wherein the sixth intermediate vector has the same dimension as the fifth intermediate vector; According to the arrangement positions of the N first image blocks in the image to be classified, adding position coding information to the N fifth intermediate vectors to obtain N seventh intermediate vectors; Adding preset position coding information to the sixth intermediate vector to obtain an eighth intermediate vector; Performing feature conversion processing on the eighth intermediate vector based on the first converter module to obtain a first vector; Based on the first converter module, feature conversion processing is performed on the N seventh intermediate vectors to obtain N second vectors.

5. The method according to claim 4, characterized in that The method further comprises: Performing feature extraction on the N first image blocks based on a convolutional feature extraction network to obtain N ninth intermediate vectors, wherein the dimension of the ninth intermediate vector is the same as the dimension of the second vector; updating the second vector corresponding to a target image block according to the ninth intermediate vector, wherein the target image block is the first image block corresponding to the ninth intermediate vector; The determining L second image blocks according to N second vectors when the first vector indicates that the image to be classified contains the target object to be classified includes: When the first vector indicates that the image to be classified contains the target object to be classified, L second image blocks are determined according to the updated N second vectors.

6. An image classification device, characterized in that: include: The first block cutting module is used to cut the image to be classified into blocks to obtain N first image blocks, where N is an integer greater than 1; A first classification module, configured to perform binary classification processing on the N first image blocks based on a first converter module to obtain a first vector and N second vectors, wherein the first vector is used to indicate whether the image to be classified contains a target object to be classified, the N second vectors correspond one-to-one to the N first image blocks, and the second vector is used to indicate whether the corresponding first image block contains the image content of the target object to be classified; A first determining module, configured to determine L second image blocks according to the N second vectors when the first vector indicates that the image to be classified contains the target object to be classified, wherein L is an integer greater than 1 and less than or equal to N, and the L second image blocks are image blocks containing image contents of the target object to be classified in the N first image blocks; A second slicing module, configured to slice the L second image blocks to obtain X third image blocks, where X is an integer greater than L; The second classification module is used to perform fine classification processing on the X third image blocks based on the second converter module to obtain a classification result of the target object to be classified.

7. The device according to claim 6, characterized in that The second classification module comprises: a first feature extraction unit, configured to perform feature extraction on the X third image blocks based on a first transformer network to obtain X third vectors, wherein the X third vectors correspond one-to-one to the X third image blocks, and the third vectors include fine feature information of the target object to be classified in the corresponding third image blocks; a feature fusion unit, configured to perform feature fusion on the second vector and the third vector corresponding to the same second image block to obtain a fourth vector; A first processing unit is configured to process the fourth vector, the classification coding vector, and the second vector corresponding to the fourth image block based on a second converter network to obtain a fifth vector and N sixth vectors, wherein the fourth image block is an image block of the N first image blocks that does not contain the image content of the target object to be classified; the classification coding vector is a random vector generated based on classification characters, and the classification coding vector contains position coding information of a preset position; the fifth vector corresponds to the classification coding vector; and the N sixth vectors correspond one-to-one to the N first image blocks; a determining unit, configured to determine a classification result of the target object to be classified according to the fifth vector; The second converter module includes the first converter network and the second converter network.

8. The device according to claim 7, characterized in that The feature fusion unit comprises: A first splicing subunit, configured to splice all the third vectors corresponding to the same second image to obtain a first intermediate vector; A first processing subunit, configured to process the first intermediate vector based on a first fully connected layer to obtain a second intermediate vector; A second splicing subunit is used to splice the second intermediate vector and the second vector corresponding to the same second image to obtain a third intermediate vector; The second processing subunit is used to process the third intermediate vector based on the second fully connected layer to obtain a fourth vector.

9. The device according to any one of claims 6 to 8, characterized in that The first classification module comprises: A first generating unit, configured to generate N fourth intermediate vectors based on image contents of the N first image blocks; A transformation unit, configured to perform linear transformation on the N fourth intermediate vectors respectively to obtain N fifth intermediate vectors; A second generating unit, configured to generate a sixth intermediate vector based on the classified characters, wherein the sixth intermediate vector has the same dimension as the fifth intermediate vector; A second processing unit is configured to add position coding information to the N fifth intermediate vectors according to arrangement positions of the N first image blocks in the image to be classified, so as to obtain N seventh intermediate vectors; a third processing unit, configured to add preset position coding information to the sixth intermediate vector to obtain an eighth intermediate vector; a fourth processing unit, configured to perform feature conversion processing on the eighth intermediate vector based on the first converter module to obtain a first vector; A fifth processing unit is used to perform feature conversion processing on the N seventh intermediate vectors based on the first converter module to obtain N second vectors.

10. The device according to claim 9, characterized in that The device also includes: a feature extraction module, configured to perform feature extraction on the N first image blocks based on a convolutional feature extraction network to obtain N ninth intermediate vectors, wherein the dimension of the ninth intermediate vector is the same as the dimension of the second vector; an updating module, configured to update the second vector corresponding to a target image block according to the ninth intermediate vector, wherein the target image block is the first image block corresponding to the ninth intermediate vector; The first determining module is specifically configured to: When the first vector indicates that the image to be classified contains the target object to be classified, L second image blocks are determined according to the updated N second vectors.

11. An electronic device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor; characterized in that: The processor is used to read the program in the memory to implement the steps in the image classification method according to any one of claims 1 to 5.

12. A computer-readable storage medium for storing a computer program, characterized in that: When the computer program is executed by a processor, the steps in the image classification method according to any one of claims 1 to 5 are implemented.

13. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, implement the steps in the image classification method according to any one of claims 1 to 5.

Citation Information

Patent Citations

  • Coarse and fine classification combined image target detection method and related device

    CN113780256A

  • Deep learning-based tomato disease and insect pest detection and identification method

    CN116630803A

  • Fine granularity identification method based on Vision Transform

    CN117409343A

  • Image classification method and apparatus

    WO2021164550A1