Image classification method, device, computer equipment and storage medium
By applying a similarity constraint relationship between image features and reducing the similarity of image features, the problem of low accuracy of traditional hierarchical image classification methods is solved, and higher image classification accuracy and multi-classification hierarchy task completion are achieved.
Patent Information
- Application Number
- CN202010303814.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-04-17
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2040-04-17
AI Technical Summary
Traditional hierarchical image classification methods have low accuracy for images, especially when the feature similarity between multiple classification levels is high.
By applying a similarity constraint relationship between image features, image features are input to the corresponding image classifier to reduce the similarity, so that image classifiers of different classification levels can pay attention to different features on the image and improve the accuracy of hierarchical classification. sex.
It improves the accuracy of image hierarchical classification, can complete classification tasks at multiple classification levels at the same time, and enhances the classification ability of images.
Smart Images

Figure CN111507403B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence technology, and in particular to an image classification method, apparatus, computer equipment and storage medium. Background Art
[0002] With the development of artificial intelligence technology, there are technologies for classifying images based on deep learning technologies such as deep neural networks. For example, an image classifier can be built based on a deep neural network to classify input images. Among them, for traditional image classification tasks, the status of each image category is generally equal, that is, no category distinction is made for all images. In this case, the image classifier is relatively easy to achieve for relatively simple classification tasks, such as distinguishing between car images and other categories such as cat and dog images. By using hierarchical image classification technology, it is possible to complete the hierarchical classification task by first determining whether the image belongs to the animal category or the non-animal category, and then further learning the cat and dog categories from the animal category, for example.
[0003] The hierarchical image classification method provided by the traditional technology often directly inputs the image into two different image classifiers, for example, one for classification training for a large category and the other for classification training for subcategories under the large category. However, the accuracy of hierarchical classification of images using this method is low. Summary of the invention
[0004] Based on this, it is necessary to provide an image classification method, device, computer equipment and storage medium to address the above technical problems.
[0005] A method for image classification, comprising:
[0006] Get the image to be classified;
[0007] Inputting at least two image features of the image to be classified into at least two image classifiers respectively; the at least two image classifiers correspond to at least two classification levels respectively; the image features input into the image classifiers corresponding to adjacent classification levels have a similarity constraint relationship, which is used to reduce the similarity between the image features;
[0008] According to the classification result of the image to be classified at the corresponding classification level output by the image classifier, a hierarchical classification result of the image to be classified is obtained.
[0009] An image classification device, comprising:
[0010] An image acquisition module, used to acquire images to be classified;
[0011] A feature input module, used for inputting at least two image features of the image to be classified into at least two image classifiers; the at least two image classifiers correspond to at least two classification levels respectively; the image features input to the image classifiers corresponding to adjacent classification levels have a similarity constraint relationship, which is used to reduce the similarity between the image features;
[0012] The result acquisition module is used to acquire the hierarchical classification result of the image to be classified according to the classification result of the image to be classified at the corresponding classification level output by the image classifier.
[0013] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:
[0014] Acquire an image to be classified; input at least two image features of the image to be classified into at least two image classifiers; the at least two image classifiers correspond to at least two classification levels respectively; there is a similarity constraint relationship between the image features input to the image classifiers corresponding to adjacent classification levels, which is used to reduce the similarity between the image features; and obtain a hierarchical classification result of the image to be classified according to the classification result of the image to be classified at the corresponding classification level output by the image classifier.
[0015] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:
[0016] Acquire an image to be classified; input at least two image features of the image to be classified into at least two image classifiers; the at least two image classifiers correspond to at least two classification levels respectively; there is a similarity constraint relationship between the image features input to the image classifiers corresponding to adjacent classification levels, which is used to reduce the similarity between the image features; and obtain a hierarchical classification result of the image to be classified according to the classification result of the image to be classified at the corresponding classification level output by the image classifier.
[0017] The above-mentioned image classification method, device, computer equipment and storage medium obtain an image to be classified, and input at least two image features of the image to be classified into at least two image classifiers, the at least two image classifiers correspond to at least two classification levels respectively, and the image features input into the image classifiers corresponding to the adjacent classification levels have a similarity constraint relationship, which is used to reduce the similarity between the image features; then, the hierarchical classification result of the image to be classified can be obtained according to the classification result of the image to be classified output by the image classifier at the corresponding classification level. The scheme imposes a similarity constraint relationship between image features, thereby reducing the similarity between image features input into the image classifiers corresponding to the adjacent classification levels as much as possible, so that the image classifiers corresponding to different classification levels can pay attention to different image features on the same image to be classified, and classify the image at their respective classification levels according to the corresponding image features, thereby improving the accuracy of hierarchical classification of images, and can simultaneously complete the classification task of the image to be classified at multiple classification levels. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] Figure 1 A diagram showing an application environment of an image classification method in one embodiment;
[0019] Figure 2 is a schematic diagram of an image classification task in one embodiment;
[0020] Figure 3 is a schematic diagram of a flow chart of an image classification method in one embodiment;
[0021] Figure 4 A schematic diagram of a flow chart of steps for constructing an image classifier in one embodiment;
[0022] Figure 5 A schematic diagram of the principle of image classification in one embodiment;
[0023] Figure 6 A schematic diagram of a flow chart of steps for obtaining sample image features in one embodiment;
[0024] Figure 7 A schematic diagram of an interface for displaying image information in an embodiment;
[0025] Figure 8 A schematic diagram of the principle of image classification in an application example;
[0026] Fig. 9 is a structural block diagram of an image classification device in one embodiment;
[0027] Fig.10 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION
[0028] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0029] The image classification method provided in this application can be applied to Figure 1 In the application environment shown, Figure 1 FIG. 1 is an application environment diagram of an image classification method in an embodiment. The terminal 110 can communicate with the server 120 through a network. The terminal 110 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, and portable wearable devices, and the server 120 can be implemented as an independent server or a server cluster composed of multiple servers.
[0030] The image classification method provided in this application relates to the field of artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that can use digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, and artificial intelligence can perceive the environment, acquire knowledge and use knowledge to obtain the best results.
[0031] Among them, artificial intelligence technology includes computer vision technology. Computer vision technology (CV) refers to the use of terminal devices such as cameras and computers to replace human eyes to identify, track and measure targets, and further perform graphic processing to make the computer processing into an image that is more suitable for human eye observation or transmission to instrument detection. Computer vision technology can include image recognition and image classification technologies, such as identifying whether the image is an image of a car, a cat or a dog.
[0032] Combining machine learning technology with computer vision technology can enable the terminal device to intelligently classify the images to be classified based on the learned image classification knowledge. Furthermore, the terminal device can perform hierarchical classification of the images to be classified.
[0033] Among them, combined Figure 2 Explain the hierarchical classification task. Figure 2 is a schematic diagram of an image classification task in one embodiment. Figure 2The difference between ordinary classification and hierarchical classification is shown. In general classification tasks, the status of each image category is equal, and the image classifier will not distinguish between each category, that is, directly identify the image as a cat image, a dog image, a bicycle image or a car image, etc., but in fact, the relationship between different categories is different. For example, in the four categories of cats, dogs, cars and bicycles, cats and dogs belong to the animal category, and the relationship is relatively close, while the relationship between the animal category and the vehicle category to which cars and bicycles belong is actually relatively far. Therefore, the hierarchical classification task can determine whether the image belongs to the animal category or a non-animal category such as the vehicle category, and then further identify cats, dogs, bicycles or cars in the animal category and the vehicle category, wherein the animal category and the vehicle category belong to the same classification level, which can be called the major classification level; cats and dogs under the animal category and bicycles and cars under the vehicle category belong to another classification level, which can be called the sub-class classification level. The image classification method provided by the present application can obtain the classification results of the image to be classified at each classification level, and obtain the hierarchical classification results of the image to be classified, for example, the hierarchical classification results of the image to be classified belonging to the animal category and the cat image can be obtained.
[0034] The image classification method provided in this application can be applied to content review and content understanding tasks involving various categories of images. For example, for video-related services, a video frame extraction strategy can be adopted to achieve content review and understanding of video services.
[0035] Specifically, the image classification method provided in the present application may be executed by the terminal 110 or the server 120 alone, or may be executed by the terminal 110 and the server 120 in cooperation with each other.
[0036] First, taking the execution of terminal 110 alone as an example, the terminal 110 can obtain an image to be classified, and input at least two image features of the image to be classified into at least two image classifiers; wherein the at least two image classifiers can be pre-configured on the terminal 110, corresponding to at least two classification levels respectively, and there is a similarity constraint relationship between the image features input to the image classifiers corresponding to adjacent classification levels, which is used to reduce the similarity between the image features; finally, the terminal 110 can obtain the hierarchical classification result of the image to be classified according to the classification result of the image to be classified output by the image classifier at the corresponding classification level.
[0037] The image classification method provided in the present application can also be executed by the terminal 110 and the server 120 in cooperation. Specifically, the terminal 110 can obtain the image to be classified and send the image to be classified to the server 120. The server 120 inputs the at least two image features of the image to be classified into at least two image classifiers; wherein the at least two image classifiers can be pre-configured on the server 120, corresponding to at least two classification levels respectively, and the image features input to the image classifiers corresponding to adjacent classification levels have a similarity constraint relationship, which is used to reduce the similarity between the image features; then the server 120 can send the classification results of the image to be classified output by each image classifier at the corresponding classification level to the terminal 110, and the terminal 110 can obtain the hierarchical classification result of the image to be classified based on the classification result.
[0038] In one embodiment, Figure 3 As shown, Figure 3 A schematic diagram of a flow chart of an image classification method in an embodiment is provided, and an image classification method is applied to Figure 1 The terminal 110 in the example is used as an example to illustrate, and the following steps are included:
[0039] Step S301, obtaining an image to be classified;
[0040] In this step, the terminal 110 may obtain an image to be classified. The image to be classified may be an image captured by the terminal 110 through an image acquisition device such as a camera, or may be an image pre-stored in an electronic gallery of the terminal 110. The image to be classified may include objects such as animals and plants. Specifically, the terminal 110 may capture an image of a cat in real time through a camera configured therein, and the image of the cat may be used as the image to be classified.
[0041] Step S302, inputting at least two image features of the image to be classified into at least two image classifiers accordingly;
[0042] In this step, the terminal 110 may input the at least two image features of the image to be classified into at least two image classifiers. The at least two image classifiers may be pre-configured on the terminal 110. The at least two image classifiers correspond to at least two classification levels, that is, different image classifiers correspond to different classification levels. Figure 2 To illustrate, in a hierarchical classification task, the terminal 110 may be configured with, for example, two image classifiers, one for classification at the level of "animals, vehicles", and the other for classification at the level of "cats and dogs under animals, etc., bicycles and cars under vehicles, etc.".
[0043] Furthermore, there is a similarity constraint relationship between the image features input to the image classifiers corresponding to the adjacent classification levels, and the similarity constraint relationship is used to reduce the similarity between the image features input to the image classifiers corresponding to the adjacent classification levels. Specifically, taking a three-level classification task as an example, let the first level of classification be: plants or animals; the second level of classification, taking animals as an example: mammals or reptiles; the third level of classification, taking reptiles as an example: lizards or snakes. In this case, two similarity constraints need to be added, the first similarity constraint relationship is applied between the image features input to the image classifiers corresponding to the first and second level classification levels, and the second similarity constraint relationship is applied between the image features input to the image classifiers corresponding to the second and third level classification levels. Among them, image features can usually be represented by vectors, so that similarity constraints can be applied to vectors by reducing the similarity between vectors to reduce the similarity between image features input to the image classifiers corresponding to the adjacent classification levels.
[0044] Exemplarily, the similarity constraint relationship may include a mutual information constraint relationship or an orthogonal constraint relationship. For the mutual information constraint relationship, the mutual information between the image features input to the image classifiers corresponding to the adjacent classification levels may be obtained to reduce the similarity between the two image features in a manner that minimizes the mutual information between the two image features. Similarly, for the orthogonal constraint relationship, the similarity between the two image features may be reduced by making the image features input to the image classifiers corresponding to the adjacent classification levels orthogonal to each other.
[0045] In the above manner, the terminal 110 can decouple the two image features input to the image classifier corresponding to adjacent classification levels, thereby eliminating the similar parts between different types of image features, so that the two image features correspond to different characteristics on the image respectively, and then enable the image classifier to pay attention to different types of characteristics on the image based on the aforementioned decoupled image features of the input, and use different image features that are more suitable for the corresponding classification level to better and accurately classify the image to be classified at the corresponding classification level, and complete the classification tasks of multiple classification levels at the same time.
[0046] Step S303, obtaining a hierarchical classification result of the image to be classified according to the classification result of the image to be classified at the corresponding classification level output by the image classifier.
[0047] In this step, the terminal 110 can obtain the classification results output by each image classifier, wherein the classification results may include the classification results of the image to be classified at each classification level. For example, in the case of three image classifiers, the terminal 110 can obtain the classification results corresponding to the three classification levels output by the three image classifiers respectively. The terminal 110 can use the three classification results as the hierarchical classification results of the image to be classified, or select the classification result of one of the classification levels as the required hierarchical classification result. Thus, the terminal 110 can simultaneously complete the classification task of the image to be classified at multiple classification levels.
[0048] In the above-mentioned image classification method, the terminal 110 obtains the image to be classified and inputs at least two image features of the image to be classified into at least two image classifiers, the at least two image classifiers correspond to at least two classification levels respectively, and the image features input into the image classifiers corresponding to the adjacent classification levels have a similarity constraint relationship, which is used to reduce the similarity between the image features; then the terminal 110 can obtain the hierarchical classification result of the image to be classified according to the classification result of the image to be classified output by the image classifier at the corresponding classification level. This scheme imposes a similarity constraint relationship between image features, thereby reducing the similarity between the image features input into the image classifiers corresponding to the adjacent classification levels as much as possible, so that the image classifiers corresponding to different classification levels can pay attention to different image features on the same image to be classified, and classify the image at their respective classification levels according to the corresponding image features, thereby improving the accuracy of hierarchical classification of images, and can simultaneously complete the classification task of the image to be classified at multiple classification levels.
[0049] In one embodiment, the step S302 of inputting the at least two image features of the image to be classified into at least two image classifiers respectively may include:
[0050] At least two image features are obtained through a pre-built feature extractor and are input into at least two image classifiers accordingly.
[0051] In this embodiment, the terminal 110 can use a pre-built feature extractor to obtain at least two image features from the image to be classified, and input them into the aforementioned at least two image classifiers. The feature extractor is constructed based on the aforementioned similarity constraint relationship with the at least two image classifiers. The feature extractor can be implemented based on a neural network model. The terminal 110 can input the image to be classified into a feature extractor based on a neural network model, divide the image features output by the last convolution layer of the feature extractor into the aforementioned at least two image features, and input them into the at least two image classifiers accordingly.
[0052] In the scheme of this embodiment, the terminal 110 can extract features of the image to be classified by pre-cooperating with at least two image classifiers and the feature extractor trained under the aforementioned similarity constraint relationship, so as to obtain a variety of image features that can be applied to each image classifier for classification. There is no need to recalculate the similarity between image features each time the image is processed, thereby improving the image classification efficiency.
[0053] In one embodiment, further, the feature extractor may further include a feature extraction network and an encoder; the step of obtaining at least two image features through the pre-built feature extractor may specifically include:
[0054] The image to be classified is input into a feature extraction network to obtain initial image features output by the feature extraction network; the initial image features are input into an encoder to obtain encoded initial image features output by the encoder; and at least two image features are obtained based on the encoded initial image features.
[0055] In this embodiment, the feature extractor may further include a feature extraction network and an encoder, wherein the feature extraction network and the encoder may be implemented based on a neural network model, such as a ResNet residual network model.
[0056] The feature extraction network can be used to preliminarily obtain the image features of the image to be classified, and the image features obtained by the feature extraction network are often of high dimension and contain redundant information. Therefore, the terminal 110 can first input the image to be classified into the feature extraction network to obtain the initial image features output by the feature extraction network, and then the terminal 110 further inputs the initial image features into the encoder, wherein the terminal 110 uses the encoder to map the image features to the coding features, which can be used to reduce the feature dimension and play a role in removing the redundant information in the initial image features. Thus, the terminal 110 obtains at least two image features based on the encoded initial image features output by the encoder. Among them, the feature extraction network, encoder and at least two classifiers are trained based on the aforementioned similarity constraint relationship. Using the solution of this embodiment, the terminal 110 can directly use the trained feature extraction network to obtain the initial image features when classifying the image. After using the encoder to perform feature dimensionality reduction, the initial image features encoded by the encoder can be divided into at least two image features, and there is a constraint relationship between the image features input into the image classifiers corresponding to adjacent classification levels, thereby improving the efficiency and accuracy of image classification.
[0057] In one embodiment, Figure 4 As shown, Figure 4The following is a flow chart of steps for constructing an image classifier in one embodiment. Before obtaining at least two image features through a pre-constructed feature extractor, the feature extractor and the image classifier may be constructed through the following steps:
[0058] Step S401, obtaining a sample image and obtaining classification labels of the sample image at at least two classification levels as true classification labels of at least two image classifiers;
[0059] In this step, the terminal 110 may obtain sample images, and the number of the sample images is generally multiple. The terminal 110 also needs to obtain classification labels corresponding to the sample images, and the classification labels include classification labels of the sample images at various classification levels. Figure 2 For illustration, the terminal 110 may obtain a cat image as a sample image, and the terminal 110 also needs to obtain classification labels at two classification levels corresponding to the cat image, namely, "animal" and "cat". Further, the terminal 110 may use the classification labels of the sample image at at least two classification levels as the true classification labels of at least two image classifiers for training the image classifiers and feature extractors.
[0060] Step S402: input the sample image into a feature extractor, and obtain at least two sample image features of the same dimension according to the image features of the sample image output by the feature extractor.
[0061] refer to Figure 5 , Figure 5 The schematic diagram of the principle of image classification in one embodiment is that the terminal 110 inputs the sample image into the feature extractor, the feature extractor outputs the image features of the sample image, and the terminal 110 further divides the sample image features into at least two sample image features of the same dimension. Specifically, the terminal 110 can divide the sample image features into image features A and image features B of the same dimension. That is, assuming that the dimension of the sample image features is 2d, the terminal 110 splits the sample image features into image features A and image features B of 1d dimensions respectively. Exemplarily, assuming that the image feature dimension of the sample image output by the acquired feature extractor is a 2048-dimensional vector, the splitting process can be to correspond the first 1 / 2-dimensional vector in the 2048-dimensional vector, that is, the 0th to 1023th dimension vectors to the image feature A, and the last 1 / 2-dimensional vector, that is, the 1024th to 2047th dimension vectors to the image feature B.
[0062] In one embodiment, inputting the sample image into the feature extractor in step S402 may specifically include: preprocessing the sample image to obtain a sample image with an image size of a preset image size; and inputting the sample image with the preset image size into the feature extractor.
[0063] The present embodiment mainly involves that the terminal 110 can pre-process the sample image before inputting it into the feature extractor, and can adjust the image size of the sample image to a preset image size by scaling the image, thereby inputting the sample image of the preset image size into the feature extractor. Specifically, for example, when training models such as image classifiers and feature extractors, images of a fixed image size are usually required. Therefore, the present embodiment can scale sample images of any image size to, for example, an image of a 256×256 image size, and then randomly crop an image of 224×224 from it as a sample image of a preset image size for training.
[0064] Step S403, inputting at least two sample image features into at least two image classifiers respectively, and obtaining predicted classification labels of the sample images output by the at least two image classifiers at corresponding classification levels;
[0065] In this step, refer to Figure 5 , the terminal 110 can input the image feature A into the image classifier A, and input the image feature B into the image classifier B. Among them, the image classifier A and the image classifier B are used to classify the to-be-classified images at different classification levels. In the process of model construction, the image classifier A can predict the classification result according to the input image feature A to obtain the predicted classification label A. Similarly, the image classifier B can obtain the predicted classification label B according to the input image feature B. The predicted classification label can correspond to the probability value of belonging to a certain category at the corresponding classification level. Specifically, it can be combined with Figure 2 To illustrate, the predicted classification label may be a probability value of the sample image belonging to an animal or a vehicle at the classification level of "animal, vehicle".
[0066] Step S404, constructing a similarity constraint relationship between sample image features input to image classifiers corresponding to adjacent classification levels;
[0067] In this step, the terminal 110 constructs a similarity constraint relationship between sample image features input to image classifiers corresponding to adjacent classification levels.
[0068] Step S405 , training a feature extractor and at least two image classifiers based on the real classification labels, the predicted classification labels, and the similarity constraint relationship, to construct the feature extractor and at least two image classifiers.
[0069] According to the technical solution of this embodiment, the terminal 110 can jointly train the feature extractor and at least two image classifiers based on the real classification labels, predicted classification labels and similarity constraint relationships of the sample images, and construct a feature extractor and at least two image classifiers, so that the trained feature extractor can obtain at least two image features with a similarity constraint relationship from the image to be classified, so as to input them into the aforementioned at least two image classifiers for classification, thereby realizing fast and accurate classification of the image to be classified.
[0070] In one embodiment, Figure 6 As shown, Figure 6 The flowchart of the step of obtaining sample image features in one embodiment is as follows. The feature extractor may include a feature extraction network and an encoder. The step of inputting the sample image into the feature extractor in the above step S402, and obtaining at least two sample image features of the same dimension according to the image features of the sample image output by the feature extractor, may include:
[0071] Step S601, inputting the sample image into the feature extraction network to obtain the initial sample image features output by the feature extraction network;
[0072] Step S602, inputting the initial sample image features into the encoder to obtain the encoded initial sample image features output by the encoder;
[0073] Step S603: split the initial sample image features into at least two sample image features with the same dimension.
[0074] This embodiment mainly involves that the terminal 110 can obtain the aforementioned at least two sample image features based on the feature extraction network and encoder included in the feature extractor. Figure 5 , Figure 5The schematic diagram of the principle of image classification in one embodiment is as follows. The terminal 110 can first input the sample image into the feature extraction network. The feature extraction network can be used to preliminarily obtain image features from the sample image. The terminal 110 obtains the initial sample image features output by the feature extraction network. As described in the above embodiment, the image features obtained by the feature extraction network often contain redundant information and the feature dimension is relatively high. Therefore, the terminal 110 further inputs the initial sample image features into the encoder. The encoder can be used to map the image features to the encoding features to reduce the feature dimension of the initial sample image features obtained by the feature extraction network and remove the redundant information in the initial sample image features. Finally, the terminal 110 splits the encoded initial sample image features output by the encoder into at least two sample image features of the same dimension as above. Using the solution of this embodiment, the feature extraction network, the encoder and at least two image classifiers can be trained together based on the similarity constraint relationship, so that the trained feature extraction network, the encoder and at least two image classifiers can be used as a whole of an image classification tool to quickly and accurately classify the image to be classified.
[0075] In one embodiment, the step of training the feature extractor and at least two image classifiers based on the real classification labels, the predicted classification labels and the similarity constraint relationship in the above step S405 may include:
[0076] According to the true classification labels and the predicted classification labels, first loss functions corresponding to the two classification levels are constructed to obtain at least two first loss functions; according to the similarity constraint relationship, a second loss function is constructed; based on the at least two first loss functions and the second loss function, a feature extractor and at least two image classifiers are trained to maximize the at least two first loss functions and the second loss function.
[0077] This embodiment provides a specific method for training a feature extractor and at least two image classifiers. Specifically, the terminal 110 can construct a first loss function based on the real classification label and the predicted classification label, wherein the first loss function may include multiple ones, corresponding to different classification levels respectively. For example, when there are three classification levels, the first loss function includes three. In addition, the terminal 110 also constructs a second loss function based on the similarity constraint relationship, that is, the second loss function is constructed by the similarity constraint relationship between the image features input to the image classifiers corresponding to the adjacent classification levels. Correspondingly, if there are two classification levels, the number of constructed second loss functions is one, and if there are three classification levels, the number of constructed second loss functions is two. Thus, the terminal 110 uses the at least two first loss functions and the second loss function to train the feature extractor and the at least two image classifiers so as to maximize the at least two first loss functions and the second loss function.
[0078] Specifically, we use an image classification task with two classification levels, corresponding to large category classification and subcategory classification, where the large category label is y super , the subcategory label is y sub , then the first loss functions corresponding to the two classification levels are:
[0079]
[0080]
[0081] Among them, L super Represents the first loss function of the large category, L sub represents the first loss function of the subcategory, C super Indicates the total amount of the super category, C sub Indicates the total amount of sub-category. Represents the true classification label of the large category, represents the true classification label of the subcategory, Represents the predicted classification label of the large category, that is, the probability value of the image to be classified belonging to category i under the large category, The predicted classification label of the subcategory is the probability value that the image to be classified belongs to category i under this subcategory.
[0082] In addition, suppose the image features input to the image classifier of the large category and subcategory are E α (x) and E β(x). The similarity constraint relationship imposed on the two image features may be a mutual information constraint relationship or an orthogonal constraint relationship. Taking the mutual information constraint as an example, the corresponding second loss function is:
[0083]
[0084] Among them, L mul represents the second loss function, where r in the mutual information constraint represents the gradient reversal layer, which multiplies the gradient by -1 when the network backpropagates the gradient, i.e., "reverses the gradient". ⊙ represents the element-by-element multiplication operation. Note that its value range can reach negative infinity. In order to avoid the image classifier and feature extractor from learning this situation, for the decoupled feature E α (x) and E β (x), L2 normalization can be used to limit the range of feature values. In this way, only when the features of two images are exactly the same, that is, E α (x)==E β (x), the second loss function L mul The minimum value is -1. When the two image features are completely orthogonal and different from each other, the second loss function L mul The maximum value is 0. Since the gradient reversal layer is used, minimizing the mutual information loss after gradient reversal is equivalent to maximizing the second loss function, which requires the two image features to reduce the similarity as much as possible.
[0085] Finally, the two first loss functions and a second loss function together are based on L = L super +L sub +L mul The entire network is maximally co-trained, including training the feature extractor and at least two image classifiers, and when the feature extractor includes a feature extraction network and an encoder, the feature extraction network and the encoder and at least two image classifiers are trained together.
[0086] In one embodiment, the step S302 of inputting the at least two image features of the image to be classified into at least two image classifiers respectively may include:
[0087] The image to be classified is sent to the server, so that the server inputs at least two image features of the image to be classified into at least two image classifiers to obtain the classification results of the image to be classified at the corresponding classification level output by the image classifier; and the classification results obtained by the server are received.
[0088] refer to Figure 1, the main process of image classification processing in this embodiment can be handed over to the server 120 for processing. Specifically, the terminal 110 can obtain the image to be classified, and then send the image to be classified to the server 120, and the server 120 can be pre-configured with at least two image classifiers mentioned above. After receiving the image to be classified, the server 120 inputs at least two image features of the image to be classified into at least two image classifiers, obtains the classification result of the image to be classified output by the image classifier at the corresponding classification level, and then sends the classification result to the terminal 110, and the terminal 110 receives the classification result sent by the server 120.
[0089] By adopting the technical solution of this embodiment, the terminal 110 can transfer the task of image classification processing to the server 120 for processing, so as to reduce the data processing pressure of the terminal 110.
[0090] In one embodiment, Figure 7 As shown, Figure 7 This is a schematic diagram of an interface for displaying image information in an embodiment. After obtaining the hierarchical classification result of the image to be classified at the corresponding classification level according to the classification result of the image to be classified output by the image classifier in step S303, the following steps may also be included:
[0091] Obtain image classification information carrying hierarchical classification results; and display the image classification information on the image to be classified.
[0092] The present embodiment mainly enables the terminal 110 to directly display the classification result of the image to be classified on the image to be classified. Figure 7 , the terminal 110 may display an image 700 to be classified. After obtaining the hierarchical classification result of the image 700 to be classified, the terminal 110 may display the image classification information carrying the hierarchical classification result in the information display area 710. The hierarchical classification result of the image 700 to be classified may include a major classification result A1 and a subclass classification result B2 of the image to be classified. Specifically, assuming that the image 700 to be classified is a cat image, the image classification information displayed by the terminal 110 may include a major classification result: animal; subclass classification result: cat. By adopting the technical solution of this embodiment, the hierarchical classification result can be superimposed and displayed on the image to be classified, thereby improving the display efficiency of the hierarchical classification result.
[0093] In order to more clearly illustrate the technical solution provided by this application, Figure 8 The principle of image classification is explained in detail. Figure 8 A schematic diagram of the principle of image classification in an application example.
[0094] In general, the input image (x) can be an image of any size, and the training of the model (including the training of the feature extraction network, encoder, and major and sub-category classifiers) generally requires the use of images of fixed image sizes. Therefore, the image of any image size can be first adjusted to an image size of 256×256, and then a 224×224 image size can be randomly cropped from it as the image to be processed. Next, the feature extraction network can be used to extract the image features f(x). Then, an encoder is used to map the image features to the encoded features E(x), which can be regarded as the image features before decoupling. The encoded features are then divided into two parts of features, corresponding to the two parts of decoupled features. Among them, the first part of the features E α (x) is used to train the large-category classifier, the second part features E β (x) is used to train the subclass classifier, and at the same time, mutual information constraints are imposed between the two decoupled features to reduce the similarity between the two decoupled features. It should be noted that the two parts of the features extracted from the image to be processed and input into the major classifier and the subclass classifier may not have any correlation characteristics, that is, it is only necessary to provide the image to be processed for the major class and subclass classifiers, without any special processing of the image features of the image in advance. The image to be processed can be input into the model to automatically complete the training of the model including the feature extraction network, the encoder, and the major class and subclass classifiers. Based on the trained model, the image to be classified can be input to realize the hierarchical classification of the image.
[0095] Specifically, suppose the input image is x and its major category label is y super , the subclass label is y sub , use a feature extraction network to extract image features. The feature extraction network is not specifically limited. For example, various neural network models can be used. Generally speaking, the output of the last convolutional layer in the neural network can be used as the image extraction feature f(x). Then, use an encoder to map the image extraction feature f(x) to the encoding feature E(x), and the dimension of the encoding feature is 2d. The structure of the encoder can use a single-layer fully connected layer. Then, the 2d-dimensional encoding feature E(x) is split into two parts of the same dimension: E(x)→[E α (x); E β (x)] where feature E α (x) and E β (x) features are all 1d-dimensional features, which are used for large category classification and subcategory classification respectively. The corresponding loss functions are:
[0096]
[0097]
[0098] Among them, L super Represents the loss function of the large category, L sub represents the loss function of the subcategory, C super Indicates the total amount of the super category, C sub Indicates the total amount of sub-category. Represents the true classification label of the large category, represents the true classification label of the subcategory, Represents the predicted classification label of the large category, that is, the probability value of the image belonging to category i under the large category, Represents the predicted classification label of the subcategory, that is, the probability value of the image belonging to category i under this subcategory.
[0099] In addition, in order to ensure the characteristic E α (x) and E β (x) can learn as different image features as possible, and impose a mutual information constraint relationship between the two image features:
[0100] Among them, L mul represents the mutual information loss function, where r in the mutual information constraint represents the gradient reversal layer, which multiplies the gradient by -1 when the network backpropagates the gradient, i.e., "reverses the gradient". ⊙ represents the element-by-element multiplication operation. Note that its value range can reach negative infinity. In order to avoid the network learning this situation during training, for the decoupled feature E α (x) and E β (x), L2 normalization can be used to limit the value range of the feature.
[0101] In this case, only when the features of the two images are exactly the same, that is, E α (x)==E β (x), the mutual information loss function achieves the minimum value -1, and when the two image features are completely orthogonal and different from each other, the second loss function L mul The maximum value is 0. Since the gradient reversal layer is used, minimizing the mutual information loss is equivalent to maximizing the mutual information loss function after gradient reversal, which requires that the two image features reduce the similarity as much as possible. Finally, the three loss functions can be used to jointly train the entire network (including the feature extraction network, encoder, and major and sub-category classifiers): L = L super +L sub +L mul .
[0102] The technical solution provided by the above application example can decouple the image features of the image into two parts of image features suitable for large category classification and subcategory classification, and at the same time use mutual information constraints to reduce the similarity between the two parts of image features, so that they can focus on different characteristics in the image as much as possible, and better complete the hierarchical classification task.
[0103] It should be understood that although Figures 3 to 6 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figures 3 to 6 At least part of the steps may include multiple steps or multiple stages. These steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed in turn or alternately with other steps or at least part of the steps or stages in other steps.
[0104] In one embodiment, Fig. 9 As shown, Fig. 9 900 is a structural block diagram of an image classification device in an embodiment. An image classification device is provided. The device can adopt a software module or a hardware module, or a combination of the two to become a part of a computer device. The device 900 specifically includes:
[0105] An image acquisition module 901 is used to acquire an image to be classified;
[0106] The feature input module 902 is used to input at least two image features of the image to be classified into at least two image classifiers; the at least two image classifiers correspond to at least two classification levels respectively; the image features input to the image classifiers corresponding to adjacent classification levels have a similarity constraint relationship, which is used to reduce the similarity between the image features;
[0107] The result acquisition module 903 is used to acquire the hierarchical classification result of the image to be classified according to the classification result of the image to be classified at the corresponding classification level output by the image classifier.
[0108] In one embodiment, the feature input module 902 is further used to obtain at least two image features through a pre-built feature extractor, and input them into at least two image classifiers accordingly; the feature extractor and at least two image classifiers are constructed based on a similarity constraint relationship.
[0109] In one embodiment, the feature extractor includes a feature extraction network and an encoder; the feature input module 902 is further used to input the image to be classified into the feature extraction network to obtain the initial image features output by the feature extraction network; input the initial image features into the encoder to obtain the encoded initial image features output by the encoder; based on the encoded initial image features, at least two image features are obtained.
[0110] In one embodiment, the apparatus 900 may further include:
[0111] A classifier construction module is used to obtain a sample image and obtain classification labels of the sample image at at least two classification levels as real classification labels for at least two image classifiers; input the sample image into a feature extractor, and obtain at least two sample image features of the same dimension based on the image features of the sample image output by the feature extractor; input the at least two sample image features into at least two image classifiers respectively, and obtain predicted classification labels of the sample image output by at least two image classifiers at corresponding classification levels; construct a similarity constraint relationship between sample image features input into image classifiers corresponding to adjacent classification levels; train the feature extractor and at least two image classifiers based on the real classification labels, predicted classification labels and similarity constraint relationships, and construct the feature extractor and at least two image classifiers.
[0112] In one embodiment, the feature extractor includes a feature extraction network and an encoder; a classifier construction module is further used to: input the sample image into the feature extraction network to obtain the initial sample image features output by the feature extraction network; input the initial sample image features into the encoder to obtain the encoded initial sample image features output by the encoder; split the initial sample image features into at least two sample image features of the same dimension.
[0113] In one embodiment, the classifier construction module is further used to: construct first loss functions corresponding to two classification levels according to the true classification labels and the predicted classification labels to obtain at least two first loss functions; construct a second loss function according to the similarity constraint relationship; based on the at least two first loss functions and the second loss function, train the feature extractor and at least two image classifiers to maximize the at least two first loss functions and the second loss function.
[0114] In one embodiment, the similarity constraint relationship includes a mutual information constraint relationship or an orthogonality constraint relationship.
[0115] In one embodiment, the classifier construction module is further used to: pre-process the sample image to obtain a sample image with an image size of a preset image size; and input the sample image with the preset image size into the feature extractor.
[0116] In one embodiment, the apparatus 900 may further include:
[0117] The information display module is used to obtain image classification information carrying hierarchical classification results; and display the image classification information on the image to be classified.
[0118] In one embodiment, the feature input module 902 is further used to send the image to be classified to the server, so that the server inputs at least two image features of the image to be classified into at least two image classifiers to obtain the classification results of the image to be classified output by the image classifier at the corresponding classification level; and receives the classification results obtained by the server.
[0119] For the specific definition of the image classification device, please refer to the definition of the image classification method above, which will not be repeated here. Each module in the above-mentioned image classification device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.
[0120] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig.10 As shown, Fig.10 It is an internal structure diagram of a computer device in an embodiment. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected by a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, an operator network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, an image classification method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a key, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.
[0121] Those skilled in the art will understand that Fig.10The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0122] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.
[0123] In one embodiment, a computer-readable storage medium is provided, storing a computer program, which implements the steps in the above method embodiments when executed by a processor.
[0124] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).
[0125] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0126] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.
Claims
1. An image classification method, It is characterized in that The method comprises: Obtaining a sample image and a classification label of a large category classification and a classification label of a subcategory classification of the sample image, as the true classification label of the image classifier for the large category classification and the true classification label of the subcategory classification, respectively; Input the sample image into a feature extraction network of a feature extractor to obtain initial sample image features output by the feature extraction network; input the initial sample image features into an encoder of the feature extractor to obtain encoded initial sample image features output by the encoder; split the initial sample image features into a first part of features and a second part of features of the same dimension; Input the first part of features and the second part of features to the image classifier for large category classification and the image classifier for subcategory classification, respectively, to obtain the predicted classification label of the sample image output by the image classifier for large category classification in the large category classification, and the predicted classification label of the sample image output by the image classifier for subcategory classification in the subcategory classification; Constructing a similarity constraint relationship between the first part of features and the second part of features; According to the true classification label and the predicted classification label, the first loss functions corresponding to the large category classification and the subcategory classification are constructed respectively, and two first loss functions are obtained; according to the similarity constraint relationship, the second loss function is constructed; the similarity constraint relationship is a mutual information constraint relationship, and the second loss function is , represents the second loss function, represents the gradient reversal layer, represents the element-wise multiplication operation, Represents the first part of the characteristics, Indicates the second part of the characteristics; Based on the two first loss functions and the second loss function, the feature extractor and the two image classifiers are trained to maximize the two first loss functions and the second loss function, and the feature extractor, the image classifier for large category classification, and the image classifier for subcategory classification are constructed; Get the image to be classified; Input the image to be classified into a pre-constructed feature extraction network of the feature extractor to obtain initial image features output by the feature extraction network, input the initial image features into an encoder of the feature extractor, and perform encoding mapping on the initial image features by the encoder to obtain encoding features after removing redundant information; the feature dimension of the encoding features is lower than the feature dimension of the initial image features; and divide the encoding features into two image features with the same dimension; Inputting the two image features into two image classifiers respectively; the two image classifiers correspond to the two classification levels of the large category classification and the subcategory classification respectively; According to the classification result of the image to be classified at the corresponding classification level output by the image classifier, a hierarchical classification result of the image to be classified is obtained.
2. The method according to claim 1, It is characterized in that The feature extractor is implemented based on a neural network model.
3. The method according to claim 1, It is characterized in that The step of inputting the sample image into a feature extraction network of a feature extractor comprises: Preprocessing the sample image to obtain a sample image with an image size that is a preset image size; The sample image of the preset image size is input into the feature extraction network of the feature extractor.
4. The method according to claim 1, It is characterized in that After obtaining the hierarchical classification result of the image to be classified according to the classification result of the image to be classified at the corresponding classification level output by the image classifier, the method comprises: Obtaining image classification information carrying the hierarchical classification result; The image classification information is displayed on the image to be classified.
5. The method according to claim 1, It is characterized in that The step of inputting the two image features into two image classifiers accordingly comprises: Sending the image to be classified to a server, so that the server inputs the two image features of the image to be classified into the two image classifiers accordingly, and obtains the classification results of the image to be classified at the corresponding classification levels output by the image classifiers; The classification result obtained by the server is received.
6. An image classification device, It is characterized in that The device comprises: A classifier construction module is used to obtain a sample image and a classification label of a large category classification and a classification label of a subcategory classification of the sample image, which are respectively used as the real classification label of the image classifier for large category classification and the real classification label of the subcategory classification; the sample image is input into a feature extraction network of a feature extractor to obtain an initial sample image feature output by the feature extraction network; the initial sample image feature is input into an encoder of the feature extractor to obtain an encoded initial sample image feature output by the encoder; the initial sample image feature is split into a first part feature and a second part feature of the same dimension; the first part feature and the second part feature are respectively input into the large category classification network. The image classifier for category classification and the image classifier for subcategory classification obtain the predicted classification label of the sample image output by the image classifier for large category classification in the large category classification, and the predicted classification label of the sample image output by the image classifier for subcategory classification in the subcategory classification; construct a similarity constraint relationship between the first part of features and the second part of features; according to the true classification label and the predicted classification label, construct the first loss function corresponding to the large category classification and the subcategory classification respectively, and obtain two first loss functions; according to the similarity constraint relationship, construct a second loss function; the similarity constraint relationship is a mutual information constraint relationship, and the second loss function is , represents the second loss function, represents the gradient reversal layer, represents the element-wise multiplication operation, Represents the first part of the characteristics, Indicates the second part of the characteristics; Based on the two first loss functions and the second loss function, the feature extractor and the two image classifiers are trained to maximize the two first loss functions and the second loss function, and the feature extractor, the image classifier for large category classification, and the image classifier for subcategory classification are constructed; An image acquisition module, used to acquire images to be classified; A feature input module is used to input the image to be classified into a pre-built feature extraction network of the feature extractor to obtain initial image features output by the feature extraction network, input the initial image features into an encoder of the feature extractor, and perform encoding mapping on the initial image features through the encoder to obtain encoding features after removing redundant information; the feature dimension of the encoding features is lower than the feature dimension of the initial image features; the encoding features are divided into two image features of the same dimension; the two image features are input into two image classifiers respectively; the two image classifiers correspond to the two classification levels of the large category classification and the subcategory classification respectively; The result acquisition module is used to acquire the hierarchical classification result of the image to be classified according to the classification result of the image to be classified at the corresponding classification level output by the image classifier.
7. The device according to claim 6, It is characterized in that The feature extractor is implemented based on a neural network model.
8. The device according to claim 6, It is characterized in that The classifier building module is also used to: Preprocessing the sample image to obtain a sample image with an image size that is a preset image size; The sample image of the preset image size is input into the feature extraction network of the feature extractor.
9. The device according to claim 6, It is characterized in that The device also includes: An information display module, used to obtain image classification information carrying the hierarchical classification result; The image classification information is displayed on the image to be classified.
10. The device according to claim 6, It is characterized in that The feature input module is further used for: Sending the image to be classified to a server, so that the server inputs the two image features of the image to be classified into the two image classifiers accordingly, and obtains the classification results of the image to be classified at the corresponding classification levels output by the image classifiers; The classification result obtained by the server is received.
11. A computer device comprising a memory and a processor, wherein the memory stores a computer program. It is characterized in that When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.
12. A computer-readable storage medium storing a computer program, It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.
