An image classification method, device and storage medium

Abstract

The application relates to the computer technical field, in particular to an image classification method and device and a storage medium, so as to improve the accuracy of the image classification result. In the method, according to the feature vector corresponding to the label, a plurality of image features of the image to be classified are obtained, and the prediction value of each image feature for each label is determined. Finally, the prediction value of the image to be classified is determined by the prediction value corresponding to each label, and the image to be classified is classified according to the label prediction value of the image to be classified. In this way, by obtaining a plurality of image features of the image to be classified, and determining the label prediction value of the image to be classified according to the prediction value of the plurality of image features for each label, the accuracy of the label prediction value in the image to be classified can be improved, thereby improving the accuracy of the image classification.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, and particularly relates to an image classification method and device and a storage medium. BACKGROUND

[0002] With the rapid development of Internet technology, the requirement for image classification is also higher and higher. At present, in order to determine which category or categories the to-be-classified image belongs to, the to-be-classified image needs to be subjected to multi-label classification. That is, whether the image belongs to one or more known categories is detected. However, most of the current technical solutions for multi-label classification ignore the semantic difference between labels, thereby resulting in that the accuracy of the classification result of the image in the prior art is low, which is not conducive to the subsequent classification task. SUMMARY

[0003] The embodiments of the present application provide an image classification method, device and storage medium, so as to improve the accuracy of the classification result of the image.

[0004] In a first aspect, an image classification method is provided, comprising:

[0005] obtaining at least two image features of a to-be-classified image; wherein different image features are obtained according to feature vectors corresponding to different labels;

[0006] for each label, obtaining a prediction value of each image feature corresponding to the label respectively;

[0007] for each label, determining a target prediction value of the label according to the prediction values of the image features corresponding to the label;

[0008] taking the target prediction values of the labels as label prediction values of the to-be-classified image;

[0009] classifying the to-be-classified image according to the label prediction values of the to-be-classified image.

[0010] In a second aspect, an image classification device is provided, comprising:

[0011] a first obtaining module, configured to obtain at least two image features of a to-be-classified image; wherein different image features are obtained according to feature vectors corresponding to different labels;

[0012] a second obtaining module, configured to obtain, for each label, a prediction value of each image feature corresponding to the label respectively;

[0013] a first determining module, configured to determine, for each label, a target prediction value of the label according to the prediction values of the image features corresponding to the label;

[0014] a second determining module, configured to take the target prediction value of each label as a label prediction value of the image to be classified;

[0015] a classifying module, configured to classify the image to be classified according to the label prediction value of the image to be classified.

[0016] Optionally, the first obtaining module comprises:

[0017] a sub-image determining unit, configured to obtain at least two sub-images from the image to be classified, and a spliced image of all the sub-images comprises all regions of the multi-label image;

[0018] a first feature extracting unit, configured to perform feature extraction on each sub-image to obtain sub-image features of the sub-image;

[0019] a first image feature determining unit, configured to take the sub-image features as image features of the image to be classified.

[0020] Optionally, the first obtaining module comprises:

[0021] a sub-image determining unit, configured to obtain at least two sub-images from the image to be classified, and a spliced image of all the sub-images comprises all regions of the multi-label image;

[0022] a second feature extracting unit, configured to perform feature extraction on the image to be classified to obtain at least two global image features of the image to be classified; and

[0023] a third feature extracting unit, configured to perform feature extraction on each sub-image to obtain at least two local image features of each sub-image; wherein the number of local image features of each sub-image is the same as that of global image features, and the local image features and the global image features correspond to each other;

[0024] a fusing unit, configured to, for each local image feature of each sub-image, fuse the local image feature with a global image feature corresponding to the local image feature in the image to be classified to obtain a fused image feature of the local image feature;

[0025] a second image feature determining unit, configured to take the fused image features of each sub-image as image features of the image to be classified.

[0026] Optionally, the fused image features are obtained by the following apparatus:

[0027] a first fusing sub-unit, configured to, for each local image feature of each sub-image, take an average value of the local image feature and a global image feature corresponding to the local image feature in the image to be classified as a fused image feature; or

[0028] a second fusion subunit configured to, for each local image feature of each sub-image, combine the local image feature with a maximum value of each dimension of a global image feature corresponding to the local image feature in the image to be classified, and take the combination result as a fusion image feature; or

[0029] a third fusion subunit configured to, for each local image feature of each sub-image, concatenate the local image feature with a global image feature corresponding to the local image feature in the image to be classified, and input the concatenation result into a fully connected network model to obtain a fusion image feature.

[0030] Optionally, the determining sub-image unit comprises:

[0031] a long-short edge determining subunit configured to determine a long edge and a short edge of the image to be classified, determine a first edge of the image to be classified as the short edge, and determine a second edge as being smaller than the long edge;

[0032] a sub-image determining subunit configured to set a split window with a size same as that of a sub-image, start from the short edge, move the split window by a set length along the long edge each time, and obtain the sub-image according to a region of the image to be classified covered by the split window.

[0033] Optionally, the first obtaining module comprises:

[0034] a fourth feature extracting unit configured to perform feature extraction on the image to be classified to obtain a feature map of the image to be classified;

[0035] a convolution unit configured to perform convolution calculation on the feature map and a feature vector corresponding to each label respectively to obtain an attention mask corresponding to each label;

[0036] a third image feature determining unit configured to perform point multiplication calculation on the feature map and the attention mask corresponding to each label respectively to obtain at least two image features of the image to be classified.

[0037] Optionally, the first determining module is specifically configured to, for each label, determine a maximum value in predicted values of each image feature corresponding to the label as a target predicted value of the label.

[0038] In a third aspect, a computing device is provided, which comprises at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program, and when the program is executed by the processing unit, the processing unit performs the steps of any of the above image classification methods.

[0039] In an embodiment, the computing device can be a server or a terminal device.

[0040] In a fourth aspect, a computer readable medium is provided, which stores a computer program executable by a terminal device, and when the program is executed on the terminal device, the terminal device is caused to perform the steps of any of the image classification methods described above.

[0041] The present application has the following advantages:

[0042] The image classification method, apparatus, electronic device and storage medium provided by the embodiments of the present application can obtain multiple image features of the image to be classified according to the feature vectors corresponding to the labels, determine the prediction values of each image feature for each label, and finally determine the prediction value of the image to be classified by the prediction values corresponding to each label, and classify the image to be classified according to the label prediction value of the image to be classified. In this way, by obtaining multiple image features of the image to be classified and determining the label prediction value of the image to be classified according to the prediction values of the multiple image features for each label, the accuracy of the label prediction value in the image to be classified can be improved, thereby improving the accuracy of image classification.

[0043] Other features and advantages of the present application will be set forth in the following description, and in part will become apparent to those skilled in the art from the description, or can be learned by practice of the present application. The objects and other advantages of the present application can be realized and achieved by the structure particularly pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF DRAWINGS

[0044] The accompanying drawings, which are included to provide a further understanding of the present application, constitute a part of the specification, illustrate the illustrative embodiments of the present application and serve to explain the principles of the present application, and do not limit the present application in any manner. In the drawings:

[0045] Figure 1 A flowchart of an image classification method in an embodiment of the present application is shown;

[0046] Figure 2 A schematic diagram of an attention mask in an embodiment of the present application is shown;

[0047] Figure 3 A framework diagram of a neural network model in an embodiment of the present application is shown;

[0048] Figure 4 A schematic diagram when a sub-image is segmented in an embodiment of the present application is shown;

[0049] Figure 5 A flowchart of an overall method in an embodiment of the present application is shown;

[0050] Figure 6 A structural schematic diagram of an image classification apparatus in an embodiment of the present application is shown;

[0051] Figure 7Figure 1 is a schematic diagram of a terminal device structure according to an embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to improve the accuracy of the classification result of the image, an image classification method, device, and storage medium are provided in the embodiments of the present application. In order to better understand the technical solutions provided in the embodiments of the present application, the basic principles of the solutions are briefly described as follows:

[0053] With the development of Internet technology, accurate image classification has been paid more and more attention. Especially in the face of multi-label classification tasks, it is very important to accurately determine the probability of the image with respect to each label. The higher the accuracy, the more conducive to subsequent classification tasks. In the prior art, most of the technical solutions for multi-label classification are based on a single-scale convolutional neural network model, that is, a given input image is input into a convolutional neural network, and then the convolutional neural network outputs a recognition result. In the training process, each label is treated as a binary classification task. When using a convolutional neural network for feature extraction, only one image feature is usually extracted for an image, and then the image feature is used for multi-label classification.

[0054] However, due to the great semantic difference between labels, in this case, if the feature extraction step only extracts one image feature, the representative features of multiple labels will be mixed together, causing ambiguity in feature representation, which is not conducive to the final classification task. For example, if an image needs to be classified to determine whether the image contains label 1 and label 2 and the probability value of each label in the image. When label 1 and label 2 appear in the image at the same time, only one image feature is extracted, which contains the content of label 1 and label 2, thus losing the specificity of feature representation, resulting in low accuracy of the classification result, which is not conducive to the subsequent classification task.

[0055] Therefore, in order to improve the accuracy of the image classification result, the embodiments of the present application provide an image classification method, device, electronic device, and storage medium. According to the feature vector corresponding to the label, a plurality of image features of the image to be classified are obtained, and the prediction value of each image feature for each label is determined. Finally, the prediction value of each label is determined by determining the prediction value of the image to be classified. In this way, by obtaining a plurality of image features of the image to be classified and determining the label prediction value of the image to be classified according to the prediction value of the plurality of image features for each label, the accuracy of the prediction value of each label in the image to be classified can be improved, thereby improving the accuracy of image classification.

[0056] Thus, if image classification is needed for an image to determine the probability value of each label in the image, the method provided in the embodiments of the present application can more accurately determine the prediction value of each label. For example, if image classification is needed for an image to determine the prediction value of the image about label 1 and label 2, the image can be feature extracted according to the feature vector corresponding to label 1 to obtain image feature 1, and the image can be feature extracted according to the feature vector corresponding to label 2 to obtain image feature 2. Thus, the prediction value of image feature 1 for label 1 is more accurate, and the prediction value of image feature 2 for label 2 is more accurate, and finally the prediction value of image feature 1 for label 1 and the prediction value of image feature 2 for label 2 are taken as the prediction value of the image about each label.

[0057] The preferred embodiments of the present application are described below in conjunction with the accompanying drawings of the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application, and the embodiments in the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0058] The image classification method provided in the embodiments of the present application is further explained below. As shown in Figure 1 , comprising the following steps:

[0059] S101: Obtain at least two image features of the image to be classified; wherein different image features are obtained according to the feature vectors corresponding to different labels.

[0060] In the embodiments of the present application, the neural network model is modified to output multiple image features.

[0061] First, the neural network model needs to be trained. The process of training is to first determine the feature vector corresponding to each label. And input the sample into the neural network model to be trained, and output the image feature of the sample. Each sample has a label and only contains one label content. Calculate the image feature of the sample and the feature vector of the label corresponding to the sample to obtain the feature error. Then adjust the parameters of the neural network model according to the feature error, so that the feature error between the image feature output by the sample and the feature vector of the label corresponding to the sample is within a predetermined range.

[0062] After the neural network model is trained, the trained neural network model can be used to extract features from the image to be classified, which specifically includes steps A1-A3:

[0063] Step A1: Feature extraction is performed on the image to be classified to obtain the feature map of the image to be classified.

[0064] In this embodiment, a feature map of the image to be classified is obtained through a feature extraction network. That is, convolution calculation is performed on the image to be classified to obtain its feature map.

[0065] Step A2: Perform convolution calculations on the feature map and the feature vector corresponding to each label to obtain the attention mask corresponding to each label.

[0066] The attention mask is a binary image, representing the approximate location of the label within the image to be classified. For example... Figure 2 The diagram shown illustrates an attention mask. The white areas represent the approximate locations of the label within the image to be classified, while the black areas represent the locations of other content within the image.

[0067] In this way, by using attention masks, the distribution of labels in the image to be classified can be determined more clearly.

[0068] Step A3: Perform dot product calculations on the feature map and the attention mask corresponding to each label to obtain at least two image features of the image to be classified.

[0069] In this embodiment, the feature map is multiplied by the attention mask and summed in the spatial domain to obtain the image features of interest to the corresponding label.

[0070] The number of labels in the neural network model corresponds to the number of feature vectors; that is, the number of labels is the same as the number of feature vectors, and they correspond one-to-one.

[0071] like Figure 3 As shown, this is a framework diagram of a neural network model. The image to be classified is processed by a feature extraction network to obtain a feature map. The feature map is then convolved with the feature vectors corresponding to each label to obtain the attention mask for each label. Finally, the attention mask for each label is applied to the feature map to obtain the image features.

[0072] As shown above, the labels are distributed in different positions on the image. Therefore, when obtaining image features of an image to be classified, the image can be first processed into multiple sub-images. Each sub-image contains different information, resulting in different image features. Specifically, this includes the following steps:

[0073] Step B1: Obtain at least two sub-images from the image to be classified, and the stitched image of all sub-images includes all regions of the multi-label image.

[0074] In this embodiment of the application, sub-images can be obtained based on the side lengths of the image to be classified. Specifically:

[0075] Determine the long and short sides of the image to be classified, and determine that the first side of the image to be classified is the short side and the second side is the long side;

[0076] Set a segmentation window with the same size as the sub-image, starting from the short side, and move the segmentation window a set length along the long side each time to obtain the sub-image based on the region of the segmentation window in the image to be classified.

[0077] Since the aspect ratios of the images to be classified vary and the positions of the objects on the images cannot be determined in advance, to make the images conform to the required size, the shorter sides of the images are usually padded with blank pixels (e.g., the padded areas are all black or white), and then scaled before being input into the model for processing. This reduces the focus of the acquired image features.

[0078] Therefore, this application solves the above problems by segmenting molecular images. When segmenting molecular images, this application first determines the long side and the short side of the image, sets the short side as the side length of the segmentation window (wherein, the segmentation window is a square, that is, all sides are the same length), and moves from one short side of the image to be classified to the other short side. During the movement, the corresponding image content in the segmentation window is used as the sub-image.

[0079] It should be noted that the number of sub-images can be selected according to the actual situation. When selecting, you can move the segmentation window by a set length to determine a sub-image, or you can obtain sub-images by random segmentation.

[0080] like Figure 4 As shown, this is a schematic diagram of molecular image cutting. Figure 4 In the diagram, 401 represents the segmentation window. The segmentation window moves from left to right, defining a sub-image by a set distance. After segmentation, three sub-images are obtained.

[0081] Step B2: Perform feature extraction for each sub-image to obtain the sub-image features of the sub-image.

[0082] In the embodiments of this application, conventional feature extraction can be performed on sub-images, that is, one image feature is extracted from one sub-image.

[0083] Step B3: Use the features of each sub-image as the image features of the image to be classified.

[0084] By processing the image to be classified to obtain sub-images and extracting the image features of each sub-image, an image is segmented so that each image has a different focus, resulting in more accurate classification results.

[0085] Of course, in order to determine the predicted label value more accurately, feature extraction can also be performed on multiple sub-images after obtaining multiple sub-images.

[0086] For example, the to-be-classified image is divided into three sub-images by a division window, and multiple image features are extracted from the three sub-images, for example, two image features are extracted from each sub-image, that is, sub-image 1 obtains feature 11 and feature 12, sub-image 2 obtains feature 21 and feature 22, and sub-image 3 obtains feature 31 and feature 32. In this way, six image features can be obtained from one to-be-classified image.

[0087] In order to further improve the accuracy of image classification, on the basis of obtaining multiple image features of multiple sub-images, the image features of the whole image are added, and the image features of the whole image are fused with the image features of the sub-images, so that the image features are more multi-scale. Specifically, steps C1-C5 are included:

[0088] Step C1: obtaining at least two sub-images from the to-be-classified image, and the spliced image of all sub-images includes all regions of the multi-label image.

[0089] Step C2: performing feature extraction on the to-be-classified image to obtain at least two global image features of the to-be-classified image.

[0090] In the embodiments of the present application, the to-be-classified image can be first padded, then scaled, and input into the model for feature extraction.

[0091] Step C3: performing feature extraction on each sub-image to obtain at least two local image features of each sub-image; wherein the number of local image features of each sub-image is the same as the number of global image features, and they correspond to each other.

[0092] It should be noted that the global image features of the to-be-classified image are the same as the image features, and the local image features of the sub-image are the same as the sub-image features.

[0093] In the embodiments of the present application, each image feature is obtained by a feature vector corresponding to a label, so the image features obtained by the feature vectors corresponding to the same label correspond to each other.

[0094] For example, the label to be classified has two, the image to be classified is divided into three sub-images after the window, the image to be classified and three sub-images are processed by feature extraction, the image to be classified obtains global feature 1 and global feature 2, sub-image 1 obtains local feature 11 and local feature 12, sub-image 2 obtains local feature 21 and local feature 22, and sub-image 3 obtains local feature 31 and local feature 32. In this way, 8 image features can be obtained from one image to be classified. Among them, global feature 1, local feature 11, local feature 21 and local feature 31 are obtained by the feature vector corresponding to label 1, and global feature 2, local feature 12, local feature 22 and local feature 32 are obtained by the feature vector corresponding to label 2.

[0095] Step C4: for each local image feature of each sub-image, the local image feature is fused with the global image feature corresponding to the local image feature in the image to be classified, to obtain the fused image feature of the local image feature.

[0096] In the embodiment of the present application, in order to improve the multi-scale nature of the image features (both global features and local features), the obtained image features are fused.

[0097] For example, an image to be classified is obtained by feature extraction 8 image features, which are: global feature 1, local feature 11, local feature 21, local feature 31, global feature 2, local feature 12, local feature 22, and local feature 32.

[0098] Fuse global feature 1 and local feature 11 to obtain fused feature 11, fuse global feature 1 and local feature 21 to obtain fused feature 21, fuse global feature 1 and local feature 31 to obtain fused feature 31, and so on.

[0099] The image features can be fused by the following methods:

[0100] 1. Average value:

[0101] For each local image feature of each sub-image, the average value of the local image feature and the global image feature corresponding to the local image feature in the image to be classified is taken as the fused image feature.

[0102] For example, the feature vector of global feature 1 is 1, 4, 8, 6, 5, 2; the feature vector of local feature 11 is 4, 5, 6, 7, 2, 1; after averaging, the fused feature 11 is 2.5, 4.5, 7, 6, 3.5, 1.5.

[0103] 2. Maximum value:

[0104] For each local image feature of each sub-image, the local image feature is combined with the maximum value of each dimension of the global image feature corresponding to the local image feature in the image to be classified, and the combination result is taken as the fusion image feature.

[0105] For example: the feature vector of global feature 1 is 1, 4, 8, 6, 5, 2; the feature vector of local feature 11 is 4, 5, 6, 7, 2, 1; after taking the maximum value, the obtained fusion feature 11 is 4, 5, 8, 7, 5, 2.

[0106] 3. Classification selection is performed through the full connection layer:

[0107] For each local image feature of each sub-image, the local image feature is combined with the maximum value of each dimension of the global image feature corresponding to the local image feature in the image to be classified, and the combination result is taken as the fusion image feature.

[0108] For example: the feature vector of global feature 1 is 1, 4, 8, 6, 5, 2; the feature vector of local feature 11 is 4, 5, 6, 7, 2, 1; the two feature vectors are connected to obtain a new feature vector: 1, 4, 8, 6, 5, 2, 4, 5, 6, 7, 2, 1; the new feature vector is input to the full connection network for classification to obtain a more complex fusion feature.

[0109] Step C5: taking the fusion image features of the sub-images as the image features of the image to be classified.

[0110] In this way, the image features of the whole image and the sub-image are combined, and the accuracy of image classification is further improved.

[0111] S102: For each label, the prediction value of each image feature corresponding to the label is obtained respectively.

[0112] In the embodiment of the present application, after obtaining the image features to be classified, the prediction values of each feature corresponding to each label are obtained through the classifier.

[0113] For example: four image features are obtained through the image to be classified, which are feature 11, feature 12, feature 21 and feature 22. Among them, feature 11 and feature 21 are obtained through label 1, and feature 12 and feature 22 are obtained through label 2. The prediction values of each image feature are obtained through the classifier. The prediction value of feature 11 is 78; the prediction value of feature 12 is 83; the prediction value of feature 21 is 15; the prediction value of feature 22 is 20; 80. Therefore, the prediction values of each image feature for label 1 are 78, 83, 15, 20 respectively; the prediction values of each image feature for label 2 are 22, 17, 85, 80 respectively.

[0114] S103: For each label, determine the target predicted value of the label based on the predicted values ​​of each image feature corresponding to that label.

[0115] In this embodiment of the application, after determining the predicted value of each image feature corresponding to the label, the determined predicted value can be used as the target predicted value of the label.

[0116] For example, the average of the predicted values ​​of each label can be used as the target predicted value of that label. For instance, if the predicted values ​​of each image feature of label 1 are 78, 83, 15, and 20, then the target predicted value of label 1 is (78+83+15+20) / 4 = 49.

[0117] Alternatively, a threshold can be set, and the average of the predicted values ​​greater than the threshold can be used as the target predicted value for the label. For example, if the threshold is set to 50, and the predicted values ​​of each image feature of label 1 are 78, 83, 15, and 20 respectively, then the target predicted value of label 1 is (78+83) / 2 = 80.5.

[0118] Of course, to make the final prediction more accurate, the maximum value corresponding to each label can be used as the predicted value for that label. Specifically: for each label, the maximum value among the predicted values ​​of each image feature corresponding to that label is determined as the target predicted value for that label.

[0119] For example, the predicted values ​​for each image feature of label 1 are 78, 83, 15, and 20, respectively; and the predicted values ​​for each image feature of label 2 are 22, 17, 85, and 80, respectively. Therefore, the target predicted value for label 1 is 83, and the target predicted value for label 2 is 85.

[0120] S104: Use the target prediction value of each label as the label prediction value of the image to be classified.

[0121] S105: Classify the image to be classified based on the predicted label values ​​of the image to be classified.

[0122] As described above, the predicted label values ​​for the image to be classified are 83 and 85, and the image is classified based on these predicted label values. That is, the image to be classified has an 83% probability of being labeled 1 and an 85% probability of being labeled 2.

[0123] In this way, by acquiring multiple image features of the image to be classified and determining the predicted label value of the image to be classified based on the predicted values ​​of the multiple image features for each label, the accuracy of the predicted label value in the image to be classified can be improved, thereby improving the accuracy of image classification.

[0124] like Figure 5 The diagram shown is an overall flowchart of an embodiment of this application. Figure 5In the embodiment, the image to be classified is divided into three sub-images, and the three sub-images are input into the feature extraction network, and two image features are obtained for each sub-image, and six image features are obtained in total. The label prediction values of the six features are obtained by the classifier, and the maximum value in each label prediction value is taken as the label prediction value of the image to be classified.

[0125] Based on the same inventive concept, the embodiment of the present application also provides an image classification device. As shown in the figure, the device comprises: Figure 6

[0126] The first obtaining module 601 is configured to obtain at least two image features of an image to be classified; wherein different image features are obtained according to feature vectors corresponding to different labels;

[0127] The second obtaining module 602 is configured to obtain, for each label, a prediction value of each image feature corresponding to the label;

[0128] The first determining module 603 is configured to determine, for each label, a maximum value in the prediction value of each image feature corresponding to the label as a target prediction value of the label;

[0129] The second determining module 604 is configured to take the target prediction values of the labels as label prediction values of the image to be classified;

[0130] The classification module 605 is configured to classify the image to be classified according to the label prediction values of the image to be classified.

[0131] Optionally, the first obtaining module 601 comprises:

[0132] The sub-image determining unit is configured to obtain at least two sub-images from the image to be classified, and the spliced image of all the sub-images comprises all regions of the multi-label image;

[0133] The first feature extraction unit is configured to perform feature extraction on each sub-image to obtain a sub-image feature of the sub-image;

[0134] The first image feature determining unit is configured to take the sub-image features as image features of the image to be classified.

[0135] Optionally, the first obtaining module 601 comprises:

[0136] The sub-image determining unit is configured to obtain at least two sub-images from the image to be classified, and the spliced image of all the sub-images comprises all regions of the multi-label image;

[0137] The second feature extraction unit is configured to perform feature extraction on the image to be classified to obtain at least two global image features of the image to be classified; and

[0138] ​The third feature extraction unit is configured to perform feature extraction on each sub-image to obtain at least two local image features of each sub-image; wherein the number of local image features of each sub-image is the same as the number of global image features, and the local image features correspond to each other;

[0139] The fusion unit is configured to, for each local image feature of each sub-image, fuse the local image feature with the global image feature corresponding to the local image feature in the image to be classified to obtain a fused image feature of the local image feature;

[0140] The second determination image feature unit is configured to take the fused image features of the sub-images as image features of the image to be classified.

[0141] Optionally, the fused image features are obtained by the following apparatus:

[0142] The first fusion sub-unit is configured to, for each local image feature of each sub-image, take the average value of the local image feature and the global image feature corresponding to the local image feature in the image to be classified as a fused image feature; or

[0143] The second fusion sub-unit is configured to, for each local image feature of each sub-image, combine the local image feature with the maximum value of each dimension of the global image feature corresponding to the local image feature in the image to be classified, and take the combination result as a fused image feature; or

[0144] The third fusion sub-unit is configured to, for each local image feature of each sub-image, connect the local image feature with the global image feature corresponding to the local image feature in the image to be classified, and input the connection result into a fully connected network model to output a fused image feature.

[0145] Optionally, the determination sub-image unit comprises:

[0146] The determination edge sub-unit is configured to determine the long edge and the short edge of the image to be classified, determine the first edge of the image to be classified as the short edge, and determine the second edge as being smaller than the long edge.

[0147] The determination sub-image sub-unit is configured to set a cutting window with the same size as the sub-image, start from the short edge, move the cutting window by a set length along the long edge each time, and obtain the sub-image according to the region of the cutting window in the image to be classified.

[0148] Optionally, the first acquisition module 601 comprises:

[0149] The fourth feature extraction unit is configured to perform feature extraction on the image to be classified to obtain a feature map of the image to be classified.

[0150] a convolution unit, configured to perform convolution calculation on the feature map and the feature vector corresponding to each label respectively to obtain an attention mask corresponding to each label;

[0151] a third determination image feature unit, configured to perform point multiplication calculation on the feature map and the attention mask corresponding to each label respectively to obtain at least two image features of the image to be classified.

[0152] Based on the same technical concept, the embodiments of the present application also provide a terminal device 700, as shown in Figure 7 , the terminal device 700 is used to implement the method described in each of the above method embodiments, for example, the embodiment shown in Figure 2 , the terminal device 700 can include a memory 701, a processor 702, an input unit 703 and a display panel 704.

[0153] The memory 701 is used to store the computer program executed by the processor 702. The memory 701 can mainly include a program storage area and a data storage area, wherein the program storage area can store the operating system, the application program required by at least one function, etc.; the data storage area can store the data created according to the use of the terminal device 700, etc. The processor 702 can be a central processing unit (CPU) or a digital processing unit, etc. The input unit 703 can be used to obtain the user instruction input by the user. The display panel 704 is used to display the information input by the user or provided to the user. In the embodiments of the present application, the display panel 704 is mainly used to display the display interface of each application program in the terminal device and the control entity displayed in each display interface. Optionally, the display panel 704 can be configured in the form of a liquid crystal display (LCD) or an OLED (organic light-emitting diode).

[0154] In the embodiments of the present application, the specific connection medium between the above-mentioned memory 701, processor 702, input unit 703 and display panel 704 is not limited. In the embodiments of the present application, the memory 701, processor 702, input unit 703 and display panel 704 are connected through a bus 705, and the bus 705 is represented by a thick line in Figure 7 , the connection mode between other components is only schematically illustrated and is not limited. The bus 705 can be divided into an address bus, a data bus, a control bus, etc. For convenience of representation, only one thick line is used in Figure 7 , but it does not mean that there is only one bus or only one type of bus. Figure 7

[0155] ​The memory 701 can be a volatile memory (volatile memory), such as a random access memory (RAM); the memory 701 can also be a non-volatile memory (non-volatile memory), such as a read-only memory, a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD), or the memory 701 is any other medium capable of carrying or storing desired program code in the form of instructions or data structures and capable of being accessed by a computer, but not limited to this. The memory 701 can be a combination of the above-mentioned memories.

[0156] The processor 702 is configured to implement various embodiments as described above. Figure 1 The processor 702 is configured to implement various embodiments as described above.

[0157] The processor 702 is configured to call the computer program stored in the memory 701 to execute the embodiments as shown in the embodiment. Figure 1 The processor 702 is configured to call the computer program stored in the memory 701 to execute the embodiments as shown in the embodiment.

[0158] The embodiments of the present application also provide a computer readable storage medium storing computer executable instructions required for the processor to execute the above-mentioned processor, which contains programs required for the processor to execute.

[0159] In some possible implementation manners, various aspects of the image classification method provided by the present application can also be implemented in the form of a program product, which includes program codes for causing the terminal device to execute the steps in the image classification method according to various exemplary embodiments of the present application described above in the specification when the program product runs on the terminal device. For example, the terminal device can execute the embodiments as shown in the embodiment. Figure 1 The processor 702 is configured to implement various embodiments as described above.

[0160] The program product can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium may, for example, be but is not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0161] The program product of the embodiments of the present application for an image classification program can employ a portable compact disc read only memory (CD-ROM) and include a program code, and can be run on a computing device. However, the program product of the present application is not limited thereto, and in the present document, a readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0162] The readable signal medium can include a data signal propagated in a baseband or as a carrier wave in a propagation medium, in which a readable program code is borne. Such a propagated data signal can take a variety of forms, including but not limited to electro-magnetic signal, optical signal, or any suitable combination thereof. The readable signal medium can also be any readable medium that is not a readable storage medium and that can be used to carry or transmit a program for use by or in connection with an instruction execution system, apparatus, or device.

[0163] The program code contained in the readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0164] The program code for performing the operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++, etc., and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's device and partly on a remote computing device or entirely on the remote computing device or server. In the latter scenario, the remote computing device can be connected to the user's device through any kind of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computing device, such as through the Internet using an Internet Service Provider.

[0165] It should be noted that although several units or sub-units of the apparatus are mentioned in the above detailed description, such division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into units for embodiment.

[0166] Moreover, although the operations of the method(s) herein can be described in a particular, sequential order, this order is not meant to be a limitation and

[0167] Those of skill in the art would understand that embodiments of the present application can be provided as a method, system or computer program product. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer readable storage media (including, but not limited to, disk memory, CD-ROMs, optical storage devices, etc.) embodying computer readable program code.

[0168] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable image classification apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the Figure 1 function specified in the flow Figure 1 block or blocks.

[0169] These computer program instructions can also be loaded onto a computer or other programmable image classification apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the Figure 1 function specified in the flow Figure 1 block or blocks.

[0170] While the preferred embodiments of the application have been described, additional variations and modifications can be made to the embodiments described and shown, and it is therefore intended that the appended claims cover all such variations and modifications that come within the scope of the application.

[0171] Obviously, numerous modifications and variations of the present application are possible in light of the above teachings. It is therefore to be understood that within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described.

Claims

1. An image classification method, characterized in that, The method includes: Obtain at least two image features of the image to be classified; wherein, different image features are obtained based on the feature vectors corresponding to different labels; For each label, obtain the predicted value of each image feature corresponding to that label; For each label, the maximum value among the predicted values ​​of each image feature corresponding to that label is determined as the target predicted value for that label; The target predicted value of each label is used as the label predicted value of the image to be classified; The image to be classified is classified based on the predicted label value of the image to be classified; The acquisition of at least two image features of the image to be classified includes: At least two sub-images are obtained from the image to be classified, and the stitched image of all sub-images includes all regions of the image to be classified; Feature extraction is performed on each sub-image to obtain the sub-image features of each sub-image; the sub-image features of each sub-image include the local image features corresponding to each label; the local image features corresponding to each label are the image features of each sub-image under each label; Each sub-image feature is used as the image feature of the image to be classified. The process of obtaining at least two image features of the image to be classified further includes: Feature extraction is performed on the image to be classified to obtain the feature map of the image to be classified; The feature map is convolved with the feature vector corresponding to each label to obtain the attention mask corresponding to each label. The feature map is multiplied by the attention mask corresponding to each label to obtain at least two image features of the image to be classified.

2. The method according to claim 1, characterized in that, Before acquiring at least two image features of the image to be classified, the method further includes: Obtain the image to be classified; The process of obtaining at least two image features of the image to be classified further includes: Feature extraction is performed on the image to be classified to obtain at least two global image features of the image to be classified; wherein the number of local image features of each sub-image is the same as the number of global image features, and they correspond to each other; For each local image feature of each sub-image, the local image feature is fused with the global image feature corresponding to the local image feature in the image to be classified to obtain the fused image feature of the local image feature; The step of using the features of each sub-image as the image features of the image to be classified includes: The fused image features of each sub-image are used as the image features of the image to be classified.

3. The method according to claim 2, characterized in that, The fused image features are obtained using the following method: For each local image feature of each sub-image, the average value of that local image feature and the global image features corresponding to that local image feature in the image to be classified is used as the fused image feature; or, For each local image feature of each sub-image, the local image feature is combined with the maximum value of each dimension of the global image feature corresponding to that local image feature in the image to be classified, and the combined result is used as the fused image feature; or, For each local image feature of each sub-image, the local image feature is concatenated with the global image feature corresponding to that local image feature in the image to be classified, and then input into the fully connected network model to output the fused image feature.

4. The method according to claim 1 or 2, characterized in that, Obtaining at least two sub-images from the image to be classified includes: Determine the long side and short side of the image to be classified, and determine the first side of the image to be classified as the short side and the second side as the long side; A segmentation window with the same size as the sub-image is set. Starting from the short side, the segmentation window is moved a set length along the long side each time. The sub-image is obtained in the region of the image to be classified according to the segmentation window.

5. An image classification device, characterized in that, The device includes: The first acquisition module is used to acquire at least two image features of the image to be classified; wherein, different image features are obtained based on feature vectors corresponding to different labels; The second acquisition module is used to obtain the predicted value of each image feature corresponding to each label for each label; The first determining module is used to determine the target predicted value of each label based on the predicted values ​​of each image feature corresponding to that label. The second determining module is used to use the target predicted value of each label as the label predicted value of the image to be classified. A classification module is used to classify the image to be classified based on the predicted label value of the image to be classified; The first acquisition module includes: A sub-image unit is determined to obtain at least two sub-images from the image to be classified, and the stitched image of all sub-images includes all regions of the image to be classified; The first feature extraction unit is used to extract features for each sub-image to obtain sub-image features for each sub-image; the sub-image features for each sub-image include local image features corresponding to each label; the local image features corresponding to each label are the image features of each sub-image under each label; The first image feature determination unit is used to use the features of each sub-image as the image features of the image to be classified. The first determining module is further configured to: for each label, determine the maximum value among the predicted values ​​of each image feature corresponding to the label as the target predicted value of the label; The first acquisition module further includes: The fourth feature extraction unit is used to extract features from the image to be classified to obtain a feature map of the image to be classified. The convolutional unit is used to perform convolution calculations on the feature map and the feature vector corresponding to each label respectively to obtain the attention mask corresponding to each label; The third image feature determination unit is used to perform dot product calculations on the feature map and the attention mask corresponding to each label to obtain at least two image features of the image to be classified.

6. The apparatus according to claim 5, characterized in that, The device further includes: Image acquisition module, used to acquire the image to be classified; The first acquisition module further includes: The second feature extraction unit is used to extract features from the image to be classified, and obtain at least two global image features of the image to be classified; wherein the number of local image features of each sub-image is the same as the number of global image features, and they correspond to each other; The fusion unit is used to fuse each local image feature of each sub-image with the global image feature corresponding to the local image feature in the image to be classified, so as to obtain the fused image feature of the local image feature. The first image feature determination unit is further configured to use the fused image features of each sub-image as the image features of the image to be classified.

7. The apparatus according to claim 6, characterized in that, The device further includes a first fusion subunit, a second fusion subunit, or a third fusion subunit; The first fusion subunit is used to take the average value of the local image feature and the global image feature corresponding to the local image feature in the image to be classified as the fused image feature for each local image feature of each sub-image. The second fusion subunit is used to combine each local image feature of each sub-image with the maximum value of each dimension of the global image feature corresponding to the local image feature in the image to be classified, and use the combination result as the fused image feature. The third fusion subunit is used to connect each local image feature of each sub-image with the global image feature corresponding to that local image feature in the image to be classified, and input it into the fully connected network model to output the fused image feature.

8. The apparatus according to claim 5 or 6, characterized in that, The determined sub-image unit includes: An edge sub-unit is defined to determine the long and short sides of the image to be classified, wherein the first side of the image to be classified is the short side and the second side is the long side; A sub-image sub-unit is determined for setting a segmentation window with the same size as the sub-image. Starting from the short side, the segmentation window is moved by a set length along the long side each time, and the sub-image is obtained in the region of the image to be classified according to the segmentation window.

9. An electronic device, characterized in that, It includes a processor and a memory, wherein the memory stores program code that, when executed by the processor, causes the processor to perform the steps of any one of the methods described in claims 1 to 4.

10. A computer-readable storage medium, characterized in that, It includes program code that, when run on an electronic device, causes the electronic device to perform the steps of the method of any one of claims 1 to 4.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by an electronic device, it implements the steps of the method according to any one of claims 1 to 4.

Images