Image classification method, apparatus, device, and medium

Abstract

The application relates to the field of data processing and provides an image classification method, device, equipment and medium, which comprises the following steps: performing encoding processing on input text to obtain a text feature matrix; performing extraction processing on an input image to obtain a visual feature set; performing matching processing on the text feature matrix and the visual feature set based on a matching strategy to obtain a similarity distribution; performing optimization processing on the visual feature set and the similarity distribution based on an optimization strategy to obtain an image feature semantic center; performing prediction processing based on the image feature semantic center and the text feature matrix to obtain a prediction result; and performing classification processing on the input image according to the prediction result. Through the matching, optimization and prediction processing on the text feature matrix and the visual feature set in the embodiment of the application, the classification accuracy can be improved, the time consumption in the test process can be reduced, the image classification can be accurately processed, and the processing efficiency can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data processing, and in particular to an image classification method, device, equipment and medium. BACKGROUND

[0002] At present, existing large visual language models (such as CLIP) have achieved excellent performance in zero-shot classification, image retrieval and other tasks by aligning images and natural language labels in a shared latent space. Such models usually use large-scale image-text paired data for pre-training, thereby having cross-modal alignment capability. However, this performance often depends on the premise that the training data and the test data have similar distributions. Once faced with Out-of-Distribution (OOD) data or containing unknown classes that do not appear in the training set, the accuracy will decrease significantly. This is mainly because the feature distribution and decision boundary formed by the model in the pre-training stage may fail under the new data distribution, resulting in unstable semantic alignment and even Spurious Correlation problems.

[0003] At the same time, the model is limited by the pre-training data distribution and cannot accurately identify when facing fine-grained classification tasks or unseen classes. Due to the decision bias in the pre-training process, the model is prone to rely on non-essential features such as background for judgment during inference, resulting in false predictions when there are spurious causal relationships in the test samples. The existing TPT (Test-time Prompt Tuning, TPT) method needs an additional optimization process and forward propagation again in addition to the regular forward propagation, significantly increasing the overall running time and affecting the inference efficiency. After the previous TPT method is enhanced by multiple views, the views are filtered according to the entropy of the view prediction distribution, and a fixed hard threshold is usually set to retain low-entropy views and discard high-entropy views. Therefore, the existing image classification solution has limited fine-grained recognition ability, spurious causal dependence, and low running efficiency, which further reduces the processing efficiency. SUMMARY

[0004] Embodiments of the present application provide an image classification method, device, equipment and medium, aiming to solve the problem of limited fine-grained recognition ability, spurious causal dependence, and low running efficiency of the existing image classification solution, which further reduces the processing efficiency.

[0005] In a first aspect, embodiments of the present application provide an image classification method, which comprises:

[0006] encoding the input text to obtain a text feature matrix;

[0007] extracting the input image to obtain a visual feature set;

[0008] match the text feature matrix and the visual feature set based on a matching strategy to obtain a similarity distribution;

[0009] optimize the visual feature set and the similarity distribution based on an optimization strategy to obtain an image feature semantic center;

[0010] perform prediction processing based on the image feature semantic center and the text feature matrix to obtain a prediction result;

[0011] perform classification processing on the input image according to the prediction result.

[0012] In a second aspect, an image classification apparatus is also provided, and the apparatus comprises:

[0013] a coding unit configured to perform coding processing on input text to obtain a text feature matrix;

[0014] an extraction unit configured to perform extraction processing on an input image to obtain a visual feature set;

[0015] a matching unit configured to match the text feature matrix and the visual feature set based on a matching strategy to obtain a similarity distribution;

[0016] an optimization unit configured to optimize the visual feature set and the similarity distribution based on an optimization strategy to obtain an image feature semantic center;

[0017] a prediction unit configured to perform prediction processing based on the image feature semantic center and the text feature matrix to obtain a prediction result;

[0018] a classification unit configured to perform classification processing on the input image according to the prediction result.

[0019] In a third aspect, an electronic device is also provided, which comprises a memory and a processor, the memory has stored thereon a computer program, and the processor implements the method of the first aspect when executing the computer program.

[0020] In a fourth aspect, a computer readable storage medium is also provided, which stores a computer program, the computer program comprises program instructions, and the program instructions can implement the method of the first aspect when executed by a processor.

[0021] The application provides an image classification method, device, equipment and medium. The method comprises: performing encoding processing on input text to obtain a text feature matrix; performing extraction processing on an input image to obtain a visual feature set; performing matching processing on the text feature matrix and the visual feature set based on a matching strategy to obtain a similarity distribution; performing optimization processing on the visual feature set and the similarity distribution based on an optimization strategy to obtain an image feature semantic center; performing prediction processing on the image feature semantic center and the text feature matrix to obtain a prediction result; and performing classification processing on the input image according to the prediction result. The text feature matrix and the visual feature set are matched, optimized and predicted in the embodiment of the application, so that the classification accuracy is improved, the time consumption of the test process is reduced, the image classification is accurately processed, and the processing efficiency is improved. BRIEF DESCRIPTION OF DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0023] Figure 1 The flowchart of the image classification method provided by the embodiment of the present application is shown.

[0024] Figure 2 The schematic block diagram of the image classification device provided by the embodiment of the present application is shown.

[0025] Figure 3 The schematic block diagram of the electronic device provided by the embodiment of the present application is shown. DETAILED DESCRIPTION

[0026] The technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the present application.

[0027] It should be understood that when used in the specification and the appended claims, the terms "comprise" and "include" indicate the presence of described features, integers, steps, operations, elements, and / or components, but do not exclude one or more other features, integers, steps, operations, elements, components, and / or sets thereof.

[0028] It should also be understood that the terms used herein are for the purpose of describing particular embodiments and are not intended to limit the application. As used in the specification and the appended claims, the singular forms "a," "an" and "the" are intended to include plural forms unless the context clearly dictates otherwise.

[0029] It should be further understood that the term "and / or" used in the specification and the appended claims means one or more of the associated listed items as well as all possible combinations of the items and includes the combinations.

[0030] Reference Figure 1 , Figure 1 The flowchart of the image classification method provided by the embodiment of the application is shown in FIG. 1. As shown in FIG. 1, the method comprises the following steps S110-S160. Figure 1

[0031] S110, encoding processing is performed on the input text to obtain a text feature matrix.

[0032] In the embodiment, the input text is a text input by a user. For example, the input text can be "visual description of a rattlesnake".

[0033] In an embodiment, the step S110, that is, the encoding processing on the input text to obtain a text feature matrix, comprises:

[0034] performing recognition processing on the input text to obtain a plurality of categories;

[0035] performing generation processing on each category to obtain a plurality of attribute descriptions;

[0036] performing first encoding processing on each attribute description by using a text encoder to obtain a feature vector;

[0037] integrating the feature vectors of the plurality of attribute descriptions corresponding to each category to obtain a text feature of each category;

[0038] performing normalization processing on the text features of all categories to obtain the text feature matrix.

[0039] ​In this embodiment, the input text can be obtained by inputting it into a large language model, a manually constructed attribute description vocabulary, a domain knowledge base (such as WordNet or ImageNet attribute annotations), or by web scraping. The input text is then processed to identify the several categories, and several attribute descriptions are generated for each category. The large language model can be GPT (Generative Pre-trained Transformer), etc.; the several categories can be K categories, where K is greater than or equal to 1; and for each category, several attribute descriptions are generated. Attribute description (the first category has) The second category has attribute descriptions. The attribute descriptions are as follows (and so on). For example, the attribute descriptions may include "horned rattlesnake is sandy yellow or light brown," "horned rattlesnake has irregular light-colored spots," "horned rattlesnake has a rattlesnake tail segment at the end and can emit a warning sound," "horned rattlesnake has rough scales on its back with weak ridges," etc.; one input text corresponds to multiple categories, and one category corresponds to multiple attribute descriptions. The total number of attribute descriptions for the various categories is... These attribute descriptions can be denoted as a set. .

[0040] Each attribute description is first encoded using a text encoder to obtain a feature vector. Specifically, several attribute descriptions for each category are input into the text encoder. In this process, the several attribute descriptions are encoded into feature vectors, and the feature vectors corresponding to the several attribute descriptions of each category are integrated to obtain the text features of each category, so as to perform text feature analysis on all categories. Normalization To obtain the text feature matrix The text encoder can be a CLIP (Contrastive Language-Image Pre-training) text encoder. , refers to the The embedding vector describes the attribute; , refers to the feature matrix describing the attributes of all categories after normalization; The dimension of the feature vector; The total number of attribute descriptions for all categories; This is the Euclidean norm, also known as the second norm.

[0041] The text encoder can also be replaced by a BERT, RoBERTa, DeBERTa, ALBERT, etc. language model.

[0042] Through the above embodiments, it can be known that the input text is subjected to recognition processing to obtain a plurality of categories; each category is subjected to generation processing to obtain a plurality of attribute descriptions; each attribute description is subjected to first encoding processing by a text encoder to obtain a feature vector; the feature vectors of the plurality of attribute descriptions corresponding to each category are integrated to obtain a text feature of each category; and the text features of all categories are subjected to normalization processing to obtain the text feature matrix. Therefore, the determination of the text feature matrix is realized through the recognition, generation, encoding, integration, and normalization processing of the input text, the fine-grained recognition performance is significantly enhanced, and the processing efficiency is improved.

[0043] In step S120, the input image is subjected to extraction processing to obtain a visual feature set.

[0044] In this embodiment, after the input image is obtained, the input image is subjected to extraction processing to obtain the visual feature set.

[0045] In an embodiment, in step S120, i.e., the extraction processing of the input image to obtain a visual feature set, includes:

[0046] A preset cropping size is obtained;

[0047] A first preset function is used to perform local cropping processing on the input image according to the cropping size to obtain a plurality of local images;

[0048] Each local image is subjected to second encoding processing by a visual encoder to obtain a local visual feature;

[0049] The local visual features of each local image form the visual feature set.

[0050] In this embodiment, for the input image , the first preset function, i.e., the local cropping function , is used to perform local cropping processing to generate a local image, i.e., a plurality of local images; the formula corresponding to the first preset function can be ; wherein is a relative size of a cropping region, i.e., a preset cropping size; represents a local region randomly cropped from an original image.

[0051] Each local image is input to the visual encoder In the embodiment, the visual features of the input image are extracted, i.e., the local visual features can be represented as .

[0052] The local visual features of each local image are combined into the visual feature set, i.e., the visual feature set includes a plurality of local visual features.

[0053] The visual encoder can be a mainstream architecture such as ViT, Swin Transformer, ResNet, or even a lightweight network such as MobileNet to adapt to low-computing scenarios.

[0054] According to the above embodiment, the preset cropping size is obtained; the first preset function is used to perform local cropping processing on the input image according to the cropping size to obtain a plurality of local images; the second encoding processing is performed on each local image according to the visual encoder to obtain the local visual features; and the local visual features of each local image form the visual feature set. Therefore, the determination of the visual feature set is realized through the local cropping and encoding processing of the input image, the fine-grained recognition performance is significantly enhanced, and the processing efficiency is improved.

[0055] In step S130, the text feature matrix and the visual feature set are matched based on a matching strategy to obtain a similarity distribution.

[0056] In the embodiment, after the encoding processing of the input text obtains the text feature matrix and the extraction processing of the input image obtains the visual feature set, the text feature matrix and the visual feature set can be matched based on a matching strategy to obtain a similarity distribution.

[0057] In an embodiment, in step S130, i.e., the text feature matrix and the visual feature set are matched based on a matching strategy to obtain a similarity distribution, which includes:

[0058] The similarity between each local visual feature in the visual feature set and each element in the text feature matrix is calculated;

[0059] The similarity and the corresponding local visual feature are calculated using a second preset function to obtain a semantic matching distribution;

[0060] A plurality of other local visual features in the visual feature set except for each local visual feature are obtained, and the semantic matching distribution of each local visual feature is taken as a first matching distribution, and the semantic matching distribution of each corresponding other local visual feature is taken as a second matching distribution;

[0061] The third preset function is used to calculate and process the first matching distribution and each corresponding second matching distribution to obtain the similarity distribution.

[0062] In the embodiment, each local visual feature in the visual feature set is projected to the text semantic space to calculate and process each local visual feature in the visual feature set to obtain the similarity ; the second preset function is used to calculate and process the similarity and the corresponding local visual feature to obtain the semantic matching distribution; the formula corresponding to the second preset function is ; wherein, is the text feature matrix; is the local visual feature , and the similarity of the local visual feature to the feature vector (element) of each attribute description is divided by is the feature normalization; is the local visual feature , and the matching score distribution of the local visual feature to the feature vector of each attribute description of all categories.

[0063] The other local visual features in the visual feature set except each local visual feature are obtained, and the semantic matching distribution of each local visual feature is taken as the first matching distribution , and the semantic matching distribution of each corresponding other local visual feature is taken as the second matching distribution ; the third preset function is used to calculate and process the first matching distribution and each corresponding second matching distribution to obtain the similarity distribution ; wherein, the formula corresponding to the third preset function is ; wherein, is to compare whether the semantic matching distributions of two local image regions are similar, which reflects the closeness of the two local image regions in the text semantic space (i.e., the text feature matrix). In simple terms: first, let each local image region "see" all the attribute descriptions to obtain its semantic direction, and then use these semantic distributions to compare the similarity of different local image regions.

[0064] In the process, represents the matching score distribution of the i-th region to the feature vector of all attribute descriptions, each component of represents the matching score of the local image region to a certain attribute description The matching score of each category. Since each category usually corresponds to multiple attribute descriptions, a single local image region is not compared with several attribute descriptions of only one category (traditional 1NN), but is compared with several attribute descriptions of multiple categories (KNN, where K is equal to the total number of attribute descriptions of all categories ).

[0065] Directly calculating the similarity between image regions in the image feature space is easily disturbed by non-essential factors such as background and posture, and the semantic similarity is unstable; after mapping the local visual features to the text semantic space, the representation of each local image region is composed of the matching results with the category attribute descriptions, which can more accurately reflect its semantic connotation.

[0066] This way can make different image regions present different fine-grained features of the same category, but their semantic matching distributions are still highly consistent. For example, for the "King Cobra" category: one region contains head features (such as the unique neck expansion pattern), and the matching points to "King Cobra"; another region contains the texture of body scales, which is obviously different in vision, but also matches to "King Cobra".

[0067] Under the similarity calculation in the semantic space, these regions will be identified as semantically consistent, thereby realizing cross-regional fine-grained feature aggregation and improving the accuracy and robustness of fine-grained classification.

[0068] Through the above embodiments, it can be known that the similarity is obtained by calculating and processing each local visual feature in the visual feature set and each element in the text feature matrix; the semantic matching distribution is obtained by calculating and processing the similarity and the corresponding local visual feature using a second preset function; a plurality of other local visual features in the visual feature set except for each local visual feature are obtained, and the semantic matching distribution of each local visual feature is taken as a first matching distribution, and the semantic matching distribution of each corresponding other local visual feature is taken as a second matching distribution; the similarity distribution is obtained by calculating and processing the first matching distribution and each corresponding second matching distribution using a third preset function. Therefore, the local visual features are mapped to the text semantic space, and the matching distribution with the attribute description is used to replace the original visual feature for inter-regional similarity measurement, which fundamentally reduces the influence of background and environmental noise on classification decision, establishes a cross-modal matching distribution between local image regions and all attribute descriptions, realizes fine-grained semantic modeling based on multi-attribute KNN matching, and through this cross-modal attribute alignment, even if different regions present different fine-grained features of the same category (such as head patterns and scale textures), their semantic distributions are still highly consistent, thereby significantly enhancing the fine-grained recognition performance and improving the processing efficiency.

[0069] S140, performing optimization processing on the set of visual features and the similarity distribution based on an optimization strategy to obtain an image feature semantic center.

[0070] In this embodiment, after the set of visual features and the similarity distribution are obtained, the set of visual features and the similarity distribution can be optimized based on an optimization strategy to obtain an image feature semantic center.

[0071] In an embodiment, the optimization processing on the set of visual features and the similarity distribution based on an optimization strategy to obtain an image feature semantic center comprises:

[0072] defining an optimization objective function and obtaining an initial semantic center and an initial weight coefficient;

[0073] updating the initial semantic center and the initial weight coefficient using a block coordinate descent algorithm and the similarity distribution to obtain an updated semantic center and an updated weight coefficient;

[0074] inputting the updated semantic center, the updated weight coefficient, and the similarity distribution into the optimization objective function to perform optimization processing on the updated semantic center and the updated weight coefficient using the optimization objective function to obtain a current semantic center and a current weight coefficient;

[0075] continuing to perform the next round of optimization processing on the current semantic center and the current weight coefficient using the block coordinate descent algorithm and the optimization objective function until the current semantic center converges, and taking the converged current semantic center as the image feature semantic center.

[0076] In an embodiment, the updating of the initial semantic center and the initial weight coefficient using a block coordinate descent algorithm and the similarity distribution to obtain an updated semantic center and an updated weight coefficient comprises:

[0077] calculating the similarity distribution, the initial semantic center, and the initial weight coefficient using a fourth preset function to obtain the updated weight coefficient;

[0078] calculating the similarity distribution, the initial semantic center, and the initial weight coefficient using a fifth preset function to obtain the updated semantic center.

[0079] In this embodiment, the optimization objective function is defined, and the initial semantic center and the initial weight coefficient are obtained. Specifically, the optimization objective function is defined as The initial semantic center and the initial weight coefficients are obtained; wherein, the initial semantic center refers to the original image, i.e., the input image; the initial weight coefficients are the weights of each local visual feature, and all initial weight coefficients are 1; The value is 0.2.

[0080] The initial semantic center and the initial weight coefficients are updated using a block coordinate descent algorithm and the similarity distribution to obtain updated semantic centers and updated weight coefficients. Specifically, firstly, the semantic center is fixed, and the updated weight coefficients are obtained by calculating the similarity distribution, the initial semantic center, and the initial weight coefficients using a fourth preset function; then, the weight coefficients are fixed, and the updated semantic center is obtained by calculating the similarity distribution, the initial semantic center, and the initial weight coefficients using a fifth preset function.

[0081] Wherein, the fourth preset function is ;in, Iterate through all local visual features; This indicates the current iteration round, i.e., the step number for weight updating; It is the first During the first iteration The weight coefficients of a local visual feature, i.e., the current weight coefficients; It is the first During the round of iteration, except for the first The weight coefficient of each of several other local visual features in addition to the local visual feature; It is based on and the current semantic center The updated weight coefficients are obtained by updating the system. Kernel similarity function, used to measure local visual features. With the current semantic center The similarity between them For each local visual feature in the set of visual features, and For each of several other local visual features in addition to each local visual feature in the visual feature set; The value of is 4; The value is 0.2; similarity distribution and This can be calculated by referring to the aforementioned S130. Each time it is fixed. Through multiple iterations, the weights will gradually converge to a distribution that reflects the semantic importance of the local view.

[0082] The fifth preset function is: ; wherein, denotes the current iteration round, i.e. the step number of semantic center updating; is the current semantic center at the first iteration round; is the current semantic center at the first iteration round; is the current weight coefficient at the first iteration round; is the updated semantic center calculated according to the current weight coefficient and the current semantic center is a kernel similarity function for measuring the similarity between the local visual feature and the current semantic center ; is each local visual feature in the visual feature set. Each fixed will be updated through multiple rounds of iterations, and the weight will gradually converge to a distribution reflecting the importance of the local view semantics; this updating process will gradually move the semantic center to the position with the highest cross-modal feature density, thereby focusing on the semantic core area and suppressing the influence of background and other non-essential features.

[0083] In particular, at the first iteration round, the current semantic center is the initial current semantic center; and the current weight coefficient is the initial weight coefficient.

[0084] The updated semantic center, the updated weight coefficient, and the similarity distribution are input into the optimization objective function to perform optimization processing on the updated semantic center and the updated weight coefficient to obtain a current semantic center and a current weight coefficient; specifically, the optimization objective function is .

[0085] wherein, is a clustering term loss function, which uses a Gaussian kernel function to measure the similarity between the current view and the updated semantic center , encouraging the feature density center to gather in the semantic prominent area; wherein, ; is the updated semantic center, indicating the target position of clustering in the feature space; is the weight coefficient of the local visual feature, indicating the confidence that the local image region is considered to be the semantic center, i.e. the updated weight coefficient; is a kernel similarity function for measuring the similarity between the local visual feature and the updated semantic center ; is a bandwidth parameter that controls the similarity decay rate; This can be viewed as a type of weighted kernel density estimation (WKDE): the idea behind kernel density estimation is to take a set of sample points as an example. By placing a kernel function (here, a Gaussian kernel) at each point The probability density in the feature space is obtained by superimposing the kernel function values; the larger the kernel function value, the higher the density in the vicinity of that point; this term encourages high weights. Local image regions should be as close as possible to the semantic center. This allows the semantic center to represent a set of the most representative semantically consistent regions. Weight It can be dynamically adjusted to reduce the impact of regions that are irrelevant to the semantic category or have high noise levels on the semantic center.

[0086] in, The consistency term loss function is used to construct a similarity matrix based on the cross-modal similarity distribution of local image regions and attribute descriptions. This encourages semantically similar regions to have similar weights. To improve semantic consistency across multiple views; among which, , The value of is 4; It is the first The weight coefficients of each local visual feature, i.e., the updated weight coefficients; Except for the first The weight coefficient of each of several other local visual features in addition to the local visual feature.

[0087] in, For weight regularization, when the weight vector When the distribution is completely concentrated in a certain region, the entropy value is minimized, exhibiting a collapse in the distribution; while when The entropy is maximized when the distribution is uniform across all regions; to avoid this excessive concentration, a regularization term is introduced; where, By maximizing entropy, the balance of regional weights is maintained, thereby improving diversity and robustness; It is the first The weight coefficients of each local visual feature, i.e., the updated weight coefficients.

[0088] During the optimization process, the above two steps—using the block coordinate descent algorithm and the optimization objective function to perform the next round of optimization on the current semantic center and the current weight coefficients—are executed iteratively until the current semantic center converges. That is, the objective condition for the alternating optimization process is... This is done to obtain the final semantic center, i.e., the semantic center of the image features. .in, The probability simplex constraint representing weight normalization ensures that the weight distribution is reasonably normalized.

[0089] Through the above embodiment, it can be known that an optimization objective function is defined, and an initial semantic center and an initial weight coefficient are obtained; the initial semantic center and the initial weight coefficient are updated by using a block coordinate descent algorithm and the similarity distribution to obtain an updated semantic center and an updated weight coefficient; the updated semantic center, the updated weight coefficient and the similarity distribution are input into the optimization objective function, so as to optimize the updated semantic center and the updated weight coefficient by using the optimization objective function to obtain a current semantic center and a current weight coefficient; the block coordinate descent algorithm and the optimization objective function are continuously used to optimize the current semantic center and the current weight coefficient in the next round, until the current semantic center converges, and the converged current semantic center is taken as the image feature semantic center. Therefore, the update of the semantic center is based on the density estimation of the cross-modal features, so that the final clustering result is focused on the semantic significant area, thereby further suppressing irrelevant features; and the block coordinate descent algorithm is adopted, and only the region weight w and the semantic center c are iteratively optimized in the test stage, and the whole process does not need back propagation and does not change the model parameters; meanwhile, only forward calculation and lightweight iterative update are involved, and the calculation efficiency is high and the applicability is strong, which is suitable for real-time or resource-limited scenarios, thereby improving the processing efficiency.

[0090] S150、based on the image feature semantic center and the text feature matrix, a prediction result is obtained by performing prediction processing.

[0091] In this embodiment, after obtaining the image feature semantic center and the text feature matrix, prediction processing can be performed based on the image feature semantic center and the text feature matrix to obtain a prediction result.

[0092] In an embodiment, in step S150, that is, the prediction result is obtained by performing prediction processing based on the image feature semantic center and the text feature matrix, including:

[0093] All elements of each category in the text feature matrix are obtained as a current element set;

[0094] Cosine calculation processing is performed on each current element set and the image feature semantic center in turn to obtain a similarity;

[0095] The similarities of all elements corresponding to each category are integrated to obtain a similarity set;

[0096] The similarity set is selected by a preset selection rule to obtain a target similarity;

[0097] obtain the category corresponding to the target similarity as the prediction result.

[0098] In the embodiment, all elements of each category in the text feature matrix are obtained as a current element set , specifically, the current element set can be calculated by , wherein the current element set is a feature vector of several attribute descriptions corresponding to each category, or the current element set can obtain all elements of each category in the text feature matrix as the current element set .

[0099] cosine calculation is performed on each current element set and the image feature semantic center in sequence to obtain a similarity , specifically, the formula corresponding to the cosine calculation can be . Wherein, the cosine calculation, i.e. cosine similarity, is replaced by Euclidean distance, inner product, Mahalanobis distance, or a learnable similarity function based on Multi-Layer Perceptron (MLP).

[0100] The similarities of all elements corresponding to each category are integrated to obtain a similarity set, and a target similarity is obtained by selecting processing the similarity set according to a preset selection rule; wherein the preset selection rule is to select the maximum value; and the formula corresponding to the selection processing is . Wherein, the preset selection rule and the selection processing can also use Top-K voting, weighted average similarity or temperature scaled Softmax output.

[0101] Through the above embodiment, it can be known that all elements of each category in the text feature matrix are obtained as a current element set; cosine calculation is performed on each current element set and the image feature semantic center in sequence to obtain a similarity; the similarities of all elements corresponding to each category are integrated to obtain a similarity set; a target similarity is obtained by selecting processing the similarity set according to a preset selection rule; and the category corresponding to the target similarity is obtained as the prediction result. Therefore, by performing prediction processing on the image feature semantic center and the text feature matrix to find a category with the highest similarity to the input image as the prediction result, accurate matching processing of the input image and the input text is realized, and the processing efficiency is improved.

[0102] S160, classifying processing the input image according to the prediction result.

[0103] In the embodiment, a category corresponding to the prediction result is taken as a description of the input image, and the classification processing of the input image is completed.

[0104] To sum up, in the embodiment, the input text is encoded to obtain a text feature matrix, the input image is extracted to obtain a visual feature set, the text feature matrix and the visual feature set are matched based on a matching strategy to obtain a similarity distribution, the visual feature set and the similarity distribution are optimized based on an optimization strategy to obtain an image feature semantic center, a prediction result is obtained based on the image feature semantic center and the text feature matrix, and the input image is classified according to the prediction result. Therefore, the text feature matrix and the visual feature set are matched, optimized and predicted, the category concept is more accurately understood, the classification accuracy is improved, the time consumption in the test process is reduced, the image classification is accurately processed, and the processing efficiency is improved.

[0105] Figure 2 The schematic block diagram of the image classification device provided by the embodiment is shown in FIG. 7. Figure 2 As shown in FIG. 7, corresponding to the above image classification method, the embodiment also provides an image classification device. Specifically, referring to FIG. 7, Figure 2 the image classification device 700 includes:

[0106] an encoding unit 701 configured to encode an input text to obtain a text feature matrix;

[0107] an extraction unit 702 configured to extract an input image to obtain a visual feature set;

[0108] a matching unit 703 configured to match the text feature matrix and the visual feature set based on a matching strategy to obtain a similarity distribution;

[0109] an optimization unit 704 configured to optimize the visual feature set and the similarity distribution based on an optimization strategy to obtain an image feature semantic center;

[0110] a prediction unit 705 configured to predict based on the image feature semantic center and the text feature matrix to obtain a prediction result;

[0111] a classification unit 706 configured to classify the input image according to the prediction result.

[0112] In some embodiments, when the encoding unit 701 performs the step of encoding the input text to obtain the text feature matrix, the encoding unit 701 is specifically configured to:

[0113] identify the input text to obtain a plurality of categories;

[0114] generating a plurality of attribute descriptions for each category;

[0115] performing first encoding processing on each attribute description by using a text encoder to obtain a feature vector;

[0116] integrating the feature vectors of the plurality of attribute descriptions corresponding to each category to obtain a text feature of each category;

[0117] performing normalization processing on the text features of all categories to obtain the text feature matrix.

[0118] In some embodiments, the extraction unit 702, when performing the step of performing extraction processing on the input image to obtain a visual feature set, is specifically configured to:

[0119] obtaining a preset cropping size;

[0120] performing local cropping processing on the input image according to the cropping size by using a first preset function to obtain a plurality of local images;

[0121] performing second encoding processing on each local image according to a visual encoder to obtain a local visual feature;

[0122] composing the visual feature set with the local visual feature of each local image.

[0123] In some embodiments, the matching unit 703, when performing the step of performing matching processing on the text feature matrix and the local visual feature based on a matching strategy to obtain a similarity distribution, is specifically configured to:

[0124] performing calculation processing on each local visual feature in the visual feature set and each element in the text feature matrix to obtain a similarity;

[0125] performing calculation processing on the similarity and the corresponding local visual feature by using a second preset function to obtain a semantic matching distribution;

[0126] obtaining a plurality of other local visual features in the visual feature set except for each local visual feature, and taking the semantic matching distribution of each local visual feature as a first matching distribution and the semantic matching distribution of each corresponding other local visual feature as a second matching distribution;

[0127] performing calculation processing on the first matching distribution and each corresponding second matching distribution by using a third preset function to obtain the similarity distribution.

[0128] In some embodiments, the optimization unit 704, in the step of performing the optimization processing on the set of visual features and the similarity distribution based on the optimization strategy to obtain the image feature semantic center, is specifically configured to:

[0129] define an optimization objective function, and obtain an initial semantic center and an initial weight coefficient;

[0130] update the initial semantic center and the initial weight coefficient using a block coordinate descent algorithm and the similarity distribution to obtain an updated semantic center and an updated weight coefficient;

[0131] input the updated semantic center, the updated weight coefficient, and the similarity distribution into the optimization objective function to perform optimization processing on the updated semantic center and the updated weight coefficient using the optimization objective function to obtain a current semantic center and a current weight coefficient;

[0132] continue to perform the next round of optimization processing on the current semantic center and the current weight coefficient using the block coordinate descent algorithm and the optimization objective function until the current semantic center converges, and take the converged current semantic center as the image feature semantic center.

[0133] In some embodiments, the optimization unit 704, in the step of performing the update processing on the initial semantic center and the initial weight coefficient using the block coordinate descent algorithm and the similarity distribution to obtain the updated semantic center and the updated weight coefficient, is further specifically configured to:

[0134] perform calculation processing on the similarity distribution, the initial semantic center, and the initial weight coefficient using a fourth preset function to obtain the updated weight coefficient;

[0135] perform calculation processing on the similarity distribution, the initial semantic center, and the initial weight coefficient using a fifth preset function to obtain the updated semantic center.

[0136] In some embodiments, the prediction unit 705, in the step of performing the prediction processing based on the image feature semantic center and the text feature matrix to obtain a prediction result, is specifically configured to:

[0137] obtain all elements of each category in the text feature matrix as a current element set;

[0138] perform cosine calculation processing on each current element set and the image feature semantic center in turn to obtain a similarity;

[0139] integrate the similarity of all elements corresponding to each category to obtain a similarity set;

[0140] The target similarity is obtained by selecting the similarity set according to a preset selection rule;

[0141] The category corresponding to the target similarity is obtained as the prediction result.

[0142] It should be noted that the specific implementation process of the image classification device and each unit can be clearly understood by those skilled in the art, and the corresponding description in the foregoing method embodiments can be referred to. For the convenience and brevity of description, it will not be repeated here.

[0143] The image classification device can be implemented in the form of a computer program, which can run on a computer device as shown in the figure. Figure 3

[0144] Please refer to Figure 3 , Figure 3 is a schematic block diagram of an electronic device provided by an embodiment of the application. The computer device 800 can be a terminal or a server, wherein the terminal can be an electronic device with communication function. The server can be a stand-alone server or a server cluster composed of multiple servers.

[0145] Please refer to Figure 3 , the electronic device 800 includes a processor 802, a memory and a network interface 805 connected through a system bus 801, wherein the memory can include a non-volatile storage medium 803 and an internal memory 804.

[0146] The non-volatile storage medium 803 can store an operating system 8031 and a computer program 8032. The computer program 8032 includes program instructions, which when executed, can cause the processor 802 to perform an image classification method.

[0147] The processor 802 is configured to provide computing and control capabilities to support the operation of the entire electronic device 800.

[0148] The internal memory 804 provides an environment for the running of the computer program 8032 in the non-volatile storage medium 803, which when executed by the processor 802, can cause the processor 802 to perform an image classification method.

[0149] The network interface 805 is configured to perform network communication with other devices. Those skilled in the art can understand that Figure 3 the structure shown in the figure is only a block diagram of part of the structure related to the scheme of the application, and does not constitute a limitation on the electronic device 800 to which the scheme of the application is applied. The specific electronic device 800 can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.​

[0150] The processor 802 is configured to run the computer program 8032 stored in the memory to implement the following steps:

[0151] The input text is encoded to obtain a text feature matrix;

[0152] The input image is extracted to obtain a visual feature set;

[0153] The text feature matrix and the visual feature set are matched based on a matching strategy to obtain a similarity distribution;

[0154] The visual feature set and the similarity distribution are optimized based on an optimization strategy to obtain an image feature semantic center;

[0155] The image feature semantic center and the text feature matrix are used for prediction to obtain a prediction result;

[0156] The input image is classified based on the prediction result.

[0157] It should be understood that, in the embodiments of the present application, the processor 802 can be a central processing unit (CPU), and the processor 802 can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0158] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments of the method can be completed by a computer program instructing related hardware. The computer program includes program instructions, and the computer program can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the above-mentioned embodiments of the method.

[0159] Therefore, the present application also provides a storage medium. The storage medium can be a computer-readable storage medium. The storage medium stores a computer program, wherein the computer program includes program instructions. The program instructions are executed by the processor to make the processor perform the following steps:

[0160] The input text is encoded to obtain a text feature matrix;

[0161] The input image is extracted to obtain a visual feature set;

[0162] The text feature matrix and the visual feature set are matched based on a matching strategy to obtain a similarity distribution;

[0163] The visual feature set and the similarity distribution are optimized based on an optimization strategy to obtain an image feature semantic center;

[0164] The image feature semantic center and the text feature matrix are used for prediction to obtain a prediction result;

[0165] The input image is classified according to the prediction result.

[0166] The storage medium can be a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk, and various computer readable storage media that can store program codes.

[0167] Those skilled in the art can realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be realized in electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been described in general terms in the above description. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0168] In several embodiments provided by the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of each unit is only a logical function division, and actual implementation can have another division manner. For example, a plurality of units or components can be combined or integrated into another system, or some features can be omitted or not executed.

[0169] The steps in the method of the embodiments of the present application can be adjusted, combined and deleted in sequence according to actual needs. The units in the device of the embodiments of the present application can be combined, divided and deleted according to actual needs. In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit.

[0170] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a storage medium. Based on such understanding, the technical solutions of the present application essentially or say the part of the prior art that contributes to the present application, or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the method described in various embodiments of the present application.

[0171] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of various equivalent modifications or replacements within the technical range disclosed by the present application, and these modifications or replacements should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims. Meanwhile, the non-company software tools or components appearing in the embodiments of the present application are only illustrative and do not represent actual use.

Claims

1. An image classification method, characterized by, The method comprises: encoding processing of input text to obtain a text feature matrix; extracting processing of an input image to obtain a visual feature set; matching processing of the text feature matrix and the visual feature set based on a matching strategy to obtain a similarity distribution; optimization processing of the visual feature set and the similarity distribution based on an optimization strategy to obtain an image feature semantic center; prediction processing based on the image feature semantic center and the text feature matrix to obtain a prediction result; classification processing of the input image according to the prediction result; the optimization processing of the visual feature set and the similarity distribution based on the optimization strategy to obtain the image feature semantic center, comprising: defining an optimization objective function, and obtaining an initial semantic center and an initial weight coefficient; updating processing of the initial semantic center and the initial weight coefficient based on the similarity distribution by using a block coordinate descent algorithm to obtain an updated semantic center and an updated weight coefficient; inputting the updated semantic center, the updated weight coefficient and the similarity distribution into the optimization objective function to optimize the updated semantic center and the updated weight coefficient by using the optimization objective function to obtain a current semantic center and a current weight coefficient; continuing to optimize the current semantic center and the current weight coefficient in the next round by using the block coordinate descent algorithm and the optimization objective function, until the current semantic center converges, and taking the converged current semantic center as the image feature semantic center.

2. The image classification method of claim 1, wherein, The encoding processing of the input text to obtain the text feature matrix comprises: performing recognition processing on the input text to obtain a plurality of categories; performing generation processing on each category to obtain a plurality of attribute descriptions; performing first encoding processing on each attribute description by using a text encoder to obtain a feature vector; integrating the feature vectors of the plurality of attribute descriptions corresponding to each category to obtain a text feature of each category; performing normalization processing on the text features of all categories to obtain the text feature matrix.

3. The image classification method of claim 2, wherein, The extracting processing of the input image to obtain the visual feature set comprises: obtaining a preset cropping size; performing local cropping processing on the input image according to the cropping size by using a first preset function to obtain a plurality of local images; performing second encoding processing on each local image according to a visual encoder to obtain a local visual feature; composing the visual feature set with the local visual features of each local image.

4. The image classification method of claim 3, wherein, The matching processing of the text feature matrix and the visual feature set based on the matching strategy to obtain the similarity distribution comprises: performing calculation processing on each local visual feature in the visual feature set and each element in the text feature matrix to obtain a similarity; performing calculation processing on the similarity and the corresponding local visual feature by using a second preset function to obtain a semantic matching distribution; obtaining a plurality of other local visual features in the visual feature set except for each local visual feature, and taking the semantic matching distribution of each local visual feature as a first matching distribution, and taking the semantic matching distribution of each corresponding other local visual feature as a second matching distribution; calculating and processing the first matching distribution and each corresponding second matching distribution by using a third preset function to obtain the similarity distribution.

5. The image classification method of claim 1, wherein, the updating processing of the initial semantic center and the initial weight coefficient by using the block coordinate descent algorithm and the similarity distribution to obtain an updated semantic center and an updated weight coefficient, comprising: calculating and processing the similarity distribution, the initial semantic center and the initial weight coefficient by using a fourth preset function to obtain the updated weight coefficient; calculating and processing the similarity distribution, the initial semantic center and the initial weight coefficient by using a fifth preset function to obtain the updated semantic center.

6. The image classification method of claim 2, wherein, the prediction processing based on the image feature semantic center and the text feature matrix to obtain a prediction result, comprising: obtaining all elements of each category in the text feature matrix as a current element set; sequentially performing cosine calculation processing on each current element set and the image feature semantic center to obtain a similarity; integrating the similarity of all elements corresponding to each category to obtain a similarity set; performing selection processing on the similarity set according to a preset selection rule to obtain a target similarity; obtaining the category corresponding to the target similarity as the prediction result.

7. An image classification apparatus characterized by comprising: The device comprises: an encoding unit configured to perform encoding processing on input text to obtain a text feature matrix; an extraction unit configured to perform extraction processing on input images to obtain a visual feature set; a matching unit configured to perform matching processing on the text feature matrix and the visual feature set based on a matching strategy to obtain a similarity distribution; an optimization unit configured to perform optimization processing on the visual feature set and the similarity distribution based on an optimization strategy to obtain an image feature semantic center; a prediction unit configured to perform prediction processing based on the image feature semantic center and the text feature matrix to obtain a prediction result; a classification unit configured to perform classification processing on the input images according to the prediction result. the optimization processing on the visual feature set and the similarity distribution based on an optimization strategy to obtain an image feature semantic center, comprising: defining an optimization objective function, and obtaining an initial semantic center and an initial weight coefficient; updating the initial semantic center and the initial weight coefficient by using a block coordinate descent algorithm and the similarity distribution to obtain an updated semantic center and an updated weight coefficient; inputting the updated semantic center, the updated weight coefficient and the similarity distribution into the optimization objective function, so as to perform optimization processing on the updated semantic center and the updated weight coefficient by using the optimization objective function to obtain a current semantic center and a current weight coefficient; A next round of optimization processing is performed on the current semantic center and the current weight coefficient by using the block coordinate descent algorithm and the optimization objective function until the current semantic center converges, and the converged current semantic center is taken as the image feature semantic center.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the image classification method according to any one of claims 1-6 when executing the computer program.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, and the computer program includes program instructions which, when executed by a processor, cause the processor to perform the image classification method according to any one of claims 1-6.

Images