Image classification method and device, equipment, computer storage medium and program product
By using a label count prediction model and a variational cross encoder, the accuracy problem of zero-shot classification technology in the case of domain offset and multiple labels is solved, and efficient and accurate classification of image data is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-17
- Publication Date
- 2026-03-13
AI Technical Summary
Zero-shot classification techniques suffer from poor performance when there is domain offset between the target and training objects, or when the target object is not unique. How can we improve its accuracy?
By introducing a label count prediction model and a variational cross encoder, the number of category labels in image data is predicted. The variational cross encoder is used to encode the image and text, and the probability is calculated by combining the target classifier to determine the target category label.
It improves the accuracy and precision of zero-shot classification, bridges the domain offset between visible and invisible classes, and enhances the coverage of unknown labels.
Smart Images
Figure CN121661422A_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of information technology, and in particular relates to an image classification method, apparatus, device, computer storage medium and program product. Background Technology
[0002] With the development of technology, deep learning techniques are constantly advancing. Traditional deep learning, such as supervised classification tasks, is heavily reliant on labeled data. However, in practical applications, obtaining high-quality labeled data often faces insurmountable bottlenecks. This has directly led to the development of zero-shot classification techniques, which can classify data even for data from categories never encountered before, through indirect knowledge associations.
[0003] However, zero-shot classification techniques often perform poorly when faced with domain shifts between the target and training objects, or when the target object is not unique. Therefore, improving the accuracy of zero-shot classification is a pressing technical problem that needs to be solved by those skilled in the art. Summary of the Invention
[0004] This application provides an image classification method, apparatus, device, computer storage medium, and program product that can improve the accuracy of zero-shot classification.
[0005] In a first aspect, embodiments of this application provide an image classification method, comprising: in response to receiving image data, inputting the image data into a label quantity prediction model, predicting the number of category labels corresponding to the image data through the label quantity prediction model, and obtaining the number of category labels corresponding to the image data; inputting the image data into a variational cross encoder, performing image encoding on the image data through the variational cross encoder to obtain a reconstructed image, and performing text encoding on the image data through the variational cross encoder to obtain predicted category labels; inputting the predicted category labels into a target classifier, and calculating the probability that the reconstructed image belongs to each category label in the target classifier through the target classifier; and determining the target category label corresponding to the image data through the number of category labels and the probability.
[0006] In one embodiment, the image data includes an image and corresponding keywords. The image data is input into a variational cross-encoder (VCE), where the VCE performs image encoding to obtain a reconstructed image. The VCE also performs text encoding on the image data to obtain a predicted category label. This process includes: performing vector transformation on the image and keywords respectively to obtain image vectors and keyword vectors; aligning the image vectors and keyword vectors according to a preset alignment algorithm to obtain an aligned target vector; inputting the target vector into the VCE, where the VCE performs image encoding to obtain a reconstructed image; and performing text encoding on the target vector to obtain a predicted category label.
[0007] In one embodiment, aligning image vectors and keyword vectors according to a preset alignment algorithm to obtain an aligned target vector includes: inputting the image vector into the image encoder of a variational cross-encoder, mapping the image vector through the image encoder to obtain a first Gaussian distribution function corresponding to the image; inputting the keyword vector into the text encoder of the variational cross-encoder, mapping the keyword vector through the text encoder to obtain a second Gaussian distribution function corresponding to the keyword; using the preset alignment algorithm to align the probability distributions of the first and second Gaussian distribution functions to obtain an aligned target Gaussian distribution function; sampling the Gaussian distribution function distribution to obtain a target vector; inputting the target vector into the variational cross-encoder, processing the target vector through the variational cross-encoder to obtain a reconstructed image, and text encoding the target vector through the variational cross-encoder to obtain a predicted category label, including: inputting the target vector into the image decoder of the variational cross-encoder, decoding the target vector through the image decoder to obtain a reconstructed image; inputting the target vector into the text decoder of the variational cross-encoder, decoding the target vector through the text decoder to obtain a predicted category label.
[0008] In one embodiment, the first Gaussian distribution function includes the image mean and the image standard deviation; the second Gaussian distribution function includes the keyword mean and the keyword standard deviation; and the preset alignment algorithm is constrained by the image mean, the image standard deviation, the keyword mean, and the keyword standard deviation.
[0009] In one implementation, the preset alignment algorithm is constrained by the following formula: in, This represents the objective function of the preset alignment algorithm. Represents the image mean. This represents the average of the keywords. Represents the image covariance. The keyword covariance is represented by the image covariance, which is calculated based on the image standard deviation, and the keyword covariance is calculated based on the keyword standard deviation.
[0010] In one implementation, the image decoder is constrained by the divergence between the text encoder and the image encoder, the divergence between the text decoder and the image decoder, the expectation of the text decoder, and the expectation of the image decoder.
[0011] In one implementation, the image encoder is constrained by the following formula: in, This represents the objective function of the image decoder. Represents an image encoder With text encoder The divergence between them; Text decoder Expectations; Image decoder Divergence between the text decoder and the text decoder; Let z represent the expectation of the image decoder, and z represent noise.
[0012] In one embodiment, image data is input into a label number prediction model, and the number of category labels corresponding to the image data is predicted by the label number prediction model to obtain the number of category labels corresponding to the image data. This includes: predicting the probability that the image data belongs to each preset category label number by the label number prediction model; and taking the preset category label number corresponding to the maximum probability as the number of category labels corresponding to the image data.
[0013] In one implementation, the target classifier is constrained by the first-order norm corresponding to the number of category labels and the first-order norm corresponding to the number of targets; wherein, the number of targets is the number of category labels whose probability is greater than a preset probability threshold.
[0014] In one implementation, the target classifier is constrained by the following formula: in, The first norm representing the number of category labels. This represents the first norm corresponding to the number of targets. The probability represents the number of category labels output by the target classifier F; T represents the preset probability threshold.
[0015] In one implementation, determining the target category label corresponding to the image data based on the number and probability of category labels includes: sorting the category labels according to the probability of each category label to obtain a category label sequence; and selecting a number of category labels from the category label sequence as the target category label.
[0016] Secondly, embodiments of this application provide an image classification apparatus, the apparatus comprising: The first prediction module is used to respond to received image data by inputting the image data into the label number prediction model, and predicting the number of category labels corresponding to the image data through the label number prediction model to obtain the number of category labels corresponding to the image data. The second prediction module is used to input image data into the variational cross encoder, perform image encoding on the image data through the variational cross encoder to obtain the reconstructed image, and perform text encoding on the image data through the variational cross encoder to obtain the predicted category label. The calculation module is used to input the predicted category labels into the target classifier and calculate the probability that the reconstructed image belongs to each category label in the target classifier. The determination module is used to determine the target category label corresponding to the image data by using the number and probability of category labels.
[0017] Thirdly, embodiments of this application provide an image classification device, the device including: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, it implements the image classification method of the first aspect or any embodiment of the first aspect.
[0018] Fourthly, a computer-readable storage medium storing computer program instructions that, when executed by a processor, implement the image classification method of the first aspect or any embodiment of the first aspect.
[0019] Fifthly, embodiments of this application provide a computer program product in which instructions, when executed by a processor of an electronic device, cause the electronic device to perform an image classification method as described in the first aspect or any embodiment of the first aspect.
[0020] The image classification method, apparatus, device, computer storage medium, and program product of this application, after receiving image data, inputs the image data into a label quantity prediction model. The label quantity prediction model predicts the number of category labels corresponding to the image data, thus obtaining the number of category labels corresponding to the image data. Further, the image data is input into a variational cross-encoder. The variational cross-encoder performs image encoding on the image data to obtain a reconstructed image, and then performs text encoding on the image data to obtain predicted category labels. The predicted category labels are input into a target classifier, and the target classifier calculates the probability that the reconstructed image belongs to each category label in the target classifier. The target category label corresponding to the image data is determined based on the number of category labels and the probabilities. The target category label corresponding to the image data is determined using multiple probabilities, including the probability for the predicted category label, and the number of category labels. It is understood that in this application embodiment, by introducing a label quantity prediction model to predict the number of category labels for the image data, a clear constraint condition is provided for the final label selection, avoiding the problem of inaccurate classification caused by uncertain label numbers in multi-labeled images. Furthermore, in this embodiment, the category label is predicted by a variational cross encoder, which enhances the coverage of unknown labels and improves classification accuracy. Attached Figure Description
[0021] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0022] Figure 1 A flowchart illustrating an image classification method provided in one embodiment of this application is shown; Figure 2 A flowchart illustrating an image classification method provided in one embodiment of this application is shown; Figure 3 A flowchart illustrating an image classification method provided in one embodiment of this application is shown; Figure 4 A schematic diagram of the architecture of a variational cross encoder provided in one embodiment of this application is shown; Figure 5 This is a schematic diagram of the structure of an image classification device provided in another embodiment of this application; Figure 6 This is a schematic diagram of the structure of an image classification device provided in another embodiment of this application. Detailed Implementation
[0023] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.
[0024] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.
[0025] With the development of technology, deep learning techniques are constantly advancing. Traditional deep learning, such as supervised classification tasks, is heavily reliant on labeled data. However, in practical applications, obtaining high-quality labeled data often faces insurmountable bottlenecks. This has directly led to the development of zero-shot classification techniques, which can classify data even for data from categories never encountered before, through indirect knowledge associations.
[0026] However, in related technologies, when using zero-shot classification techniques, the domain offset between the training and test datasets must be considered. Specifically, in existing technologies, there is a data distribution difference between visible and invisible classes, leading to low accuracy and poor classification efficiency. Furthermore, existing technologies also perform poorly when the objects to be classified are not unique. Therefore, improving the accuracy of zero-shot classification techniques is a pressing technical problem that needs to be solved by those skilled in the art.
[0027] To address the problems of the prior art, embodiments of this application provide an image classification method, apparatus, device, computer storage medium, and program product. The image classification method provided in this application embodiment will be described first below.
[0028] Figure 1 A schematic flowchart of an image classification method provided in one embodiment of this application is shown. Figure 1 As shown, the image classification method includes the following steps: S110. In response to receiving image data, input the image data into the label number prediction model, predict the number of category labels corresponding to the image data through the label number prediction model, and obtain the number of category labels corresponding to the image data.
[0029] S120. Input the image data into the variational cross encoder, perform image encoding on the image data through the variational cross encoder to obtain the reconstructed image, and perform text encoding on the image data through the variational cross encoder to obtain the predicted category label.
[0030] S130. Input the predicted category label into the target classifier, and calculate the probability that the reconstructed image belongs to each category label in the target classifier through the target classifier.
[0031] S140. Determine the target category label corresponding to the image data by using the number and probability of category labels.
[0032] In some embodiments, in S110, upon receiving image data, the received image data can be input into a label prediction model. The label prediction model predicts the number of category labels corresponding to the image data, thereby obtaining the number of category labels corresponding to the image data.
[0033] For example, the category label corresponding to the image data can represent the category to which the image belongs, wherein the image data can have one or more category labels.
[0034] In some optional embodiments, the probability of image data belonging to each preset category label number is predicted by the label number prediction model; the preset category label number corresponding to the maximum probability is taken as the category label number corresponding to the image data.
[0035] For example, the preset number of category labels can represent the number of labels corresponding to the image data. For instance, the number of labels can include 1, 2, or 3, etc.
[0036] For example, by inputting image data into a label number prediction model, the probability of the image data belonging to a preset category label for each label is predicted by the label number prediction model, and the preset category label with the highest probability value is taken as the category label number corresponding to the image data.
[0037] For example, the label count prediction model can be represented as a vector. For instance, a label count prediction model outputting [0.3, 0.6, 0.1] indicates that the probability of the image data having one category label is 0.3, the probability of having two category labels is 0.6, and the probability of having three category labels is 0.1. Furthermore, the category label with the highest probability can be taken as the number of category labels corresponding to the image data; that is, the image data corresponds to two category labels.
[0038] For example, a label count prediction model may include a multi-layered structure. For instance, a label count prediction model may include an image classification layer, a normalization layer, and a probabilistic layer.
[0039] In one example, each set of image data can consist of an image (I) and a category label (L), that is, the image data can be represented as N represents the number of image data points. For the images, the raw image pixel values can be used as input. For the category labels, they can be converted into binary vectors, where 0 indicates that the image does not belong to the category corresponding to the vector element, and 1 indicates that the image belongs to the category corresponding to the vector element. Furthermore, an image can belong to multiple categories simultaneously.
[0040] Furthermore, the number of labels in the predicted image data is determined, i.e., the cardinality of the predicted class label vectors. The image classification layer can be a VGG16 (Visual Geometry Group 16-layer) deep convolutional neural network. This neural network can be characterized as follows: ,in This is the parameter set of the neural network. A normalization layer and a probabilistic layer are then added after the output layer of the neural network. The cardinality of the class label vector represents the number of component elements in the class label vector.
[0041] The normalization layer can be characterized by the following formula (1): in, for Minimum output value, for Maximum output value.
[0042] The probabilistic layer can be characterized by the following formula (2): in, This is the output value of the normalized layer. Therefore, the output of the label count prediction model is: .
[0043] Furthermore, in order to train the label count prediction model, the cardinality of the category labels can be... As the fitting target (i.e., This represents the number of true class labels corresponding to the image. For example, when the number of class labels is 1, it indicates that the image belongs to only one class, and the fitting target can be represented as... If the number of category labels is 2, it indicates that the image belongs to two categories simultaneously, and the fitting objective can be represented as... And so on. Therefore, the objective function for training the label number prediction model is represented by the following formula (3): Furthermore, the Adam algorithm can be used to minimize the objective function of the label count prediction model to obtain the neural network parameters. After training the cardinality predictor, inputting a new image will yield the expected cardinality, as shown in formula (4) below: Where j = 1, 2, ..., D. D represents the maximum possible number of labels for the image. The expected value of the cardinality can be used as the probability of the image data for the number of class labels.
[0044] In this embodiment of the application, by inputting image data into the label number prediction model, the probability of the image data belonging to the number of preset category labels for each label is predicted by the label number prediction model, and the number of preset category labels with the highest probability value is taken as the number of category labels corresponding to the image data, thereby enabling the rapid and accurate determination of the number of category labels corresponding to the image data.
[0045] In some embodiments, in S120, image data can be input into a variational cross encoder, the image data can be image encoded by the variational cross encoder to obtain a reconstructed image, and the image data can be text encoded by the variational cross encoder to obtain a predicted category label.
[0046] For example, a variational cross encoder can be used to encode image data as well as text data to obtain a reconstructed image and a predicted category label.
[0047] The predicted class label can be the class label predicted by the variational cross encoder based on the image data. The reconstructed image can be the image obtained by the variational cross encoder reconstructing the image data according to different requirements.
[0048] In one example, the process of reconstructing an image can be a process of compressing and decompressing the image data, resulting in a reconstructed image similar to the original image data. For instance, a variational cross encoder can include an encoder and a decoder. The input image data is a 1024x1024 image, which can be compressed to a 12x12 size by the encoder. Arranging this data in rows or columns creates a 144-dimensional vector, representing the image. Further, the decoder decodes this 144-dimensional vector to obtain a 1024x1024 reconstructed image.
[0049] In some embodiments, in S130, the predicted category label is input into the target classifier, and the probability that the reconstructed image belongs to each category label in the target classifier is calculated by the target classifier.
[0050] For example, the target classifier can be used to predict the category label to which the reconstructed image belongs and to calculate the probability of the reconstructed image belonging to each category label.
[0051] For example, the target classifier can be a fully connected neural network.
[0052] In one example, the target classifier can be characterized as Where I' is the reconstructed image, This is the set of parameters for the neural network in the target classifier.
[0053] In some optional embodiments, the target classifier can be constrained by the first-order norm corresponding to the number of class labels and the first-order norm corresponding to the target number; wherein, the target number is the number of class labels whose probability is greater than a preset probability threshold. It is understood that by constraining the target classifier by the first-order norm corresponding to the number of class labels and the first-order norm corresponding to the target number, the target classifier can quickly and accurately output the probability that the reconstructed image belongs to each class label.
[0054] Furthermore, in some optional embodiments, the target classifier is constrained by the following formula (5): in, The first norm representing the number of category labels. This represents the first norm corresponding to the number of targets. The probability represents the number of category labels output by the target classifier F; T represents the preset probability threshold.
[0055] Understandably, constrained by the number of class labels, the objective function of the target classifier is as follows (6): Where T is a preset probability threshold, i.e. a threshold value, usually taken as 0.5. The 0th norm represents the norm of a vector, which is equal to its cardinality. However, the 0th norm makes the objective function of a target classifier an NP-hard problem, preventing the direct application of gradient descent-based methods (such as the Adam algorithm). Therefore, the 1st norm can be used instead. approximate And using the first-order norm. approximate Where ReLU represents the corrected linear unit, its expression can be expressed as: After using the above approximation, the constraint conditions are continuously differentiable. Therefore, the objective function of the target classifier can be optimized using the Lagrange multiplication method or the Adam algorithm to obtain the target classifier parameters. .in, To reconstruct the image.
[0056] In this embodiment, the first-order norm is used to approximate the zero-order norm, ensuring that the constraint conditions are continuously differentiable. This enables the optimization of the objective function of the target classifier, facilitating efficient training to obtain classifier parameters.
[0057] In some embodiments, in S140, the target category label corresponding to the image data can be determined based on the number of category labels and the probability that the image data of the target classifier data belongs to each category label.
[0058] For example, the target category label can be the category label corresponding to the image data, wherein the number of target category labels is consistent with the number of category labels output by the performance quantity prediction model.
[0059] For example, a number of category labels can be selected as the target category labels for the image data based on the probability that the image data belongs to each category label in the target classifier data.
[0060] In one example, the output of the label prediction model is [0.3, 0.6, 0.1], indicating that the image data has the highest probability of having two class labels. Further, the output of the target classifier is [0.1, 0.2, 0.3, 0.4]. Since the image data is predicted to have two class labels, the two highest probabilities are selected as the target class labels for the image, thus determining that the image data belongs to class 3 and class 4.
[0061] In some optional embodiments, the category labels are sorted according to the probability of each category label to obtain a category label sequence, and a certain number of category labels are selected from the category label sequence as the target category label. It is understood that by sorting the category labels according to the probability of each category label, category labels with higher probabilities can be selected first, thus ensuring that the selected target category label has high confidence in the model prediction, improving the accuracy and reliability of the target category label.
[0062] For example, the category labels can be sorted in descending order of probability to obtain a category label sequence; or, the category labels can be sorted in ascending order of probability to obtain a category label sequence.
[0063] In one example, the probability of each category label output by the target classifier may include the probability that the image data belongs to the predicted category label, and the target category label corresponding to the image data is determined by multiple probabilities, including the probability of the predicted category label, and the number of category labels.
[0064] In another example, the probability of each category label output by the target classifier may not include the probability of the image data belonging to the predicted category label. Here, the predicted category label can be used as one of the target category labels, and further, other target category labels are determined based on the probability of each category label output by the target classifier and the number of category labels.
[0065] exist Figure 1In the illustrated embodiment, after receiving image data, the image data is input into a label quantity prediction model. This model predicts the number of category labels corresponding to the image data, thus obtaining the number of category labels for the image data. Further, the image data is input into a variational cross-encoder. The variational cross-encoder encodes the image data to obtain a reconstructed image, and then performs text encoding on the image data to obtain predicted category labels. The predicted category labels are input into a target classifier, which calculates the probability that the reconstructed image belongs to each category label in the target classifier. The target category label corresponding to the image data is determined based on the number of category labels and their probabilities. The target category label corresponding to the image data is determined using multiple probabilities, including the probability for the predicted category label, and the number of category labels. It is understood that in this embodiment, by introducing a label quantity prediction model to predict the number of category labels for the image data, a clear constraint is provided for the final label selection, avoiding the classification inaccuracy problem caused by uncertain label numbers in multi-labeled images. Furthermore, in this embodiment, the variational cross-encoder is used to predict category labels, enhancing the coverage of unknown labels and improving classification accuracy.
[0066] In order to obtain the reconstructed image and predict the label category, as another implementation of this application, this application also provides another implementation of the image classification method, as detailed in the following embodiments.
[0067] Figure 2 A schematic flowchart of an image classification method provided in one embodiment of this application is shown. Figure 2 As shown, the image classification method includes the following steps: S210. Perform vector transformation on the image and keywords respectively to obtain image vector and keyword vector.
[0068] S220. Align the image vector and keyword vector according to the preset alignment algorithm to obtain the aligned target vector.
[0069] S230. Input the target vector into the variational cross encoder, process the target vector through the variational cross encoder to obtain the reconstructed image, and then perform text encoding on the target vector through the variational cross encoder to obtain the predicted category label.
[0070] In some embodiments, in S210, the image data includes an image and keywords corresponding to the image. The image and keywords can be vectorized to obtain image vectors and keyword vectors, respectively.
[0071] For example, images and keywords can be vectorized in different ways.
[0072] In one example, an image can be vectorized using a neural network, where the neural network used for vector transformation could be a neural network in a label count prediction model. In another example, a keyword semantic vector extraction module, i.e., word embedding, performs vector transformation.
[0073] For example, an image vector can be a vector obtained by vectorizing an image; a keyword vector can be a vector obtained by vectorizing keywords corresponding to an image.
[0074] In some embodiments, in S220, the image vector and the keyword vector can be aligned according to a preset alignment algorithm to obtain the aligned target vector.
[0075] For example, the preset alignment algorithm can be used to align the image vector and the keyword vector to obtain the aligned target vector.
[0076] In some optional embodiments, the image vector is input into the image encoder in the variational cross encoder, and the image encoder maps the image vector to obtain the first Gaussian distribution function corresponding to the image; the keyword vector is input into the text encoder in the variational cross encoder, and the text encoder maps the keyword vector to obtain the second Gaussian distribution function corresponding to the keyword; the first Gaussian distribution function and the second Gaussian distribution function are aligned in probability distribution using a preset alignment algorithm to obtain the aligned target Gaussian distribution function; the Gaussian distribution function is sampled to obtain the target vector.
[0077] For example, a variational cross encoder may include an image encoder and a text encoder. Both the image encoder and the text encoder can be fully connected neural networks, and the image encoder and the text encoder are used to output the Gaussian distribution functions corresponding to the image and the keywords, respectively.
[0078] The image encoder and text encoder can each include two output layers: one output layer can be used to output the mean vector of the Gaussian distribution function, and the other output layer can be used to output the standard deviation vector of the Gaussian distribution function.
[0079] For example, an image encoder can be used to map image vectors to obtain a first Gaussian distribution function corresponding to the image. A text encoder can be used to map keyword vectors to obtain a second Gaussian distribution function corresponding to the keywords.
[0080] The first Gaussian distribution function and the second Gaussian distribution function can be aligned using a preset alignment algorithm to obtain the aligned target Gaussian distribution function.
[0081] In some optional embodiments, the first Gaussian distribution function includes the image mean and the image standard deviation; the second Gaussian distribution function includes the keyword mean and the keyword standard deviation. Furthermore, the preset alignment algorithm is constrained by the image mean, the image standard deviation, the keyword mean, and the keyword standard deviation.
[0082] Furthermore, in some optional embodiments, the preset alignment algorithm is constrained by the following formula (7): in, This represents the objective function of the preset alignment algorithm. Represents the image mean. This represents the average of the keywords. Represents the image covariance. The keyword covariance is represented by the image covariance, which is calculated based on the image standard deviation, and the keyword covariance is calculated based on the keyword standard deviation.
[0083] For example, the parameters in the image encoder and text encoder can be optimized using the above formula, so that the first Gaussian distribution function and the second Gaussian distribution function output by the image encoder and the text encoder have aligned characteristics.
[0084] For example, the target Gaussian distribution function can be sampled to obtain the target vector.
[0085] In one example, the target Gaussian function can be an N-dimensional Gaussian distribution, with its sampled values being an N-dimensional real vector. Programming languages typically have built-in functions for generating this distribution, such as MATLAB's `randn(1,N)`, which samples a 1xN random vector. The Gaussian distribution function can be implemented using the `std::normal_distrition` algorithm in the C++ standard library, or it can be generated using Java's `java.util.Random` algorithm.
[0086] It is understandable that by constraining the preset alignment algorithm with image mean, image standard deviation, keyword mean, and keyword standard deviation, constraints can be imposed on both the center point (i.e., mean) and shape (standard deviation) of the two Gaussian distribution functions. This allows for better capture of the similarity of feature distributions between images and keywords, thereby enabling better utilization of cross-modal association knowledge learned from visible classes and its smooth transfer to the recognition of invisible classes. This significantly improves the model's generalization ability to unknown categories.
[0087] In some embodiments, in S230, the target vector can be input into a variational cross encoder, the image can be encoded by the variational cross encoder to obtain a reconstructed image, and the target vector can be text encoded by the variational cross encoder to obtain a predicted category label.
[0088] For example, the aligned target vector can be image-encoded and text-encoded using a variational cross encoder to obtain a reconstructed image and a predicted category label.
[0089] In some optional embodiments, the target vector is input into the image decoder in the variational cross encoder, and the image decoder decodes the target vector to obtain a reconstructed image; and the target vector is input into the text decoder in the variational cross encoder, and the text decoder decodes the target vector to obtain a predicted class label.
[0090] For example, a variational cross-encoder may include an image decoder and a text decoder. Both the image decoder and the text decoder can be fully connected neural networks. The image decoder and the text decoder are used to decode the target vector to obtain a reconstructed image and a predicted class label.
[0091] In some alternative embodiments, the image decoder is constrained by the divergence between the text encoder and the image encoder, the divergence between the text decoder and the image decoder, the expectation of the text decoder, and the expectation of the image decoder.
[0092] Furthermore, in some optional embodiments, the image encoder is constrained by the following formula (8): in, This represents the objective function of the image decoder. Indicates the image encoder With the text encoder The divergence between them; This indicates the text decoder Expectations; Image decoder The divergence between the text decoder and the text decoder; Let z represent the expectation of the image decoder, and z represent noise.
[0093] The expression for divergence is given by the following formula (9): in, It can represent the distribution of noise z in an image encoder given an image I; This can represent the distribution of noise z in a text encoder given a keyword L. This can represent the distribution of the image I reconstructed by the image decoder under noise z. This can represent the distribution of the predicted category label I by the text decoder in the presence of noise z. , This represents the prior distribution of noise z in both the image decoder and the text decoder.
[0094] It is understood that, in the embodiments of this application, the divergence between the text encoder and the image encoder, the divergence between the text decoder and the image decoder, the expectation of the text decoder, and the expectation of the image decoder constrain the image encoder, which can force the latent distributions of images and keywords to align to the same regular space. This can effectively bridge the domain differences between modalities and between visible and invisible classes, significantly improve the model's generalization ability, and ensure that high-quality labels can be stably generated from latent variables, achieving high-precision zero-shot multi-label classification.
[0095] In this embodiment, by aligning the keywords and the corresponding Gaussian distributions of the images, the model can learn the mapping relationship between modalities in the images and text. This constrains the representations of all categories into the same regular space, so the semantic vectors of unknown categories can be naturally mapped to the corresponding positions in this space and correctly decoded. This can bridge the domain offset between visible and invisible categories and improve the generalization ability for unknown categories.
[0096] exist Figure 2 In the illustrated embodiment, image vectors and keyword vectors are obtained by performing vector transformations on the image and keywords respectively. The image vectors and keyword vectors are then aligned according to a preset alignment algorithm to obtain the aligned target vector. The target vector is input into a variational cross-encoder, which encodes the image to obtain a reconstructed image. Finally, the target vector is text-encoded using the variational cross-encoder to obtain the predicted category label. It can be understood that by utilizing both the image and its corresponding keywords as modalities, and by forcing the feature distributions of the image and keywords to align through the variational cross-encoder, the domain offset between visible and invisible classes is effectively bridged, improving the generalization ability for unknown categories.
[0097] Below, in conjunction with Figure 3 , Figure 4 The following examples illustrate image classification methods.
[0098] Figure 3 A flowchart illustrating an image classification method provided in one embodiment of this application is shown; Figure 4 A schematic diagram of the architecture of a variational cross encoder provided in one embodiment of this application is shown.
[0099] like Figure 3 As shown, in step S310, images and keywords are acquired. In one example, the image can be a picture uploaded by a user on a social media platform, and the keywords can be the keywords given to the image by the user who uploaded it, which can be represented as tags. Users typically use "#" and accompanying text to explain the image. However, such tags are quite noisy; for example, a user might use "#" to refer to an event, but the accompanying text might be an emoji expressing their attitude towards that event. Therefore, in image classification, user tags and accompanying text cannot be directly used as categories. Instead, labels, or tags, need to be manually selected and generated for machine learning.
[0100] In step S320, the image is input into the label number prediction model to predict the number of category labels corresponding to the image.
[0101] In step S330, the image and keywords are input into the variational cross encoder to obtain the reconstructed image and predicted category label.
[0102] exist Figure 4 In this process, images can be input into the CNN network 401, where the CNN network performs vector transformation to obtain the corresponding image vector. Keywords can also be input into the semantic vector extraction module 402, where the semantic vector extraction module performs vector transformation to obtain keyword vectors. Further, the image vectors are input into the image encoder 403 of the variational cross-encoder to obtain the first Gaussian distribution function corresponding to the image, and the keyword vectors are input into the text encoder 403 of the variational cross-encoder to obtain the second Gaussian distribution function corresponding to the keywords. By aligning the first and second Gaussian distribution functions, an aligned target Gaussian distribution function is obtained. This aligned target Gaussian distribution function is then decoded using the image decoder 405 and text decoder 406 of the variational cross-encoder to obtain the reconstructed image and predicted category labels, respectively.
[0103] In one example, a text encoder can encode most English words, generating a vector. Words with similar meanings or that frequently appear together tend to have similar encodings. Therefore, by encoding and aligning images and keywords in the same space, finding similar text encodings, and then decoding them, new labels are obtained. For instance, when training a target classifier, there is no "Sky" label. When applying the target classifier and a sky image appears, a variational cross-encoder can find the encodings of one or more words closest to this image in the encoding space, and use these as the image's label.
[0104] In step S340, the reconstructed image and the predicted category label are input into the target classifier, which predicts the probability that the image belongs to each category label.
[0105] In step S350, the probabilities of multiple category labels are sorted to obtain a category label sequence.
[0106] In step S360, the category label sequence is truncated based on the number of category labels, and the category label with the highest probability is taken as the target category label corresponding to the image.
[0107] In one example, the image is input into a trained label count prediction model to calculate its expected cardinality, denoted as C. Then, the image is input into an image encoder to obtain the image mean and standard deviation vector corresponding to the first Gaussian distributed image. The corresponding tube detection is then input into a keyword encoder to obtain the keyword mean and standard deviation vector corresponding to the second Gaussian distributed image. The probability distributions of the first and second Gaussian distributed images are aligned and sampled. Further, the sampled vectors are input into an image decoder to obtain a reconstructed image. This sampled vector is then input into a text decoder to generate new label vectors. The reconstructed image is input into a classifier to obtain the probability of the image belonging to each label. The C categories with the highest probabilities are selected as the image's category labels.
[0108] In this embodiment, a cross-variable encoder is used to bridge the data distribution differences between visible and invisible classes during model training and application (or testing). Furthermore, a label count prediction model is used to predict and constrain the number of labels in an image, thereby improving the accuracy of multi-label image classification.
[0109] First see Figure 5 The image classification device provided in this application embodiment includes the following modules: The first prediction module 501 is used to respond to received image data by inputting the image data into the label number prediction model, predicting the number of category labels corresponding to the image data through the label number prediction model, and obtaining the number of category labels corresponding to the image data. The second prediction module 502 is used to input image data into the variational cross encoder, perform image encoding on the image data through the variational cross encoder to obtain a reconstructed image, and perform text encoding on the image data through the variational cross encoder to obtain a predicted category label. The calculation module 503 is used to input the predicted category label into the target classifier and calculate the probability that the reconstructed image belongs to each category label in the target classifier through the target classifier; The determination module 504 is used to determine the target category label corresponding to the image data by using the number and probability of category labels.
[0110] In one implementation of this application, the image data includes an image and keywords corresponding to the image. The second prediction module 502 inputs the image data into a variational cross encoder in the following manner: the variational cross encoder performs image encoding on the image data to obtain a reconstructed image, and performs text encoding on the image data to obtain a predicted category label. The image and keywords are transformed into vectors respectively to obtain image vectors and keyword vectors. The image vector and keyword vector are aligned according to a preset alignment algorithm to obtain an aligned target vector. The target vector is input into the variational cross encoder, which performs image encoding on the image to obtain a reconstructed image, and performs text encoding on the target vector to obtain a predicted category label.
[0111] As one implementation of this application, the second prediction module 502 aligns the image vector and keyword vector according to a preset alignment algorithm to obtain the aligned target vector: The image vector is input into the image encoder in the variational cross-encoder, and the image vector is mapped to obtain the first Gaussian distribution function corresponding to the image; the keyword vector is input into the text encoder in the variational cross-encoder, and the keyword vector is mapped to obtain the second Gaussian distribution function corresponding to the keyword; the first Gaussian distribution function and the second Gaussian distribution function are aligned in probability distribution using the preset alignment algorithm to obtain the aligned target Gaussian distribution function; the Gaussian distribution function is sampled to obtain the target vector; the second prediction module 502 inputs the target vector into the variational cross-encoder, and the target vector is processed by the variational cross-encoder to obtain the reconstructed image, and the target vector is text-encoded by the variational cross-encoder to obtain the predicted category label: The target vector is input into the image decoder in the variational cross-encoder, and the target vector is decoded by the image decoder to obtain the reconstructed image; the target vector is input into the text decoder in the variational cross-encoder, and the target vector is decoded by the text decoder to obtain the predicted category label.
[0112] As one implementation of this application, the first Gaussian distribution function includes the image mean and the image standard deviation; the second Gaussian distribution function includes the keyword mean and the keyword standard deviation; and the preset alignment algorithm is constrained by the image mean, the image standard deviation, the keyword mean, and the keyword standard deviation.
[0113] As one implementation of this application, the preset alignment algorithm is constrained by the following formula: in, This represents the objective function of the preset alignment algorithm. Represents the image mean. This represents the average of the keywords. Represents the image covariance. The keyword covariance is represented by the image covariance, which is calculated based on the image standard deviation, and the keyword covariance is calculated based on the keyword standard deviation.
[0114] As one implementation of this application, the image decoder is constrained by the divergence between the text encoder and the image encoder, the divergence between the text decoder and the image decoder, the expectation of the text decoder, and the expectation of the image decoder.
[0115] As one implementation of this application, the image encoder is constrained by the following formula: in, This represents the objective function of the image decoder. Represents an image encoder With text encoder The divergence between them; Text decoder Expectations; Image decoder Divergence between the text decoder and the text decoder; Let z represent the expectation of the image decoder, and z represent noise.
[0116] As one implementation of this application, the first prediction module 501 inputs the image data into the label number prediction model in the following manner, and predicts the number of category labels corresponding to the image data through the label number prediction model to obtain the number of category labels corresponding to the image data: predicting the probability that the image data belongs to each preset number of category labels through the label number prediction model. The number of preset category labels corresponding to the maximum probability is taken as the number of category labels corresponding to the image data.
[0117] As one implementation of this application, the target classifier is constrained by the first-order norm corresponding to the number of category labels and the first-order norm corresponding to the number of targets; wherein, the number of targets is the number of category labels whose probability is greater than a preset probability threshold.
[0118] As one implementation of this application, the target classifier is constrained by the following formula: in, The first norm representing the number of category labels. This represents the first norm corresponding to the number of targets. The probability represents the number of category labels output by the target classifier F; T represents the preset probability threshold.
[0119] As one implementation of this application, the determining module 504 determines the target category label corresponding to the image data by using the number and probability of category labels as follows: sorting the category labels according to the probability of each category label to obtain a category label sequence; selecting a number of category labels from the category label sequence as the target category label.
[0120] Figure 6 A schematic diagram of the hardware structure of the image classification device provided in an embodiment of this application is shown.
[0121] The image classification device may include a processor 601 and a memory 602 storing computer program instructions.
[0122] Specifically, the processor 601 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.
[0123] Memory 602 may include mass storage for data or instructions. For example, and not limitingly, memory 602 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 602 may include removable or non-removable (or fixed) media. Where appropriate, memory 602 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, memory 602 is non-volatile solid-state memory.
[0124] Memory may include read-only memory (ROM), random access memory (RAM), disk storage media devices, optical storage media devices, flash memory devices, and electrical, optical, or other physical / tangible memory storage devices. Therefore, typically, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the methods according to one aspect of this disclosure.
[0125] The processor 601 implements any of the image classification methods described in the above embodiments by reading and executing computer program instructions stored in the memory 602.
[0126] In one example, the image classification device may also include a communication interface 603 and a bus 610. Wherein, as Figure 6 As shown, the processor 601, memory 602, and communication interface 603 are connected through bus 610 and complete communication with each other.
[0127] The communication interface 603 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.
[0128] Bus 610 includes hardware, software, or both, that couples components of an online data traffic metering device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 610 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, any suitable bus or interconnect is contemplated herein.
[0129] This image classification device can execute the image classification method described in this application based on image data, thereby achieving a combination of... Figure 1 and Figure 2 Describe the image classification method.
[0130] Furthermore, in conjunction with the image classification methods described in the above embodiments, this application embodiment can provide a computer storage medium for implementation. This computer storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the image classification methods described in the above embodiments.
[0131] This application also provides a computer program product, including a computer program, which, when executed, implements any of the image classification methods described in the above embodiments.
[0132] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.
[0133] The functional blocks shown in the above-described structural diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.
[0134] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.
[0135] The aspects of this disclosure have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by special-purpose hardware performing the specified functions or actions, or can be implemented by a combination of special-purpose hardware and computer instructions.
[0136] The above description is merely a specific implementation of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.
Claims
1. An image classification method, characterized in that, include: In response to receiving image data, the image data is input into a label number prediction model, and the number of category labels corresponding to the image data is predicted by the label number prediction model to obtain the number of category labels corresponding to the image data; The image data is input into a variational cross encoder, which encodes the image data to obtain a reconstructed image. The image data is then encoded using the variational cross encoder to obtain a predicted category label. The predicted category label is input into the target classifier, and the probability of the reconstructed image belonging to each category label in the target classifier is calculated through the target classifier; The target category label corresponding to the image data is determined by the number of category labels and the probability.
2. The method according to claim 1, characterized in that, The image data includes images and keywords corresponding to the images; The process of inputting the image data into a variational cross encoder, performing image encoding on the image data using the variational cross encoder to obtain a reconstructed image, and performing text encoding on the image data using the variational cross encoder to obtain a predicted category label includes: The image and the keyword are respectively transformed into vectors to obtain image vectors and keyword vectors; According to a preset alignment algorithm, the image vector and the keyword vector are aligned to obtain the aligned target vector; The target vector is input into the variational cross encoder, which performs image encoding to obtain a reconstructed image. The target vector is then text-encoded using the variational cross encoder to obtain a predicted category label.
3. The method according to claim 2, characterized in that, The step of aligning the image vector and the keyword vector according to a preset alignment algorithm to obtain the aligned target vector includes: The image vector is input into the image encoder in the variational cross encoder, and the image vector is mapped by the image encoder to obtain the first Gaussian distribution function corresponding to the image; The keyword vector is input into the text encoder in the variational cross encoder, and the keyword vector is mapped by the text encoder to obtain the second Gaussian distribution function corresponding to the keyword; Using a preset alignment algorithm, the first Gaussian distribution function and the second Gaussian distribution function are aligned in probability distribution to obtain the aligned target Gaussian distribution function; The target vector is obtained by sampling the Gaussian distribution function. The process of inputting the target vector into a variational cross encoder, performing image encoding on the target vector using the variational cross encoder to obtain a reconstructed image, and performing text encoding on the target vector using the variational cross encoder to obtain a predicted category label includes: The target vector is input into the image decoder in the variational cross encoder, and the image decoder decodes the target vector to obtain the reconstructed image; The target vector is input into the text decoder in the variational cross encoder, and the text decoder decodes the target vector to obtain the predicted class label.
4. The method according to claim 2 or 3, characterized in that, The first Gaussian distribution function includes the image mean and image standard deviation; the second Gaussian distribution function includes the keyword mean and keyword standard deviation. The preset alignment algorithm is constrained by the image mean, the image standard deviation, the keyword mean, and the keyword standard deviation.
5. The method according to claim 4, characterized in that, The preset alignment algorithm is constrained by the following formula: in, Let represent the objective function of the preset alignment algorithm. Represents the image mean. This represents the average of the keywords. Represents the image covariance. The keyword covariance is represented by the image covariance, which is calculated based on the image standard deviation, and the keyword covariance is calculated based on the keyword standard deviation.
6. The method according to claim 3, characterized in that, The image decoder is constrained by the divergence between the text encoder and the image encoder, the divergence between the text decoder and the image decoder, the expectation of the text decoder, and the expectation of the image decoder.
7. The method according to claim 6, characterized in that, The image encoder is constrained by the following formula: in, This represents the objective function of the image decoder. Indicates the image encoder With the text encoder The divergence between them; This indicates the text decoder Expectations; Image decoder The divergence between the text decoder and the text decoder; Let z represent the expectation of the image decoder, and z represent noise.
8. The method according to claim 1, characterized in that, The step of inputting the image data into the label quantity prediction model, and using the label quantity prediction model to predict the number of category labels corresponding to the image data, to obtain the number of category labels corresponding to the image data, includes: The probability of the image data belonging to the number of labels in each preset category is predicted using the label number prediction model. The number of preset category labels corresponding to the maximum probability is taken as the number of category labels corresponding to the image data.
9. The method according to claim 1, characterized in that, The target classifier is constrained by the first-order norm corresponding to the number of category labels and the first-order norm corresponding to the target number; wherein, the target number is the number of category labels whose probability is greater than a preset probability threshold.
10. The method according to claim 9, characterized in that, The target classifier is constrained by the following formula: in, The first norm representing the number of category labels. This represents the first norm corresponding to the number of targets. The probability represents the number of category labels output by the target classifier F; T represents the preset probability threshold.
11. The method according to claim 1, characterized in that, Determining the target category label corresponding to the image data based on the number of category labels and the probability includes: The category labels are sorted according to the probability of each category label to obtain a category label sequence; Select the specified number of category labels from the category label sequence as the target category label.
12. An image classification device, characterized in that, The device includes: The first prediction module is used to respond to received image data by inputting the image data into a label number prediction model, and using the label number prediction model to predict the number of category labels corresponding to the image data, thereby obtaining the number of category labels corresponding to the image data. The second prediction module is used to input the image data into the variational cross encoder, perform image encoding on the image data through the variational cross encoder to obtain a reconstructed image, and perform text encoding on the image data through the variational cross encoder to obtain a predicted category label. The calculation module is used to input the predicted category label into the target classifier and calculate the probability that the reconstructed image belongs to each category label in the target classifier through the target classifier; The determination module is used to determine the target category label corresponding to the image data based on the number of category labels and the probability.
13. An image classification device, characterized in that, The device includes: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, it implements the image classification method as described in any one of claims 1-11.
14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions that, when executed by a processor, implement the image classification method as described in any one of claims 1-11.
15. A computer program product, characterized in that, When the instructions in the computer program product are executed by the processor of the electronic device, the electronic device performs the image classification method as described in any one of claims 1-11.