Image classification method and device, equipment and storage medium

Abstract

The invention belongs to the technical field of artificial intelligence, and discloses an image classification method, device and equipment and a storage medium, and the classification method comprises the steps: obtaining a target image and a target text, the target text being a text description corresponding to the target image; inputting the target image into an image encoder to obtain an image encoding sequence corresponding to the target image; inputting the target text into a text encoder to obtain a text encoding sequence corresponding to the target text; splicing the image coding sequence and the text coding sequence to obtain a spliced sequence; inputting the splicing sequence into a preset encoder, and outputting a target image vector representing global information of the target image and a target text vector representing global information of the target text; and inputting the target image vector and the target text vector into a classification module to obtain a classification result of the target image. The problem that in the prior art, a classification mode based on image recognition cannot classify details of images is solved.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and in particular relates to an image classification method, device, equipment and storage medium. Background Art

[0002] Image recognition refers to the process of using computers to automatically analyze and understand objects, scenes, text, and other content in an image, thereby determining the category of the image or extracting specific targets in the image. In simple terms, it allows computers to "understand" what is in an image like humans. For example, identifying whether a photo is a cat or a dog, or identifying specific targets such as traffic lights and buildings in street photos.

[0003] In order to perform image recognition, Contrastive Language-Image Pre-Training (CLIP) came into being. The training process of CLIP uses a large amount of image-text data pairs from the Internet, as many as 400 million pairs. In this process, it cleverly uses the descriptive text as the label of the image. In specific operations, the image is first processed by the image encoder, and the corresponding descriptive text is processed by the text encoder, so that the image vector and text vector can be obtained respectively.

[0004] In the pre-training stage, through a large number of different image-text data pairs, CLIP's image encoder and text encoder work together to learn a rich and diverse representation of the overall image features, and the mapping relationship between these features and the corresponding text descriptions shows a relatively good classification ability for the global image content. When processing image recognition tasks, CLIP can demonstrate a relatively good classification ability for the global image content. For example, when inputting an everyday street scene image containing a person wearing a T-shirt, CLIP can accurately identify the object category of the T-shirt in the image by relying on the association between its learned image features and language descriptions. It can distinguish T-shirts from other similar types of clothing such as shirts and jackets. This ability to accurately judge large categories of objects enables CLIP to play a good role in many conventional image recognition scenarios.

[0005] However, when it comes to classifying local details of an image, CLIP has obvious shortcomings. During its training, despite having a huge amount of image-text data pairs, CLIP focuses more on learning the general semantic information conveyed by the image as a whole and the category characteristics of the main objects, while the local subtle features of the objects in the image are not learned and strengthened in a detailed and targeted manner. For example, although CLIP can recognize the overall category of T-shirts in an image, once it focuses on the local details such as the color and style of the T-shirt, CLIP's performance is not satisfactory.

[0006] In summary, the classification method based on image recognition in the prior art cannot classify the details of the image. Summary of the invention

[0007] The present invention provides an image classification method, device, equipment and storage medium to solve the problem that the classification method based on image recognition in the prior art cannot classify the details of the image.

[0008] In a first aspect, the present invention provides an image classification method, comprising:

[0009] Acquire a target image and a target text, wherein the target text is a text description corresponding to the target image;

[0010] Inputting the target image into an image encoder to obtain an image coding sequence corresponding to the target image;

[0011] Inputting the target text into a text encoder to obtain a text encoding sequence corresponding to the target text;

[0012] splicing the image coding sequence and the text coding sequence to obtain a spliced ​​sequence;

[0013] Input the spliced ​​sequence into a preset encoder, make the vector at the first position in the spliced ​​sequence pay attention to the vector at the second position in the spliced ​​sequence, and output a target image vector representing the global information of the target image and a target text vector representing the global information of the target text;

[0014] The target image vector and the target text vector are input into a classification module to obtain a classification result of the target image.

[0015] In a second aspect, the present invention provides an image classification device, comprising:

[0016] An acquisition module, used to acquire a target image and a target text, wherein the target text is a text description corresponding to the target image;

[0017] A first encoding module, used for inputting the target image into an image encoder to obtain an image encoding sequence corresponding to the target image;

[0018] A second encoding module, used for inputting the target text into a text encoder to obtain a text encoding sequence corresponding to the target text;

[0019] A splicing module, used for splicing the image coding sequence and the text coding sequence to obtain a spliced ​​sequence;

[0020] A third encoding module is used to input the spliced ​​sequence into a preset encoder, so that the vector at the first position in the spliced ​​sequence pays attention to the vector at the second position in the spliced ​​sequence, and output a target image vector representing the global information of the target image and a target text vector representing the global information of the target text;

[0021] The classification module is used to input the target image vector and the target text vector into the classification module to obtain the classification result of the target image.

[0022] In a third aspect, the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned image classification method when executing the computer program.

[0023] In a fourth aspect, the present invention provides a computer-readable storage medium storing a computer program, which implements the steps of the above-mentioned image classification method when executed by a processor.

[0024] In the scheme implemented by the above-mentioned image classification method, after inputting the target image into the image encoder and the target text into the text encoder, what is obtained is the entire image coding sequence output by the image encoder and the entire text coding sequence of the text encoder, while when CLIP encodes the target image and the target text through the encoder, it outputs only the [CLS] vector representing the global semantics of the target image and the target text, so the adoption of this scheme can retain the detailed features in the target image and the target text. Furthermore, when the scheme obtains the entire image coding sequence output by the image encoder and the entire text coding sequence of the text encoder, and inputs the concatenated sequence of the image coding sequence and the text coding sequence into the preset encoder to obtain the target image vector and the target text vector representing the global information, the vectors at different positions in the concatenated sequence in the preset encoder pay attention to each other, thereby ensuring that all image coding sequences and text coding sequences can be related to each other, so that even if the final output is the target image vector and the target text vector representing the global information, the detailed features in the target image and the target text can also be included.

[0025] In summary, the present invention can solve the problem in the prior art that the classification method based on image recognition cannot classify the details of the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0027] Figure 1 is a flow chart of an image classification method in one embodiment of the present invention;

[0028] Figure 2 is another schematic flow chart of an image classification method according to an embodiment of the present invention;

[0029] Figure 3 is another schematic flow chart of an image classification method according to an embodiment of the present invention;

[0030] Figure 4 is another flow chart of an image classification method in one embodiment of the present invention;

[0031] Figure 5 is a flow chart of step S60 in one embodiment of the present invention;

[0032] Figure 6 is a flow chart of step S623 in one embodiment of the present invention;

[0033] Figure 7 is a structural schematic diagram of an image classification device in one embodiment of the present invention;

[0034] Figure 8 is a schematic diagram of a structure of a computer device in one embodiment of the present invention;

[0035] Fig. 9 It is another structural schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0036] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0037] See also Figure 1 As shown, Figure 1 A schematic flow chart of an image classification method provided by an embodiment of the present invention includes the following steps:

[0038] Step S10, obtaining a target image and a target text, wherein the target text is a text description corresponding to the target image.

[0039] It should be noted that in step S10, the target image can come from a variety of channels, such as being selected from an image database, or being a real scene image acquired in real time by an image acquisition device (such as a camera, scanner, etc.). The target text is a text description of the target image. Its function is similar to adding a semantic label to the image, which can help the subsequent processing flow to better understand the meaning expressed by the image. It can be described specifically according to the specific needs of the application. For example, if the target image is a photo depicting a sunset at the seaside, the corresponding target text may be "a beautiful view of the sun setting at the seaside in the evening". This text description can be generated by manual annotation or obtained from some data sets with image descriptions. Its purpose is to establish an association between the image and the text, laying the foundation for subsequent operations such as using text information to assist image classification.

[0040] Step S20: input the target image into an image encoder to obtain an image coding sequence corresponding to the target image.

[0041] It should be noted that in step S20, the image encoder is a module specially designed to extract image features and convert them into specific vector representations. For example, the image encoder can be a convolutional neural network based on deep learning, or a vision transformer (ViT). When the target image is input into the image encoder, the encoder will perform a series of operations according to the network structure and parameters pre-set inside it. For example, the convolution layer in the convolutional neural network will slide the convolution kernel on the image to perform a convolution operation, extracting the features of each local area of ​​the image, such as feature information such as edges, textures, and colors; then the pooling layer will perform a dimensionality reduction operation to further screen and condense these features, reducing the amount of data while retaining the main feature information. As the network continues to propagate forward, the image encoding sequence that is finally output is a vector representation of the original target image features. This encoding sequence contains various key features of the image, and further analysis and processing can be performed based on these features in the future, so that the computer can understand the content of the image from a numerical level.

[0042] It should be noted that the image coding sequence in step S20 is all coding sequences obtained after the target image is input into the image encoder.

[0043] Step S30: input the target text into a text encoder to obtain a text encoding sequence corresponding to the target text.

[0044] It should be noted that the text encoder in step S30 is also implemented based on a specific deep learning architecture, for example, it can be a Transformer, or a recurrent neural network and its variants. When the target text is input into the text encoder, the encoder will analyze and extract features from the words and other language units in the text according to its own operation logic. Taking the Transformer architecture as an example, it will first embed the text (convert the words into vector representations), and then capture the semantic associations and contextual relationships between the words in the text through structures such as multi-head attention mechanisms. After a series of calculations, the final output text encoding sequence is a vector representation of the semantic features of the target text. The sequence can reflect the semantic information of the text, such as the subject object, scene characteristics, emotional tendencies, etc. described in the text, and prepare for subsequent operations such as fusion with image information.

[0045] It should be noted that the text encoding sequence in step S30 is all encoding sequences obtained after the target text is input into the text encoder.

[0046] Step S40, splicing the image coding sequence and the text coding sequence to obtain a spliced ​​sequence.

[0047] It is understandable that in step S40, after respectively obtaining the image coding sequence corresponding to the target image and the text coding sequence corresponding to the target text, in order to allow the information of the two to be integrated and work together, they need to be spliced ​​together to form a new sequence, namely a spliced ​​sequence. The splicing method can be determined according to the specific coding representation and subsequent processing requirements. For example, the image coding sequence and the text coding sequence can be spliced ​​in dimension in sequence. Through this splicing operation, the feature information of the image and the text is integrated into the same vector sequence, which is convenient for subsequent further processing, so that the model can consider the features of both the image and the text at the same time to perform operations such as classifying the target image.

[0048] Step S50, inputting the spliced ​​sequence into a preset encoder, making the vector at the first position in the spliced ​​sequence focus on the vector at the second position in the spliced ​​sequence, and outputting a target image vector representing global information of the target image and a target text vector representing global information of the target text.

[0049] It should be noted that in step S50, the preset encoder can be a module built based on a specific deep learning architecture, and its core function is to realize the interaction and feature fusion between vectors at different positions in the splicing sequence, and to allow vectors at specific positions to pay attention to vectors at other positions, so as to mine deeper association information between images and texts. For example, the preset encoder can be a Transformer architecture. After a series of operations and feature fusion inside the preset encoder, two vectors can be output in the end. One is a target image vector representing the global information of the target image. This vector integrates the image features after interaction and fusion with text information, and more comprehensively reflects the information of the image at the overall semantic level; the other is a target text vector representing the global information of the target text. It also integrates information related to the image, more accurately reflects the descriptive semantics of the text for the image, and provides a more relevant and semantically rich feature representation for subsequent classification operations.

[0050] In one embodiment of the present invention, Figure 2 Another flow chart provided for an embodiment of the present invention is as follows: Figure 2 As shown, the step S50 of “inputting the spliced ​​sequence into a preset encoder so that the vector at the first position in the spliced ​​sequence pays attention to the vector at the second position in the spliced ​​sequence” includes:

[0051] Step S511, inputting the spliced ​​sequence into a preset encoder;

[0052] Step S512, for the vector at each position in the concatenated sequence, respectively generate a query vector, a key vector and a value vector;

[0053] Step S513, calculating the similarity between the query vector of the target position and the key vector corresponding to each position, and obtaining the attention weight corresponding to each position, wherein the attention weight is proportional to the similarity;

[0054] Step S514, weighted summing the attention weight corresponding to each position and the value vector corresponding to each position to obtain an output vector of the target position;

[0055] Step S515: Update the output vector to the vector of the target position.

[0056] In step S512, for the vectors at each position in the spliced ​​sequence, query vectors, key vectors and value vectors are generated respectively. The query vector, key vector and value vector are obtained through specific linear transformations (such as multiplying the vectors at the corresponding positions by a preset trainable weight matrix). The query vector is used to "inquire" about the degree of relevance of vectors at other positions; the key vector is used to cooperate with the query vector to determine the degree of association between the two; the value vector actually carries the specific feature information of the corresponding position, and the corresponding value vector will be extracted and fused based on the degree of association (i.e., attention weight) to achieve the redistribution and integration of features between vectors at different positions.

[0057] It should be noted that in step S513, the method for calculating the similarity can be any feasible method, such as using a dot product operation (the dot product of the query vector and the key vector), or other normalized dot products. The query vector of the target position is calculated in turn with the key vectors of all other positions, and the result obtained reflects the degree of association between them. The higher the degree of association, the greater the similarity value obtained, and the greater the attention weight assigned to the value vector of the corresponding position. For example, if the similarity calculated between the key vector of a certain position and the query vector of the target position is very high, it means that the information of this position is more important to the target position, and it will be given a higher weight to participate in the weighted summation operation later.

[0058] It should be noted that in step S514, based on the attention weights of each position obtained in the previous step, they are multiplied by the value vectors of the corresponding positions, and then all the product results are added to obtain a new output vector of the target position. This process can reintegrate the feature values ​​carried by each position according to the degree of correlation between different position vectors and the target position, so that the vector of the target position integrates the feature information of other positions with different degrees of correlation in the entire splicing sequence, highlighting the features of important positions and weakening the features of positions with weak correlation. It should be noted that the target position can be any position in the splicing sequence.

[0059] It should be noted that in step S515, the output vector is updated to the vector of the target position, that is, the new output vector obtained by weighted summation in step S514 replaces the vector of the original target position. In this way, the vector of each position in the entire splicing sequence has been updated in this way, so that the features contained in the vector of each position are no longer limited to its initial single representation, but integrate the features of other positions related to it in the entire sequence, better mining the interactive relationship between the image coding sequence and the text coding sequence, and laying the foundation for the subsequent output of the target image vector representing the global information of the target image and the target text vector representing the global information of the target text.

[0060] In one embodiment of the invention, Figure 3 FIG. 1 is another flow chart provided by an embodiment of the present invention, wherein the step S50 of “inputting the spliced ​​sequence into a preset encoder so that the vector at the first position in the spliced ​​sequence pays attention to the vector at the second position in the spliced ​​sequence” includes:

[0061] Step S521, inputting the spliced ​​sequence into a preset encoder;

[0062] Step S522, calculating the similarity between the image vector at the target position in the splicing sequence and each text vector;

[0063] Step S523, when the similarity between the image vector at the target position and the target text vector is greater than a preset threshold, the image vector at the target position and the target text vector are fused to obtain a target fused vector;

[0064] Step S524: Update the image vector of the target position to the target fusion vector.

[0065] It should be noted that in step S522, in the splicing sequence, the image vector and the text vector respectively characterize the target image and its corresponding description text from different angles. The similarity between them is calculated to measure the degree of correlation between the features presented by the image and the semantics expressed by the text. The similarity can be calculated in any feasible way, such as by cosine similarity calculation or dot product similarity calculation. By performing corresponding calculations on the image vector of the target position and each text vector in turn, a set of values ​​reflecting their similarity can be obtained. The higher the value, the more consistent the semantics represented by the text vector is with the image features corresponding to the image vector of the target position, which provides a basis for subsequent judgment on whether to perform fusion and how to fuse.

[0066] It should be noted that in step S523, the preset threshold is a pre-set measurement standard, which can be set according to the specific needs of the application, and can be used to determine whether the degree of correlation between the image vector and the text vector has reached a level worthy of fusion. When the similarity calculated by step S522 exceeds this preset threshold, it means that the corresponding image features and text semantics have a strong correlation, and the image vector and the target text vector at the target position are fused at this time. The fusion method can be any feasible method, such as weighted averaging (adding the corresponding elements of the two vectors according to the preset weights), splicing and then linear transformation, etc. The purpose is to integrate the useful information carried by the image vector and the text vector together to form a new target fusion vector. This vector combines the key features of both the image and the text, and can more comprehensively reflect the intrinsic connection between the target image and the corresponding text description.

[0067] It should be noted that in step S524, after the target fusion vector is obtained after the fusion is completed, the new target fusion vector is used to replace the image vector of the original target position. Through such an update, the image vector of the target position is no longer limited to the feature representation originally obtained from the image encoder, but incorporates the text feature information that is strongly associated with it. After the image vectors in the entire splicing sequence are updated through such selective fusion, the synergy between the image vector and the text vector is stronger, which paves the way for the subsequent output of the vector that can accurately reflect the global information of the target image and the global information of the target text, and helps to better utilize the comprehensive features of the image and text to serve related tasks such as image classification.

[0068] It should be noted that the target position in steps S521 to 524 is any position where the image vector is located.

[0069] Step S60: input the target image vector and the target text vector into a classification module to obtain a classification result of the target image.

[0070] It should be noted that, in step S60, the classification module may be any module that can determine the category to which the target image belongs based on the input target image vector and target text vector, and is not limited in detail here.

[0071] In one embodiment of the present invention, Figure 4 Another flow chart of the embodiment of the present invention is shown in FIG. Figure 4 As shown, step S60 includes:

[0072] Step S61, inputting the target image vector and the target text vector into multiple fully connected layers to obtain the classification result of the target image.

[0073] It should be noted that the fully connected layer is a layer structure widely used in neural networks. In step S61, the target image vector and the target text vector are fed into multiple fully connected layers as input. Each neuron in the fully connected layer is connected to all neurons in the previous layer. For example, if the dimension of the input vector is, there are neurons in the first fully connected layer, then there will be a connection weight from the input layer to the first fully connected layer. These connection weights are continuously adjusted and optimized by the back propagation algorithm during the model training process. When the input vector enters the fully connected layer, it will perform matrix multiplication with these connection weights, and then add a bias term (also learned during the training process), and the result will be processed by the activation function to generate a new output vector. The role of the activation function is to introduce nonlinear factors so that the neural network can fit various complex functional relationships. In step S61, it is mentioned that using multiple fully connected layers can gradually perform higher-level abstraction and feature extraction on the input vector. The first fully connected layer will perform preliminary linear combination and nonlinear transformation on the target image vector and the target text vector to obtain a new intermediate vector. The dimension and semantic information of this intermediate vector are different from those of the input vector, and it represents the combined features of the image and text in a more abstract way. Then, this intermediate vector will be used as the input of the next fully connected layer to continue similar operations. As the fully connected layers progress, the features contained in the vector become more and more abstract, and can increasingly reflect the key information related to image classification. For example, in a simple image classification task, the initial input vector may contain basic features such as the color, texture, and vocabulary of the image description. After passing through multiple fully connected layers, the output vector may represent more advanced semantic features of the category to which the image belongs, such as "is it an animal or a plant?", "is it a means of transportation or a building?", etc. The output dimension of the last fully connected layer usually corresponds to the number of categories to be classified. In this way, using multiple fully connected layers to process the target image vector and the target text vector can comprehensively consider the information of the image and text, and effectively classify the target image.

[0074] In one embodiment of the present invention, Figure 5 is a flow chart of step S60 in an embodiment of the present invention, as shown in Figure 5 As shown, step S60 includes:

[0075] Step S621, setting a text prompt sentence, and setting a text vector position and a category label position in the text prompt sentence, wherein the text vector position is used to fill the target text vector, and the category label position is used to fill the category label;

[0076] Step S622, extracting the target text vector and category label in the text prompt sentence;

[0077] Step S623, inputting the target text vector, the target image vector and the category label into a classification module to obtain the category label to which the target image belongs.

[0078] It should be noted that in step S621, the text prompt sentence is a text template, which is designed to better guide the model to classify the target image using the target text vector and combining the category label. By setting a specific format, the text vector position and the category label position are reserved to make it versatile and flexible. The reserved text vector position will later embed the target text vector obtained in the previous steps into this position, so that the text prompt sentence can describe the image based on specific text feature information; and the category label position can be filled with the corresponding specific category (such as "cat", "dog", "car", etc.) in different classification scenarios to assist the model in determining whether the image belongs to this category. This setting method is equivalent to building a bridge, connecting the target text vector, the target image and the category judgment, and building a reasonable framework for subsequent classification operations.

[0079] It should be noted that in step S622, after the corresponding content is filled in according to the set text prompt sentence format, it is necessary to accurately extract key information from the sentence, that is, the target text vector and category label.

[0080] It should be noted that in step S623, the classification module can be constructed based on a preset neural network architecture, and has powerful data analysis and classification decision-making capabilities. After the target text vector extracted from the text prompt sentence, the target image vector obtained by the previous processing, and the clear category label are input into the classification module, the classification module will comprehensively consider this information. It will analyze the image features contained in the target image vector, the text description semantics expressed by the target text vector, and the given category label, and judge whether the target image really belongs to the category represented by the category label through internal complex calculations and learned patterns. For example, the classification module may compare the given category label (such as "dog") based on the visual features of the object shape, color, etc. reflected by the target image vector, combined with the semantic information such as animal habits and object functions described by the target text vector, to finally determine whether the target image does belong to the category of "dog". After such a comprehensive judgment, the category label to which the target image belongs is output as the final classification result.

[0081] In one embodiment of the present invention, Figure 6 is a flow chart of step S623 in an embodiment of the present invention, as shown in Figure 6 As shown, step S623 includes:

[0082] Step S6231, inputting the target text vector, the target image vector and the category label into a classification module;

[0083] Step S6232, based on the target text vector and the target image vector, calculating the probability that the target image belongs to each category label;

[0084] Step S6233, when the probability that the target image belongs to the target category label is the largest, the category label to which the target image belongs is determined as the target category label.

[0085] It should be noted that, in step S6232, the relationship between the target text vector and the target image vector can be used to quantify the possibility that the target image belongs to different category labels.

[0086] In step S6233, after the probability of the target image belonging to each category label is calculated in step S6232, these probability values ​​are compared. Among all the calculated probabilities, the category label corresponding to the one with the largest probability value is identified as the category label to which the target image belongs. This is a classification decision-making method based on the principle of probability maximization, which conforms to the logic of selecting the most likely situation under uncertainty. For example, if the calculated probability that the target image belongs to the category label "cat" is 0.7, the probability that it belongs to the category label "dog" is 0.2, and the probability that it belongs to the category label "rabbit" is 0.1, then according to the principle of maximum probability, the category label to which the target image belongs is determined to be "cat", thereby completing the classification operation of the target image.

[0087] In one embodiment of the present invention, step S6232 includes:

[0088] Based on the following formula, the probability that the target image belongs to each category label is calculated:

[0089]

[0090] Wherein, x is any of the category labels, i is the total number of the category labels, P(y=i│x) is the probability that the target image belongs to the category label x, f is the target image vector, g(t i ) is the target text vector, cos(g(t i ),f is the cosine similarity between the target text vector and the target image vector.

[0091] It can be understood that the meaning of the above formula is that for each possible category label x (where i represents the total number of category labels, traversing all possible categories), by calculating the target text vector g(t i)(corresponding to the text vector representation of each category label) and the cosine similarity of the target image vector f cos(g(t i ),f, and then perform an exponential operation on the cosine similarity exp(cos(g(t i ),f)), and use it as the numerator. The denominator is the sum of the cosine similarities after exponential operation corresponding to all category labels, that is, ∑exp(cos(g(t i ),f)). Through such a combination of numerator and denominator, we can finally get the probability P(y=i│x) that the target image belongs to the category label x. The essence of this calculation method is to measure the possibility of the image belonging to a certain category based on the similarity between the text vector and the image vector. The higher the similarity (i.e. the higher the cosine similarity, the larger the proportion in the numerator after the exponential operation), the greater the probability of the corresponding category, and vice versa.

[0092] It can be understood that the denominator of the above formula realizes the normalization of the calculation results of all category labels, ensuring that for any given target image, the sum of the probabilities corresponding to all category labels calculated is 1, which conforms to the basic definition of probability, so that each probability value can be reasonably interpreted as the possibility that the target image belongs to the category, which facilitates classification decisions.

[0093] In this way, after the target image is input into the image encoder and the target text is input into the text encoder, what is obtained is the entire image coding sequence output by the image encoder and the entire text coding sequence output by the text encoder. When CLIP encodes the target image and target text through the encoder, it only outputs the [CLS] vector representing the global semantics of the target image and target text. Therefore, the use of this scheme can retain the detailed features in the target image and target text. Furthermore, when the scheme obtains the entire image coding sequence output by the image encoder and the entire text coding sequence output by the text encoder, and inputs the concatenated sequence of the image coding sequence and the text coding sequence into the preset encoder to obtain the target image vector and target text vector representing global information, the vectors at different positions in the concatenated sequence in the preset encoder pay attention to each other, thereby ensuring that all image coding sequences and text coding sequences can be related to each other, so that even if the final output is the target image vector and target text vector representing global information, they can also contain the detailed features in the target image and target text.

[0094] In summary, the present invention can solve the problem in the prior art that the classification method based on image recognition cannot classify the details of the image.

[0095] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0096] In one embodiment, a device for classifying images is provided, and the device for classifying images corresponds one-to-one to the method for classifying images in the above embodiment. Figure 7 As shown, the image classification device includes an acquisition module 710, a first encoding module 720, a second encoding module 730, a splicing module 740, a third encoding module 750 and a classification module 760. The functional modules are described in detail as follows:

[0097] An acquisition module 710 is used to acquire a target image and a target text, wherein the target text is a text description corresponding to the target image;

[0098] A first encoding module 720, configured to input the target image into an image encoder to obtain an image encoding sequence corresponding to the target image;

[0099] The second encoding module 730 is used to input the target text into a text encoder to obtain a text encoding sequence corresponding to the target text;

[0100] A splicing module 740 is used to splice the image coding sequence and the text coding sequence to obtain a spliced ​​sequence;

[0101] A third encoding module 750 is used to input the spliced ​​sequence into a preset encoder, so that the vector at the first position in the spliced ​​sequence pays attention to the vector at the second position in the spliced ​​sequence, and output a target image vector representing the global information of the target image and a target text vector representing the global information of the target text;

[0102] The classification module 760 is used to input the target image vector and the target text vector into the classification module to obtain the classification result of the target image.

[0103] In one embodiment, the third encoding module 750 is specifically used to:

[0104] Inputting the spliced ​​sequence into a preset encoder;

[0105] For the vector at each position in the concatenated sequence, respectively generate a query vector, a key vector and a value vector;

[0106] Calculate the similarity between the query vector of the target position and the key vector corresponding to each position, and obtain the attention weight corresponding to each position, wherein the attention weight is proportional to the similarity;

[0107] The attention weight corresponding to each position and the value vector corresponding to each position are weighted and summed to obtain an output vector of the target position;

[0108] Update the output vector to the vector of the target position.

[0109] In one embodiment, the third encoding module 750 is specifically used to:

[0110] Inputting the spliced ​​sequence into a preset encoder;

[0111] Calculate the similarity between the image vector of the target position in the spliced ​​sequence and each text vector;

[0112] When the similarity between the image vector at the target position and the target text vector is greater than a preset threshold, fusing the image vector at the target position and the target text vector to obtain a target fused vector;

[0113] The image vector at the target position is updated to the target fusion vector.

[0114] In one embodiment, the classification module 760 is specifically used to:

[0115] The target image vector and the target text vector are input into multiple fully connected layers to obtain a classification result of the target image.

[0116] In one embodiment, the classification module 760 is specifically used to:

[0117] Setting a text prompt sentence, and setting a text vector position and a category label position in the text prompt sentence, wherein the text vector position is used to fill the target text vector, and the category label position is used to fill the category label;

[0118] Extracting target text vectors and category labels from the text prompt sentence;

[0119] The target text vector, the target image vector and the category label are input into a classification module to obtain the category label to which the target image belongs.

[0120] In one embodiment, the classification module 760 is further used to:

[0121] Inputting the target text vector, the target image vector and the category label into a classification module;

[0122] Based on the target text vector and the target image vector, calculating the probability that the target image belongs to each category label;

[0123] When the probability that the target image belongs to the target category label is the largest, the category label to which the target image belongs is determined as the target category label.

[0124] In one embodiment, the classification module 760 is further configured to:

[0125] Based on the following formula, the probability that the target image belongs to each category label is calculated:

[0126]

[0127] Wherein, x is any of the category labels, i is the total number of the category labels, P(y=i│x) is the probability that the target image belongs to the category label x, f is the target image vector, g(t i ) is the target text vector, cos(g(t i ),f is the cosine similarity between the target text vector and the target image vector.

[0128] The present invention provides an image classification device. After inputting a target image into an image encoder and inputting a target text into a text encoder, what is obtained is the entire image coding sequence output by the image encoder and the entire text coding sequence output by the text encoder. When CLIP encodes the target image and the target text through the encoder, it outputs only the [CLS] vector representing the global semantics of the target image and the target text. Therefore, the use of this scheme can retain the detailed features in the target image and the target text. Further, when the scheme obtains the entire image coding sequence output by the image encoder and the entire text coding sequence output by the text encoder, and inputs the concatenated sequence of the image coding sequence and the text coding sequence into a preset encoder to obtain the target image vector and the target text vector representing global information, the vectors at different positions in the concatenated sequence in the preset encoder pay attention to each other, thereby ensuring that all image coding sequences and text coding sequences can be related to each other, so that even if the target image vector and the target text vector representing global information are finally output, they can also contain the detailed features in the target image and the target text.

[0129] In summary, this solution can solve the problem that the classification method based on image recognition in the prior art cannot classify the details of the image.

[0130] For the specific definition of the image classification device, please refer to the definition of the image classification method above, which will not be repeated here. Each module in the above-mentioned image classification device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0131] Based on the above image classification method, such as Figure 8As shown, an embodiment of the present invention further provides a structural schematic diagram of an image classification device, the classification device includes a processor 81 and a memory 82 coupled to the processor 81. The memory 82 stores a computer program, and when the computer program is executed by the processor 81, the processor 81 executes the steps of the image classification method in the above embodiment.

[0132] For other details about how the processor 81 in the above-mentioned image classification device implements the above-mentioned technical solution, please refer to the description of the image classification method provided in the above-mentioned invention embodiment, which will not be repeated here.

[0133] Among them, the processor 81 can also be called a CPU (Central Processing Unit), and the processor 81 may be an integrated circuit chip with signal processing capabilities; the processor 81 can also be a general-purpose processor, DSP (Digital Signal Process), ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, among which the general-purpose processor can be a microprocessor or the processor 81 can also be any conventional processor, etc.

[0134] like Fig. 9 As shown, the embodiment of the present invention also provides a schematic diagram of the structure of a computer-readable storage medium, on which a readable computer program 91 is stored; wherein the computer program 91 can be stored in the above storage medium in the form of a software product, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, magnetic disk or optical disk, ROM (Read-Only Memory), RAM (Random Access Memory) and other media that can store program codes, or terminal devices such as computers, servers, mobile phones, and tablets.

[0135] In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0136] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0137] In addition, each functional module in each embodiment of the present invention may be integrated into a processing module, or each module may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may be stored in a computer-readable storage medium.

[0138] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0139] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium), or a semiconductor medium (e.g., an SSD (solid state disk, solid state hard disk)).

[0140] The technical solution provided by the present invention is introduced in detail above. The present invention applies specific examples to illustrate the principle and implementation mode of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation mode and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

[0141] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, optical storage, etc.) containing computer-usable program code.

[0142] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0143] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0144] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0145] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations. The non-Company software tools or components that appear in the embodiments of the present invention are merely examples and do not represent actual use.

Claims

1. A method for classifying an image, characterized in that: include: Acquire a target image and a target text, wherein the target text is a text description corresponding to the target image; Inputting the target image into an image encoder to obtain an image coding sequence corresponding to the target image; Inputting the target text into a text encoder to obtain a text encoding sequence corresponding to the target text; splicing the image coding sequence and the text coding sequence to obtain a spliced ​​sequence; Input the spliced ​​sequence into a preset encoder, make the vector at the first position in the spliced ​​sequence pay attention to the vector at the second position in the spliced ​​sequence, and output a target image vector representing the global information of the target image and a target text vector representing the global information of the target text; The target image vector and the target text vector are input into a classification module to obtain a classification result of the target image.

2. The image classification method according to claim 1, characterized in that: The step of inputting the spliced ​​sequence into a preset encoder so that a vector at a first position in the spliced ​​sequence pays attention to a vector at a second position in the spliced ​​sequence includes: Inputting the spliced ​​sequence into a preset encoder; For the vector at each position in the concatenated sequence, respectively generate a query vector, a key vector and a value vector; Calculate the similarity between the query vector of the target position and the key vector corresponding to each position, and obtain the attention weight corresponding to each position, wherein the attention weight is proportional to the similarity; The attention weight corresponding to each position and the value vector corresponding to each position are weighted and summed to obtain an output vector of the target position; Update the output vector to the vector of the target position.

3. The image classification method according to claim 1, characterized in that: The step of inputting the spliced ​​sequence into a preset encoder so that a vector at a first position in the spliced ​​sequence pays attention to a vector at a second position in the spliced ​​sequence includes: Inputting the spliced ​​sequence into a preset encoder; Calculate the similarity between the image vector of the target position in the spliced ​​sequence and each text vector; When the similarity between the image vector at the target position and the target text vector is greater than a preset threshold, fusing the image vector at the target position and the target text vector to obtain a target fused vector; The image vector at the target position is updated to the target fusion vector.

4. The image classification method according to claim 1, characterized in that: The step of inputting the target image vector and the target text vector into a classification module to obtain a classification result of the target image includes: The target image vector and the target text vector are input into multiple fully connected layers to obtain a classification result of the target image.

5. The image classification method according to claim 1, characterized in that: The step of inputting the target image vector and the target text vector into a classification module to obtain a classification result of the target image includes: Setting a text prompt sentence, and setting a text vector position and a category label position in the text prompt sentence, wherein the text vector position is used to fill the target text vector, and the category label position is used to fill the category label; Extracting target text vectors and category labels from the text prompt sentence; The target text vector, the target image vector and the category label are input into a classification module to obtain the category label to which the target image belongs.

6. The image classification method according to claim 5, characterized in that: The step of inputting the target text vector, the target image vector and the category label into a classification module to obtain the category label to which the target image belongs includes: Inputting the target text vector, the target image vector and the category label into a classification module; Based on the target text vector and the target image vector, calculating the probability that the target image belongs to each category label; When the probability that the target image belongs to the target category label is the largest, the category label to which the target image belongs is determined as the target category label.

7. The image classification method according to claim 6, characterized in that: The calculating the probability that the target image belongs to each category label based on the target text vector and the target image vector includes: Based on the following formula, the probability that the target image belongs to each category label is calculated: Where x is any of the category labels, i is the total number of category labels, P(y=i|x) is the probability that the target image belongs to category label x, f is the target image vector, g(t i ) is the target text vector, cos(g(t i ),f is the cosine similarity between the target text vector and the target image vector.

8. An image classification device, characterized in that: include: An acquisition module, used to acquire a target image and a target text, wherein the target text is a text description corresponding to the target image; A first encoding module, used for inputting the target image into an image encoder to obtain an image encoding sequence corresponding to the target image; A second encoding module, used for inputting the target text into a text encoder to obtain a text encoding sequence corresponding to the target text; A splicing module, used for splicing the image coding sequence and the text coding sequence to obtain a spliced ​​sequence; A third encoding module is used to input the spliced ​​sequence into a preset encoder, so that the vector at the first position in the spliced ​​sequence pays attention to the vector at the second position in the spliced ​​sequence, and output a target image vector representing the global information of the target image and a target text vector representing the global information of the target text; The classification module is used to input the target image vector and the target text vector into the classification module to obtain the classification result of the target image.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the image classification method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the image classification method according to any one of claims 1 to 7 are implemented.

Images