Image classification method, electronic equipment and computer readable storage medium

Abstract

The embodiment of the invention provides an image classification method, electronic equipment and a computer readable storage medium, and relates to the technical field of artificial intelligence. The method comprises the steps of inputting a first sample image into a pre-training visual language model for text generation to obtain a first sample image text description; adding a problem guide language to the first sample image text description, and then inputting the first initial large language model to carry out category prediction to obtain a first sample prediction image category; performing fine tuning on the initial large language model based on the real image category and the first sample prediction image category to obtain a first target large language model; inputting the first target image into a pre-training visual language model for text generation to obtain a first target image text description; and adding a problem guide language to the first target image text description, and then inputting the first target large language model to carry out category prediction to obtain a first target prediction image category. The image classification accuracy is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an image classification method, an electronic device, and a computer-readable storage medium. Background Art

[0002] In various computer vision applications, we often face the challenge of distribution differences between the training domain and the test domain (also known as domain shift). For example, self-driving cars need to be able to handle bad weather conditions that did not appear during the training phase, which places great demands on the generalization ability of the model. To alleviate domain shift, cross-domain learning aims to extract domain-invariant knowledge between the source domain and the target domain.

[0003] At present, some works on large-scale visual language models have shown that image classification performance has been significantly improved by learning from a large number of paired image-text samples. However, learning domain-invariant features based on large-scale visual language models still faces huge challenges of domain shift, resulting in poor cross-domain performance. On the other hand, large language models have strong zero-shot generalization capabilities in text-based tasks. However, due to the inherent differences between modalities and task structures, the success of large language models on text cannot be directly extended to pure vision and visual-language tasks.

[0004] In summary, the accuracy of image classification of current related technologies is poor. Summary of the invention

[0005] The main purpose of the embodiments of the present application is to propose an image classification method, an electronic device and a computer-readable storage medium, thereby improving the accuracy of image classification.

[0006] To achieve the above objective, a first aspect of an embodiment of the present application proposes an image classification method, the method comprising:

[0007] Obtain a first sample image and a real image category;

[0008] Inputting the first sample image into a pre-trained visual language model for text generation to obtain a text description of the first sample image;

[0009] Adding a question guide to the first sample image text description to generate a first sample guide text;

[0010] Inputting the first sample guide text into a first initial large language model for category prediction to obtain a first sample predicted image category;

[0011] Fine-tune the initial large language model based on the real image category and the first sample predicted image category to obtain a first target large language model;

[0012] Acquire a first target image, and input the first target image into a pre-trained visual language model for text generation to obtain a text description of the first target image;

[0013] Adding the question guide words to the first target image text description to generate a first target guide text;

[0014] The first target guiding text is input into the first target large language model for category prediction to obtain a first target predicted image category.

[0015] Optionally, the question guide includes an information provision template and a category question template; and the step of adding the question guide to the first sample image text description to generate the first sample guide text includes:

[0016] Acquire a category set; the category set includes multiple categories, and the multiple categories include the first sample prediction image category;

[0017] The first sample image text description is embedded in the information provision template, and the category set is embedded in the category question template to obtain the first sample guide text.

[0018] Optionally, the first sample image text description includes a first sample image label, and the pre-trained visual language model includes an image label model; and the step of inputting the first sample image into the pre-trained visual language model for text generation to obtain the first sample image text description includes:

[0019] Obtain a tag set, the tag set including a plurality of tags;

[0020] Embed each of the labels into a preset label template to generate label text;

[0021] Inputting the first sample image and the label text into the image label model to perform similarity calculation to obtain a first image label similarity;

[0022] The plurality of labels are screened according to the first image label similarity to obtain the first sample image label.

[0023] Optionally, the first sample image text description further includes first sample image attributes, and the pre-trained visual language model includes an image attribute model; and the step of inputting the first sample image into the pre-trained visual language model for text generation to obtain the first sample image text description further includes:

[0024] Embed each of the tags into a preset attribute guide template to generate attribute guide text;

[0025] Input each of the attribute guide texts into a pre-trained language model to generate attribute texts, thereby obtaining candidate attribute texts;

[0026] Inputting the first sample image and the candidate attribute text into the image attribute model to perform similarity calculation to obtain a first image attribute similarity;

[0027] The plurality of candidate attribute texts are screened according to the first image attribute similarity to obtain the first sample image attribute.

[0028] Optionally, the first sample image text description further includes a first sample image caption, and the pre-trained visual language model includes an image caption model; and the step of inputting the first sample image into the pre-trained visual language model for text generation to obtain the first sample image text description further includes:

[0029] The first sample image is input into the image caption model to generate captions, thereby obtaining the first sample image captions.

[0030] Optionally, fine-tuning the initial large language model based on the real image category and the first sample predicted image category to obtain a first target large language model includes:

[0031] Screening a plurality of first sample prediction image categories according to the real image category to obtain a first sample target prediction category, and taking the prediction probability of the first sample target prediction category as a first prediction probability;

[0032] Performing logarithm calculation on the first predicted probability to obtain a first logarithm;

[0033] Reverse the first logarithm to obtain first loss data;

[0034] Parameters of the initial large language model are adjusted according to the first loss data to obtain the first target large language model.

[0035] Optionally, after fine-tuning the initial large language model based on the real image category and the first sample predicted image category to obtain a first target large language model, the method further includes:

[0036] Acquire a second sample image; wherein the first sample image and the second sample image are in different domains;

[0037] Using the first target large language model to perform category prediction on the second sample image input to obtain a pseudo image category;

[0038] Inputting the second sample image into the pre-trained visual language model to generate text, thereby obtaining a text description of the second sample image;

[0039] Adding a question guide to the second sample image text description to generate a second sample guide text;

[0040] Inputting the second sample guide text into the first target large language model for category prediction to obtain a second sample predicted image category;

[0041] Fine-tune the first target large language model according to the real image category, the first sample predicted image category, the pseudo image category, and the second sample predicted image category to obtain a second target large language model;

[0042] Acquire a second target image, and input the second target image into the pre-trained visual language model for text generation to obtain a text description of the second target image;

[0043] Adding the question guide words to the second target image text description to generate a second target guide text;

[0044] The second target guiding text is input into the second target large language model for category prediction to obtain a second target predicted image category.

[0045] Optionally, fine-tuning the first target large language model according to the real image category, the first sample predicted image category, the pseudo image category, and the second sample predicted image category to obtain a second target large language model includes:

[0046] Screening a plurality of first sample prediction image categories according to the real image category to obtain a first sample target prediction category, and taking the prediction probability of the first sample target prediction category as a first prediction probability;

[0047] Performing logarithmic calculation on the first predicted probability to obtain a first logarithm, and negating the first logarithm to obtain first loss data;

[0048] Screening a plurality of second sample prediction image categories according to the pseudo image category to obtain a second sample target prediction category, and using the prediction probability of the first sample target prediction category as the second prediction probability;

[0049] Performing logarithmic calculation on the second predicted probability to obtain a second logarithm, and negating the second logarithm to obtain second loss data;

[0050] Adding the first loss data and the second loss data to obtain target loss data;

[0051] Parameters of the first target large language model are adjusted according to the target loss data to obtain the second target large language model.

[0052] To achieve the above-mentioned purpose, the second aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0053] To achieve the above objectives, the third aspect of an embodiment of the present application proposes a storage medium, which is a computer-readable storage medium and stores a computer program. When the computer program is executed by a processor, the method described in the first aspect is implemented.

[0054] The present application proposes an image classification method, electronic device, and computer-readable storage medium, which uses a visual language model to extract text descriptions of image modalities, and fine-tunes the designed question guides on a large language model, thereby applying the generalization capability of the large language model to the cross-domain transfer learning problem on the image modality, and can effectively improve the image classification accuracy.

[0055] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 is an optional flow chart of the image classification method provided in the embodiment of the present application;

[0057] Figure 2 is a general framework diagram of an image classification method according to an embodiment of the present application;

[0058] Figure 3 yes Figure 1 Flowchart of step 102;

[0059] Figure 4 yes Figure 1 Flowchart of step 102;

[0060] Figure 5 yes Figure 1 Flow chart of step 103;

[0061] Figure 6 yes Figure 1 Flow chart of step 105;

[0062] Figure 7is an optional flow chart of an image classification method provided by another embodiment of the present application;

[0063] Figure 8 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0065] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0066] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0067] First, some nouns involved in this application are analyzed:

[0068] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0069] Large Language Model (LLM): refers to a deep learning model trained with a large amount of text data that can generate natural language text or understand the meaning of language text. Large language models can handle a variety of natural language tasks, such as text classification, question answering, and dialogue, and are an important path to artificial intelligence.

[0070] Back propagation: The general principle of back propagation is: input the training set data into the input layer of the neural network, pass through the hidden layer of the neural network, and finally reach the output layer of the neural network and output the result; since there is an error between the output result of the neural network and the actual result, the error between the estimated value and the actual value is calculated, and the error is backpropagated from the output layer to the hidden layer until it propagates to the input layer; in the process of back propagation, the values ​​of various parameters are adjusted according to the error; the above process is continuously iterated until convergence.

[0071] There are many domain generalization and unsupervised domain adaptation methods that alleviate the problem of domain shift by learning domain-invariant features in the image modality. However, these methods only use the image modality for cross-domain learning, and the lack of use of the text modality limits further performance improvement.

[0072] Different from using only the image modality, some recent works on large-scale visual language models (VLMs) have shown significant improvements in image classification performance by learning from a large number of paired image-text samples. For example, CLIP consists of an image encoder and a text encoder, which are trained on 400 million image-text pairs via contrastive learning. However, learning domain-invariant features based on CLIP still faces huge challenges of domain shift, resulting in poor cross-domain performance.

[0073] On the other hand, Large Language Models (LLMs) have strong zero-shot generalization capabilities in text-based tasks. However, due to the inherent differences between modalities and task structures, the success of LLMs on text cannot be directly extended to pure vision and vision-language tasks. There is currently no related work combining LLMs with visual cross-domain learning.

[0074] Based on this, the present application provides an image classification method, the purpose of which is to alleviate the domain shift problem in cross-domain learning of vision through the inherent generalization ability of a large language model, thereby achieving better object recognition effects.

[0075] The image classification method of the embodiment of the present application can be executed by the server alone, or by the terminal alone, or by the terminal and the server together. In addition, the image classification method provided by the embodiment of the present application can also be software running on the server. The server can be configured as an independent physical server, or as a server cluster or distributed system composed of multiple physical servers, or as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the image classification method, etc., but is not limited to the above forms.

[0076] The embodiments of the present application provide an image classification method, an electronic device, and a computer-readable storage medium, which are specifically described through the following embodiments. First, the image classification method in the embodiments of the present application is described.

[0077] See also Figure 1 According to an embodiment of the present application, an image classification method includes:

[0078] Step 101, obtaining a first sample image and a real image category;

[0079] Step 102, inputting the first sample image into a pre-trained visual language model to generate text, and obtaining a text description of the first sample image;

[0080] Step 103, adding a question guide word to the first sample image text description to generate a first sample guide text;

[0081] Step 104, inputting the first sample guide text into the first initial large language model for category prediction to obtain a first sample predicted image category;

[0082] Step 105, fine-tuning the initial large language model based on the real image category and the first sample predicted image category to obtain a first target large language model;

[0083] Step 106, obtaining a first target image, and inputting the first target image into a pre-trained visual language model for text generation to obtain a text description of the first target image;

[0084] Step 107, adding a question guide word to the first target image text description to generate a first target guide text;

[0085] Step 108: input the first target guiding text into the first target large language model for category prediction to obtain the first target predicted image category.

[0086] Reference Figure 2 , Figure 2 It is an overall framework diagram of the image classification method according to an embodiment of the present application.

[0087] Figure 2 The upper left side of is the first sample image in step 101. The first sample image is an image used to train the model. The first sample image is marked with a real image category in advance to indicate the category to which the first sample image belongs. For example, if the first sample image is an image of a dog, the real image category is "dog". The real image category can be marked manually.

[0088] It should be noted that Figure 2The Trainable shown indicates that the LLM model needs fine-tuning, Frozen indicates that VLMs do not need fine-tuning, DG indicates domain generalization tasks, and UDA indicates unsupervised domain adaptation tasks. For the domain generalization task, the first sample image in the source domain is converted into a description in the text modality through VLMs, and then the extracted description is used to fine-tune the LLM. For the unsupervised domain adaptation task, the second sample image in the target domain with the pseudo image category is further used to fine-tune the LLM.

[0089] like Figure 2 As shown, this application proposes a classification framework for visual cross-domain learning based on a large language model (VLLaVO). Assume that there is a set of source domains with a number S Each domain has training data and labels where n i represents the number of samples in each domain. We let Represents all the source domain training data, and expects the model to have better performance on the target domain T. In the setting of domain generalization tasks, many works assume that there is a global label mapping function that can handle different domains. First, VLLaVO uses a pre-trained visual language model (VLM) to generate text descriptions of images. These obtained text descriptions still have domain offsets. Therefore, these descriptions and their category labels are combined by fine-tuning the large language model (LLM) using the designed question guide. Through fine-tuning, LLM improves the ability to follow the question guide and focuses on domain-invariant information related to classification. The fine-tuning process is described in detail below.

[0090] In one example, the first sample image is obtained by: (1) downloading the first sample image from a public data set; (2) collecting images from the Internet using crawler technology to obtain the first sample image.

[0091] In step 102, the pre-trained visual language model (VLM) is a model that can generate text descriptions based on images. The first sample image is input into the pre-trained visual language model for text generation, and the text description of the first sample image can be obtained. For example, for each first sample image x i , using the pre-trained VLM to generate the corresponding first sample image text description The first sample image text description consists of three parts, namely, first sample image tags (Tags), first sample image attributes (Attributes), and first sample image captions (Captions).

[0092] like Figure 2 As shown, the pre-trained visual language models (VLMs) include image label models (VLMTag ), Image Attribute Model (VLM Att ) and the Image Captioning Model (VLM Cap ). The first sample image text description includes a first sample image label, a first sample image attribute, and a first sample image subtitle.

[0093] The image labeling model is a model that can generate labels based on images. For example, if the first sample image with the real image category of "house" is input into the image labeling model, the first image labels obtained include "schoolhouse", "birdhouse", etc.

[0094] The image attribute model is a model that can generate attributes based on an image. For example, if the first sample image with the real image category of "house" is input into the image label model, the first image attributes obtained include "house which is a building with walls and a roof", "residence which has windows", etc.

[0095] The image captioning model is a model that can generate captions based on images. For example, if the first sample image with the real image category of "house" is input into the image labeling model, the first image captions obtained include "a picture of a house with a bird on the roof", "a cartoon house with a lot of windows", etc.

[0096] It should be noted that the image label model and the image attribute model may be a contrastive language-image pre-training (CLIP) model, and the image caption model may be a bilingual language-image pre-training (BLIP) model.

[0097] In one embodiment, referring to Figure 3 , step 102 comprises:

[0098] Step 301, obtaining a tag set, the tag set including a plurality of tags;

[0099] Step 302, embed each label into a preset label template to generate label text;

[0100] Step 303, inputting the first sample image and the label text into the image label model to perform similarity calculation to obtain image label similarity;

[0101] Step 304 , screening multiple labels according to image label similarity to obtain a first sample image label.

[0102] Specifically, a comprehensive label dictionary, i.e., a label set, is first constructed using labels from a variety of different data sets. Afterwards, each label is embedded into a standard label template, such as "A photo of {tag}", and the pre-trained CLIP model is used to obtain the label features of each label after embedding the template. For each picture, the CLIP model is also used to obtain its image features, and the cosine similarity is calculated using the image features and label features. Based on the cosine similarity corresponding to each label, the K labels closest to the first sample image can be obtained. The benefit of this embodiment is that the accuracy of generating the first sample image label is improved.

[0103] In one embodiment, referring to Figure 4 , step 102 further includes:

[0104] Step 401, embed each tag into a preset attribute guide template to generate attribute guide text;

[0105] Step 402, input each attribute guide text into the pre-trained language model to generate attribute text, and obtain candidate attribute text;

[0106] Step 403, inputting the first sample image and the candidate attribute text into the image attribute model to perform similarity calculation to obtain the first image attribute similarity;

[0107] Step 404 , screening multiple candidate attribute texts according to the first image attribute similarity to obtain the first sample image attribute.

[0108] Specifically, the first sample image label is usually relatively short, so that it cannot include all the detailed descriptions of the first sample image. To this end, a pre-trained language model (such as GPT-3) is used to generate the attributes required to distinguish each label to obtain the first candidate attribute text. Specifically, by inputting the first attribute guide text as "What are the useful features for distinguishing a{tag}in a photo?" to prompt GPT-3 to generate key attributes for distinguishing visual features. Similar to the label, for each first sample image, the cosine similarity between the first sample image and the first candidate attribute text is calculated using the pre-trained CLIP model to obtain the closest M first sample image attributes. The benefit of this embodiment is that it improves the accuracy of generating the first sample image attributes.

[0109] In one embodiment, step 102 further includes: inputting the first sample image into an image caption model to generate captions to obtain first sample image captions.

[0110] Specifically, labels and attributes are all conceptual knowledge about images, lacking a specific description of the image. Therefore, it is necessary to further expand the extracted image information through image captions. For each first sample image, the image caption model (BLIP) is used to generate N first sample image captions that are closest to the first sample image.

[0111] After obtaining the first sample image label, the first sample image attribute and the first sample image caption, the first sample image text description It can be expressed as follows:

[0112]

[0113] In step 103, a question guide is added to the first sample image text description to generate a first sample guide text. For example, the question guide is "'<first sample image text description>', what is the image category corresponding to the text description". In this embodiment, by adding a question guide to the first sample image text description, the first sample guide text is more suitable as the text of the category to be identified without the need for annotated data, so as to subsequently guide the first initial large language model to output a more accurate category.

[0114] In one embodiment, the question guide includes an information provision template and a category question template. Figure 5 , step 103, comprising:

[0115] Step 501, obtaining a category set; the category set includes multiple categories, and the multiple categories include the first sample prediction image category;

[0116] Step 502: embed the first sample image text description into the information providing template, and embed the category set into the category question template to obtain the first sample guide text.

[0117] For example, the information provision field is “Give the information about an image:{}”, and the category question field is “According to the infoimation, choose the most similar category from the given options:{},###Answer:”. After embedding the information providing template and embedding the category set (Category Set) into the category question template, the first sample guide text obtained is “Give the information about an image: ", "According to the infoimation, choose the most similar category from the given options: {Category Set}, ###Answer: ".

[0118] In this embodiment, the first initial large language model is fine-tuned by embedding the text description and the category into the designed template as the corpus. After the first initial large language model is fine-tuned, its ability to follow the template can be improved, and it can pay attention to the information related to the classification.

[0119] After obtaining the text description of the first sample image, first in step 104, the first sample guide text is input into the first initial large language model for category prediction to obtain the first sample predicted image category. Then in step 105, the initial large language model is fine-tuned based on the real image category and the first sample predicted image category to obtain the first target large language model.

[0120] In one embodiment, referring to Figure 6 , step 105 comprises:

[0121] Step 601, screening multiple first sample prediction image categories according to the real image category to obtain a first sample target prediction category, and taking the prediction probability of the first sample target prediction category as the first prediction probability;

[0122] Step 602, performing logarithm calculation on the first predicted probability to obtain a first logarithm;

[0123] Step 603, negate the first logarithm to obtain first loss data;

[0124] Step 604: Adjust parameters of the initial large language model according to the first loss data to obtain a first target large language model.

[0125] By using question guides, the cross-domain image classification task can be effectively transformed into a text classification task. A simple method is to use zero-shot fine-tuning LLM (hereinafter referred to as ZS-LLM) to convert The LLM is input and the sum of the log-likelihood of each token in each category is calculated to apply rank classification to select the category with the highest log-likelihood as the predicted category. However, ZS-LLM performs poorly in cross-domain tasks because: (1) The distribution of text descriptions still varies between different domains. LLMs are easily affected by irrelevant context in the descriptions (e.g., domain-specific words), which may negatively affect their classification accuracy. (2) The output of LLM, while related to the true category, is still not fully aligned.

[0126] In order to solve the above problems and further utilize the ability of LLM in classifying samples based on text descriptions, this application can formalize the classification problem as a question-answering problem. Where y is the category label of x, represented by a series of LLM tokens, and used as the question The real answer.

[0127] In the setting of domain generalization, many works assume that there is a global label mapping function that can handle different domains. Following this setting, all source domain data can be used to instruct fine-tuning of the large language model. Since the large language model has hundreds of millions of parameters, fully fine-tuning the large language model will have a high computational complexity. The parameter efficient fine-tuning method LoRA can be used for fine-tuning. The calculation process of the first loss data is as follows:

[0128]

[0129] Among them, θ represents the parameters of the LoRA module, L dg (θ) refers to the first loss data, refers to the first predicted probability, y refers to the real image category, refers to the first sample image text description, x refers to the first sample image, and D refers to the source domain.

[0130] The benefit of the above embodiment is that the parameters of the large language model are adjusted through the first loss data, which further improves the model performance and thus improves the image classification accuracy.

[0131] In step 106, a first target image is acquired, and the first target image is input into a pre-trained visual language model for text generation to obtain a text description of the first target image.

[0132] The first target image is an image to be classified after model training. In one example, the first target image is obtained by: (1) reading the first target image from the local storage space of the terminal. (2) receiving the first target image from another terminal or server.

[0133] The first target image is input into the pre-trained visual language model for text generation, and a first target image text description can be obtained. For example, for each first target image, the pre-trained VLMs are used to generate its corresponding first target image text description. The first target image text description consists of three parts, namely, the first target image label, the first target image attribute, and the first target image caption.

[0134] It should be noted that the process of generating the first target image text description is similar to the process of generating the first sample image text description described above, and will not be repeated here.

[0135] In step 107, a question guide is added to the first target image text description to generate a first target guide text. For example, a category set is obtained; the category set includes multiple categories; the first target image text description is embedded in the information providing template, and the category set is embedded in the category question template to obtain the first target guide text.

[0136] In step 108, the first target guiding text is input into the first target large language model for category prediction to obtain the first target predicted image category. After fine-tuning the first target large language model, the first target large language model can make predictions based on a given category set, reflecting its generalization ability in visual modality.

[0137] In one embodiment, after step 105, refer to Figure 7 , the image classification method also includes:

[0138] Step 701, obtaining a second sample image;

[0139] Step 702, using the first target large language model to perform category prediction on the second sample image input to obtain a pseudo image category;

[0140] Step 703, inputting the second sample image into the pre-trained visual language model to generate text, and obtaining a text description of the second sample image;

[0141] Step 704, adding a question guide word to the second sample image text description to generate a second sample guide text;

[0142] Step 705, inputting the second sample guide text into the first target large language model for category prediction to obtain a second sample predicted image category;

[0143] Step 706, fine-tuning the first target large language model according to the real image category, the first sample predicted image category, the pseudo image category, and the second sample predicted image category to obtain a second target large language model;

[0144] Step 707, obtaining a second target image, and inputting the second target image into a pre-trained visual language model for text generation to obtain a text description of the second target image;

[0145] Step 708, adding a question guide word to the second target image text description to generate a second target guide text;

[0146] Step 709: input the second target guiding text into the second target large language model for category prediction to obtain the second target predicted image category.

[0147] Figure 2The lower left side of is the second sample image in step 701. The second sample image is an image used to train the model. The first sample image and the second sample image are in different domains. For example, the first sample image is in the source domain, and the second sample image is in the target domain.

[0148] In one example, the second sample image is obtained by: (1) downloading the second sample image from a public data set; (2) collecting images from the Internet using crawler technology to obtain the second sample image.

[0149] In step 702, there is no need to annotate the second sample image in advance, but the category of the second sample image is predicted by the first target large language model to obtain a pseudo image category.

[0150] In step 703, the second sample image is input into the pre-trained visual language model for text generation to obtain a text description of the second sample image. For example, for each second sample image x i , using pre-trained VLMs to generate the corresponding second sample image text description The second sample image text description consists of three parts, namely, second sample image tags (Tags), second sample image attributes (Attributes), and second sample image captions (Captions).

[0151] In one embodiment, step 703 includes: obtaining a label set, the label set including multiple labels; embedding each label into a preset label template to generate a label text; inputting the second sample image and the label text into the image label model to perform similarity calculation to obtain the second image label similarity; screening multiple labels according to the second image label similarity to obtain the second sample image label. This embodiment is similar to the process of generating the first sample image label above, and can improve the accuracy of generating the second sample image label.

[0152] In one embodiment, step 703 further includes: embedding each tag into a preset attribute guide template to generate attribute guide text; inputting each attribute guide text into a pre-trained language model to generate attribute text to obtain candidate attribute text; inputting the second sample image and the candidate attribute text into an image attribute model to perform similarity calculation to obtain the second image attribute similarity; screening multiple candidate attribute texts according to the second image attribute similarity to obtain the second sample image attribute. This embodiment is similar to the process of generating the first sample image attribute in the above text, and can improve the accuracy of generating the second sample image attribute.

[0153] In one embodiment, step 703 further includes: inputting the second sample image into an image caption model to generate captions to obtain second sample image captions.

[0154] In one embodiment, step 704 includes: obtaining a category set; the category set includes multiple categories, and the multiple categories include the second sample predicted image category; embedding the second sample image text description into the information providing template, and embedding the category set into the category question template to obtain the second sample guide text.

[0155] In one embodiment, step 706 includes:

[0156] Screening multiple first sample prediction image categories according to the real image category to obtain a first sample target prediction category, and using the prediction probability of the first sample target prediction category as the first prediction probability;

[0157] Performing logarithmic calculation on the first predicted probability to obtain a first logarithm, and negating the first logarithm to obtain first loss data;

[0158] Screening a plurality of second sample prediction image categories according to the pseudo image category to obtain a second sample target prediction category, and using the prediction probability of the first sample target prediction category as the second prediction probability;

[0159] Performing logarithmic calculation on the second predicted probability to obtain a second logarithm, and negating the second logarithm to obtain second loss data;

[0160] Adding the first loss data and the second loss data to obtain target loss data;

[0161] The parameters of the first target large language model are adjusted according to the target loss data to obtain a second target large language model.

[0162] The calculation process of the target loss data is as follows:

[0163]

[0164] Among them, θ represents the parameters of the LoRA module, L uda (θ) refers to the target loss data, is the first predicted probability, refers to the second predicted probability, y refers to the real image category, refers to the pseudo image category, D refers to the source domain, and T refers to the target domain.

[0165] In step 707, the second target image is an image to be classified after model training. In one example, the second target image is obtained by: (1) reading the second target image from the local storage space of the terminal. (2) receiving the second target image from another terminal or server.

[0166] The second target image is input into the pre-trained visual language model for text generation, and a second target image text description can be obtained. For example, for each second target image, the corresponding second target image text description is generated using the pre-trained VLMs. The second target image text description consists of three parts, namely, the second target image label, the second target image attribute, and the second target image caption.

[0167] It should be noted that the process of generating the second target image text description is similar to the process of generating the first sample image text description described above, and will not be repeated here.

[0168] In step 708, a question guide is added to the second target image text description to generate a second target guide text. For example, a category set is obtained; the category set includes multiple categories; the second target image text description is embedded in the information providing template, and the category set is embedded in the category question template to obtain the second target guide text.

[0169] In step 709, after fine-tuning to obtain the second target large language model, the second target large language model can make predictions based on a given set of categories, reflecting its generalization ability in the visual modality.

[0170] The benefit of the above embodiment is that, after fine-tuning, the second target large language model is able to make predictions based on a given set of categories, reflecting its generalization ability in the visual modality.

[0171] This application uses a visual language model to extract text descriptions of image modalities, and fine-tunes the large language model for the designed question guide, thereby applying the generalization ability of the large language model to the cross-domain transfer learning problem on the image modality, which can effectively improve the classification accuracy. In order to verify the effectiveness of the model of this application (which can be called VLLaVO), the performance of VLLaVO was tested on PACS, OfficeHome, DomainNet and other datasets under the domain generalization setting, proving the effectiveness of the proposed method.

[0172] The embodiment of the present application also provides an electronic device, which includes: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory, and the program is executed by the processor to realize the above-mentioned image classification method. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0173] See also Figure 8 , Figure 8 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0174] The processor 801 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0175] The memory 802 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 802 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 802, and the processor 801 calls and executes the image classification method of the embodiment of this application;

[0176] Input / output interface 803, used to implement information input and output;

[0177] The communication interface 804 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0178] A bus 805 that transmits information between the various components of the device (e.g., the processor 801, the memory 802, the input / output interface 803, and the communication interface 804);

[0179] The processor 801 , the memory 802 , the input / output interface 803 and the communication interface 804 are connected to each other in communication within the device via a bus 805 .

[0180] An embodiment of the present application also provides a storage medium, which is a computer-readable storage medium used for computer-readable storage. The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the above-mentioned image classification method.

[0181] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0182] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0183] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0184] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0185] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0186] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0187] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0188] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0189] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable an electronic device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0190] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. An image classification method, characterized in that: The method comprises: Obtain a first sample image and a real image category; Inputting the first sample image into a pre-trained visual language model for text generation to obtain a text description of the first sample image; Adding a question guide to the first sample image text description to generate a first sample guide text; Inputting the first sample guide text into a first initial large language model for category prediction to obtain a first sample predicted image category; Fine-tune the initial large language model based on the real image category and the first sample predicted image category to obtain a first target large language model; Acquire a first target image, and input the first target image into a pre-trained visual language model for text generation to obtain a text description of the first target image; Adding the question guide words to the first target image text description to generate a first target guide text; The first target guiding text is input into the first target large language model for category prediction to obtain a first target predicted image category.

2. The method according to claim 1, characterized in that The question guide includes an information provision template and a category question template; The step of adding a question guide to the first sample image text description to generate a first sample guide text includes: Acquire a category set; the category set includes multiple categories, and the multiple categories include the first sample prediction image category; The first sample image text description is embedded in the information provision template, and the category set is embedded in the category question template to obtain the first sample guide text.

3. The method according to claim 1, characterized in that The first sample image text description includes a first sample image label, and the pre-trained visual language model includes an image label model; The step of inputting the first sample image into a pre-trained visual language model for text generation to obtain a text description of the first sample image includes: Obtain a tag set, the tag set including a plurality of tags; Embed each of the labels into a preset label template to generate label text; Inputting the first sample image and the label text into the image label model to perform similarity calculation to obtain a first image label similarity; The plurality of labels are screened according to the first image label similarity to obtain the first sample image label.

4. The method according to claim 3, characterized in that The first sample image text description also includes a first sample image attribute, and the pre-trained visual language model includes an image attribute model; The step of inputting the first sample image into a pre-trained visual language model to generate text to obtain a text description of the first sample image further includes: Embed each of the tags into a preset attribute guide template to generate attribute guide text; Input each of the attribute guide texts into a pre-trained language model to generate attribute texts, thereby obtaining candidate attribute texts; Inputting the first sample image and the candidate attribute text into the image attribute model to perform similarity calculation to obtain a first image attribute similarity; The plurality of candidate attribute texts are screened according to the first image attribute similarity to obtain the first sample image attribute.

5. The method according to claim 4, characterized in that The first sample image text description further includes a first sample image caption, and the pre-trained visual language model includes an image caption model; The step of inputting the first sample image into a pre-trained visual language model to generate text to obtain a text description of the first sample image further includes: The first sample image is input into the image caption model to generate captions, thereby obtaining the first sample image captions.

6. The method according to any one of claims 1 to 5, characterized in that: The step of fine-tuning the initial large language model based on the real image category and the first sample predicted image category to obtain a first target large language model includes: Screening a plurality of first sample prediction image categories according to the real image category to obtain a first sample target prediction category, and taking the prediction probability of the first sample target prediction category as a first prediction probability; Performing logarithm calculation on the first predicted probability to obtain a first logarithm; Reverse the first logarithm to obtain first loss data; Parameters of the initial large language model are adjusted according to the first loss data to obtain the first target large language model.

7. The method according to any one of claims 1 to 5, characterized in that: After fine-tuning the initial large language model based on the real image category and the first sample predicted image category to obtain a first target large language model, the method further includes: Acquire a second sample image; wherein the first sample image and the second sample image are in different domains; Using the first target large language model to perform category prediction on the second sample image input to obtain a pseudo image category; Inputting the second sample image into the pre-trained visual language model to generate text, thereby obtaining a text description of the second sample image; Adding a question guide to the second sample image text description to generate a second sample guide text; Inputting the second sample guide text into the first target large language model for category prediction to obtain a second sample predicted image category; Fine-tune the first target large language model according to the real image category, the first sample predicted image category, the pseudo image category, and the second sample predicted image category to obtain a second target large language model; Acquire a second target image, and input the second target image into the pre-trained visual language model for text generation to obtain a text description of the second target image; Adding the question guide words to the second target image text description to generate a second target guide text; The second target guiding text is input into the second target large language model for category prediction to obtain a second target predicted image category.

8. The method according to claim 7, characterized in that According to the real image category, the first sample predicted image category, the pseudo image category, and the second sample predicted image category, fine-tuning the first target large language model to obtain a second target large language model includes: Screening a plurality of first sample prediction image categories according to the real image category to obtain a first sample target prediction category, and taking the prediction probability of the first sample target prediction category as a first prediction probability; Performing logarithmic calculation on the first predicted probability to obtain a first logarithm, and negating the first logarithm to obtain first loss data; Screening a plurality of second sample prediction image categories according to the pseudo image category to obtain a second sample target prediction category, and using the prediction probability of the first sample target prediction category as the second prediction probability; Performing logarithmic calculation on the second predicted probability to obtain a second logarithm, and negating the second logarithm to obtain second loss data; Adding the first loss data and the second loss data to obtain target loss data; Parameters of the first target large language model are adjusted according to the target loss data to obtain the second target large language model.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 8 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images