Image classification method and device, equipment and storage medium

Abstract

The invention relates to the technical field of computer vision, and discloses an image classification method, device and equipment and a storage medium, and the method comprises the steps: obtaining a to-be-classified image, and obtaining a to-be-detected text sequence based on the pixel value conversion of the to-be-classified image; obtaining a training image data set, converting pixel values of training images in the training image data set to obtain a training text sequence, and training the large language model based on the training text sequence to obtain a pre-trained large language model; the to-be-tested text sequence is input into the pre-trained large language model, a classification result of the to-be-classified image is obtained, and the classification result represents the category of the to-be-classified image. The text sequence prediction problem is solved through the pre-trained large language model, and the advantages of the pre-trained large language model on text sequence data processing are explicitly utilized; and revealing the difference of different types of images and identifying the types of the images.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method, device, equipment and storage medium for image classification. Background Art

[0002] Image classification models in the field of computer vision have made significant progress in classification models trained on datasets such as ImageNet and CUB, such as residual networks (ResidualNetwork, referred to as Resnet), deep learning models (InceptionV3), and recent classification network backbones such as Transformer-based vision models (Vision Transformer, referred to as ViT). However, although these image classification models perform well in traditional computer vision tasks, they often rely on carefully designed architectures and the support of a large number of annotated image datasets to capture the spatial features of images. Such models may encounter new challenges when generalizing to some fields such as cross-modal classification. Summary of the invention

[0003] In view of this, the present invention provides a method, apparatus, device and storage medium for image classification to solve the problem of inaccurate classification of cross-modal images by image classification models in related technologies.

[0004] In a first aspect, the present invention provides a method for image classification, the method comprising: obtaining an image to be classified, converting the pixel values ​​of the image to be classified to obtain a text sequence to be tested; obtaining a training image data set, converting the pixel values ​​of the training images in the training image data set to obtain a training text sequence, training a large language model based on the training text sequence to obtain a pre-trained large language model; inputting the text sequence to be tested into the pre-trained large language model to obtain a classification result of the image to be classified, the classification result representing the category of the image to be classified.

[0005] In an optional embodiment, the aforementioned method for image classification also includes: performing low-rank decomposition on the weight matrix in the pre-trained large language model, and characterizing the weight matrix by the pre-trained weight matrix obtained by the low-rank decomposition and the newly added trainable matrix; adjusting and training the pre-trained large language model, and updating the newly added trainable matrix; until the condition for the end of the training of the pre-trained large language model is met, and an updated newly added trainable matrix is ​​obtained; characterizing the weight matrix based on the pre-trained weight matrix and the updated newly added trainable matrix to obtain a fine-tuned pre-trained large language model; inputting the text sequence to be tested into the fine-tuned pre-trained large language model to obtain a classification result of the image to be classified.

[0006] In an optional embodiment, the adjusting training of the pre-trained large language model and updating the newly added trainable matrix include: performing the adjustment training based on the training image dataset, wherein the training image dataset includes the training images and actual labels corresponding to the training images; calculating the difference between the prediction result of the pre-trained large language model and the actual labels through a cross entropy loss function, and updating the newly added trainable matrix based on the difference.

[0007] In an optional embodiment, the method of converting the pixel values ​​of the image to be classified to obtain the text sequence to be tested includes: converting each row of image pixels in the image to be classified to obtain corresponding each row of text sequence; and combining the each row of text sequence to obtain the text sequence to be tested.

[0008] In an optional implementation, the converting of the image pixels in each row in the image to be classified to obtain the corresponding text sequences in each row includes: converting the image pixels into RGB pixel values ​​to obtain the text sequences in each row.

[0009] In an optional implementation, the aforementioned method for image classification further includes: downsampling the image to be classified to obtain a downsampled image, and converting pixel values ​​of the downsampled image to obtain the text sequence to be tested.

[0010] In a second aspect, the present invention provides a device for image classification, the device comprising: a sequence acquisition module, used to acquire an image to be classified, and obtain a text sequence to be tested based on pixel value conversion of the image to be classified; a model training module, used to acquire a training image data set, convert the pixel values ​​of the training images in the training image data set to obtain a training text sequence, and train a large language model based on the training text sequence to obtain a pre-trained large language model; a classification module, used to input the text sequence to be tested into the pre-trained large language model to obtain a classification result of the image to be classified, and the classification result represents the category of the image to be classified.

[0011] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the method for image classification of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0012] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the method for image classification of the first aspect or any corresponding embodiment thereof.

[0013] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions for causing a computer to execute the method for image classification according to the first aspect or any corresponding embodiment thereof.

[0014] The pixel values ​​of the image to be tested are encoded into a text sequence, and the deep semantic understanding ability of the pre-trained large language model is used to distinguish between real images and fake images generated based on the image generation model. Instead of directly processing the visual features of the image to be classified, the image classification task is converted into a text sequence prediction problem, and the text sequence prediction problem is processed by the pre-trained large language model. The advantages of the pre-trained large language model in processing text sequence data are explicitly used to reveal the subtle differences between real images and fake images and identify the authenticity of the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the related technologies, the drawings required for use in the specific embodiments or the related technical descriptions will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0016] Figure 1 A schematic diagram of a process for image classification provided by an embodiment of the present invention is shown;

[0017] Figure 2 A schematic diagram of converting an image to be classified into a text sequence to be tested is shown;

[0018] Figure 3 A schematic flow chart of another method for image classification provided by an embodiment of the present invention is shown;

[0019] Figure 4 A schematic diagram showing an application scenario of the method for image classification provided by an embodiment of the present invention is shown;

[0020] Figure 5 A schematic diagram of the structure of an apparatus for image classification provided by an embodiment of the present invention is shown;

[0021] Figure 6 A schematic diagram of the hardware structure of a computer device provided in an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0023] Natural language processing (NLP) is a core technology in computer science that aims to enable computers to understand, process and generate human language. Since humans rely on language to communicate, express thoughts and emotions, it is extremely valuable to develop systems that can efficiently process natural language. NLP has a wide range of applications, covering text analysis, speech recognition, sentiment analysis, and machine translation. In these applications, text, as the main medium for carrying information, contains rich semantics. When performing tasks such as sentiment analysis, computer systems must understand the meaning, structure, and context of the text in order to accurately judge the emotional tendency expressed by the text. Machine translation is an important means to promote international communication and cooperation by converting text from one language to another.

[0024] Among the many branches of NLP, text generation occupies a core position. Text generation models are specifically designed to automatically generate text content, including automatic responses, news summaries, and creative writing. These text generation models are trained by analyzing and learning language patterns and contextual relationships in large text corpora. For example, large language models (LLMs), such as the Generative Pre-Trained Transformer (GPT) series, have demonstrated their outstanding ability to understand and generate natural language through training on large data sets. The core design principle of these text generation models is to grasp the sequence dependency in natural language and predict the next words or characters that may appear. This ability makes them outstanding in dealing with tasks with obvious sequence dependency characteristics. After proper adjustment and fine-tuning, these text generation models can even demonstrate their capabilities in non-linguistic tasks, such as in coding sequence prediction, bioinformatics, and music composition.

[0025] Image classification is a computer vision task that aims to assign input images to predefined categories. Image classification models can identify the main elements in an image, such as objects and scenes, by learning visual features from a large number of annotated samples. Commonly used image classification models include convolutional neural network (CNN) architectures, such as Residual Network (Resnet) and deep learning models (Inception), as well as Transformer-based vision models (such as Vision Transformer, ViT), which have achieved remarkable results in processing standard datasets (such as ImageNet).

[0026] Image classification models in the field of computer vision have made significant progress in classification models trained on datasets such as ImageNet and CUB, such as residual networks (ResidualNetwork, referred to as Resnet), deep learning models (InceptionV3), and recent classification network backbones such as Transformer-based vision models (Vision Transformer, referred to as ViT). However, although these image classification models perform well in traditional computer vision tasks, they often rely on carefully designed architectures and the support of a large number of annotated image datasets to capture the spatial features of images. Such models may encounter new challenges when generalizing to some fields such as cross-modal classification.

[0027] The related technology discloses a multimodal recognition method, device, electronic device and storage medium based on a large language model. The method is a method based on a large language model, including collecting text data and different types of modal data, preprocessing the data, combining the text data with the modal data, using the large language model to extract the semantic representation features of the text data, and extracting features from the modal data, performing multi-stage multimodal fusion, selecting a multimodal recognition task model architecture, building a multimodal recognition model, designing a loss function, using the loss function and an optimizer to train the multimodal recognition model, using a validation set and a test set to evaluate the performance of the multimodal recognition model, adjusting the model hyperparameters and architecture, optimizing the model, realizing intelligent communication, and visually interacting with users to meet user needs.

[0028] The modeling process of cross-modal tasks in related technologies is relatively complex. Explore the application of large language models in non-traditional text processing fields, and at the same time, consider converting image classification problems into sequence prediction problems for processing. According to an embodiment of the present invention, a method embodiment for image classification is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that here.

[0029] In this embodiment, a method for image classification is provided, which can be used in mobile terminals, such as mobile phones, tablet computers, servers, desktop computers, or laptop computers. Figure 1 A schematic diagram of a process for image classification provided by an embodiment of the present invention is shown. Figure 1 As shown, the process includes the following steps:

[0030] Step S101, obtaining an image to be classified, and converting the pixel values ​​of the image to be classified to obtain a text sequence to be tested.

[0031] In this step, the pixel value of each point in the image to be classified can be expressed in the (R, G, B) format. Figure 2 A schematic diagram of converting the image to be classified into the text sequence to be tested is shown in FIG. Figure 2 As shown, the first text sequence 23 to be tested obtained based on the pixel value conversion of the first image to be tested 21 is 255, 255, ..., 118, 38, ..., 35, ..., 172, 51, 147, 255, and the second text sequence 24 to be tested obtained based on the pixel value conversion of the second image to be tested 22 is 24, 11, 26, 13, 5 ..., 43, 25, 12, ..., 37, 20, 156, 33.

[0032] Step S102, obtaining a training image data set, converting pixel values ​​of training images in the training image data set to obtain a training text sequence, training a large language model based on the training text sequence, and obtaining a pre-trained large language model.

[0033] In this step, the training image data set may include images of multiple categories. Specifically, it includes images of categories 0 to N. After obtaining the training image data set, the training text sequence is obtained based on the training image conversion. The specific conversion method can refer to the method in step S101.

[0034] Large Language Model (LLM) refers to an artificial intelligence model that has been trained with a large amount of text data and has strong text understanding and generation capabilities. Large language models can perform a variety of language-related tasks such as language translation, question answering, text summarization, and writing assistance.

[0035] The large language model you can choose is the Qianwen Large Language Model (QWen). QWen includes models with multiple parameter values. QWen is a large-scale language model based on the Transformer architecture. It is pre-trained by predicting the next word. It can achieve knowledge understanding and task execution in a wide range of fields through powerful natural language processing capabilities and cross-modal interaction.

[0036] Step S103, inputting the text sequence to be tested into the pre-trained large language model to obtain a classification result of the image to be classified, and the classification result represents the category of the image to be classified.

[0037] In this step, the classification result output by the pre-trained large language model can be a number 0 to N, such as 0, 1, 2 ... N, representing the prediction of the pre-trained large language model on the category of the image to be classified, wherein 0 represents that the pre-trained large language model predicts that the image to be classified is a first category image, and 1 represents that the pre-trained large language model predicts that the image to be classified is a second category image. Using the natural language processing capability of the pre-trained large language model, the category of the image to be classified can be identified.

[0038] The method for image classification provided in this embodiment maps the pixel values ​​or features of the image to be tested to a text sequence, and uses the deep semantic understanding ability of the pre-trained large language model to distinguish the category of the image. It does not directly process the visual features of the image to be classified, but converts the image classification task into a text sequence prediction problem. The text sequence prediction problem is processed by the pre-trained large language model, and the processing advantage of the pre-trained large language model on text sequence data is explicitly used to reveal the differences between images of different categories, so as to achieve effective detection of image categories under low pixels; the pre-trained large language model has the ability to naturally model long-distance dependencies in the sequence, and can capture complex feature associations in image data, such as subtle changes in texture and color, which is particularly critical in detecting images of similar categories or high-detail classification tasks; the serialization conversion allows a single model framework to process multiple input types, such as text or images, which can simplify the modeling process of cross-modal tasks.

[0039] In some optional embodiments, the aforementioned method for image classification also includes: performing low-rank decomposition on the weight matrix in the pre-trained large language model, and characterizing the weight matrix by the pre-trained weight matrix obtained by the low-rank decomposition and the newly added trainable matrix; adjusting and training the pre-trained large language model, and updating the newly added trainable matrix; until the conditions for the end of the training of the pre-trained large language model are met, and an updated newly added trainable matrix is ​​obtained; characterizing the weight matrix based on the pre-trained weight matrix and the updated newly added trainable matrix, to obtain a fine-tuned pre-trained large language model; inputting the text sequence to be tested into the fine-tuned pre-trained large language model to obtain a classification result of the image to be classified.

[0040] In this implementation, a low-rank adaptation strategy (LoRA) can be used to fine-tune the pre-trained large language model. LoRA is based on the idea of ​​an adapter, which generalizes knowledge for downstream specific tasks by adjusting the learning external module, and can reduce the overhead of model fine-tuning and storage through the design of a learnable rank decomposition matrix.

[0041] The weight matrix W∈R in the pre-trained large language model can be m×n Perform low-rank decomposition and reparameterize to W = W 0 + BA, where m×n is the dimension of the weight matrix W, W 0 is the pre-trained weight matrix, which is the original pre-trained weight of the pre-trained large language model. It is a fixed full-rank matrix and does not accept gradient updates. New trainable matrices include: A and B, B∈R m×r , A∈R r ×n , r is the low-rank dimension r<<min(m,n).

[0042] The newly added trainable matrices A and B can be initialized using smaller random numbers, which can ensure that the newly added trainable matrices have less impact on the pre-trained large language model at the beginning of training.

[0043] During the adjustment training of the pre-trained large language model, that is, during the fine-tuning training of the pre-trained large language model, only the newly added trainable matrices A and B are updated, and the pre-trained weight matrix W of the pre-trained large language model is maintained. 0 This can be achieved by including only A and B parameters in the optimizer.

[0044] Until the conditions for the end of the pre-trained large language model training are met, the updated newly added trainable matrix is ​​obtained. The conditions for the end of the pre-trained large language model training can be set based on a variety of factors, including: fixed number of training rounds, set a fixed number of training rounds, and stop training when the number of rounds is reached; the performance index reaches the preset value, and a performance index (such as accuracy, F1-score or mAP, etc.) can be set. When the performance of the pre-trained large language model reaches or exceeds the preset value, the training is stopped, which helps to ensure that the performance of the model on a specific task reaches the expected level; the validation set error no longer decreases. During the training process, the validation set is used to monitor the performance of the pre-trained large language model. If the validation set error no longer decreases in multiple consecutive training cycles (epochs), it may indicate that the pre-trained large language model is close to the optimal state. At this time, the training can be stopped to avoid overfitting; use a callback function, such as the EarlyStopping callback function in TensorFlow, to automatically stop training according to the set conditions (such as the performance index on the validation set no longer improves more than a certain number of times).

[0045] Exemplarily, the QWen model can be fine-tuned by LoRA, and the weights obtained by fine-tuning are merged with the initial model weights to obtain a fine-tuned pre-trained large language model. The text sequence to be tested is input into the fine-tuned pre-trained large language model to obtain the classification result of the image to be classified.

[0046] In this way, the pre-trained large language model is fine-tuned through the LoRA strategy, which introduces only a minimum of trainable parameters, thereby preserving the essence of the pre-trained weights, while giving the fine-tuned pre-trained large language model the ability to learn task-specific nuances. The efficiency of LoRA facilitates targeted updates to the model's attention mechanism, making it sensitive to the subtleties of pixel value sequences that distinguish between images of different categories.

[0047] You can also use a Generative Pre-Trained Transformer (GPT), a specific large language model architecture that learns the common patterns and structures of the language through large-scale corpus pre-training, and can be fine-tuned on specific downstream tasks to complete specific text generation or understanding tasks.

[0048] In some optional embodiments, the pre-trained large language model is adjusted and trained, and a newly added trainable matrix is ​​updated, including: adjusting and training based on a training image dataset, the training image dataset including training images and actual labels corresponding to the training images; calculating the difference between the prediction results of the pre-trained large language model and the actual labels through a cross entropy loss function, and updating the newly added trainable matrix based on the difference.

[0049] In this embodiment, the training image dataset includes a large number of image representations and labels corresponding to the image representations, where 1 represents the first category image, 2 represents the second category image, ..., and N represents the Nth category image. Through a supervised fine-tuning strategy, a cross entropy loss function is used to calculate the difference between the model prediction and the actual label, and the difference is used to guide the LoRA-based fine-tuning.

[0050] In this way, the performance and accuracy of the fine-tuned pre-trained large language model can be effectively improved by predicting the difference between the probability distribution and the true label distribution through the cross entropy loss function.

[0051] In this embodiment, a method for image classification is also provided, which can be used in mobile terminals, such as mobile phones, tablet computers, servers, desktop computers or notebook computers, etc. Figure 3 A schematic diagram of a process flow of another method for image classification provided by an embodiment of the present invention is shown. Figure 3 As shown, the process includes the following steps:

[0052] Step S301, obtain the image to be classified, and convert the pixel values ​​of the image to be classified to obtain the text sequence to be tested. Figure 1 Step S101 of the illustrated embodiment will not be described in detail here.

[0053] Specifically, the above step S301 includes:

[0054] Step S3011, converting each row of image pixels in the image to be classified into corresponding each row of text sequences.

[0055] In this step, the natural row-by-row expansion of the image can be simulated. After each row of image pixels is converted into pixel values, a carriage return character or a line feed character is added at the end of each row of pixel values.

[0056] Step S3012, combining the text sequences of each line to obtain the text sequence to be tested.

[0057] In this step, the text sequences of each line can be concatenated and combined to obtain the text sequence to be tested.

[0058] Step S302, obtain a training image dataset, convert the pixel values ​​of the training images in the training image dataset to obtain a training text sequence, train a large language model based on the training text sequence, and obtain a pre-trained large language model. Figure 1 Step S102 of the illustrated embodiment will not be described in detail here.

[0059] Step S303: Input the text sequence to be tested into the pre-trained large language model to obtain the classification result of the image to be classified, and the classification result represents the category of the image to be classified. Figure 1Step S103 of the illustrated embodiment will not be described in detail here.

[0060] The pixel values ​​of the image are converted into text form, and the concept of "rows" of image pixels is retained. This meticulous pixel transcription method can retain the spatial structure of the image, so that the fine-tuned pre-trained large language model can learn and utilize the spatial structure of real and fake images while processing text information. It fully combines the advantages of image processing and natural language processing, and improves the ability of the fine-tuned pre-trained large language model to capture the characteristics of fake images.

[0061] In some optional implementations, obtaining corresponding lines of text sequences based on image pixel conversion of each line in the image to be classified includes: converting image pixels into RGB pixel values ​​to obtain corresponding lines of text sequences.

[0062] In this embodiment, by converting image pixels into RGB pixel values, the fine-tuned pre-trained large language model can learn and utilize the spatial and color distribution patterns in different categories of images while processing text information. This fully combines the advantages of image processing and natural language processing, and improves the ability of the fine-tuned pre-trained large language model to capture false image features.

[0063] In some optional implementations, the aforementioned method for image classification further includes: downsampling the image to be classified to obtain a downsampled image, and converting pixel values ​​of the downsampled image to obtain a text sequence to be tested.

[0064] In this embodiment, the images in the classified image and training image data set are downsampled to 28*28, and the pixel values ​​of the downsampled images are converted to RGB pixel values, which can speed up the processing speed and the training speed of the model and improve the computing efficiency.

[0065] During the testing phase of the fine-tuned pre-trained large language model, the same prompt and input structure can be used to evaluate the generalization ability of the fine-tuned pre-trained large language model for unknown data. The output of the fine-tuned pre-trained large language model can be a real value, such as 0, 1, 2...N, and 0~N are used to represent the prediction of the fine-tuned pre-trained large language model for the category of the image to be classified.

[0066] Figure 4 FIG. 1 is a schematic diagram showing an application scenario of the method for image classification provided by an embodiment of the present invention. Figure 4As shown, the fine-tuned pre-trained large language model is novel adapted for image classification tasks. Given a textual representation of the pixel values ​​of an image, the fine-tuned pre-trained large language model 401 receives the following prompt: "You are a trained image classification discriminator. Based on the string input representing the RGB pixel values ​​of the image, determine which category this image belongs to, represented by numbers 0 to N."

[0067] Exemplarily, based on the first text sequence 23 to be tested obtained by pixel value conversion of the first image to be tested 21, the first text sequence 23 to be tested is input into the fine-tuned pre-trained large language model 401, and the fine-tuned pre-trained large language model 401 obtains the classification result 1 of the first image to be tested 21, that is, the first category image 402, based on the above prompts and the first text sequence 23 to be tested. Based on the second text sequence 24 to be tested obtained by pixel value conversion of the second image to be tested 22, the second text sequence 24 to be tested is input into the fine-tuned pre-trained large language model 401, and the fine-tuned pre-trained large language model 401 obtains the classification result 2 of the second image to be tested 22, that is, the second category image 403, based on the above prompts and the second text sequence 24 to be tested.

[0068] The method for image classification provided by the present invention proposes a new framework based on text sequence processing to explore the problem of image classification. It takes advantage of the inherent advantages of large language models in capturing long-distance dependencies in sequence data, which correspond to complex texture patterns and color distributions in the pixel-level representation of images. It not only expands the application scope of large language models in the field of non-traditional text sequence processing, but also provides a new perspective, namely, identifying image categories by analyzing the sequence patterns of image pixel values. It brings new perspectives and methods to the field of image classification and provides new directions for future research work.

[0069] In this embodiment, a device for image classification is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0070] This embodiment provides a device for image classification. Figure 5 FIG. 1 is a schematic diagram showing the structure of an apparatus for image classification provided by an embodiment of the present invention. Figure 5 As shown, including:

[0071] The sequence acquisition module 501 is used to acquire the image to be classified, and obtain the text sequence to be tested based on the pixel value conversion of the image to be classified.

[0072] The model training module 502 is used to obtain a training image data set, convert the pixel values ​​of the training images in the training image data set to obtain a training text sequence, and train a large language model based on the training text sequence to obtain a pre-trained large language model.

[0073] The classification module is used to input the text sequence to be tested into the pre-trained large language model to obtain the classification result of the image to be classified, and the classification result represents the category of the image to be classified.

[0074] A device for image classification based on a large language model is proposed, which transforms the basic paradigm of image classification tasks into sequence prediction tasks, explicitly completes the combination of multimodal tasks, and explores the application of large language models in more non-text fields. After converting the pixel values ​​of the image into text, this scheme uses the original sequence prediction processing capabilities of the large language model to identify patterns and anomalies. The device uses the long sequence understanding advantage of the large language model to achieve effective detection of image categories.

[0075] In some optional implementations, the aforementioned apparatus for image classification further includes:

[0076] The model fine-tuning module is used to perform low-rank decomposition on the weight matrix in the pre-trained large language model, and represent the weight matrix by the pre-trained weight matrix obtained by the low-rank decomposition and the newly added trainable matrix; adjust the training of the pre-trained large language model, and update the newly added trainable matrix; until the conditions for the end of the training of the pre-trained large language model are met, an updated newly added trainable matrix is ​​obtained; the weight matrix is ​​represented based on the pre-trained weight matrix and the updated newly added trainable matrix to obtain a fine-tuned pre-trained large language model; the text sequence to be tested is input into the fine-tuned pre-trained large language model to obtain the classification result of the image to be classified.

[0077] In some optional implementations, the model fine-tuning module includes:

[0078] The model fine-tuning unit is used to adjust the training based on the training image dataset, where the training image dataset includes training images and actual labels corresponding to the training images; the difference between the prediction results of the pre-trained large language model and the actual labels is calculated through the cross entropy loss function, and the newly added trainable matrix is ​​updated based on the difference.

[0079] In some optional implementations, the sequence acquisition module includes:

[0080] The first sequence acquisition unit is used to convert each row of image pixels in the image to be classified to obtain corresponding each row of text sequences; and combine each row of text sequences to obtain a text sequence to be tested.

[0081] In some optional implementations, the sequence acquisition unit includes:

[0082] The sequence acquisition subunit is used to convert image pixels into RGB pixel values ​​to obtain each line of text sequence.

[0083] In some optional implementations, the sequence acquisition module further includes:

[0084] The second sequence acquisition unit is used to downsample the image to be classified to obtain a downsampled image, and convert the pixel values ​​of the downsampled image to obtain a text sequence to be tested.

[0085] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0086] The device for image classification in this embodiment is presented in the form of a functional unit, where the unit refers to an application specific integrated circuit (ASIC) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0087] The pixel values ​​of the image to be tested are encoded into a text sequence, and the deep semantic understanding ability of the pre-trained large language model is used to distinguish between real images and fake images generated based on the image generation model. Instead of directly processing the visual features of the image to be classified, the image classification task is converted into a text sequence prediction problem, and the text sequence prediction problem is processed by the pre-trained large language model. The processing advantages of the pre-trained large language model for text sequence data are explicitly utilized to reveal the subtle differences between real images and fake images and identify the authenticity of the image.

[0088] The embodiment of the present invention also provides a computer device having the above Figure 5 The apparatus for image classification is shown.

[0089] See also Figure 6 , Figure 6 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 6As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the computer device, including instructions stored in or on the memory to display graphical information of a graphical user interface on an external input / output device (such as a display device coupled to an interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 6 A processor 10 is taken as an example.

[0090] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0091] The aforementioned memory 20 stores instructions executable by at least one processor 10, so that the aforementioned at least one processor 10 executes the method shown in the above embodiment.

[0092] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0093] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0094] The computer device also includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Figure 6 The example of connecting through bus is taken in the following.

[0095] The input device 30 can receive input digital or character information, and generate key signal input related to the user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator rod, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (such as a light emitting diode) and a tactile feedback device (such as a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device can be a touch screen.

[0096] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0097] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0098] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A method for image classification, characterized in that The method comprises: Acquire an image to be classified, and convert the pixel values ​​of the image to be classified to obtain a text sequence to be tested; Acquire a training image data set, convert pixel values ​​of training images in the training image data set to obtain a training text sequence, and train a large language model based on the training text sequence to obtain a pre-trained large language model; The text sequence to be tested is input into the pre-trained large language model to obtain a classification result of the image to be classified, wherein the classification result represents the category of the image to be classified.

2. The method according to claim 1, characterized in that The method further comprises: Performing low-rank decomposition on the weight matrix in the pre-trained large language model, and representing the weight matrix by the pre-trained weight matrix obtained by the low-rank decomposition and the newly added trainable matrix; Adjust and train the pre-trained large language model, and update the newly added trainable matrix; Until the condition for completing the training of the pre-trained large language model is met, an updated newly added trainable matrix is ​​obtained; Characterizing the weight matrix based on the pre-trained weight matrix and the updated newly added trainable matrix to obtain a fine-tuned pre-trained large language model; The text sequence to be tested is input into the fine-tuned pre-trained large language model to obtain a classification result of the image to be classified.

3. The method according to claim 2, characterized in that The adjusting and training the pre-trained large language model and updating the newly added trainable matrix include: Performing the adjustment training based on the training image dataset, wherein the training image dataset includes the training images and actual labels corresponding to the training images; The difference between the prediction result of the pre-trained large language model and the actual label is calculated by a cross entropy loss function, and the newly added trainable matrix is ​​updated based on the difference.

4. The method according to claim 1 or 2, characterized in that: The step of converting the pixel values ​​of the image to be classified to obtain a text sequence to be tested includes: Convert each row of image pixels in the image to be classified to obtain corresponding rows of text sequences; The text sequence to be tested is obtained by combining the text sequences of each line.

5. The method according to claim 4, characterized in that The step of converting pixels of each row of the image in the image to be classified to obtain corresponding text sequences of each row includes: The image pixels are converted into RGB pixel values ​​to obtain the text line sequences.

6. The method according to claim 1, characterized in that The method further comprises: The image to be classified is downsampled to obtain a downsampled image, and pixel values ​​of the downsampled image are converted to obtain the text sequence to be tested.

7. A device for image classification, characterized in that The device comprises: A sequence acquisition module is used to acquire an image to be classified, and obtain a text sequence to be tested based on pixel value conversion of the image to be classified; A model training module is used to obtain a training image data set, convert pixel values ​​of training images in the training image data set to obtain a training text sequence, and train a large language model based on the training text sequence to obtain a pre-trained large language model; The classification module is used to input the text sequence to be tested into the pre-trained large language model to obtain a classification result of the image to be classified, wherein the classification result represents the category of the image to be classified.

8. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the method for image classification according to any one of claims 1 to 6 by executing the computer instructions.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the method for image classification according to any one of claims 1 to 6.

10. A computer program product, characterized in that The method comprises computer instructions for causing a computer to execute the method for image classification according to any one of claims 1 to 6.

Images