Image classification method and apparatus, device, storage medium, and program product

Abstract

An image classification method, comprising: acquiring an original image and a plurality of prompt texts, each prompt text being generated according to one image label, and each image label being one preset image classification category (202); for each prompt text, inputting the original image, the prompt text, and a random noise image into a trained diffusion model, generating a noisy image according to the original image and the random noise image by means of the diffusion model, generating a predicted noise image according to the noisy image and the prompt text, and calculating a difference between the generated predicted noise image and the random noise image (204); according to a difference calculated for each predicted noise image, selecting a predicted noise image with the minimum corresponding difference, and acquiring a prompt text on which the generation of the selected predicted noise image is dependent (206); and taking, as the image label of the original image, an image label corresponding to the dependent prompt text (208).

Description

Image classification method, device, equipment, storage medium and program product

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on June 21, 2023, with application number 202310746237.3 and application name “Image classification method, device, equipment, storage medium and program product”, all contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of computer technology, and in particular to an image classification method, apparatus, device, storage medium, and program product. Background Art

[0003] With the rapid development of artificial intelligence and computer technology, image processing technology is being applied to various business scenarios. Image classification technology utilizes image features to quantitatively analyze the image, classifying the image or each pixel or region within it into one of several categories (or labels), replacing human visual interpretation.

[0004] Image classification has a wide range of applications. For example, image classification can be used in image recognition to identify animals, plants, car models, fruits, vegetables, and more. Another example is the automatic classification of photos taken on smartphones. Another example is image classification used in image content retrieval on e-commerce platforms. E-commerce backends can categorize product images and build a database, enabling more accurate image searches. Image classification can also be used in scenarios such as garbage sorting.

[0005] At present, the accuracy of image classification in a certain business field often depends on a large amount of manually annotated image data in that business field. The classification effect will only be significantly improved with the increase in the amount of manually annotated image data. However, the quality of manually annotated image data varies greatly. At the same time, the workload of manual annotation is extremely large, which requires a lot of manual annotation costs and is not efficient, which will affect the ability to launch image classification in a certain business field in a timely manner.

[0006] Summary of the Invention

[0007] Based on this, according to various embodiments provided in this application, a method for image classification is provided. The method includes:

[0008] Obtain an original image and multiple prompt texts, each prompt text is generated according to an image label, and each image label is a preset image classification category;

[0009] For each prompt text, the original image, the prompt text, and the random noise image are input into a trained diffusion model, a noisy image is generated based on the original image and the random noise image using the diffusion model, a predicted noise image is generated based on the noisy image and the prompt text, and a difference between the generated predicted noise image and the random noise image is calculated;

[0010] Selecting a predicted noise image with the smallest corresponding difference based on the differences calculated for each predicted noise image, and obtaining a prompt text relied upon when generating the selected predicted noise image;

[0011] The image label corresponding to the dependent prompt text is used as the image label of the original image.

[0012] This application also provides an image classification device. The device includes:

[0013] An acquisition module is used to acquire an original image and multiple prompt texts, each prompt text is generated according to an image label, and each image label is a preset image classification category;

[0014] a noise prediction module, configured to, for each prompt text, input the original image, the prompt text, and the random noise image into a trained diffusion model, generate a noisy image based on the original image and the random noise image using the diffusion model, generate a predicted noise image based on the noisy image and the prompt text, and calculate a difference between the generated predicted noise image and the random noise image;

[0015] A determination module, used to determine the prompt text that is relied upon to generate the predicted noise image with the minimum difference;

[0016] The label classification module is used to use the image label corresponding to the dependent prompt text as the image label of the original image.

[0017] The present application also provides a computer device, which includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the image classification method when executing the computer-readable instructions.

[0018] The present application also provides a computer-readable storage medium having computer-readable instructions stored thereon, which implement the steps of the image classification method when executed by a processor.

[0019] The present application also provides a computer program product, which includes computer-readable instructions, and when the computer-readable instructions are executed by a processor, implements the steps of the image classification method.

[0020] The present application also provides a method for processing a diffusion model, the method comprising:

[0021] Obtain multiple sample images, each sample image corresponds to an image label;

[0022] For each sample image, generating a corresponding sample prompt text according to the image label of the sample image, generating a noisy image according to the sample image and a sample random noise image using an initial diffusion model, generating a predicted noise image according to the noisy image and the sample prompt text, and performing denoising on the noisy image according to the predicted noise image to obtain a predicted image;

[0023] A sample loss is constructed according to the difference between the predicted image and the sample image, the initial diffusion model is updated according to the sample loss, and the trained diffusion model obtained after the update is used for image classification.

[0024] The present application also provides a diffusion model processing device, which includes: a sample acquisition module, a sample training module and a model update module, wherein:

[0025] A sample acquisition module is used to acquire multiple sample images, each of which has an image label.

[0026] a sample training module configured to generate, for each sample image, corresponding sample prompt text according to the image label of the sample image, generate a noisy image according to the sample image and a sample random noise image using an initial diffusion model, generate a predicted noise image according to the noisy image and the sample prompt text, and denoise the noisy image according to the predicted noise image to obtain a predicted image;

[0027] The model updating module is used to construct a sample loss according to the difference between the predicted image and the sample image, update the initial diffusion model according to the sample loss, and use the trained diffusion model obtained after the update for image classification.

[0028] The present application also provides a computer device, which includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the diffusion model processing method when executing the computer-readable instructions.

[0029] In a ninth aspect, the present application further provides a computer-readable storage medium having computer-readable instructions stored thereon, which, when executed by a processor, implement the steps of the diffusion model processing method.

[0030] In a tenth aspect, the present application further provides a computer program product, comprising computer-readable instructions, which, when executed by a processor, implement the steps of the method for processing a diffusion model.

[0031] The details of one or more embodiments of the present application are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the present application will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or traditional technical descriptions. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the disclosed drawings without any creative work.

[0033] FIG1 is a diagram illustrating an application environment of an image classification method according to an embodiment;

[0034] FIG2 is a schematic diagram of a flow chart of an image classification method according to an embodiment;

[0035] FIG3 is a schematic structural diagram of a diffusion model;

[0036] FIG4 is a schematic diagram of the overall framework of the image classification method;

[0037] FIG5 is a schematic diagram of the structure of a noise predictor in one embodiment;

[0038] FIG6 is a schematic diagram of the training process of the CLIP model in one embodiment;

[0039] FIG7 is a schematic diagram of a data set format used for training in a diffusion model according to one embodiment;

[0040] FIG8 is a schematic diagram of a training process of a first training stage of an initial diffusion image in one embodiment;

[0041] FIG9 is a schematic diagram of the training process of the second training stage of the initial diffusion image;

[0042] FIG10 is a block diagram of an image classification apparatus according to an embodiment;

[0043] FIG11 is a block diagram of a device for processing a diffusion model according to an embodiment;

[0044] FIG12 is a diagram showing the internal structure of a server in one embodiment. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0046] The diffusion model is a conditional model that relies on priors. In image generation tasks, the priors are usually text, images, or semantic maps. That is, the diffusion model generates corresponding images based on the input text, images, or semantic maps.

[0047] The image classification method provided in the embodiment of the present application can be applied in the application environment shown in Figure 1. In which, the server 104 obtains the original image and multiple prompt texts from the terminal 102, each prompt text is generated according to an image label, and each image label is a preset image classification category; for each prompt text, the server 104 inputs the original image, prompt text and random noise image into a trained diffusion model, generates a noisy image based on the original image and the random noise image through the diffusion model, generates a predicted noise image based on the noisy image and the prompt text, and calculates the difference between the generated predicted noise image and the random noise image; based on the calculated difference for each predicted noise image, the server 104 selects the predicted noise image with the smallest corresponding difference, obtains the prompt text that is relied upon when generating the selected predicted noise image; the server 104 uses the image label corresponding to the dependent prompt text as the image label of the original image. The data storage system can store multiple prompt texts that the server 104 needs to process. The data storage system can be integrated on the server 104, or placed on the cloud or other servers.

[0048] In other embodiments, the above-mentioned image classification method can also be executed by the terminal 102, which obtains the original image and multiple prompt texts. For each prompt text, the original image, the prompt text and the random noise image are input into a trained diffusion model, and then the image label of the original image is determined.

[0049] The terminal 102 may be, but is not limited to, various desktop computers, laptops, smartphones, tablet computers, IoT devices, and portable wearable devices. IoT devices may be smart car devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server 104 may be implemented as a standalone server or a server cluster consisting of multiple servers.

[0050] In one embodiment, as shown in FIG2 , a method for image classification is provided. The method is described by taking the server in FIG1 as an example, and includes the following steps:

[0051] Step 202: Obtain an original image and a plurality of prompt texts, each prompt text is generated according to an image label, and each image label is a preset image classification category.

[0052] The prompt text is the prior content of the image generated by the diffusion model, that is, the diffusion model generates the image based on the prompt text. The prompt text contains the image label, which is a preset image classification category. For example, the image label can be scenery, food, architecture, animals, people, etc. Multi-label image classification is the process of classifying an image into one or more of multiple image labels. In this embodiment, by setting different weights for image labels or by mixing image labels, prompt text is generated according to different image labels, and each prompt text is different. The format of the prompt text is generally A photo of a{class}, where class is the image label. For example, the prompt text is {A photo of a T-shirt}.

[0053] In some embodiments, the original image may be a product image, and the image label may be a product category, for example, the product category may be daily necessities, maternal and child products, clothing products, cosmetics products, etc. Multi-label image classification is to classify product images into product categories.

[0054] In some embodiments, the original image may be a video cover, and the image label may be a video category, for example, the video category may be comedy, action, horror, science fiction, etc. Multi-label image classification is to classify the video cover.

[0055] The server can retrieve each image tag in a preset image classification tag system and a preset prompt text template. It then iterates through each image tag in the preset image classification tag system, populating the prompt text template with the traversed image tags to obtain the corresponding prompt text for each image tag. As can be seen, the prompt text is a sentence describing the image, also known as a prompt. The preset image classification tag system is a collection of image tags for each business image that can be involved in a specific application scenario.

[0056] In step 204, for each prompt text, the original image, the prompt text, and the random noise image are input into a trained diffusion model. The diffusion model generates a noisy image based on the original image and the random noise image, and a predicted noise image is generated based on the noisy image and the prompt text. The difference between the generated predicted noise image and the random noise is calculated.

[0057] Noise often appears in images as isolated pixels or pixel blocks that cause strong visual effects. These pixels or pixel blocks are factors that hinder people from receiving their information. In layman's terms, noise makes the image unclear, so the image generated based on these noises can be called a noisy image. Random noise image is used to represent Gaussian noise, that is, random noise. Random noise can be randomly determined through Gaussian distribution; random noise image is denoted as sampleε ～ N(0,1), N(0,1) represents a Gaussian distribution, and ε represents random noise. The generation of a random noise image is related to the amount of random noise, denoted as t. Different amounts of random noise t can be used to simulate a perturbation process that gradually increases over time. Each amount of random noise represents a perturbation process that gradually changes the image distribution by applying noise multiple times, starting from an initial state. Therefore, a smaller amount of random noise represents a weaker noise perturbation, while a larger amount represents a stronger noise perturbation.

[0058] The computer device calls the diffusion model and generates an image based on the prompt text through the diffusion model. Figure 3 is a structural diagram of the diffusion model. As shown in Figure 3, the diffusion model includes a CLIP (Contrastive Language-Image Pre-training) model, a diffuser, a noise predictor, and an image decoder. The CLIP model includes an image encoder and a text encoder. The process of generating a predicted image by the diffusion model is as follows: after the original image X is encoded by the image encoder, the image encoding representation of the original image in the latent space is obtained, which is denoted as Z; the image encoding representation Z is input into the diffuser, and the image encoding representation Z is mixed with the random noise image sampleε through the diffuser. ～ N(0,1) is superimposed to produce a noisy image Z T ; The prompt text is semantically encoded through the text encoder to obtain the text semantic representation corresponding to the prompt text, which is recorded as τ θ ; The noisy image Z T , text semantic representation τ θ The coding information of the random noise amount t is input into the noise predictor, and the noise predictor generates a predicted noise image; the noisy image Z is calculated according to the preset formula. T Subtract the predicted noise image from the predicted noise image to obtain the predicted noise image corresponding to the previous step of random noise amount t, which is recorded as Z T-1 ; Predict the noise image Z T-1 , text semantic representation τ θ The coded information of the random noise amount t-1 is input into the noise predictor, and the noise predictor generates the predicted noise image Z T-2; And so on, until the noise predictor generates the predicted noisy image Z0, that is, the image coding representation Z corresponding to the image without superimposed random noise is obtained. The predicted noisy image Z0 is decoded by the image decoder to obtain the predicted image

[0059] In this embodiment, each prompt text is processed in the same manner: the original image, a prompt text, and a random noise image are input into the trained diffusion model. After the diffusion model outputs a predicted noise image, the original image, the next prompt text, and the random noise image are then input into the trained diffusion model, until the original image, the last prompt text, and the random noise image are input into the trained diffusion model. As each prompt text is sequentially input into the diffusion model, the random noise image used in each round of the diffusion model's generation of the predicted noise image based on the prompt text can be the same or different.

[0060] Specifically, the server reads a prompt text from multiple prompt texts, and inputs the original image, prompt text and random noise image into a trained diffusion model. After encoding the original image through the image encoder, the image coding representation of the original image in the latent space is obtained; the image coding representation and the random noise image are input into the diffuser, and the image coding representation and the random noise image are superimposed on each other through the diffuser to generate a noisy image; the prompt text is semantically encoded through the text encoder to obtain the text semantic representation corresponding to the prompt text; the encoding information of the noisy image, text semantic representation and random noise amount is input into the noise predictor, and a predicted noise image is generated through the noise predictor, and the difference between the generated predicted noise image and the random noise image is calculated.

[0061] The server extracts another prompt text from the multiple prompt texts that has not been input into the diffusion model, and returns to execute the step of inputting the original image, prompt text, and random noise image into the trained diffusion model. The server continues to execute until all the prompt texts are input into the diffusion model, and obtains the difference between the predicted noise image and the random noise image corresponding to each prompt text. The difference is the difference between the noises of the two.

[0062] Step 206 : Based on the differences calculated for each predicted noise image, a predicted noise image with the smallest corresponding difference is selected, and a prompt text relied upon when generating the selected predicted noise image is obtained.

[0063] The predicted noise image with the smallest difference refers to the predicted noise image corresponding to the smallest difference between the predicted noise image and the random noise image. The prompt text on which the predicted noise image with the smallest difference depends refers to the prompt text required to generate the predicted noise image. For example, there are M prompt texts, and M corresponding predicted noise images are generated. The prompt text on which the predicted noise image with the smallest difference between the M predicted noise images and the random noise image depends is Mi, where Mi represents the i-th prompt text. The image label corresponding to the prompt text Mi is used as the image label of the original image.

[0064] In some embodiments, multiple prompt texts that are relied upon to generate predicted noise images with relatively small differences can also be determined, and the image labels corresponding to the dependent prompt texts can be used as the image labels of the original image, that is, the original image corresponds to multiple image labels.

[0065] Among them, relatively small differences refer to arranging the differences between each predicted noise image and the random noise image in order from small to large, selecting prompt texts on which multiple predicted noise images with relatively small differences depend in order from small to large, and using the image labels corresponding to the dependent prompt texts as the image labels of the original images.

[0066] In some embodiments, FIG4 is a schematic diagram of the overall framework of the image classification method. As can be seen from FIG4 , after step 204, the difference between the predicted noise image and the random noise image corresponding to each prompt text can be obtained by using the following calculation formula, and the prompt text on which the predicted noise image with the smallest difference depends is selected. The corresponding calculation formula is as follows:

[0067] Among them, ε represents a random noise image; ε θ represents the predicted noise image; x t represents a noisy image; E t represents the predicted image; t represents the amount of random noise.

[0068] Specifically, the server calculates the predicted noise image ε θ The difference between the random noise image ε and the predicted noise image ε is determined θ The hint text that the predicted noise image with the smallest difference from the random noise image ε depends on.

[0069] Step 208: Use the image tag corresponding to the dependent prompt text as the image tag of the original image.

[0070] Specifically, after the server determines the prompt text on which the predicted noise image with the smallest corresponding difference is generated, the server uses the image label corresponding to the prompt text as the image label of the original image.

[0071] In the above image classification method, multiple prompt texts are obtained, and each prompt text is generated according to a different image label. For each prompt text, the original image, the prompt text, and the random noise image are input into a trained diffusion model, and a predicted noise image is generated by the diffusion model. The difference between the generated predicted noise image and the random noise image is calculated, that is, each prompt text corresponds to a random noise image, and the image label corresponding to the prompt text on which the predicted noise image with the smallest corresponding difference is generated is used as the image label of the original image. The above method can directly transfer the capabilities of the diffusion model to the multi-label classification task, and directly use the diffusion model to classify business images (i.e., original images) in a specific application scenario. There is no need to rely on manually annotated image data in the application scenario to train the image classification model and then use the trained images for classification, because training requires a large amount of manually annotated image data. In this way, the workload of manual annotation can be reduced, a large amount of manual annotation costs can be reduced, and the efficiency of image multi-label classification can be improved.

[0072] In one embodiment, the original image, the prompt text, and the random noise image are input into a trained diffusion model, and a noisy image is generated based on the original image and the random noise image by the diffusion model, including the following steps:

[0073] The image encoder of the diffusion model is used to encode the original image to obtain an image coding representation of the original image; the diffuser of the diffusion model is used to superimpose noise information corresponding to the random noise image onto the image coding information to obtain a noisy image.

[0074] The image encoder in the CLIP model encodes the original image so that it can be represented in a latent space. The resulting image encoding is represented as an image embedding vector. Latent space is a common term in the generative field, representing the high-dimensional information of an image and is generally used for feature alignment of the generated results.

[0075] The purpose of superimposing the noise information corresponding to the random noise image onto the image coding information is to destroy the original image and obtain a noisy image, and to regenerate the predicted image in the denoising process of the noisy image.

[0076] Specifically, the server encodes the original image through the image encoder in the CLIP model to obtain the image coding representation of the original image, which is the image representation of the original image in the latent space; the server uses the diffuser of the diffusion model to superimpose the noise information corresponding to the random noise image on the image coding information to obtain a noisy image.

[0077] In this embodiment, the original image is encoded by the image encoder in the CLIP model so that the original image can be represented in the latent space, and the noise information corresponding to the random noise image is superimposed on the image encoding information in the latent space. By diffusing in the latent space, it is possible to maintain a high generation quality while reducing computational resource consumption.

[0078] In one embodiment, generating a predicted noise image based on a noisy image and a prompt text includes the following steps:

[0079] The prompt text is semantically encoded through the text encoder of the diffusion model to obtain the text semantic representation corresponding to the prompt text; the noisy image and the text semantic representation are input into the noise predictor of the diffusion model, and the predicted noise image is output through the noise predictor.

[0080] The text encoder of the diffusion model is the same as that of the CLIP model. The purpose of semantically encoding the prompt text through the text encoder is to represent the prompt text in the latent space. The semantic representation of text is generally a text embedding vector.

[0081] Specifically, the server semantically encodes the prompt text through the text encoder of the CLIP model in the diffuser to obtain the text semantic representation corresponding to the prompt text; the noisy image, text semantic representation and random noise amount encoding information are input into the noise predictor of the diffusion model, and the predicted noise image is output through the noise predictor, where the random noise amount encoding information refers to the vector representation obtained by encoding the random noise amount.

[0082] In one embodiment, the noise predictor includes a plurality of alternatingly connected residual networks and attention layers; a noisy image and a text semantic representation are input into the noise predictor of the diffusion model, and the noise predictor outputs a predicted noisy image, including:

[0083] 1. Input the random noise amount encoding information corresponding to the noisy image and the random noise image into the first residual network, and output the predicted noise information through the first residual network; input the predicted noise information and the text semantic representation into the first attention layer, and output the attention information through the first attention layer.

[0084] Optionally, the noise predictor can use the UNet model. In order to add text semantic representation to the noise prediction process, the structural diagram of the noise predictor of this embodiment is shown in Figure 5. Referring to Figure 5, it can be seen that for a prompt text, the prompt text is first compressed into a text semantic expression using the text encoder of the CLIP model. The text semantic expression can be a text embedding vector. In the denoising process of the UNet model, the attention mechanism is continuously used to inject text embedding vectors into the denoising process. Each residual network is no longer directly connected to the adjacent residual network, but an attention layer is added in the middle. The text embedding vector obtained by the text encoder in the CLIP model is processed using this attention layer. In this way, text semantic expression can be continuously injected.

[0085] The random noise encoding information refers to the vector representation obtained by encoding the random noise. The predicted noise information and attention information represent multi-dimensional arrays of different dimensions. Through the sequential processing of multiple alternating residual networks and attention layers in the UNet model, a predicted noise image can be obtained.

[0086] Specifically, the server encodes the random noise amount of the random noise image through an encoder to obtain random noise amount encoding information, inputs the noisy image and the random noise amount encoding information into the first residual network, and outputs the predicted noise information through the first residual network; inputs the predicted noise information and the text semantic representation into the first attention layer, and outputs the attention information through the first attention layer.

[0087] 2. Starting from the second residual network, the next residual network is used as the current residual network, the next attention layer is used as the current attention layer, the previous attention information output by the previous attention layer connected to the current residual network and the random noise amount encoding information are input into the current residual network, and the predicted noise information is output through the current residual network; the predicted noise information and text semantic representation output by the current residual network are input into the current attention layer, and the attention information is output through the current attention layer.

[0088] Specifically, starting from the second residual network, the server uses the second residual network as the current residual network, the second attention layer as the current attention layer, and inputs the previous attention information output by the previous attention layer connected to the current residual network and the random noise amount encoding information into the current residual network, and outputs the predicted noise information through the current residual network; inputs the predicted noise information and text semantic representation output by the current residual network into the current attention layer, and outputs the attention information through the current attention layer; uses the next residual network connected to the current attention layer as the current residual network, and the next attention layer as the current attention layer, and returns to execute the step of inputting the previous attention information output by the previous attention layer connected to the current residual network and the random noise amount encoding information into the current residual network, and continues to execute until the current residual network is the last residual network and the current attention layer is the last attention layer.

[0089] 3. Use the attention information output by the last attention layer as the predicted noise image.

[0090] Among them, the attention information output by the last attention layer is also the output of the noise predictor. Therefore, the attention information output by the last attention layer is used as the predicted noise image.

[0091] In this embodiment, in the process of generating a predicted noise image based on a noisy image and a prompt text, multiple attention layers are introduced into the noise predictor. The text semantic expression is added to the predicted noise information output by the residual network through the attention layer. The text semantic expression can be continuously injected into the noise predictor, so that the predicted noise image output by the noise predictor has a higher degree of matching with the original image.

[0092] In one embodiment, the training process of the CLIP model specifically includes the following steps:

[0093] 1. Obtain training samples, which include sample text, sample images, and annotation information used to indicate whether the sample text and sample image match.

[0094] The sample text includes image labels, which are preset image classification categories, and sample text, which is a piece of text describing the image. The goal of CLIP model training is to enable the model to match text with images.

[0095] The annotation information includes whether the sample text and the sample image match or whether the sample text and the sample image do not match, which can be represented by 0 or 1. Of course, other forms can also be used to quantify whether the sample text and the sample image match.

[0096] Specifically, the server randomly selects a training sample from the training set. The training sample includes a sample text and a sample image, and also includes annotation information indicating whether the sample text and the sample image match.

[0097] 2. Perform image encoding on the sample image through the initial image encoder to obtain an image encoding representation of the sample image.

[0098] The initial image encoder is the image encoder in the initial state of the CLIP model. After the sample image is encoded by the initial image encoder, the resulting image encoding is represented as an image embedding vector.

[0099] 3. Use the initial text encoder to semantically encode the sample text and obtain the text semantic representation corresponding to the sample text.

[0100] The initial text encoder is the initial state text encoder in the CLIP model. After the sample text is semantically encoded by the initial text encoder, the obtained text semantics is represented as a text embedding vector.

[0101] 4. Calculate the similarity between the image encoding representation and the text semantic representation, and determine the prediction result as to whether the sample text matches the sample image based on the similarity.

[0102] In this embodiment, the similarity between the image coding representation and the text semantic representation, that is, the prediction result, can be calculated by cosine similarity. The larger the prediction result, the more the model predicts that the sample text matches the sample image, and the smaller the prediction result, the less the model predicts that the sample text matches the sample image.

[0103] 5. Based on the difference between the labeled information and the predicted results, a sample loss is constructed. After updating the initial image encoder and the initial text encoder based on the sample loss, the training is continued by returning to the step of obtaining training samples to obtain the image encoder of the diffusion model and the text encoder of the diffusion model.

[0104] Among them, when the CLIP model starts training, even if the sample text and sample image are matched, the initial image encoder and initial text encoder in the CLIP model have just been initialized and their parameters are chaotic. Therefore, the image encoding representation and the text semantic representation must also be chaotic, and the calculated similarity is often close to 0. This will result in the situation shown in Figure 6, where the sample text and sample image are matched, but the cosine similarity indicates that the sample text and sample image do not match. At this time, the annotation information and the prediction results are different and the difference is large. It is necessary to reversely update the parameters of the initial image encoder and initial text encoder based on the comparison results of the annotation information and the prediction results.

[0105] By repeatedly repeating the backpropagation process, the initial image encoder and initial text encoder are trained, resulting in the diffusion model's image encoder and text encoder. If a 1 in the annotation information indicates a match between the sample text and the sample image, and a 0 indicates a mismatch, the trained CLIP model should be able to: for paired sample images and sample text, the diffusion model's image encoder and text encoder will ultimately output similar embedding vectors, resulting in a cosine similarity close to 1, indicating a match. For mismatched sample images and sample text, the diffusion model's image encoder and text encoder will output very different embedding vectors, resulting in a cosine similarity close to 0, indicating a mismatch. For example, if a picture of a puppy is fed into the CLIP model's image encoder and the text description "puppy photo" is fed into the CLIP model's text encoder, the CLIP model will generate two similar embedding vectors, thus confirming that the sample text and sample image match. The CLIP model now connects the previously unrelated concepts of computer vision and human language, giving them a unified mathematical representation. The prompt text can be converted into an image representation through a text encoder, or the prompt text can be converted into a semantic representation through an image encoder, and the two can interact with each other.

[0106] Construct sample loss. Specifically, the server can calculate the relative distance between the annotation information and the prediction result through the loss function. If the relative distance is less than the preset value, the server stops updating the parameters of the CLIP model and obtains the trained diffusion model image encoder and diffusion model text encoder.

[0107] Specifically, the server constructs a sample loss based on the difference between the labeled information and the prediction results, updates the parameters of the initial image encoder and the initial text encoder, and returns to the step of obtaining training samples to continue training until the sample loss is less than the preset value. The training of the CLIP model is stopped, and the image encoder of the diffusion model and the text encoder of the diffusion model are obtained.

[0108] In this embodiment, by training the CLIP model, computer vision and human language, two originally unrelated information, are connected together through the CLIP model, and the two have a unified mathematical representation; the prompt text can be converted into an image representation through a text encoder, and the prompt text can also be converted into a semantic representation through an image encoder. The two can interact with each other, providing a basis for the diffusion model to predict images based on the prompt text.

[0109] In one embodiment, obtaining multiple prompt texts includes the following steps:

[0110] Obtain a prompt text template and multiple image tags; fill each of the multiple image tags into the prompt text template to obtain multiple prompt texts corresponding to the corresponding image tags.

[0111] In this embodiment, the multiple image label sets include a first image label set and a second image label set. The first image label in the first image label set and the second image label in the second image label set are combined to obtain the multiple image labels. The first image label in the first image label set and the second image label in the second image label set are derived from different image application scenarios. For example, the first image label is derived from the image label of a training sample in the original training data of the diffusion model, and the second image label is derived from the image label of a business image described by the business. The business refers to a specific application scenario in different fields.

[0112] The general format of the prompt text is a photo of {class}. The prompt text corresponding to M image labels is represented as: A photo of class1, A photo of class2, ..., A photo of classM. Figure 7 shows the dataset format used for training in the diffusion model. The text shown in the prompt text column in Figure 7 is denoted as the prompt text.

[0113] Specifically, the server reads multiple first image tags from a pre-stored first image tag set, reads multiple second image tags from a pre-stored second image tag set, fills the multiple first image tags and the multiple second image tags into the prompt text template respectively, and obtains multiple prompt texts corresponding to the corresponding image tags.

[0114] In this embodiment, image labels from different image application scenarios are mixed to obtain multiple image labels required by this embodiment, so that when the diffusion model generates a predicted image based on the prompt text, the prompt text determined to generate the predicted noise image with the smallest corresponding difference includes the image label of this business, thereby ensuring that the image label assigned to the original image includes the image label of this business, thereby improving the label annotation accuracy of the original image.

[0115] In one embodiment, since the features on the original image are within the size range of the original image and occupy a relatively small area, in order to accurately classify the features into image labels, in this embodiment, the method further includes:

[0116] The original image is divided into multiple sub-images.

[0117] Specifically, as shown in FIG4 , the server divides the original image into N*N grids. For example, N can be 3, and the original image is divided into 9 sub-images.

[0118] Accordingly, step 204 may include:

[0119] 1. Obtain one prompt text from the multiple prompt texts in sequence. For each obtained prompt text, input the sub-image, the obtained prompt text, and the random noise image into a trained diffusion model, generate a noisy sub-image based on the sub-image and the random noise image using the diffusion model, generate a sub-predicted noise image based on the noisy sub-image and the prompt text, and calculate the difference between the generated sub-predicted noise image and the random noise image.

[0120] Each of the multiple divided sub-images is processed in the same manner: one sub-image, one prompt text, and a random noise image are input into the trained diffusion model. After the diffusion model outputs a sub-predicted noise image, the sub-image, the next prompt text, and the random noise image are then input into the trained diffusion model. This process repeats until the sub-image, the last prompt text, and the random noise image are input into the trained diffusion model, resulting in multiple sub-predicted noise images corresponding to the sub-image. The process continues by inputting the next sub-image, one prompt text, and a random noise image into the trained diffusion model until multiple sub-predicted noise images corresponding to all sub-images are obtained.

[0121] It should be noted that, for each sub-image, the size of the corresponding random noise image should be the same as the sub-image size; and for the original image, the size of the corresponding random noise image should be the same as the original image.

[0122] Specifically, the server extracts a subgraph from multiple subgraphs, reads a prompt text from multiple prompt texts, and feeds the subgraph, prompt text, and random noise image into a trained diffusion model. The subgraph is encoded using an image encoder to obtain an image encoding representation of the subgraph in the latent space. The image encoding representation and the random noise image are fed into a diffuser, which superimposes the image encoding representation and the random noise image to generate a noisy subgraph. The prompt text is semantically encoded using a text encoder to obtain a text semantic representation corresponding to the prompt text. The noisy subgraph, text semantic representation, and the encoded information of the random noise amount are fed into a noise predictor, which generates a sub-predicted noise image and calculates the difference between the generated sub-predicted noise image and the random noise image. The server then extracts another prompt text from the multiple prompt texts that has not been fed into the diffusion model and returns to the step of feeding the subgraph, prompt text, and random noise image into the trained diffusion model. This process continues until all prompt texts have been fed into the diffusion model. The difference between the sub-predicted noise image corresponding to each prompt text in the subgraph and the random noise image is obtained, which is the difference between the noises of the two.

[0123] The server extracts another subgraph from the multiple subgraphs that has not been input into the diffusion model, and returns to execute the step of inputting the subgraph, prompt text, and random noise image into the trained diffusion model, and continues to execute until the difference between the sub-prediction noise image corresponding to each prompt text corresponding to all subgraphs and the random noise image is obtained. This difference is the difference between the noises of the two.

[0124] In some embodiments, the sub-image, the acquired prompt text, and the random noise image are input into a trained diffusion model, and a noisy sub-image is generated according to the sub-image and the random noise image by the diffusion model, including:

[0125] The sub-image is encoded by the image encoder of the diffusion model to obtain the image coding representation of the sub-image; the noise information corresponding to the random noise image is superimposed on the image coding information by the diffuser of the diffusion model to obtain the noisy sub-image.

[0126] The image encoder is the one in the CLIP model, which encodes the sub-image so that it can be represented in the latent space. The resulting image encoding is represented as an image embedding vector. Latent space is a common term in the generative field, representing the high-dimensional information of an image and is generally used for feature alignment of the generated results.

[0127] The purpose of superimposing the noise information corresponding to the random noise image onto the image coding information is to destroy the original image and obtain a noisy sub-image. The predicted sub-image is then regenerated during the denoising process of the noisy sub-image.

[0128] Specifically, the server encodes the sub-image through the image encoder in the CLIP model to obtain the image coding representation of the sub-image, which is the image representation of the sub-image in the latent space; the server uses the diffuser of the diffusion model to superimpose the noise information corresponding to the random noise image onto the image coding information to obtain a noisy image.

[0129] In this embodiment, the sub-image is encoded by the image encoder in the CLIP model so that the sub-image can be represented in the latent space, and the noise information corresponding to the random noise image is superimposed on the image encoding information in the latent space. By diffusing in the latent space, it is possible to maintain a high generation quality while reducing computational resource consumption.

[0130] In some embodiments, generating a sub-predicted noise image according to the noisy sub-image and the prompt text includes:

[0131] The prompt text is semantically encoded through the text encoder of the diffusion model to obtain the text semantic representation corresponding to the prompt text; the noisy sub-graph and the text semantic representation are input into the noise predictor of the diffusion model, and the sub-predicted noise image is output through the noise predictor.

[0132] The text encoder of the diffusion model is the same as the text encoder of the CLIP model. The purpose of semantically encoding the prompt text through the text encoder is to represent the prompt text in the latent space. The semantic representation of text is generally a text embedding vector.

[0133] Specifically, the server semantically encodes the prompt text through the text encoder of the CLIP model in the diffuser to obtain the text semantic representation corresponding to the prompt text; the noisy sub-graph, text semantic representation and random noise amount encoding information are input into the noise predictor of the diffusion model, and the sub-predicted noise image is output through the noise predictor, where the random noise amount encoding information refers to the vector representation obtained by encoding the random noise amount.

[0134] Second, based on the differences calculated for each sub-prediction noise image, a sub-prediction noise image with the smallest corresponding difference is selected, and a prompt text relied upon when generating the selected sub-prediction noise image is obtained.

[0135] After obtaining the difference between the sub-predicted noise image and the random noise image corresponding to each prompt text corresponding to all sub-images, the server determines the prompt text on which the sub-predicted noise image with the smallest difference is generated in each sub-image.

[0136] For example, as shown in Figure 4, there are nine sub-images and nine prompt texts. For sub-image 1, the nine sub-predicted noise images generated based on sub-image 1 and the nine prompt texts depend on prompt text 3, and the image label corresponding to prompt text 3 is used as the image label of sub-image 1. For sub-image 2, the nine sub-predicted noise images generated based on sub-image 2 and the nine prompt texts depend on prompt text 7, and the image label corresponding to prompt text 7 is used as the image label of sub-image 2. The remaining sub-images are processed in the same manner, and the prompt text that the sub-predicted noise image with the smallest difference in each sub-image depends on is determined. The image label corresponding to the prompt text that each sub-image depends on is used as the image label of each sub-image.

[0137] 3. Use the image label corresponding to the dependent prompt text as the image label of the sub-image.

[0138] Specifically, for each sub-image, after determining the prompt text that is relied upon to generate the corresponding predicted noise sub-image with the smallest difference in each sub-image, the server uses the image label corresponding to the relied upon prompt text as the image label of each sub-image.

[0139] Finally, the server can obtain the image label of the original image based on the image labels of the multiple sub-images.

[0140] Optionally, the image labels of the multiple sub-images can all be used as the image labels of the original image;

[0141] Optionally, voting may be performed based on the image labels of the multiple sub-images, and an image label may be selected from the various image labels of the multiple sub-images as the image label of the original image.

[0142] For example, if the original image is divided into 9 sub-images, where the image labels corresponding to 3 sub-images are image label L1, the image labels corresponding to 4 sub-images are image label L2, and the image labels corresponding to 2 sub-images are image label L3, then image labels L1, L2, and L3 can all be used as image labels of the original image, or voting can be performed based on the image labels corresponding to each sub-image, and the image label of the original image can be determined to be L2 based on the voting results.

[0143] In this embodiment, the original image is divided into multiple sub-images. For each sub-image, based on multiple prompt texts and a diffusion model, multiple sub-prediction noise images corresponding to the prompt texts are generated, and the prompt text on which the sub-prediction noise image with the smallest difference is generated is determined; the image label corresponding to the dependent prompt text is used as the image label of the sub-image; the image labels of the multiple sub-images are used as the image labels of the original image, and image labels can be assigned to image features in various areas of the original image, thereby increasing the image labels of the original image, so that the image labels of the original image have more categories and the annotation accuracy of the original image is higher.

[0144] In one embodiment, this embodiment provides detailed steps of an image classification method, specifically including the following steps:

[0145] 1. Get the prompt text template and multiple image tags.

[0146] Second, each of the multiple image tags is filled into the prompt text template to obtain multiple prompt texts corresponding to the corresponding image tags.

[0147] 3. For each prompt text, the original image, prompt text and random noise image are input into the trained diffusion model. The original image is encoded through the image encoder of the diffusion model to obtain the image encoding representation of the original image.

[0148] Fourth, through the diffuser of the diffusion model, the noise information corresponding to the random noise image is superimposed on the image coding information to obtain a noisy image.

[0149] 5. Through the text encoder of the diffusion model, the prompt text is semantically encoded to obtain the text semantic representation corresponding to the prompt text.

[0150] 6. Input the random noise amount encoding information corresponding to the noisy image and the random noise image into the first residual network of the noise predictor, and output the predicted noise information through the first residual network; input the predicted noise information and the text semantic representation into the first attention layer of the noise predictor, and output the attention information through the first attention layer.

[0151] 7. Starting from the second residual network of the noise predictor, the next residual network is used as the current residual network, the next attention layer of the noise predictor is used as the current attention layer, the previous attention information output by the previous attention layer connected to the current residual network and the random noise amount encoding information are input into the current residual network, and the predicted noise information is output through the current residual network; the predicted noise information and text semantic representation output by the current residual network are input into the current attention layer, and the attention information is output through the current attention layer.

[0152] 8. Use the attention information output by the final attention layer of the noise predictor as the predicted noise image.

[0153] 9. Calculate the difference between the generated predicted noise image and the random noise image.

[0154] 10. Based on the differences calculated for each predicted noise image, select a predicted noise image with the smallest corresponding difference, and obtain a prompt text relied upon when generating the selected predicted noise image.

[0155] 11. The image label corresponding to the dependent prompt text is used as the image label of the original image.

[0156] In this embodiment, multiple prompt texts are obtained, each generated based on a different image label. For each prompt text, the original image, the prompt text, and a random noise image are input into a trained diffusion model. A predicted noise image is generated using the diffusion model, and the difference between the generated predicted noise image and the random noise image is calculated. That is, each prompt text corresponds to a random noise image, and the image label corresponding to the prompt text that generates the predicted noise image with the smallest difference is used as the image label of the original image. This method can directly transfer the capabilities of the diffusion model to multi-label classification, reducing the amount of annotation while eliminating the need to retrain the diffusion model. The final label of the original image can be directly obtained using the existing diffusion model, significantly reducing the workload.

[0157] In one embodiment, this embodiment provides a diffusion model processing method that can be applied in the application environment shown in FIG1 . The server 104 obtains multiple sample images from the terminal 102 or a data storage system, each sample image corresponding to an image label. For each sample image, the server 104 generates corresponding sample prompt text based on the sample image's image label. The server 104 generates a noisy image based on the sample image and a sample random noise image using an initial diffusion model. The server 104 generates a predicted noise image based on the noisy image and the sample prompt text. The server 104 denoises the noisy image based on the predicted noise image to obtain a predicted image. The server 104 constructs a sample loss based on the difference between the predicted image and the sample image. The server 104 updates the initial diffusion model based on the sample loss. The updated trained diffusion model is then used for image classification.

[0158] In one embodiment, a method for processing a diffusion model is provided. The method is described by taking the server in FIG. 1 as an example, and specifically includes the following steps:

[0159] 1. Obtain multiple sample images, each of which has a corresponding image label.

[0160] Currently, there are significant domain differences between the training data used by the diffusion model and the business data used in the application scenario, resulting in low image label classification accuracy for the diffusion model. Therefore, to address this issue, this embodiment requires obtaining a diffusion model suitable for the application scenario before performing image classification. To obtain a diffusion model suitable for this business, the sample images in this embodiment include the business images required by the application scenario and the general images used by the diffusion model. Using both business and general images to train the diffusion model allows the diffusion model's capabilities to be applied to multi-label classification for this business, improving the diffusion model's ability to classify business images for the application scenario. Furthermore, using both business and general images to train the diffusion model eliminates the need for a large number of manually annotated business images for the application scenario, requiring only a small number of these images. This significantly reduces manual annotation costs, accelerates diffusion model training, and enables timely launch of image classification functionality for business images in the business scenario. General images refer to sample images from the diffusion model's general training sample set, which encompasses sample images from various fields. For example, general images include sample images from the food field, animal field, and landscape field. Business images refer to images involved in the specific application scenarios of this application, such as product image classification scenarios and video cover image classification scenarios.

[0161] Specifically, the server extracts multiple general images from the original training sample set used by the diffusion model, extracts multiple business images from the business data required by the application scenario, and uses the general images and business images as sample images required for training the diffusion model applicable to this business.

[0162] 2. For each sample image, generate the corresponding sample prompt text according to the image label of the sample image, generate a noisy image according to the sample image and the sample random noise image through the initial diffusion model, generate a predicted noise image according to the noisy image and the sample prompt text, and denoise the noisy image according to the predicted noise image to obtain a predicted image.

[0163] In some embodiments, generating corresponding sample prompt text according to the image tag of the sample image includes:

[0164] A sample prompt text template and multiple image labels of a sample image are obtained; each of the multiple image labels is filled into the sample prompt text template to obtain multiple sample prompt texts corresponding to the corresponding image labels.

[0165] Among them, the sample prompt text template can adopt a prompt general template. This embodiment includes a first image label set and a second image label set, and the first image label in the first image label set and the second image label in the second image label set are combined to obtain multiple image labels, and each obtained image label includes at least one first image label or a second image label, wherein the first image label in the first image label set and the second image label in the second image label set are derived from different image application scenarios. For example, the first image label is derived from the image label of the training sample in the original training data of the diffusion model, and the second image label is derived from the image label of the business image described in the business. Among them, business refers to specific application scenarios in different fields.

[0166] The general format of the sample prompt text is a photo of{class}. The prompt texts corresponding to M image labels are represented as: A photo of class1, A phot of class2, ..., A photo of classM.

[0167] Specifically, the server reads multiple first image tags from a pre-stored first image tag set, reads multiple second image tags from a pre-stored second image tag set, and fills the multiple first image tags and the multiple second image tags into the sample prompt text template respectively to obtain multiple sample prompt texts corresponding to the corresponding image tags.

[0168] In some embodiments, generating a noisy image according to a sample image and a sample random noise image using an initial diffusion model includes:

[0169] The sample image is encoded by the image encoder of the initial diffusion model to obtain an image coding representation of the sample image; the noise information corresponding to the sample random noise image is superimposed on the image coding information by the diffuser of the initial diffusion model to obtain a noisy image.

[0170] The image encoder, as used in the CLIP model, encodes the sample image so that it can be represented in a latent space. The resulting image encoding is represented as an image embedding vector. Latent space is a common term in the generative field, representing the high-dimensional information of an image and is generally used for feature alignment of the generated results.

[0171] The noise information corresponding to the sample random noise image is superimposed on the image encoding information to destroy the sample image, resulting in a noisy image. The predicted image is then regenerated during the denoising process of the noisy image. The image encoder is the same as that in the CLIP model. During the training of the initial diffusion model, the image encoder of the CLIP model is frozen. That is, the image encoder of the CLIP model is already trained during the training of the diffusion model. Therefore, when the parameters of the initial diffusion model are updated, the parameters of the CLIP model are not updated.

[0172] Specifically, the server encodes the sample image through the image encoder in the CLIP model to obtain the image coding representation of the sample image, which is the image representation of the sample image in the latent space; the server uses the diffuser of the initial diffusion model to superimpose the noise information corresponding to the sample random noise image onto the image coding information to obtain a noisy image.

[0173] In some embodiments, generating a predicted noise image based on a noisy image and a sample prompt text includes:

[0174] The sample prompt text is semantically encoded through the text encoder of the initial diffusion model to obtain the sample text semantic representation corresponding to the sample prompt text; the noisy image and the sample text semantic representation are input into the noise predictor of the initial diffusion model, and the predicted noise image is output through the noise predictor.

[0175] The text encoder of the initial diffusion model is the text encoder of the CLIP model. The purpose of semantically encoding the sample prompt text using the text encoder is to represent the sample prompt text in the latent space. The semantic representation of the sample text is generally a text embedding vector. The text encoder is the text encoder of the CLIP model. During the training of the initial diffusion model, the text encoder of the CLIP model is frozen. That is, the text encoder of the CLIP model is already trained during the training of the diffusion model. Therefore, when updating the parameters of the initial diffusion model, the parameters of the CLIP model are not updated.

[0176] Specifically, the server semantically encodes the sample prompt text through the text encoder of the CLIP model in the initial diffuser to obtain the sample text semantic representation corresponding to the sample prompt text; the noisy image, the sample text semantic representation and the sample random noise amount encoding information are input into the noise predictor of the initial diffusion model, and the predicted noise image is output through the noise predictor, wherein the sample random noise amount encoding information refers to the vector representation obtained by encoding the sample random noise amount.

[0177] In some embodiments, denoising the noisy image according to the predicted noise image to obtain the predicted image includes:

[0178] According to the preset formula, the noisy image and the predicted noise image are subtracted to obtain the predicted noisy image Z corresponding to the previous step of the random noise amount t. T-1 ; Input the predicted noisy image corresponding to the previous step, the semantic representation of the sample text and the encoded information of the random noise amount t-1 into the noise predictor, and generate the predicted noisy image Z through the noise predictor T-2 ; And so on, until the noise predictor generates a predicted noisy image Z0, the predicted noisy image Z0 is decoded by the image decoder to obtain a predicted image.

[0179] 3. Based on the difference between the predicted image and the sample image, a sample loss is constructed, and the initial diffusion model is updated according to the sample loss. The trained diffusion model obtained after the update is used for image classification.

[0180] Among them, the sample loss ensures that the pixel distance between the predicted image and the sample image is less than a preset value. The sample loss is used to calculate the pixel distance between the predicted image and the sample image. If the pixel distance is less than the preset value, the update of the parameters of the initial diffusion model is stopped, and the trained diffusion model is obtained. The sample loss can be calculated using the following formula:

[0181] Where, l g is the sample loss; N is the number of pixels in the sample image; G i is the i-th pixel in the predicted image; J i is the i-th pixel in the sample image.

[0182] Specifically, the server constructs a sample loss based on the pixel distance between the predicted image and the sample image, updates the parameters of the initial diffusion model, and returns to the step of obtaining multiple sample images to continue training until the sample loss is less than a preset value. The training of the initial diffusion model is stopped to obtain a trained diffusion model.

[0183] In this embodiment, by training a diffusion model suitable for the application scenario, the problem of low image label classification accuracy of the diffusion model caused by large domain differences between the training data used by the current diffusion model and the business data required by the application scenario is avoided, thereby improving the accuracy of image label classification.

[0184] In one embodiment, in order to transfer the capabilities of the diffusion model to the multi-label classification work of this service, a second sample image derived from the service image is added to the sample image, and the diffusion model is trained using the second sample image, so that the diffusion model can have the recognition capability of this service. Specifically, the training steps of the diffusion model include:

[0185] 1. Obtain training samples, where the training samples include general images and business images. Multiple first sample images are derived from general images, and multiple second sample images are derived from business images. An image label set formed by image labels of first sample images in the multiple first sample images is the same as an image label set formed by image labels of second sample images in the multiple second sample images.

[0186] Among them, there is a large domain difference between the training data used by the current diffusion model and the business data required by the application scenario, which leads to low accuracy of image label classification by the diffusion model. Therefore, to solve the above problem, this embodiment needs to migrate the capabilities of the diffusion model to the multi-label classification work of this business before performing image classification.

[0187] The training samples of this embodiment include general images and business images. A plurality of first sample images are derived from general images, and a plurality of second sample images are derived from business images. The plurality of first sample images and the plurality of second sample images each correspond to a plurality of image labels of their own. The image label set formed by the image labels of the first sample images in the plurality of first sample images is the same as the image label set formed by the image labels of the second sample images in the plurality of second sample images. For example, there are 1,000 first sample images in the plurality of first sample images, and the image label sets formed by the image labels of each first sample image are 10, namely L1, L2, L3, ..., L10. There are 100 second sample images in the plurality of second sample images, and the image label sets formed by the image labels of each second sample image are also 10, and are also L1, L2, L3, ..., L10. By training the diffusion model with business images and general images together, the capabilities of the diffusion model can be transferred to the multi-label classification work of this business.

[0188] Specifically, the server can first select some images from the business images as second sample images, and based on the image labels of the second sample images, select some images with any one or more of the above image labels from the general images as first sample images, and obtain training samples based on the first sample images and the second sample images.

[0189] The domain of the first sample image determined by the above method is closer to the domain of the second sample image. This method is used to reduce the domain discrepancy between the training data used by the current diffusion model and the business data required for the application scenario. Adding a second sample image derived from business images to the sample images and using it to train the diffusion model can transfer the model's capabilities to the multi-label classification work of this business, giving the diffusion model the recognition capabilities of this business.

[0190] 2. In the first training stage, a plurality of first sample images and a plurality of second sample images are used to perform first-stage model training on the initial diffusion model to obtain a diffusion model trained in the first stage.

[0191] In the first training stage, the initial diffusion model is trained using the first sample image derived from the general image and the second sample image derived from the business image. The diffusion model trained in the first stage has the ability to recognize the business image of this business.

[0192] FIG8 is a schematic diagram of the training process of the first training stage of the initial diffusion image. As can be seen from FIG8 , the first sample image and the second sample image are image encoded by the image encoder of the initial diffusion model, and the image encoding representation of the first sample image is obtained as shown in FIG8 e.opt As shown, the image coding representation of the second sample image is obtained as shown in Figure 8e tpt In the first training phase of the initial diffusion model, the reconstruction loss is used to ensure that e tpt As close as possible to e opt , so that the diffusion model trained in the first stage has the ability to recognize the business image of this business.

[0193] Specifically, for each first sample image, the server generates a corresponding sample prompt text according to the image label of the first sample image, the server inputs the first sample image, the sample prompt text and the sample random noise image into the initial diffusion model, the server generates a noisy image according to the first sample image and the sample random noise image through the initial diffusion model, generates a first predicted noise image according to the noisy image and the sample prompt text, denoises the noisy image according to the first predicted noise image to obtain a first predicted image; the server constructs a sample loss according to the difference between the first predicted image and the first sample image, and the server updates the initial diffusion model according to the sample loss; for each second sample image, the server generates a noisy image according to the first predicted image and the sample prompt text, and updates the initial diffusion model according to the sample loss. The corresponding sample prompt text is generated according to the image label of the second sample image, and the second sample image, the sample prompt text and the sample random noise image are input into the updated initial diffusion model. The server generates a noisy image based on the second sample image and the sample random noise image through the updated initial diffusion model, and generates a second predicted noise image based on the noisy image and the sample prompt text. The server denoises the noisy image based on the second predicted noise image to obtain a second predicted image; the server constructs a sample loss based on the difference between the second predicted image and the second sample image, and the server continues to update the initial diffusion model based on the sample loss until the training stop condition is met, thereby obtaining the diffusion model trained in the first stage. The training stop condition can be that the sample damage is less than a preset value, or that the number of iterations reaches a preset number. The training method for each second sample image adopts the training method for the sample image in the above embodiment. Therefore, the specific training process of the second sample image will not be repeated here.

[0194] 3. In the second training stage, the diffusion model trained in the first stage is further trained in the second stage using the general image to obtain the diffusion model trained in the second stage.

[0195] Figure 9 is a diagram of the training process of the second training phase of the initial diffusion image. As shown in Figure 9, in the second training phase, the second sample image derived from the business image is eliminated, and only the first sample image derived from the general image is used to train the diffusion model trained in the first phase. This is to avoid the overfitting caused by the introduction of too many business images, thereby ensuring the generalization of the diffusion model. Generalization refers to the ability of the model to make accurate predictions when applied to new data after training. In the second training phase, the reconstruction loss is used to opt Directly reach the capabilities of the original diffusion model to avoid the need to re-enter the image labels of the business images for alignment each time.

[0196] Specifically, for each general image, the server generates corresponding sample prompt text based on the image label of the general image, and inputs the general image, sample prompt text and sample random noise image into the initial diffusion model. The server generates a noisy image based on the general image and the sample random noise image through the initial diffusion model, and generates a first predicted noise image based on the noisy image and the sample prompt text. The server denoises the noisy image based on the first predicted noise image to obtain a first predicted image; the server constructs a sample loss based on the difference between the first predicted image and the general image, and updates the diffusion model trained in the second stage based on the sample loss.

[0197] 4. The diffusion model trained in the second stage is used as the trained diffusion model.

[0198] In this embodiment, by using first sample images derived from general images and second sample images derived from business images to jointly train the initial diffusion model in the first training phase, the diffusion model trained in the first phase is capable of recognizing business images for the business. In the second training phase, the second sample images derived from business images are eliminated, and only the first sample images derived from general images are used to train the diffusion model trained in the first phase. This is to avoid overfitting caused by the introduction of too many business images, thereby ensuring the generalization of the diffusion model. The diffusion model trained using the above method can avoid the problem of low image label classification accuracy caused by the large domain difference between the training data used by the current diffusion model and the business data of the application scenario, thereby improving the accuracy of image label classification.

[0199] In one embodiment, detailed steps of a method for processing a diffusion model are provided, specifically including the following steps:

[0200] 1. Obtain training samples, where the training samples include general images and business images. Multiple first sample images are derived from general images, and multiple second sample images are derived from business images. An image label set formed by image labels of first sample images in the multiple first sample images is the same as an image label set formed by image labels of second sample images in the multiple second sample images.

[0201] 2. In the first training stage, for each first sample image, a corresponding sample prompt text is generated according to the image label of the first sample image, the first sample image, the sample prompt text and the sample random noise image are input into the initial diffusion model, a noisy image is generated according to the first sample image and the sample random noise image through the initial diffusion model, a first predicted noise image is generated according to the noisy image and the sample prompt text, the noisy image is denoised according to the first predicted noise image to obtain a first predicted image; a sample loss is constructed according to the difference between the first predicted image and the first sample image, and the initial diffusion model is updated according to the sample loss.

[0202] 3. For each second sample image, generate corresponding sample prompt text according to the image label of the second sample image, input the second sample image, sample prompt text and sample random noise image into the updated initial diffusion model, generate a noisy image according to the second sample image and the sample random noise image through the updated initial diffusion model, generate a second predicted noise image according to the noisy image and the sample prompt text, denoise the noisy image according to the second predicted noise image to obtain a second predicted image; construct a sample loss according to the difference between the second predicted image and the second sample image, and continue to update the initial diffusion model according to the sample loss until the training stop condition is met, thereby obtaining the diffusion model trained in the first stage.

[0203] 4. In the second training stage, the second sample images derived from the business images are eliminated, and only the first sample images derived from the general images are used to train the initial diffusion model. For each first sample image, a corresponding sample prompt text is generated according to the image label of the first sample image. The first sample image, the sample prompt text and the sample random noise image are input into the initial diffusion model. A noisy image is generated according to the first sample image and the sample random noise image through the initial diffusion model. A first predicted noise image is generated according to the noisy image and the sample prompt text. The noisy image is denoised according to the first predicted noise image to obtain a first predicted image. A sample loss is constructed according to the difference between the first predicted image and the first sample image. The diffusion model trained in the first stage is updated according to the sample loss. The diffusion model trained in the second stage is used as the trained diffusion model, and the trained diffusion model is used for image classification.

[0204] In this embodiment, by using the first sample image derived from the general image and the second sample image derived from the business image to jointly train the initial diffusion model in the first training stage, the diffusion model trained in the first stage has the ability to recognize the business image of this business; in the second training stage, the second sample image derived from the business image is eliminated, and only the first sample image derived from the general image is used to train the diffusion model trained in the first stage, in order to avoid the introduction of too many business images leading to overfitting, thereby ensuring the generalization of the diffusion model.

[0205] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0206] Based on the same inventive concept, embodiments of the present application also provide an image classification device for implementing the aforementioned image classification method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations in one or more of the following image classification device embodiments can be found in the above-described limitations on the image classification method and will not be further elaborated here.

[0207] In one embodiment, as shown in FIG10 , an image classification apparatus is provided, comprising: an acquisition module 1001 , a noise prediction module 1002 , a determination module 1003 , and a label classification module 1004 , wherein:

[0208] The acquisition module 1001 is used to acquire an original image and a plurality of prompt texts, each prompt text is generated according to an image tag, and each image tag is a preset image classification category.

[0209] The noise prediction module 1002 is used to input the original image, the prompt text and the random noise image into a trained diffusion model for each prompt text, generate a noisy image based on the original image and the random noise image through the diffusion model, generate a predicted noise image based on the noisy image and the prompt text, and calculate the difference between the generated predicted noise image and the random noise image.

[0210] The determination module 1003 is configured to select a predicted noise image with the smallest corresponding difference based on the differences calculated for each predicted noise image, and obtain a prompt text relied upon when generating the selected predicted noise image.

[0211] The label classification module 1004 is configured to use the image label corresponding to the dependent prompt text as the image label of the original image.

[0212] In one embodiment, the noise prediction module 1002 is further used to perform image encoding on the original image through the image encoder of the diffusion model to obtain an image encoding representation of the original image; and to superimpose the noise information corresponding to the random noise image onto the image encoding information through the diffuser of the diffusion model to obtain a noisy image.

[0213] In one embodiment, the noise prediction module 1002 is further used to perform semantic encoding on the prompt text through the text encoder of the diffusion model to obtain a text semantic representation corresponding to the prompt text; input the noisy image and the text semantic representation into the noise predictor of the diffusion model, and output a predicted noise image through the noise predictor.

[0214] In one embodiment, the noise predictor includes multiple alternatingly connected residual networks and attention layers; the noise prediction module 1002 is also used to input the random noise amount encoding information corresponding to the noisy image and the random noise image into the first residual network, and output the predicted noise information through the first residual network; the predicted noise information and the text semantic representation are input into the first attention layer, and the attention information is output through the first attention layer; starting from the second residual network, the next residual network is sequentially used as the current residual network, the next attention layer is used as the current attention layer, the previous attention information output by the previous attention layer connected to the current residual network and the random noise amount encoding information are input into the current residual network, and the predicted noise information is output through the current residual network; the predicted noise information and the text semantic representation output by the current residual network are input into the current attention layer, and the attention information is output through the current attention layer; the attention information output by the last attention layer is used as the predicted noise image.

[0215] In one embodiment, the acquisition module 1001 is further used to acquire a prompt text template and multiple image tags; each of the multiple image tags is filled into the prompt text template to obtain multiple prompt texts corresponding to the corresponding image tags.

[0216] In one embodiment, the acquisition module 1001 is also used to divide the original image to obtain multiple sub-images; the noise prediction module 1002 is also used to obtain prompt text from multiple prompt texts for each sub-image in turn, and for each prompt text obtained, the sub-image, the obtained prompt text and the random noise image are input into a trained diffusion model, and a noisy sub-image is generated according to the sub-image and the random noise image through the diffusion model, and a sub-prediction noise image is generated according to the noisy sub-image and the prompt text, and the difference between the generated sub-prediction noise image and the random noise image is calculated; the determination module 1003 is also used to select the sub-prediction noise image with the smallest corresponding difference based on the difference calculated for each sub-prediction noise image, and obtain the generated selected sub-prediction noise image; the label classification module 1004 is also used to use the image label corresponding to the dependent prompt text as the image label of the sub-image; and obtain the image label of the original image based on the image labels of the multiple sub-images.

[0217] In one embodiment, the apparatus further comprises: a first training module 1005;

[0218] The first training module 1005 is used to obtain training samples, which include sample text, sample images, and annotation information used to indicate whether the sample text and the sample image match; perform image encoding on the sample image through an initial image encoder to obtain an image encoding representation of the sample image; perform semantic encoding on the sample text through an initial text encoder to obtain a text semantic representation corresponding to the sample text; calculate the similarity between the image encoding representation and the text semantic representation, and determine a prediction result on whether the sample text and the sample image match based on the similarity; construct a sample loss based on the difference between the annotation information and the prediction result, update the initial image encoder and the initial text encoder based on the sample loss, and then return to the step of obtaining training samples to continue training, so as to obtain an image encoder of the diffusion model and a text encoder of the diffusion model.

[0219] In one embodiment, the apparatus further includes: a second training module 1006;

[0220] The second training module 1006 is used to obtain training samples, which include general images and business images. The multiple first sample images are all derived from general images, and the multiple second sample images are all derived from business images. The image label set formed by the image labels of the first sample images in the multiple first sample images is the same as the image label set formed by the image labels of the second sample images in the multiple second sample images. In the first training stage, the multiple first sample images and the multiple second sample images are used to perform first-stage model training on the initial diffusion model to obtain a diffusion model trained in the first stage. In the second training stage, the general images are used to continue to perform second-stage model training on the diffusion model trained in the first stage to obtain a diffusion model trained in the second stage. The diffusion model trained in the second stage is used as the trained diffusion model.

[0221] In one embodiment, the second training module 1006 is also used to generate corresponding sample prompt text for each sample image based on the image label of the sample image, generate a noisy image based on the sample image and the sample random noise image through the initial diffusion model, generate a predicted noise image based on the noisy image and the sample prompt text, denoise the noisy image based on the predicted noise image to obtain a predicted image; construct a sample loss based on the difference between the predicted image and the sample image, and update the initial diffusion model based on the sample loss.

[0222] Each module in the above-mentioned image classification device can be implemented in whole or in part through software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor of the computer device in hardware form, or can be stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each of the above modules.

[0223] Based on the same inventive concept, embodiments of the present application also provide a diffusion model processing device for implementing the aforementioned diffusion model processing method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations of the one or more diffusion model processing device embodiments provided below can be found in the aforementioned limitations of the diffusion model method and are not further elaborated here.

[0224] In one embodiment, as shown in FIG11 , a diffusion model processing device is provided, comprising: a sample acquisition module 1101 , a sample training module 1102 , and a model updating module 1103 , wherein:

[0225] The sample acquisition module 1101 is used to acquire multiple sample images, each of which corresponds to an image label;

[0226] The sample training module 1102 is configured to generate, for each sample image, corresponding sample prompt text based on the image label of the sample image, generate a noisy image based on the sample image and the sample random noise image using an initial diffusion model, generate a predicted noise image based on the noisy image and the sample prompt text, and perform denoising on the noisy image based on the predicted noise image to obtain a predicted image;

[0227] The model updating module 1103 is used to construct a sample loss according to the difference between the predicted image and the sample image, and to update the initial diffusion model according to the sample loss. The trained diffusion model obtained after the update is used for image classification.

[0228] In one embodiment, the sample images include a plurality of first sample images and a plurality of second sample images, the first sample images are derived from general images, the second sample images are derived from business images, and an image label set formed by image labels of the first sample images in the plurality of first sample images is the same as an image label set formed by image labels of the second sample images in the plurality of second sample images. The sample training module 1102 is further configured to, in a first training phase, use the plurality of first sample images and the plurality of second sample images to perform a first phase model training on an initial diffusion model to obtain a diffusion model trained in the first phase; and, in a second training phase, use the plurality of first sample images to continue to perform a second phase model training on the diffusion model trained in the first phase to obtain a diffusion model trained in the second phase.

[0229] The model updating module 1103 is further configured to use the diffusion model trained in the second stage as the trained diffusion model.

[0230] In one embodiment, the sample training module 1102 is further configured to generate, for each first sample image, a corresponding sample prompt text according to the image label of the first sample image, input the first sample image, the sample prompt text, and the sample random noise image into the initial diffusion model, generate a noisy image according to the first sample image and the sample random noise image through the initial diffusion model, generate a first predicted noise image according to the noisy image and the sample prompt text, perform denoising on the noisy image according to the first predicted noise image, and obtain a first predicted image; construct a sample loss according to the difference between the first predicted image and the first sample image, and update the initial diffusion model according to the sample loss; for each second sample image, generate a first predicted noise image according to the initial diffusion model, and generate a first predicted noise image according to the initial diffusion model; For this image, a corresponding sample prompt text is generated according to the image label of the second sample image, the second sample image, the sample prompt text and the sample random noise image are input into the updated initial diffusion model, a noisy image is generated according to the second sample image and the sample random noise image through the updated initial diffusion model, a second predicted noise image is generated according to the noisy image and the sample prompt text, the noisy image is denoised according to the second predicted noise image to obtain a second predicted image; a sample loss is constructed according to the difference between the second predicted image and the second sample image, and the initial diffusion model is continuously updated according to the sample loss until the training stop condition is met, thereby obtaining the diffusion model trained in the first stage.

[0231] Each module in the aforementioned diffusion model processing device may be implemented in whole or in part through software, hardware, or a combination thereof. Each module may be embedded in or independent of a processor within a computer device in the form of hardware, or may be stored in a computer device memory in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0232] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be shown in Figure 12. The computer device includes a processor, a memory, an input / output interface (I / O), and a communication interface. The processor, memory, and I / O interface are connected via a system bus, and the communication interface is connected to the system bus via the I / O interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, computer-readable instructions, and a database. The internal memory provides an environment for the operation of the operating system and computer-readable instructions in the non-volatile storage medium. The database of the computer device is used to store prompt text. The I / O interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with an external terminal via a network connection. When the computer-readable instructions are executed by the processor, an image classification method is implemented.

[0233] Those skilled in the art will understand that the structure shown in FIG12 is merely a block diagram of a portion of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0234] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores computer-readable instructions, and the processor implements the steps in the above method embodiment when executing the computer-readable instructions.

[0235] In one embodiment, a computer-readable storage medium is provided, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor, the steps in the above method embodiment are implemented.

[0236] In one embodiment, a computer program product is provided, comprising computer-readable instructions, which implement the steps of the above method embodiment when executed by a processor.

[0237] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions.

[0238] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through computer-readable instructions. The computer-readable instructions can be stored in a non-volatile computer-readable storage medium. When the computer-readable instructions are executed, they can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, and the like.

[0239] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0240] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. An image classification method, performed by a computer device, comprising: Obtain an original image and multiple prompt texts, each prompt text is generated according to an image label, and each image label is a preset image classification category; For each prompt text, the original image, the prompt text and the random noise image are input into a trained diffusion model, a noisy image is generated according to the original image and the random noise image by the diffusion model, a predicted noise image is generated according to the noisy image and the prompt text, and a difference between the generated predicted noise image and the random noise image is calculated; Selecting a predicted noise image with the smallest corresponding difference according to the difference calculated for each predicted noise image, and obtaining a prompt text relied upon when generating the selected predicted noise image; and The image label corresponding to the dependent prompt text is used as the image label of the original image.

2. The method according to claim 1, wherein the inputting the original image, the prompt text and the random noise image into a trained diffusion model, and generating a noisy image according to the original image and the random noise image through the diffusion model, comprises: Performing image encoding on the original image by using the image encoder of the diffusion model to obtain an image encoding representation of the original image; and The noise information corresponding to the random noise image is superimposed on the image coding information through the diffuser of the diffusion model to obtain the noisy image.

3. The method according to claim 2, wherein generating a predicted noise image according to the noisy image and the prompt text comprises: The prompt text is semantically encoded by the text encoder of the diffusion model to obtain a text semantic representation corresponding to the prompt text; and The noisy image and the text semantic representation are input into the noise predictor of the diffusion model, and the noise predictor outputs a predicted noise image.

4. The method according to claim 3, wherein the noise predictor comprises a plurality of alternately connected residual networks and attention layers; wherein the noisy image and the text semantic representation are input into the noise predictor of the diffusion model, and the predicted noise image is outputted by the noise predictor, comprising: Inputting the random noise amount encoding information corresponding to the noisy image and the random noise image into a first residual network, and outputting the predicted noise information through the first residual network; inputting the predicted noise information and the text semantic representation into a first attention layer, and outputting the attention information through the first attention layer; and Starting from the second residual network, the next residual network is sequentially used as the current residual network, the next attention layer is used as the current attention layer, the previous attention information output by the previous attention layer connected to the current residual network and the random noise amount encoding information are input into the current residual network, and the predicted noise information is output through the current residual network; Inputting the predicted noise information output by the current residual network and the text semantic representation into the current attention layer, and outputting the attention information through the current attention layer; and The attention information output by the last attention layer is used as the predicted noise image.

5. The method according to claim 3, further comprising: Acquire a training sample, wherein the training sample includes a sample text, a sample image, and annotation information indicating whether the sample text and the sample image match; Performing image encoding on the sample image by an initial image encoder to obtain an image encoding representation of the sample image; Performing semantic encoding on the sample text by an initial text encoder to obtain a text semantic representation corresponding to the sample text; Calculating the similarity between the image coding representation and the text semantic representation, and determining a prediction result as to whether the sample text matches the sample image according to the similarity; and According to the difference between the annotation information and the prediction result, a sample loss is constructed. After the initial image encoder and the initial text encoder are updated according to the sample loss, the step of obtaining training samples is returned to continue training to obtain the image encoder of the diffusion model and the text encoder of the diffusion model.

6. The method according to any one of claims 1 to 5, wherein obtaining a plurality of prompt texts comprises: Get the prompt text template and multiple image tags; and Each of the multiple image tags is filled into the prompt text template to obtain multiple prompt texts corresponding to the corresponding image tags.

7. The method according to any one of claims 1 to 6, further comprising: Dividing the original image to obtain multiple sub-images; and For each prompt text, the original image, the prompt text and the random noise image are input into a trained diffusion model, a noisy image is generated according to the original image and the random noise image through the diffusion model, a predicted noise image is generated according to the noisy image and the prompt text, a difference between the generated predicted noise image and the random noise image is calculated, and according to the calculated difference of each predicted noise image, a predicted noise image with the smallest corresponding difference is selected, and the prompt text relied on when generating the selected predicted noise image is obtained; Using the image label corresponding to the dependent prompt text as the image label of the original image includes: For each sub-image, the following steps are respectively performed: obtaining a prompt text from the multiple prompt texts in turn, for each obtained prompt text, inputting the sub-image, the obtained prompt text and the random noise image into a trained diffusion model, generating a noisy sub-image according to the sub-image and the random noise image through the diffusion model, generating a sub-prediction noise image according to the noisy sub-image and the prompt text, and calculating the difference between the generated sub-prediction noise image and the random noise image; selecting the sub-prediction noise image with the smallest corresponding difference according to the calculated difference of each sub-prediction noise image, and obtaining the prompt text relied on when generating the selected sub-prediction noise image; using the image label corresponding to the relied on prompt text as the image label of the sub-image; The image label of the original image is obtained according to the image labels of each of the multiple sub-images.

8. The method according to any one of claims 1 to 7, wherein the step of training the diffusion model comprises: Acquire training samples, the training samples comprising general images and business images, a plurality of first sample images all originate from general images, a plurality of second sample images all originate from business images, and an image label set formed by image labels of first sample images in the plurality of first sample images is the same as an image label set formed by image labels of second sample images in the plurality of second sample images; In a first training stage, the plurality of first sample images and the plurality of second sample images are used to perform a first stage model training on the initial diffusion model to obtain a diffusion model trained in the first stage; In the second training stage, the general image is used to continue the model training of the second stage for the diffusion model trained in the first stage, so as to obtain the diffusion model trained in the second stage; The diffusion model trained in the second stage is used as the trained diffusion model.

9. The method according to claim 8, wherein the step of training the diffusion model in each training stage comprises: For each sample image, a corresponding sample prompt text is generated according to the image label of the sample image, a noisy image is generated according to the sample image and the sample random noise image through an initial diffusion model, a predicted noise image is generated according to the noisy image and the sample prompt text, and a denoising process is performed on the noisy image according to the predicted noise image to obtain a predicted image; A sample loss is constructed according to the difference between the predicted image and the sample image, and the initial diffusion model is updated according to the sample loss.

10. A method for processing a diffusion model, executed by a computer device, the method comprising: Obtain multiple sample images, each sample image corresponds to an image label; For each of the sample images, a corresponding sample prompt text is generated according to the image label of the sample image, a noisy image is generated according to the sample image and the sample random noise image through an initial diffusion model, a predicted noise image is generated according to the noisy image and the sample prompt text, and a denoising process is performed on the noisy image according to the predicted noise image to obtain a predicted image; and According to the difference between the predicted image and the sample image, a sample loss is constructed, and the initial diffusion model is updated according to the sample loss. The trained diffusion model obtained after the update is used for image classification.

11. An image classification device, comprising: An acquisition module is used to acquire an original image and multiple prompt texts, each prompt text is generated according to an image label, and each image label is a preset image classification category; a noise prediction module, for each prompt text, inputting the original image, the prompt text and the random noise image into a trained diffusion model, generating a noisy image according to the original image and the random noise image through the diffusion model, generating a predicted noise image according to the noisy image and the prompt text, and calculating the difference between the generated predicted noise image and the random noise image; A determination module, used for selecting a predicted noise image with the smallest corresponding difference according to the difference calculated for each predicted noise image, and obtaining a prompt text relied upon when generating the selected predicted noise image; The label classification module is used to use the image label corresponding to the dependent prompt text as the image label of the original image.

12. A device for processing a diffusion model, the device comprising: A sample acquisition module is used to acquire multiple sample images, each of which has an image label. A sample training module is used to generate a corresponding sample prompt text for each sample image according to the image label of the sample image, generate a noisy image according to the sample image and the sample random noise image through an initial diffusion model, generate a predicted noise image according to the noisy image and the sample prompt text, and perform denoising on the noisy image according to the predicted noise image to obtain a predicted image; and The model updating module is used to construct a sample loss according to the difference between the predicted image and the sample image, and to update the initial diffusion model according to the sample loss. The trained diffusion model obtained after the update is used for image classification.

13. A computer device, comprising a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the method according to any one of claims 1 to 10 when executing the computer-readable instructions.

14. A computer-readable storage medium having computer-readable instructions stored thereon, wherein the computer-readable instructions, when executed by a processor, implement the steps of the method according to any one of claims 1 to 10.

15. A computer program product comprising computer readable instructions, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 10.

Images