Image classification method and device based on hidden space purification of multi-modal model CLIP

Abstract

The application provides an image classification method and device based on a multi-modal model CLIP hidden space purification, comprising: inputting a picture to be classified into an image encoder of a graph-text multi-modal contrast pre-training model to obtain a representation vector of the picture to be classified on a hidden space; using a text encoder of the graph-text multi-modal contrast pre-training model to obtain a null category text embedding vector represented by a null category text; calculating a probability likelihood value of the representation vector and the null category text embedding vector; extracting a vector length and a direction vector of the representation vector, calculating a gradient of the likelihood probability value on the direction vector, updating the direction vector by using a gradient ascent method while keeping the vector length unchanged to obtain a purified vector; and selecting a category text embedding vector with the highest similarity to the final purified vector, taking an image category corresponding to the category text embedding vector as a classification result of the category text embedding vector.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of zero-shot image classification task adversarial defense, and in particular to a method and device for realizing adversarial robust zero-shot image classification by purifying samples in a multi-modal latent space. BACKGROUND

[0002] Traditional image classifiers are usually supervised trained using specific class training data, and zero-shot image classifiers can perform image classification without seeing specific class training samples. The CLIP (Contrastive Language-Image Pre-training) model is a visual language pre-training model that can effectively complete the zero-shot image classification task by aligning the latent space information of the text and image two modalities.

[0003] In the image classification task, adversarial attack refers to the behavior of making the model make wrong predictions by applying a tiny perturbation that is difficult to detect with the naked eye to the input data. Adversarial robustness refers to the ability of a deep neural network to make correct predictions in the face of carefully designed adversarial attacks. Existing defense methods for improving adversarial robustness mainly include adversarial training and data purification: adversarial training enhances the model's ability to defend against the same known attacks by introducing adversarial samples during training; data purification eliminates adversarial perturbations by optimizing adversarial samples to make them closer to the distribution of clean samples, thereby enhancing the model's ability to defend against various adversarial attacks.

[0004] Although the existing CLIP model has superior classification accuracy on clean samples, the classification accuracy on adversarial samples may drop to zero. Existing adversarial training methods can enhance the model's defense capability against known attacks, but the defense effect against unknown attacks is not good. For example, two existing adversarial training methods integrated with the CLIP model, FARE and TeCoA methods, can enhance the robustness of zero-shot classification, but significantly damage the zero-shot classification accuracy of the model on clean samples, and are difficult to generalize to defense against unknown adversarial attacks. Existing data purification methods usually need a generative model to simulate the distribution of clean samples, which can defend against unknown adversarial attacks, but has low inference efficiency. In addition, existing data purification methods are usually performed in the pixel space of a single visual modality, but the high dimensionality and sparsity of the pixel space may produce a peak gradient distribution, resulting in higher purification risk. At the same time, theoretically, there is a gap in quality between single-modal and multi-modal latent space representations, which together limits the effectiveness of data purification methods. SUMMARY

[0005] The purpose of the present application is to solve the problem of poor defense effect and low efficiency of the existing data purification method. To this end, the present application proposes a novel method of purifying adversarial sample data using multi-modal information in latent space, thereby realizing robust zero-shot classification. This method aligns the multi-modal information of clean samples in the latent space through the CLIP model to guide the purification of adversarial samples, and introduces a discriminative model instead of a generative model, significantly improving the adversarial robustness and reasoning efficiency of the model for zero-shot classification. In addition, the present application also provides a complete system implementation scheme, including the detailed steps of data processing, model training and inference process, to ensure the effectiveness and practicability of the method. For example, when a neural network classification model is faced with an adversarial attack, the attacker adds a human invisible perturbation to an image of a "panda", and the neural network will incorrectly classify it as a "gibbon" category. The present application proposes a method of purifying adversarial samples using multi-modal information in latent space, which eliminates the influence of adversarial perturbation in the input sample latent space, so that the model can correctly classify its category as "panda".

[0006] In view of the deficiencies of the prior art, as shown in Figure 4 The present application proposes an image classification method based on latent space purification of multi-modal model CLIP, which includes:

[0007] In the initial step, the picture to be classified is input into the image encoder of the graph-text multi-modal contrast pre-training model to obtain the representation vector of the picture to be classified in the latent space; the text encoder of the graph-text multi-modal contrast pre-training model is used to obtain the empty class text embedding vector represented by the empty class text.

[0008] In the calculation step, the probability likelihood value of the representation vector and the empty class text embedding vector is calculated.

[0009] In the iteration step, the vector length and direction vector of the representation vector are extracted, the gradient of the likelihood probability value with respect to the direction vector is calculated, and the gradient ascent method is used to update the direction vector while keeping the vector length unchanged to obtain the purified vector; the calculation step and the iteration step are executed again until the probability likelihood value converges, and the current purified vector is saved as the final vector.

[0010] In the classification step, the graph-text multi-modal contrast pre-training model selects the class text embedding vector with the highest similarity to the final vector, and takes the image class corresponding to the class text embedding vector as the classification result of the class text embedding vector.

[0011] The image classification method based on latent space purification of multi-modal model CLIP, wherein the calculation step estimates the probability likelihood value of the representation vector using a text-to-image diffusion model; the calculation step includes:

[0012] The text-to-image diffusion model takes the empty category text embedding vector as a prior condition, calculates a noise prediction value of the representation vector in a conditional diffusion inverse process, and estimates the likelihood probability value of the representation vector based on the noise prediction value.

[0013] The image classification method based on the CLIP hidden space purification of a multi-modal model, wherein the calculation step estimates the probability likelihood value of the representation vector by using the cosine similarity between the representation vector and the empty category embedding vector; and the calculation step includes:

[0014] The cosine similarity between the space representation and the empty category text embedding vector is calculated, and the cosine similarity is taken as the likelihood probability value.

[0015] The image classification method based on the CLIP hidden space purification of a multi-modal model, wherein the iteration step includes obtaining the gradient based on the chain rule of composite function derivation according to the vector length and the direction vector of the representation vector.

[0016] As shown in Figure 5 The application further provides an image classification device based on the CLIP hidden space purification of a multi-modal model, which comprises:

[0017] An initial module is configured to input a picture to be classified into an image encoder of a graph-text multi-modal contrast pre-training model to obtain a representation vector of the picture to be classified in a hidden space, and use a text encoder of the graph-text multi-modal contrast pre-training model to obtain an empty category text embedding vector represented by empty category text.

[0018] A calculation module is configured to calculate a probability likelihood value of the representation vector and the empty category text embedding vector.

[0019] An iteration module is configured to extract a vector length and a direction vector of the representation vector, calculate a gradient of the likelihood probability value with respect to the direction vector, update the direction vector by using a gradient ascent method while keeping the vector length unchanged to obtain a purified vector, execute the calculation step and the iteration step again until the probability likelihood value converges, and save the current purified vector as a final vector.

[0020] A classification module is configured to select, by the graph-text multi-modal contrast pre-training model, a category text embedding vector with the highest similarity to the final vector, and take an image category corresponding to the category text embedding vector as a classification result of the category text embedding vector.

[0021] The image classification device based on the CLIP hidden space purification of a multi-modal model, wherein the calculation module estimates the probability likelihood value of the representation vector by using a text-to-image diffusion model; and the calculation module includes:

[0022] The text-based diffusion model takes the empty category text embedding vector as a prior condition, calculates the noise prediction value of the representation vector in the conditional diffusion inverse process, and estimates the likelihood probability value of the representation vector based on the noise prediction value.

[0023] The image classification device based on the CLIP hidden space purification method of the multi-modal model, wherein the calculation module estimates the probability likelihood value of the representation vector by using the cosine similarity between the representation vector and the empty category embedding vector; the calculation module comprises:

[0024] The cosine similarity between the space representation and the empty category text embedding vector is calculated, and the cosine similarity is used as the likelihood probability value.

[0025] The application further provides an electronic device comprising the image classification device based on the CLIP hidden space purification method of the multi-modal model, and the electronic device is connected with an information display device for displaying the classification result by using display parameters, attributes set by a user or an artificial intelligence model.

[0026] The application further provides a computer readable storage medium having a computer program stored thereon, and the computer program is executed by a processor to realize the steps of the image classification method based on the CLIP hidden space purification method of the multi-modal model.

[0027] The application further provides a computer program product comprising a computer program, and the computer program is executed by a processor to realize the steps of the image classification method based on the CLIP hidden space purification method of the multi-modal model.

[0028] From the above scheme, the application has the following advantages:

[0029] The CLIP hidden space purification method based on the multi-modal model provides a novel data purification method by utilizing multi-modal information in the hidden space, thereby significantly improving the adversarial robustness and reasoning efficiency of zero-shot classification of the model. Compared with the prior art, the method not only overcomes the limitations of single-modal information representation, but also avoids the purification risks brought by high-dimensional and sparse pixel space, significantly improves the adversarial robustness of the model under various attack behaviors, and greatly improves the reasoning efficiency by using a discriminative model instead of a traditional generative model, and exhibits excellent performance in complex attack environments and efficiency-sensitive tasks.

[0030] The models included in the comparative evaluation of the technical effects of the present application include CLIP, a pre-trained model for image-text multimodal contrast, LM-EDM, a maximum likelihood data purification method based on Euler discretization momentum diffusion model, DiffPure, an adversarial sample data purification method based on diffusion model, and the proposed CLIPure, a hidden space purification method based on multimodal model CLIP, CLIPure-Diff, a generative variant based on diffusion model, and CLIPure-Cos, a discriminative variant based on cosine similarity.

[0031] Table 1: Performance test results of the hidden space purification method based on multimodal model CLIP on the CIFAR-10 dataset

[0032]

[0033] The clean sample accuracy of each model without any attack and the adversarial robustness under the listed attack behaviors are shown in Tables 1 and 2 to reflect the defense capabilities of the models in different attack methods. The average inference time test results of each model on the CIFAR10 dataset for image classification of 100 samples are shown in Table 3 to reflect the inference efficiency of different types of adversarial defense models.

[0034] Table 1 shows a series of adversarial test results on the CIFAR-10 dataset using AutoAttack (AA), which evaluates the robustness of different models when facing advanced adversarial attacks. AutoAttack is an automated adversarial attack tool that integrates multiple attack methods to comprehensively test the defense of the model. The attack methods listed in Table 1 include:

[0035] AA Linf (perturbation range of 8 / 255): AutoAttack using L-infinity norm, which limits the maximum allowed perturbation of each pixel to no more than 8 / 255, testing the sensitivity of the model to extreme pixel perturbations;

[0036] AA L2 (perturbation range of 0.5): AutoAttack using L2 norm, which evaluates the sensitivity to overall picture perturbations (Euclidean distance of all pixel perturbations does not exceed 0.5).

[0037] Table 2: Performance test results of the hidden space purification method based on multimodal model CLIP on the ImageNet dataset

[0038]

[0039] Table 2 demonstrates a series of adversarial testing results using AutoAttack (AA) on the ImageNet dataset, which evaluate the robustness of different models when facing advanced adversarial attacks. AutoAttack is an automated adversarial attack tool that integrates multiple attack methods to comprehensively test the defense of models. The attack methods listed in Table 2 include:

[0040] AA Linf (perturbation range of 4 / 255): AutoAttack using L-infinity norm, which limits the maximum allowed perturbation of each pixel to no more than 8 / 255, testing the sensitivity of the model to extreme pixel perturbations.

[0041] These results in Table 1 and Table 2 clearly show the potential of the hidden space purification method based on the multi-modal model CLIP to improve the adversarial robustness of the model, especially how it helps the model to maintain high clean sample accuracy while showing excellent adversarial robustness when facing diversified adversarial attacks. This comprehensive evaluation highlights the defense performance of the hidden space purification method based on the multi-modal model CLIP when facing unseen attack methods, and it is worth mentioning that since the hidden space purification method based on the multi-modal model CLIP does not use any adversarial samples during training, any attack method is unknown to the model during testing.

[0042] Table 3: Inference efficiency test results of the model on the CIFAR10 dataset for image classification of 100 samples

[0043]

[0044] Table 3 shows the inference efficiency of a series of different categories of adversarial defense models on the CIFAR10 dataset. The experiment was run on a single 4090 graphics card, measuring the average inference time of each model for image classification on 100 samples in the CIFAR-10 dataset, and taking the inference time of the image-text multimodal contrast pre-training model (CLIP) without using adversarial defense measures as the benchmark, comparing the inference efficiency of each adversarial defense model. The results in Table 3 clearly show that the two variants of the CLIPure method based on the multimodal model CLIP proposed in the present application, the generative variant based on the diffusion model (CLIPure-Diff) and the discriminative variant based on the cosine similarity (CLIPure-Cos), significantly improve the inference efficiency compared to existing data purification methods based on generative models, thus having excellent advantages in efficient and robust inference. Among them, the discriminative variant based on cosine similarity (CLIPure-Cos) further innovatively uses discriminative methods for adversarial sample purification, achieving inference time comparable to discriminative model baselines, making this variant method perform more outstandingly in efficiency-sensitive tasks. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 Flowchart of the zero-shot robust classification inference process based on the multimodal model CLIP hidden space data purification of the present application;

[0046] Figure 2 Flowchart of the method for estimating image representation probability likelihood values in the hidden space based on a diffusion model;

[0047] Figure 3 Flowchart of the method for estimating image representation probability likelihood values in the hidden space based on a discriminative model;

[0048] Figure 4 Flowchart of the overall method of the present application;

[0049] Figure 5 Module diagram of the device of the present application;

[0050] Figure 6 Structure schematic diagram of the first electronic device of the present application;

[0051] Figure 7 Structure schematic diagram of the application environment of the first electronic device of the present application;

[0052] Figure 8 Structure schematic diagram of the second electronic device of the present application.

[0053] Reference signs:

[0054] A - first electronic device;

[0055] B - Image classification apparatus based on hidden space purification of a multi-modal model CLIP

[0056] C - Data acquisition device

[0057] D - Information display device

[0058] 1000 - Second electronic device

[0059] I - Computing unit

[0060] II - ROM

[0061] III - RAM

[0062] IV - Bus

[0063] V - Interface

[0064] VI - Input unit

[0065] VII - Output unit

[0066] VIII - Storage medium

[0067] IX - Communication unit DETAILED DESCRIPTION

[0068] It should be noted that the relational terms herein, such as first and second, and the like, are used solely to distinguish one from another entity or action without necessarily requiring or implying any actual relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus.

[0069] Without further limitation, an element defined by the phrase "comprising a" does not exclude the presence of additional identical elements in the process, method, article, or apparatus including the element.

[0070] The processor of the present application is the control center of the electronic device, which can be one processor or a collective term of multiple processing elements. For example, it is one or more central processing units (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present application, such as one or more digital signal processors (DSP), or one or more field programmable gate arrays (FPGA).

[0071] Optionally, the processor can execute various functions of the electronic device by running or executing software programs stored in the memory and calling data stored in the memory.

[0072] In a specific implementation, as an embodiment, the processor can include one or more CPUs. Each of these processors can be a single-CPU or a multi-CPU. The processor here can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions). The electronic device can include a server, a desktop computer, a notebook computer, a smartphone, a tablet computer, an embedded computer, etc., wherein the embedded computer includes vehicles and robots, etc.

[0073] The memory is used to store software programs for implementing the solutions of the present application, and is controlled by the processor to execute. The specific implementation can refer to the above method embodiments, which will not be repeated here.

[0074] It should be noted that the structure of the electronic device shown in the drawings of the present application does not constitute a limitation, and the actual knowledge structure recognition device can include more or fewer components than shown, or combine certain components, or different component arrangements.

[0075] The above-described embodiments can be implemented in whole or in part by software, hardware (e.g., circuitry), firmware, or any combination thereof. When implemented in software, the above-described embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are wholly or partially generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center through wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible by a computer or a data storage device such as a server, data center, etc. containing one or more available medium collections. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. The semiconductor medium can be a solid-state disk.

[0076] It should also be understood that the term "and / or" used herein is merely an association relationship between the associated objects, which means that there can be three relationships, for example, A and / or B can represent the following three cases: A exists alone, A and B exist together, and B exists alone, where A and B can be singular or plural. In addition, the character " / " herein generally represents an "or" relationship between the associated objects before and after it, but it can also represent an "and / or" relationship, which can be understood in the context before and after it.

[0077] In the present application, "at least one" means one or more, and "multiple" means two or more. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c can represent a, b, c, a-b, a-c, b-c, or a-b-c, where a, b, and c can be single or multiple.

[0078] It should also be understood that in various embodiments of the present application, the size of the sequence number of the above-described processes does not mean the order of execution, and the execution order of the processes should be determined by their functions and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0079] In several embodiments provided by the present application, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other manners. For example, the embodiments of the apparatus described above are merely schematic. For example, the division of the units is only a logical function division. There can be another division manner for the actual implementation. For example, a plurality of units or components can be combined or integrated into another device, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections can be indirect couplings or communication connections through some interfaces, devices or units, and can be in electrical, mechanical or other forms.

[0080] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., can be located in one place, or can be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiments.

[0081] In addition, each functional unit in the various embodiments of the present application can be integrated into a processing unit, or each unit can be a physically independent unit, or two or more units can be integrated into a unit.

[0082] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the parts of the technical solutions that essentially contribute to the prior art can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0083] During the research, the inventors found that although existing methods for enhancing adversarial robustness, such as purification methods, can deal with unknown attacks, they are less effective than adversarial training in defending against known attacks and have lower inference efficiency. Through further research and theoretical analysis, the inventors found that the reason for the poor defense effect is that the high dimensionality and sparsity of the pixel space lead to a high purification risk, and the low quality of single-modal information representation makes it difficult to fully cope with diverse adversarial perturbations. Purification risk refers to the difference between the purified sample and the original clean sample. The most ideal purification is to perfectly restore the original clean sample after purification, and the difference between the clean sample and the purified sample is the purification risk. The inventors deduced that using multi-modal information for purification in the more dense and smooth latent space can effectively reduce the purification risk and improve the adversarial robustness. The low inference efficiency is because existing purification methods usually rely on generative models (such as diffusion models), which involve a large number of complex inference processes.

[0084] To solve these problems, the inventors propose a latent space purification method based on a multi-modal model CLIP (CLIPure), which is a method for robust zero-shot classification by using multi-modal information for adversarial sample data purification in the latent space. This method uses the CLIP model to align the multi-modal information of clean samples in the latent space, guiding the purification process of adversarial samples, thereby enhancing the adversarial robustness of the model. In addition, the inventors also propose a variant that uses a discriminative model instead of a generative model for purification, greatly improving the inference efficiency and making the method perform better in efficiency-sensitive tasks.

[0085] Specifically, the inventors propose the following key technical points:

[0086] Key point 1: A framework for improving the adversarial robustness of zero-shot image classification tasks by using multi-modal information for data purification in the latent space. Use an existing CLIP image encoder to obtain the latent space representation vector of the input image; estimate the likelihood probability of the latent space representation vector; normalize the latent space representation vector to a unit vector to eliminate the influence of the modulus; use the chain rule and gradient ascent method to optimize the direction of the latent space representation vector; and use an existing CLIP model to perform zero-shot classification based on the purified latent space representation vector to obtain the predicted class.

[0087] This framework enables the model to maintain high classification accuracy and robustness while maintaining excellent inference efficiency when performing zero-shot image classification tasks in an adversarial environment, even in the face of complex and diverse attack forms.

[0088] Key point 2: A method for estimating the probability likelihood value of an image representation in the latent space based on a diffusion model.

[0089] The empty category text embedding vector represented by the text template "a picture of a < > category" is obtained using the existing CLIP text encoder, and the latent space representation vector of the input image is obtained; the noise prediction value of the conditional diffusion inverse process of the input image latent space representation vector is calculated under the condition of the empty category text embedding vector using the prior diffusion module (DiffusionPrior) in the text-to-image diffusion model (DaLLE-2) pre-trained based on image-text contrast; and the likelihood probability of the input image latent space representation vector is estimated according to the noise prediction value of the conditional diffusion inverse process.

[0090] Key point 3: A method for estimating the probability likelihood value of an image representation in the latent space based on a discriminative model. The empty category text embedding vector represented by the text template "a picture of a < > category" is obtained using the existing CLIP text encoder, and the latent space representation vector of the input image is obtained; the cosine similarity between the input image latent space representation vector and the empty category text embedding vector is calculated; and the likelihood probability of the input image latent space representation vector is estimated according to the cosine similarity.

[0091] To make the above features and effects of the present application more clear and easy to understand, the following embodiments are described in detail below, and the drawings are described as follows. The present application discloses one or more embodiments containing the features of the present application. The disclosed embodiments are only for illustration. The protection scope of the present application is not limited to the disclosed embodiments, and the present application is defined by the appended claims.

[0092] Figure 1 A flowchart of a zero-shot robust classification inference process based on a latent space purification method (CLIPure) based on a multi-modal model CLIP is shown, which can purify the representation of an adversarial sample image in the latent space with the help of multi-modal information, thereby realizing robust zero-shot image classification. The following are the detailed steps of the process:

[0093] Step 1: encode the latent space representation vector of the input adversarial sample image (S101)

[0094] The trained image-text multi-modal contrast pre-training model (CLIP) includes an image encoder and a text encoder, and this step does not require additional training. The image encoder of the image-text multi-modal contrast pre-training model (CLIP) is directly used to encode the input adversarial sample image into a representation vector in the latent space. The image-text multi-modal contrast pre-training model (CLIP) as a visual language alignment encoder model has shown its superiority in many tasks, and the present application utilizes the multi-modal information aligned in the latent space to realize the adversarial sample data purification process.

[0095] Step 2: determine whether to use a generative model (S102)

[0096] The method of the present application includes two versions, one using a generative model and the other not requiring a generative model. Users can choose flexibly whether to use a generative model to calculate the likelihood probability of the image latent space representation vector in the data purification process. If yes, go to step 3, otherwise go to step 4. Since the generative model requires a more complex inference process, it will reduce the inference efficiency to some extent. The calculation of the likelihood function is a method already available in the diffusion model. The innovation of the present application lies in using the null category text to calculate the likelihood function.

[0097] Step 3: Estimate the likelihood probability of the latent space representation vector based on the prior diffusion module (S103)

[0098] If the generative model is selected in data purification, the diffusion model is used to estimate the probability likelihood value of the image representation in the latent space. This likelihood function value can be represented as a function of the input image latent space representation vector and the null category text embedding vector, for example, by using the cosine similarity of the image latent space vector and the null category text vector as the likelihood value. The detailed calculation process of the estimation method of this part will be introduced in the flow Figure 2 .

[0099] Step 4: Estimate the likelihood probability of the latent space representation vector based on the cosine similarity (S104)

[0100] If the generative model is not selected in data purification, the discriminative model is directly used to estimate the probability likelihood value of the image representation in the latent space. This likelihood function value can be represented as a function of the input image latent space representation vector and the null category text embedding vector. The probability improves the inference efficiency while ensuring that the accuracy does not decrease. The detailed calculation process of the estimation method of this part will be introduced in the flow Figure 3 . Directly using the discriminative model to attack the probability likelihood estimation can significantly improve the inference efficiency.

[0101] Step 5: Obtain the length and direction of the latent space representation vector respectively (S105)

[0102] The latent space representation vector is normalized to a unit vector to represent the vector direction, and the unit vector is obtained by dividing the vector by its vector length (i.e., the modulus of the vector), and the length of the vector is a unit length, which is called a unit vector. The unit vector eliminates the influence of the vector length and only retains the vector direction information, so that the influence of the modulus length can be eliminated in the subsequent purification process. This is because the image-text multimodal contrast pre-training model (CLIP) aligns the image-text multimodal information through cosine similarity, and the modulus length is not important to reflect the relationship between vectors. Therefore, when performing maximum likelihood optimization in the latent space purification method (CLIPure) based on the multi-modal model CLIP, the latent space representation vector needs to be normalized to eliminate the influence of the modulus length. This is crucial to the method proposed in the present application, because our experiments show that the data purification process can be hindered by changes in the modulus length of the latent space representation vector and even ultimately fail.

[0103] Step 6: Optimize the latent space representation vector by maximizing the likelihood probability (S106)

[0104] Data purification eliminates adversarial perturbations by optimizing adversarial samples to approach the clean sample distribution, and the latent space representation vector is optimized to maximize the likelihood probability. The likelihood probability value of the image latent space representation vector output in step 3 or step 4 can be represented as a function of the input image latent space representation vector, and the image latent space representation vector is decoupled into a modulus length scalar and a direction vector after normalization in step 5. In this step 6, according to the chain rule of composite function derivation, the gradient of the likelihood probability value with respect to the direction vector of the latent space representation vector (the direction of the latent space representation vector) can be calculated, and then the gradient ascent method is used to update the direction vector of the latent space representation vector, thereby realizing data purification in the latent space.

[0105] Step 7: Repeat the purification process until convergence (S107)

[0106] Repeat the vector optimization process of steps 2 to 6 until the vector likelihood probability converges. To obtain the purified vector, the modulus length of the original vector (which does not change during the purification process) needs to be multiplied by the purified unit vector (which represents the direction of the vector) to obtain the purified vector.

[0107] Step 8: Perform zero-shot classification using the purified latent space representation vector (S108)

[0108] The purified latent space representation vector is input into the image-text multimodal contrast pre-training model (CLIP) for zero-shot classification to obtain the predicted category. In this process, the ready-made image-text multimodal contrast pre-training model (CLIP) completes the zero-shot classification task without seeing specific category training data by selecting the category text embedding vector with the highest similarity to the image latent space representation vector. The method of the present application purifies the image representation vector in the latent space to improve the robustness of the model.

[0109] Figure 2 A method flowchart for estimating the image representation probability likelihood value in the latent space using the diffusion generation model is shown. This flow accepts the latent space representation vector of the input space as input, estimates the likelihood probability of the latent space representation vector using the prior diffusion module (DiffusionPrior), and outputs it as the content of step 3 (S103) described in the above. Figure 1 The detailed steps of this flow are as follows:

[0110] Step 1: Encode the embedding vector of the empty category text template (S201)

[0111] The text template "a picture of a < > category" is directly encoded into an empty category text embedding vector using the ready-made text encoder of the image-text multimodal contrast pre-training model (CLIP). The trained image-text multimodal contrast pre-training model (CLIP) includes an image encoder and a text encoder, and this step does not require additional training. The empty category text embedding vector obtained by encoding will be used as a prior condition for the diffusion process in the subsequent steps.

[0112] Step 2: Use the prior diffusion module to calculate the noise prediction value of the conditional diffusion inverse process of the input image latent space representation vector (S202)

[0113] The prior diffusion module (DiffusionPrior) in the text-to-image diffusion model (DaLLE-2) based on image-text contrast pre-training is used to calculate the noise prediction value of the conditional diffusion inverse process of the input image latent space representation vector given the empty category text embedding vector. The text-to-image diffusion model (DaLLE-2) based on image-text contrast pre-training used in this step is trained based on the image-text multimodal contrast pre-training model (CLIP) and models the generation process of the latent space representation vector in the image-text multimodal contrast pre-training model (CLIP). Therefore, the prior diffusion module (DiffusionPrior) used in the former is used as the generative model for modeling the latent space representation vector in the data purification process.

[0114] Step 3: Estimate the likelihood probability of the input image latent space representation vector according to the noise prediction value of the conditional diffusion inverse process (S303)

[0115] After obtaining the noise prediction value of the conditional diffusion inverse process, the likelihood probability of the image latent space representation vector can be estimated based on the variational lower bound theory. The image latent space representation vector likelihood probability value output in this step will be used as the input of the process Figure 1 The maximization target in the process of optimizing the latent space representation vector in step 6 (S106). The variational lower bound theory is the algorithm of the diffusion model itself. The details of the variational lower bound are provided when the diffusion model is proposed, and are used as the loss function for training the diffusion model.

[0116] Figure 3 A flowchart of a method for estimating the image representation probability likelihood value in the latent space using a discriminative model is shown. This flowchart accepts the latent space representation vector of the input space as input, estimates the likelihood probability of the latent space representation vector using the cosine similarity between the image representation vector and the empty category embedding vector, and outputs it as the flow Figure 1 The content of step 4 (S104) described in the above section. The detailed steps of this flowchart are as follows:

[0117] Step 1: Encode the embedding vector of the empty category text template (S301)

[0118] The text template "a picture of a <category>" is directly encoded into an empty category text embedding vector using the ready-to-use text encoder of the image-text multimodal contrast pre-training model (CLIP). The trained image-text multimodal contrast pre-training model (CLIP) includes an image encoder and a text encoder. This step does not require additional training, and the empty category text embedding vector obtained by encoding will be used as a prior condition for the diffusion process in the subsequent steps.

[0119] Step 2: Calculate the cosine similarity between the input image latent space representation vector and the empty category text embedding vector (S302)

[0120] The cosine similarity between the input image latent space representation vector and the empty category text embedding vector is calculated as an approximate estimate of the likelihood probability. It is worth noting that this variant of the estimation method does not rely on the generative model, thereby significantly improving the inference efficiency by two to three orders of magnitude, and achieving an inference speed comparable to that of the discriminative model for image classification (without data purification).

[0121] Step 3: Estimate the likelihood probability of the input image latent space representation vector based on the cosine similarity (S303)

[0122] The calculated cosine similarity is directly used as an approximate estimate of the likelihood probability. The image latent space representation vector likelihood probability value output in this step will be used as the input of the process Figure 1The maximization objective in the optimization of the latent space representation vector process in step 6 (S106).

[0123] The following is a system embodiment corresponding to the above method embodiment. The technical details mentioned in the above embodiments are still valid in this embodiment. To avoid repetition, they are not described here. Correspondingly, the technical details mentioned in this embodiment can also be applied to the above embodiments.

[0124] As shown in Figure 5 The application also provides an image classification device based on latent space purification of a multi-modal model CLIP, which includes:

[0125] An initial module inputs a picture to be classified into an image encoder of a graph-text multi-modal contrast pre-training model to obtain a representation vector of the picture to be classified in a latent space; and uses a text encoder of the graph-text multi-modal contrast pre-training model to obtain a null category text embedding vector represented by a null category text.

[0126] A calculation module calculates a probability likelihood value of the representation vector and the null category text embedding vector.

[0127] An iteration module extracts a vector length and a direction vector of the representation vector, calculates a gradient of the likelihood probability value with respect to the direction vector, updates the direction vector using a gradient ascent method while keeping the vector length unchanged to obtain a purified vector; and executes the calculation step and the iteration step again until the probability likelihood value converges, and saves the current purified vector as a final vector.

[0128] A classification module selects a category text embedding vector with the highest similarity to the final vector by the graph-text multi-modal contrast pre-training model, and takes an image category corresponding to the category text embedding vector as a classification result of the category text embedding vector.

[0129] The image classification device based on latent space purification of a multi-modal model CLIP, wherein the calculation module estimates the probability likelihood value of the representation vector using a text-to-image diffusion model; and the calculation module includes:

[0130] The text-to-image diffusion model takes the null category text embedding vector as a prior condition to calculate a noise prediction value of the representation vector in a conditional diffusion inverse process; and estimates the likelihood probability value of the representation vector based on the noise prediction value according to a variational lower bound theory.

[0131] The image classification device based on latent space purification of a multi-modal model CLIP, wherein the calculation module estimates the probability likelihood value of the representation vector using a cosine similarity between the representation vector and the null category embedding vector; and the calculation module includes:

[0132] calculating a cosine similarity between the space representation and the empty class text embedding vector; and taking the cosine similarity as the likelihood probability value.

[0133] As shown in Figure 6 The present application also proposes a first electronic device A comprising the image classification device based on the CLIP hidden space purification in another embodiment.

[0134] As shown in Figure 7 The first electronic device A can be connected to a data acquisition device C and an information display device D through wired or wireless information transmission schemes, the data acquisition device C is used to acquire images to be identified and classified, such as the animal images described in the embodiments of the present application, and the information display device D is used to display the image classification results obtained by the analysis of the present application.

[0135] The information display device D can process the data output by the first electronic device A based on an information display mechanism to improve the readability of the data output by the first electronic device A. The information display mechanism can be manually preset, for example, the data output by the first electronic device A is visually displayed, which can display parameters and / or attributes set by the user, such as display data range, display font, color, whether to scroll, etc. The user can more timely understand the information without accessing secondary pages or scrolling the page, saving the user's operation. Or the information display mechanism can be an artificial intelligence AI display model, which can learn the user's focus information according to the user's previous usage habits, such as viewing time, click times, editing times, etc., and then automatically present the user with rich and necessary focus information.

[0136] The present application also provides a computer program product, the computer program product comprising a computer program, the computer program can be stored on a readable storage medium, when the computer program is executed by a processor, the computer can execute the image classification device method based on the CLIP hidden space purification of the multi-modal model provided by the above-mentioned method.

[0137] The present application also proposes, in another implementation, a storage medium VIII for storing a computer program for implementing the image classification device method based on the multi-modal model CLIP. It should be understood that the storage medium in the embodiments of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM) or a flash memory. The volatile memory can be a random access memory (RAM) used as an external cache. By way of example, and not limitation, many forms of random access memory (RAM) are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchlink DRAM (SLDRAM), and direct Rambus RAM (DR RAM).

[0138] Figure 8 A schematic block diagram of a second electronic device 1000 that can be used to implement embodiments of the present application is shown. The second electronic device 1000 is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The second electronic device 1000 can also represent various forms of mobile devices such as personal digital assistants, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit implementations of the present application described and / or claimed in this document. The second electronic device 1000 can be the same as or different from the first electronic device A.

[0139] The second electronic device 1000 includes a computing unit I that can perform various appropriate actions and processes in accordance with a computer program stored in a read-only memory II (ROM) or a computer program loaded into a random access memory (RAM) III from a storage medium VIII. In the RAM III, various programs and data required for the operation of the device 1000 can also be stored. The computing unit I, the ROM II, and the RAM III are connected to each other through a bus IV. An input / output (I / O) interface V is also connected to the bus IV.

[0140] A plurality of components in the second electronic device 1000 are connected to the I / O interface V, including an input unit VI such as a keyboard, a mouse, and the like, an output unit VII such as various types of displays, a speaker, and the like, a storage medium VIII such as a magnetic disk, an optical disk, and the like, and a communication unit IX such as a network card, a modem, a wireless communication transceiver, and the like. The communication unit IX allows the second electronic device 1000 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0141] The computing unit I can be various general and / or special-purpose processing components having processing and computing capabilities. Some examples of the computing unit I include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, and the like. The computing unit I performs various methods and processes described above, such as the method steps S1-S4. For example, in some embodiments, the methods can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage medium VIII. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 1000 via the ROM II and / or the communication unit IX. When the computer program is loaded into the RAM III and executed by the computing unit I, one or more steps of the methods described above can be performed. Alternatively, in other embodiments, the computing unit I can be configured to perform the methods by any other appropriate means, such as by means of firmware.

[0142] While the embodiments of the present application have been disclosed as above, they are not limited to only the applications listed in the specification and the embodiments, and can be fully applied to various fields suitable for the present application, and additional modifications can be easily made by those skilled in the art, and thus the present application is not limited to specific details and the figures shown and described herein, without departing from the general concept defined by the claims and the equivalent scope.

Claims

1. An image classification method based on latent space purification using the multimodal model CLIP, characterized in that, include: The initial step involves inputting the image to be classified into the image encoder of the image-text multimodal contrast pre-trained model to obtain the representation vector of the image in the latent space. Using the text encoder of the pre-trained image-text multimodal contrast model, the empty category text embedding vector represented by the empty category text is obtained; The calculation steps involve calculating the likelihood probability values ​​of the representation vector and the empty category text embedding vector. The iterative steps involve extracting the vector length and direction vector of the representation vector, calculating the gradient of the likelihood probability value with respect to the direction vector, updating the direction vector using the gradient ascent method while keeping the vector length unchanged, and obtaining the cleaned vector. The calculation and iterative steps are executed again until the likelihood probability value converges, and the current cleaned vector is saved as the final vector. In the classification step, the image-text multimodal contrast pre-trained model selects the category text embedding vector with the highest similarity to the final vector, and uses the image category corresponding to the category text embedding vector as the classification result of the category text embedding vector; The calculation step employs a text-based image diffusion model to estimate the likelihood probability value of the representation vector. This calculation step includes: the text-based image diffusion model uses the empty-class text embedding vector as a prior condition to calculate the noise prediction value of the representation vector in the inverse conditional diffusion process; based on the noise prediction value, the likelihood probability value of the representation vector is estimated using variational lower bound theory; or The calculation step uses the cosine similarity between the representation vector and the empty category text embedding vector to estimate the likelihood probability value of the representation vector; the calculation step includes: calculating the cosine similarity between the spatial representation and the empty category text embedding vector; and using the cosine similarity as the likelihood probability value.

2. The image classification method based on latent space cleanup using the multimodal model CLIP as described in claim 1, characterized in that, The iterative steps include obtaining the gradient based on the chain rule of derivative of the composite function, according to the vector length and direction vector of the representation vector.

3. An image classification device based on latent space purification using the multimodal model CLIP, characterized in that, include: The initial module inputs the image to be classified into the image encoder of the image-text multimodal contrast pre-trained model to obtain the representation vector of the image in the latent space. Using the text encoder of the pre-trained image-text multimodal contrast model, the empty category text embedding vector represented by the empty category text is obtained; The calculation module calculates the likelihood probability values ​​of the representation vector and the empty category text embedding vector; The iterative module extracts the vector length and direction vector of the representation vector, calculates the gradient of the likelihood probability value with respect to the direction vector, updates the direction vector using the gradient ascent method while keeping the vector length unchanged, and obtains the cleaned vector; the calculation module and the iterative module are executed again until the likelihood probability value converges, and the current cleaned vector is saved as the final vector. In the classification module, the image-text multimodal contrast pre-trained model selects the category text embedding vector with the highest similarity to the final vector, and uses the image category corresponding to the category text embedding vector as the classification result of the category text embedding vector; The calculation module employs a text-based image diffusion model to estimate the likelihood probability of the representation vector. This module includes: using the text-based image diffusion model as a prior condition for the empty-class text embedding vector, calculating the noise prediction value of the representation vector in the inverse conditional diffusion process; and estimating the likelihood probability value of the representation vector based on the variational lower bound theory according to the noise prediction value. The calculation module uses the cosine similarity between the representation vector and the empty category text embedding vector to estimate the likelihood probability value of the representation vector; the calculation module includes: calculating the cosine similarity between the spatial representation and the empty category text embedding vector; and using the cosine similarity as the likelihood probability value.

4. An electronic device, characterized in that, The image classification device for latent space purification based on the CLIP multimodal model as described in claim 3 includes an electronic device or an information display device connected to it, which is used to display the classification result using user-set display parameters, attributes, or through an artificial intelligence model.

5. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the image classification method based on latent space purification according to claim 1 or 2.

6. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the latent space purification image classification method based on the multimodal model CLIP as described in claim 1 or 2.

Images