Image classification method and device for hidden space purification based on multi-modal model CLIP

Abstract

The invention provides an image classification method and device for hidden space purification based on a multi-modal model CLIP, and the method comprises the steps: inputting a to-be-classified image into an image encoder of an image-text multi-modal comparison pre-training model, and obtaining a representation vector of the to-be-classified image in a hidden space; using a text encoder of an image-text multi-modal comparison pre-training model to obtain an empty category text embedding vector represented by the empty category text; calculating probability likelihood values of the representation vector and the empty category text embedding vector; extracting a vector length and a direction vector representing the vector, calculating a gradient of a likelihood probability value to the direction vector, updating the direction vector by adopting a gradient rising method while keeping the vector length unchanged, and obtaining a purified vector; and selecting a category text embedding vector with the highest similarity with the final purification vector, and taking an image category corresponding to the category text embedding vector as a classification result of the category text embedding vector.

Description

Technical Field

[0001] The present invention relates to the field of adversarial defense technology for zero-shot image classification tasks, and in particular to a method and device for zero-shot image classification that achieves adversarial robustness by purifying samples in a multimodal latent space. Background Art

[0002] Traditional image classifiers usually use specific category training data for supervised training, while zero-shot image classifiers can classify images without ever seeing training samples of a specific category. CLIP (Visual Language Pre-training) is a visual language pre-training model that can effectively complete zero-shot image classification tasks by aligning the latent space information of the two modalities of text and image.

[0003] In the image classification task, adversarial attack refers to the behavior of making the model make incorrect predictions by applying tiny perturbations to the input data that are imperceptible to the naked eye. Adversarial robustness refers to the ability of a deep neural network to make correct predictions in the face of carefully designed adversarial attacks. Existing defense methods to improve adversarial robustness mainly include adversarial training and data purification: adversarial training enhances the model's ability to defend against the same known attacks by introducing adversarial samples during the training process; data purification eliminates adversarial perturbations by optimizing adversarial samples to make them close to the clean sample distribution. Among them, existing data purification methods all act on the pixel space, usually using generative models, denoising autoencoders, or optimization techniques based on stochastic differential equations to restore the perturbed pixel distribution to a pixel distribution that is more consistent with the clean sample image, thereby enhancing the model's ability to defend against various adversarial attacks.

[0004] Although the existing CLIP model has superior classification accuracy on clean samples, the classification accuracy on adversarial samples may drop to zero. Although existing adversarial training methods can enhance the model's defense against known attacks, they are not effective in defending against unknown attacks. For example, the two existing adversarial training methods integrated with the CLIP model - FARE and TeCoA methods - can enhance the robustness of zero-shot classification, but have significant damage to the model's zero-shot classification accuracy on clean samples, and are difficult to generalize to defense against unknown adversarial attacks. Existing data purification methods usually require a generative model to simulate the distribution of clean samples. Although it can defend against unknown adversarial attacks, it has low inference efficiency. In addition, existing data purification methods are usually performed in the pixel space of a single visual modality. However, the high dimensionality and sparsity of the pixel space may produce peak gradient distributions, resulting in higher purification risks. At the same time, in theory, there is also a gap in quality between single-modal and multi-modal latent space representations. These factors jointly limit the effectiveness of data purification methods. Summary of the invention

[0005] The purpose of the present invention is to solve the problem that the above-mentioned existing data purification methods have poor defense effect and low efficiency. To this end, the present invention proposes a novel method for purifying adversarial sample data using multimodal information in latent space, thereby achieving robust zero-sample classification. The method guides the purification of adversarial samples by aligning the multimodal information of clean samples in latent space through the CLIP model, and introduces a discriminant model instead of a generative model, which significantly improves the zero-sample classification adversarial robustness and reasoning efficiency of the model. In addition, the present invention also provides a complete set of system implementation plans, including detailed steps of data processing, model training and inference process, to ensure the effectiveness and practicality of the method. For example, when a neural network classification model faces adversarial attacks, the attacker adds perturbations that are invisible to humans to an image of a "panda", and the neural network will misclassify it as a "gibbon" category. The present invention proposes a method for purifying adversarial samples using multimodal information in latent space, eliminating the influence of adversarial perturbations in the latent space of the input sample, so that the model can correctly classify its category as "panda".

[0006] In view of the shortcomings of existing technologies, such as Figure 4 As shown, the present invention proposes an image classification method based on latent space purification of a multimodal model CLIP, which includes:

[0007] In the initial step, the image to be classified is input into the image encoder of the image-text multimodal comparison pre-training model to obtain the representation vector of the image to be classified in the latent space; the text encoder of the image-text multimodal comparison pre-training model is used to obtain the empty category text embedding vector represented by the empty category text;

[0008] A calculation step, calculating the probability likelihood value of the representation vector and the embedding vector of the empty category text;

[0009] Iteration step, extracting the vector length and direction vector of the representation vector, calculating the gradient of the likelihood probability value to the direction vector, updating the direction vector using the gradient ascent method while keeping the vector length unchanged, and obtaining a purified vector; executing the calculation step and the iteration step again until the probability likelihood value converges, and saving the current purified vector as the final vector;

[0010] In the classification step, the image-text multimodal comparison pre-training model selects the category text embedding vector with the highest similarity to the final vector, and uses the image category corresponding to the category text embedding vector as the classification result of the category text embedding vector.

[0011] The image classification method based on latent space purification of the multimodal model CLIP, wherein the calculation step uses the Vincent graph diffusion model to estimate the probability likelihood value of the representation vector; the calculation step includes:

[0012] The text graph diffusion model takes the empty category text embedding vector as a priori condition and calculates the noise prediction value of the representation vector in the inverse process of conditional diffusion; according to the noise prediction value, the likelihood probability value of the representation vector is estimated based on the variational lower bound theory.

[0013] The image classification method based on latent space purification of the multimodal model CLIP, wherein the calculation step uses the cosine similarity between the representation vector and the empty category embedding vector to estimate the probability likelihood value of the representation vector; the calculation step includes:

[0014] Calculate the cosine similarity between the spatial representation and the empty category text embedding vector; and use the cosine similarity as the likelihood probability value.

[0015] The image classification method based on latent space purification of the multimodal model CLIP, wherein the iterative step includes obtaining the gradient based on the chain rule of composite function derivation according to the vector length and direction vector of the representation vector.

[0016] like Figure 5 As shown, the present invention also proposes an image classification device based on latent space purification of the multimodal model CLIP, which includes:

[0017] The initial module inputs the image to be classified into the image encoder of the image-text multimodal comparison pre-training model to obtain the representation vector of the image to be classified in the latent space; uses the text encoder of the image-text multimodal comparison pre-training model to obtain the empty category text embedding vector represented by the empty category text;

[0018] A calculation module calculates the probability likelihood value of the representation vector and the empty category text embedding vector;

[0019] The iterative module extracts the vector length and direction vector of the representation vector, calculates the gradient of the likelihood probability value to the direction vector, updates the direction vector using the gradient ascent method while keeping the vector length unchanged, and obtains a purified vector; executes the calculation step and the iterative step again until the probability likelihood value converges, and saves the current purified vector as the final vector;

[0020] The classification module uses the image-text multimodal comparison pre-training model to select the category text embedding vector with the highest similarity to the final vector and use the image category corresponding to the category text embedding vector as the classification result of the category text embedding vector.

[0021] The image classification device based on latent space purification of the multimodal model CLIP, wherein the calculation module uses the Vincent graph diffusion model to estimate the probability likelihood value of the representation vector; the calculation module includes:

[0022] The text graph diffusion model takes the empty category text embedding vector as a priori condition and calculates the noise prediction value of the representation vector in the inverse process of conditional diffusion; according to the noise prediction value, the likelihood probability value of the representation vector is estimated based on the variational lower bound theory.

[0023] The image classification device based on latent space purification of the multimodal model CLIP, wherein the calculation module uses the cosine similarity between the representation vector and the empty category embedding vector to estimate the probability likelihood value of the representation vector; the calculation module includes:

[0024] Calculate the cosine similarity between the spatial representation and the empty category text embedding vector; and use the cosine similarity as the likelihood probability value.

[0025] The present invention also proposes an electronic device, which includes an image classification device based on latent space purification of the multimodal model CLIP. The electronic device may be connected to an information display device, and the information display device is used to display the classification result according to display parameters and attributes set by the user or through an artificial intelligence model.

[0026] The present invention also proposes a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the image classification method based on latent space purification of the multimodal model CLIP are implemented.

[0027] The present invention also proposes a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of the image classification method based on latent space purification of the multimodal model CLIP are implemented.

[0028] It can be seen from the above scheme that the advantages of the present invention are:

[0029] The latent space purification method based on the multimodal model CLIP proposed in this invention provides a novel data purification method by utilizing multimodal information in the latent space, thereby significantly improving the model's zero-shot classification adversarial robustness and reasoning efficiency. The latent space purification method based on the multimodal model CLIP operates in the latent space of the CLIP model and can enhance the adversarial robustness of zero-shot classification without additional training. Compared with the prior art, this method not only overcomes the limitations of single-modal information representation, but also avoids the purification risks brought by high-dimensional and sparse pixel space, significantly improves the adversarial robustness of the model under various attack behaviors, and uses discriminative models to replace traditional generative models to greatly improve reasoning efficiency, showing excellent performance in complex attack environments and efficiency-sensitive tasks.

[0030] The models included in the comparison during the technical effect evaluation stage of the present invention include a multimodal comparative pre-training model of images and texts (CLIP) without using adversarial defense measures, a maximum likelihood data purification method based on the Euler discretized momentum diffusion model (LM-EDM), an adversarial sample data purification method based on a diffusion model (DiffPure), and a latent space purification method (CLIPure) based on the multimodal model CLIP proposed in the present invention, a generative variant based on the diffusion model (CLIPure-Diff) and a discriminant variant based on cosine similarity (CLIPure-Cos).

[0031] Table 1: Performance test results of the latent space purification method based on the multimodal model CLIP on the CIFAR-10 dataset

[0032]

[0033] Tables 1 and 2 show the clean sample accuracy of each model in the CIFAR-10 and ImageNet datasets when no attack is encountered, as well as the adversarial robustness under the listed attack behaviors, to reflect the defense capabilities of the model in different attack methods. Table 3 shows the average inference time test results of each model for image classification of 100 samples on the CIFAR10 dataset, to reflect the inference efficiency of different types of adversarial defense models.

[0034] Table 1 shows a series of adversarial test results using AutoAttack (AA) on the CIFAR-10 dataset, which evaluate the robustness of different models in the face of advanced adversarial attacks. AutoAttack is an automated adversarial attack tool that integrates multiple attack methods to comprehensively test the defense of the model. The attack methods listed in Table 1 include:

[0035] AA Linf (perturbation range is 8 / 255): AutoAttack using the L-infinity norm. This attack limits the maximum perturbation allowed for each pixel to no more than 8 / 255, thereby testing the model's sensitivity to extreme pixel perturbations.

[0036] AA L2 (perturbation range is 0.5): AutoAttack using the L2 norm, which evaluates the sensitivity to global image perturbations (the Euclidean distance of all pixel perturbations does not exceed 0.5).

[0037] Table 2: Performance test results of the latent space purification method based on the multimodal model CLIP on the ImageNet dataset

[0038]

[0039] Table 2 shows a series of adversarial test results using AutoAttack (AA) on the ImageNet dataset, which evaluates the robustness of different models in the face of advanced adversarial attacks. AutoAttack is an automated adversarial attack tool that integrates multiple attack methods to comprehensively test the defense of the model. The attack methods listed in Table 2 include:

[0040] AA Linf (perturbation range is 4 / 255): AutoAttack using the L-infinity norm. This attack limits the maximum perturbation allowed for each pixel to no more than 8 / 255, thereby testing the model's sensitivity to extreme pixel perturbations.

[0041] These results in Tables 1 and 2 clearly show the potential of CLIP-based latent space purification to improve the adversarial robustness of models, especially how it helps models maintain high clean sample accuracy while showing excellent adversarial robustness in the face of diverse adversarial attacks. This comprehensive evaluation emphasizes the defensive performance of CLIP-based latent space purification against unseen attack methods. It is worth mentioning that since CLIP-based latent space purification does not use any adversarial samples during training, any attack method is unknown to the model during testing.

[0042] Table 3: Results of the model’s inference efficiency test on 100 samples of image classification on the CIFAR10 dataset

[0043]

[0044] Table 3 shows the image classification inference efficiency of a series of different types of adversarial defense models on the CIFAR10 dataset. The experiment was run on a single 4090 graphics card, measuring the average inference time of each model for image classification of 100 samples in the CIFAR-10 dataset, and comparing the inference efficiency of each adversarial defense model with the inference time of the text-image multimodal comparison pre-training model (CLIP) without adversarial defense measures as a benchmark. The results in Table 3 clearly show that the two variants of the latent space purification method (CLIPure) based on the multimodal model CLIP proposed in the present invention - the generative variant based on the diffusion model (CLIPure-Diff) and the discriminative variant based on cosine similarity (CLIPure-Cos) - have significantly improved the inference efficiency compared with the existing data purification methods based on generative models, and therefore have excellent advantages in efficient and robust inference. Among them, the discriminative variant based on cosine similarity (CLIPure-Cos) achieves an inference time comparable to the discriminative model baseline by innovatively using discriminative methods to purify adversarial samples, making this variant method perform better in efficiency-sensitive tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 This is a flowchart of zero-shot robust classification reasoning based on CLIP latent space data purification based on the multimodal model of the present invention;

[0046] Figure 2 The flowchart of the method for estimating the probability likelihood value of image representation in latent space based on the diffusion model;

[0047] Figure 3 It is a flow chart of a method for estimating the probability likelihood value of image representation in latent space based on a discriminant model;

[0048] Figure 4 The overall flow chart of the method of the present invention is as follows;

[0049] Figure 5 It is a module diagram of the device of the present invention;

[0050] Figure 6 This is a schematic diagram of the structure of a first electronic device of the present invention;

[0051] Figure 7 This is a schematic diagram of the application environment structure of the first electronic device of the present invention;

[0052] Figure 8 It is a schematic diagram of the structure of a second electronic device of the present invention.

[0053] Reference numerals:

[0054] A-First electronic device;

[0055] B-Image classification device based on latent space purification of multimodal model CLIP;

[0056] C-data acquisition equipment;

[0057] D-information display device;

[0058] 1000 - second electronic device;

[0059] Ⅰ-computational unit;

[0060] II-ROM;

[0061] III-RAM;

[0062] IV-bus;

[0063] V-interface;

[0064] VI - input unit;

[0065] VII-output unit;

[0066] VIII- Storage medium;

[0067] Ⅸ-Communication unit. DETAILED DESCRIPTION

[0068] It should be noted that, in this application, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0069] Without more constraints, an element defined by the phrase "comprising a..." does not exclude the existence of other identical elements in the process, method, article or apparatus comprising the element.

[0070] The processor described in the present invention is the control center of the electronic device, which can be a processor or a general term for multiple processing elements. For example, it can be one or more central processing units (CPUs), or application specific integrated circuits (ASICs), or one or more integrated circuits configured to implement the embodiments of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (FPGAs).

[0071] Optionally, the processor can perform various functions of the electronic device by running or executing a software program stored in the memory, and calling data stored in the memory.

[0072] In a specific implementation, as an embodiment, the processor may include one or more CPUs. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (such as computer program instructions). Electronic devices may include: servers, desktop computers, laptops, smart phones, tablet computers, embedded computers, etc., wherein the embedded computers include vehicles and robots, etc.

[0073] The memory is used to store the software program for executing the solution of the present invention, and the execution is controlled by the processor. The specific implementation method can refer to the above method embodiment, which will not be repeated here.

[0074] It should be noted that the structure of the electronic device shown in the drawings of the present invention does not constitute a limitation thereto, and the actual knowledge structure recognition device may include more or fewer components than shown in the drawings, or a combination of certain components, or a different arrangement of components.

[0075] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0076] It should also be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0077] In the present invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can be represented by: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0078] It should also be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0079] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0080] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0081] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0082] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0083] During the research process, the inventors found that although the existing methods for enhancing adversarial robustness (such as purification methods) can cope with unknown attacks, the defense effect is not as good as adversarial training when facing known attacks, and the reasoning efficiency is low. After further research and theoretical analysis, the inventors found that the reason for the poor defense effect is that the high dimensionality and sparsity of the pixel space lead to a large purification risk. At the same time, the quality of unimodal information representation is low, and it is difficult to fully cope with diverse adversarial perturbations; purification risk refers to the difference between the purified sample and the original clean sample. The most ideal purification is that the purified sample perfectly restores the original clean sample, and the difference from the clean sample is the purification risk. The inventors deduced that using multimodal information for purification in a denser and smoother latent space can effectively reduce the purification risk and improve adversarial robustness. The reasoning efficiency is low because the existing purification methods usually rely on generative models (such as diffusion models) and involve a large number of complex reasoning processes.

[0084] In order to solve these problems, the inventors proposed a latent space purification method (CLIPure) based on the multimodal model CLIP, which is a method for purifying adversarial sample data using multimodal information in latent space to achieve robust zero-sample classification. This method uses the CLIP model to align the multimodal information of clean samples in latent space to guide the purification process of adversarial samples, thereby enhancing the adversarial robustness of the model. In addition, the present invention also proposes a variant that uses a discriminant model instead of a generative model for purification, which greatly improves the inference efficiency and makes this method perform better in efficiency-sensitive tasks.

[0085] Specifically, the present invention proposes the following key technical points:

[0086] Key point 1: A framework for improving the robustness of zero-shot image classification tasks by using multimodal information to purify data in latent space. Use the existing CLIP image encoder to obtain the latent space representation vector of the input image; estimate the likelihood probability of the latent space representation vector; normalize the latent space representation vector to a unit vector to eliminate the influence of the modulus length; use the chain rule and gradient ascent method to perform maximum likelihood optimization on the direction of the latent space representation vector; and use the existing CLIP model to perform zero-shot classification based on the purified latent space representation vector to obtain the predicted category.

[0087] This framework enables the model to demonstrate excellent reasoning efficiency and maintain a high level of classification accuracy and robustness when performing zero-shot image classification tasks in an adversarial environment, even in the face of complex and diverse attack forms.

[0088] Key point 2: A method for estimating the probability likelihood of image representation in latent space based on a diffusion model.

[0089] The existing CLIP text encoder is used to obtain the empty category text embedding vector represented by the text template "a picture of the category <>", and the latent space representation vector of the input image is obtained; the prior diffusion module (DiffusionPrior) in the text-image diffusion model (DaLLE-2) based on image-text comparison pre-training is used to calculate the noise prediction value of the conditional diffusion inverse process of the input image latent space representation vector under the condition of a given empty category text embedding vector; the likelihood probability of the input image latent space representation vector is estimated based on the noise prediction value of the conditional diffusion inverse process.

[0090] Key point 3, a method for estimating the probability likelihood value of image representation in latent space based on a discriminant model. Use the existing CLIP text encoder to obtain the empty category text embedding vector represented by the text template "a picture of the <> category", and obtain the latent space representation vector of the input image; calculate the cosine similarity between the latent space representation vector of the input image and the empty category text embedding vector; and estimate the likelihood probability of the latent space representation vector of the input image based on the cosine similarity.

[0091] In order to make the above features and effects of the present invention more clearly and understandably described, embodiments are given below and described in detail with reference to the accompanying drawings. This specification discloses one or more embodiments that include the features of the present invention. The disclosed embodiments are only for illustration. The scope of protection of the present invention is not limited to the disclosed embodiments, and the present invention is defined by the attached claims.

[0092] Figure 1 The flowchart of zero-shot robust classification reasoning based on CLIPure, a latent space purification method based on the multimodal model CLIP, is presented. The adversarial sample image representation can be purified in the latent space with the help of multimodal information, thereby achieving robust zero-shot image classification. The following are the detailed steps of the process:

[0093] Step 1: Encode the latent space representation vector of the input adversarial sample image (S101)

[0094] The trained CLIP model includes an image encoder and a text encoder. This step does not require additional training. The ready-made image encoder of CLIP is directly used to encode the input adversarial sample image into a representation vector in the latent space. As a visual language aligned encoder model, CLIP has demonstrated the superiority of its multimodal embedding in many tasks. The present invention uses the multimodal information aligned in the latent space to realize the adversarial sample data purification process.

[0095] Step 2: Determine whether to use the generative model (S102)

[0096] The method of the present invention includes two versions, one using a generative model and the other not requiring a generative model. The user can flexibly choose whether to use a generative model to calculate the likelihood probability of the image latent space representation vector during the data purification process. If so, proceed to step 3, otherwise proceed to step 4. Since the generative model requires a more complex reasoning process, this will reduce the reasoning efficiency to a certain extent. The calculation of the likelihood function is an existing method in the diffusion model. The innovation of the present invention is to use empty category text to calculate the likelihood function.

[0097] Step 3: Estimate the likelihood probability of the latent space representation vector based on the prior diffusion module (S103)

[0098] If you choose to use a generative model in data purification, the diffusion model is used to estimate the probability likelihood of the image representation in the latent space. This likelihood function value can be expressed as a function of the input image latent space representation vector and the empty category text embedding vector, for example, by using the cosine similarity between the image latent space vector and the empty category text vector as the likelihood value. The detailed calculation process of this part of the estimation method will be described in the process Figure 2 Introduced in.

[0099] Step 4: Estimate the likelihood probability of the latent space representation vector based on cosine similarity (S104)

[0100] If you choose not to use a generative model in data purification, you can directly use the discriminative model to estimate the probability likelihood of the image representation in the latent space. This likelihood function value can be expressed as a function of the latent space representation vector of the input image and the embedding vector of the empty category text. The probability improves the reasoning efficiency while ensuring that the accuracy does not decrease. The detailed calculation process of this part of the estimation method will be described in the process Figure 3 It is introduced in , that directly using the discriminant model to attack the probability likelihood estimation can significantly improve the inference efficiency.

[0101] Step 5: Obtain the length and direction of the latent space representation vector respectively (S105)

[0102] The latent space representation vector is normalized to a unit vector to represent the vector direction. The unit vector is obtained by dividing the vector by its vector length (i.e., the modulus of the vector). A vector with a length of unit length is called a unit vector. The unit vector eliminates the influence of the vector length and only retains the vector direction information, thereby being able to eliminate the influence of the modulus length in the subsequent purification process. This is because the image-text multimodal comparison pre-training model (CLIP) aligns the image-text multimodal information through cosine similarity, and the modulus length is not important for reflecting the relationship between vectors. Therefore, when performing maximum likelihood optimization in the latent space purification method (CLIPure) based on the multimodal model CLIP, the latent space representation vector needs to be normalized to eliminate the influence of the modulus length. This is crucial to the method proposed in the present invention, because our experiments show that the data purification process may be hindered by changes in the modulus length of the latent space representation vector and may even ultimately fail.

[0103] Step 6: Optimize the latent space representation vector by maximizing the likelihood probability (S106)

[0104] Data purification eliminates adversarial perturbations by optimizing adversarial samples to make them close to the distribution of clean samples, and optimizes the latent space representation vector with the goal of maximizing the likelihood probability. The likelihood probability value of the image latent space representation vector output in step 3 or step 4 can be expressed as a function of the input image latent space representation vector, and the image latent space representation vector in step 5 is decoupled into a modulus scalar and a direction vector after normalization. In this step 6, according to the chain rule of composite function derivation, the gradient of the likelihood probability value with respect to the direction vector of the latent space representation vector (the direction of the latent space representation vector) can be calculated, and then the gradient ascent method is used to update the direction vector of the latent space representation vector, thereby achieving data purification on the latent space.

[0105] Step 7: Repeat the purification process until convergence (S107)

[0106] Repeat the vector optimization process from step 2 to step 6 until the vector likelihood probability converges. In order to obtain the purified vector, it is necessary to multiply the modulus of the original vector (unchanged during the purification process) by the purified unit vector (indicating the direction of the vector) to obtain the purified vector.

[0107] Step 8: Use the purified latent space representation vector for zero-shot classification (S108)

[0108] The purified latent space representation vector is input into the text-image multimodal comparison pre-training model (CLIP) for zero-shot classification to obtain the predicted category. In this process, the existing text-image multimodal comparison pre-training model (CLIP) completes the zero-shot classification task without seeing any specific category training data by selecting the category text embedding vector with the highest similarity to the image latent space representation vector. The method of the present invention achieves the purpose of improving the robustness of the model by purifying the image representation vector in the latent space.

[0109] Figure 2 The flowchart of the method of estimating the probability likelihood value of image representation in latent space using the diffusion generation model is shown. This process accepts the latent space representation vector of the input space as input, estimates the likelihood probability of the latent space representation vector using the prior diffusion module (DiffusionPrior), and uses its output as the process Figure 1 The following are the detailed steps of the process:

[0110] Step 1: Encode the embedding vector of the empty category text template (S201)

[0111] Directly use the existing text encoder of CLIP to encode the text template "a picture of the <> category" into an empty category text embedding vector. The trained CLIP includes an image encoder and a text encoder. This step does not require additional training. The encoded empty category text embedding vector will be used as a priori condition for the diffusion process in subsequent steps.

[0112] Step 2: Use the prior diffusion module to calculate the noise prediction value of the inverse process of conditional diffusion of the input image latent space representation vector (S202)

[0113] Using the prior diffusion module (DiffusionPrior) in the text-image diffusion model (DaLLE-2) based on pre-training of text-image comparison, the noise prediction value of the inverse process of conditional diffusion of the latent space representation vector of the input image is calculated under the condition of a given empty category text embedding vector. The text-image diffusion model (DaLLE-2) based on pre-training of text-image comparison used in this step is trained based on the text-image multimodal comparison pre-training model (CLIP), and models the generation process of the latent space representation vector in the text-image multimodal comparison pre-training model (CLIP). Therefore, the method proposed in the present invention uses the prior diffusion module (DiffusionPrior) used in the former as a generative model for modeling the latent space representation vector in the data purification process.

[0114] Step 3: Estimate the likelihood probability of the latent space representation vector of the input image based on the noise prediction value of the inverse process of conditional diffusion (S303)

[0115] After obtaining the noise prediction value of the inverse process of conditional diffusion, the likelihood probability of the image latent space representation vector can be estimated based on the variational lower bound theory. The likelihood probability value of the image latent space representation vector output in this step will be used as the process Figure 1 The maximization target in the process of optimizing the latent space representation vector in step 6 (S106) is as follows. The variational lower bound theory is the algorithm of the diffusion model itself. When the diffusion model was proposed, the variational lower bound details were included as the loss function for diffusion model training.

[0116] Figure 3 The flowchart of the method for estimating the probability likelihood value of image representation in latent space using a discriminative model is shown. This process accepts the latent space representation vector of the input space as input, uses the cosine similarity between the image representation vector and the empty category embedding vector, estimates the likelihood probability of the latent space representation vector without the need for complex reasoning of the generative model, and outputs it as the process Figure 1 The following are the detailed steps of the process:

[0117] Step 1: Encode the embedding vector of the empty category text template (S301)

[0118] Directly use the existing text encoder of CLIP to encode the text template "a picture of the <> category" into an empty category text embedding vector. The trained CLIP includes an image encoder and a text encoder. This step does not require additional training. The encoded empty category text embedding vector will be used as a priori condition for the diffusion process in subsequent steps.

[0119] Step 2: Calculate the cosine similarity between the latent space representation vector of the input image and the embedding vector of the empty category text (S302)

[0120] The cosine similarity between the latent space representation vector of the input image and the embedding vector of the empty category text is calculated as an approximate estimate of the likelihood probability. It is worth noting that this variant of the estimation method does not need to rely on the generative model, which greatly improves the reasoning efficiency by two to three orders of magnitude and achieves a reasoning speed comparable to that of the discriminative model directly performing image classification (without data purification).

[0121] Step 3: Estimate the likelihood probability of the latent space representation vector of the input image based on cosine similarity (S303)

[0122] The calculated cosine similarity is directly used as an approximate estimate of the likelihood probability. The image latent space representation vector likelihood probability value output in this step will be used as the process Figure 1The maximization target when optimizing the latent space representation vector process in step 6 (S106).

[0123] The following is a system embodiment corresponding to the above method embodiment. This embodiment can be implemented in conjunction with the above embodiment. The relevant technical details mentioned in the above embodiment are still valid in this embodiment. In order to reduce repetition, they are not repeated here. Accordingly, the relevant technical details mentioned in this embodiment can also be applied in the above embodiment.

[0124] like Figure 5 As shown, the present invention also proposes an image classification device based on latent space purification of the multimodal model CLIP, which includes:

[0125] The initial module inputs the image to be classified into the image encoder of the image-text multimodal comparison pre-training model to obtain the representation vector of the image to be classified in the latent space; uses the text encoder of the image-text multimodal comparison pre-training model to obtain the empty category text embedding vector represented by the empty category text;

[0126] A calculation module calculates the probability likelihood value of the representation vector and the empty category text embedding vector;

[0127] The iterative module extracts the vector length and direction vector of the representation vector, calculates the gradient of the likelihood probability value to the direction vector, updates the direction vector using the gradient ascent method while keeping the vector length unchanged, and obtains a purified vector; executes the calculation step and the iterative step again until the probability likelihood value converges, and saves the current purified vector as the final vector;

[0128] The classification module uses the image-text multimodal comparison pre-training model to select the category text embedding vector with the highest similarity to the final vector and use the image category corresponding to the category text embedding vector as the classification result of the category text embedding vector.

[0129] The image classification device based on latent space purification of the multimodal model CLIP, wherein the calculation module uses the Vincent graph diffusion model to estimate the probability likelihood value of the representation vector; the calculation module includes:

[0130] The text graph diffusion model takes the empty category text embedding vector as a priori condition and calculates the noise prediction value of the representation vector in the inverse process of conditional diffusion; according to the noise prediction value, the likelihood probability value of the representation vector is estimated based on the variational lower bound theory.

[0131] The image classification device based on latent space purification of the multimodal model CLIP, wherein the calculation module uses the cosine similarity between the representation vector and the empty category embedding vector to estimate the probability likelihood value of the representation vector; the calculation module includes:

[0132] Calculate the cosine similarity between the spatial representation and the empty category text embedding vector; and use the cosine similarity as the likelihood probability value.

[0133] like Figure 6 As shown, the present invention further proposes a first electronic device A in another embodiment, including the image classification device based on latent space purification of the multimodal model CLIP.

[0134] like Figure 7 As shown, the first electronic device A can also be connected to the data acquisition device C and the information display device D through a wired or wireless information transmission scheme. The data acquisition device C is used to collect and obtain images to be identified and classified, such as the animal images described in the embodiment of the present invention, and the information display device D is used to display the image classification results obtained by the analysis of the present invention.

[0135] The information display device D can sort and process the data output by the first electronic device A based on the information display mechanism to improve the readability of the data output by the first electronic device A. The information display mechanism can be manually preset, for example, the data output by the first electronic device A is visually displayed, which can be based on the display parameters and / or attributes set by the user. The display parameters can be, for example, the display data range, and the display attributes can be, for example, the display font, color, whether to scroll and play, etc. The user is presented with the key information specified by the user, for example, focusing on displaying specific classification results, etc. The user can understand this information more timely without having to access the secondary page or scroll the page, saving the user's operation. Or the information display mechanism can be an artificial intelligence AI display model, which can learn the user's key information based on the user's previous usage habits, such as viewing time, number of clicks, number of edits, etc., and then automatically present rich and necessary key information to the user.

[0136] The present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a readable storage medium. When the computer program is executed by a processor, the computer can execute the image classification device method based on latent space purification of the multimodal model CLIP provided by the above methods.

[0137] The present invention also proposes a storage medium VIII in another embodiment for storing a computer program for executing the image classification device method of latent space purification based on the multimodal model CLIP. It should be understood that the storage medium in the embodiment of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DR RAM).

[0138] Figure 8 A schematic block diagram of a second electronic device 1000 that can be used to implement an embodiment of the present invention is shown. The second electronic device 1000 electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The second electronic device 1000 can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples, and are not intended to limit the implementation of the present invention described and / or required herein. The second electronic device 1000 may be the same or different from the first electronic device A.

[0139] The second electronic device 1000 includes a computing unit I, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory II (ROM) or a computer program loaded from a storage medium VIII into a random access memory (RAM) III. In RAM III, various programs and data required for the operation of the device 1000 can also be stored. The computing unit I, ROM II, and RAM III are connected to each other via a bus IV. An input / output (I / O) interface V is also connected to the bus IV.

[0140] A plurality of components in the second electronic device 1000 are connected to the I / O interface V, including: an input unit VI, such as a keyboard, a mouse, etc.; an output unit VII, such as various types of displays, speakers, etc.; a storage medium VIII, such as a disk, an optical disk, etc.; and a communication unit IX, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit IX allows the second electronic device 1000 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0141] The computing unit I may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of computing unit I include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit I performs the various methods and processes described above, such as method steps S1-S4. For example, in some embodiments, the method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage medium VIII. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1000 via ROM II and / or a communication unit IX. When the computer program is loaded into RAM III and executed by the computing unit I, one or more steps of the method described above may be performed. Alternatively, in other embodiments, the computing unit I may be configured to execute the method in any other appropriate manner (e.g., by means of firmware).

[0142] Although the embodiments of the present invention have been disclosed as above, they are not limited to the applications listed in the specification and the implementation modes, and they can be fully applied to various fields suitable for the present invention. For those familiar with the art, additional modifications can be easily implemented. Therefore, without departing from the general concept defined by the claims and the scope of equivalents, the present invention is not limited to the specific details and the illustrations shown and described herein.

Claims

1. An image classification method based on latent space purification of multimodal model CLIP, characterized in that: include: In the initial step, the image to be classified is input into the image encoder of the image-text multimodal comparison pre-training model to obtain the representation vector of the image to be classified in the latent space; Using the text encoder of the image-text multimodal comparison pre-training model, a null category text embedding vector represented by the null category text is obtained; A calculation step, calculating the probability likelihood value of the representation vector and the embedding vector of the empty category text; Iteration step, extracting the vector length and direction vector of the representation vector, calculating the gradient of the likelihood probability value to the direction vector, updating the direction vector using the gradient ascent method while keeping the vector length unchanged, and obtaining a purified vector; executing the calculation step and the iteration step again until the probability likelihood value converges, and saving the current purified vector as the final vector; In the classification step, the image-text multimodal comparison pre-training model selects the category text embedding vector with the highest similarity to the final vector, and uses the image category corresponding to the category text embedding vector as the classification result of the category text embedding vector.

2. The image classification method based on latent space purification of multimodal model CLIP as claimed in claim 1, characterized in that: The calculation step uses the Vincent graph diffusion model to estimate the probability likelihood value of the representation vector; the calculation step includes: The text graph diffusion model takes the empty category text embedding vector as a priori condition and calculates the noise prediction value of the representation vector in the inverse process of conditional diffusion; according to the noise prediction value, the likelihood probability value of the representation vector is estimated based on the variational lower bound theory.

3. The image classification method based on latent space purification of multimodal model CLIP as claimed in claim 1, characterized in that: The calculation step uses the cosine similarity between the representation vector and the empty category embedding vector to estimate the probability likelihood value of the representation vector; the calculation step includes: Calculate the cosine similarity between the spatial representation and the empty category text embedding vector; and use the cosine similarity as the likelihood probability value.

4. The image classification method based on latent space purification of multimodal model CLIP as claimed in claim 1, characterized in that: The iterative step includes obtaining the gradient based on the chain rule of composite function derivation according to the vector length and the direction vector of the representation vector.

5. An image classification device based on latent space purification of multimodal model CLIP, characterized in that: include: The initial module inputs the image to be classified into the image encoder of the image-text multimodal comparison pre-training model to obtain the representation vector of the image to be classified in the latent space; Using the text encoder of the image-text multimodal comparison pre-training model, a null category text embedding vector represented by the null category text is obtained; A calculation module calculates the probability likelihood value of the representation vector and the empty category text embedding vector; The iterative module extracts the vector length and direction vector of the representation vector, calculates the gradient of the likelihood probability value to the direction vector, updates the direction vector using the gradient ascent method while keeping the vector length unchanged, and obtains a purified vector; executes the calculation step and the iterative step again until the probability likelihood value converges, and saves the current purified vector as the final vector; The classification module uses the image-text multimodal comparison pre-training model to select the category text embedding vector with the highest similarity to the final vector and use the image category corresponding to the category text embedding vector as the classification result of the category text embedding vector.

6. The image classification device based on latent space purification of multimodal model CLIP as claimed in claim 5, characterized in that: The calculation module estimates the probability likelihood value of the representation vector using the Wensheng graph diffusion model; the calculation module includes: The text graph diffusion model takes the empty category text embedding vector as a priori condition and calculates the noise prediction value of the representation vector in the inverse process of conditional diffusion; according to the noise prediction value, the likelihood probability value of the representation vector is estimated based on the variational lower bound theory.

7. The image classification device based on latent space purification of multimodal model CLIP as claimed in claim 5, characterized in that: The calculation module estimates the probability likelihood value of the representation vector by using the cosine similarity between the representation vector and the empty category embedding vector; the calculation module includes: Calculate the cosine similarity between the spatial representation and the empty category text embedding vector; and use the cosine similarity as the likelihood probability value.

8. An electronic device, characterized in that: An image classification device comprising a latent space purification based on a multimodal model CLIP as described in claims 5-7, wherein the electronic device is connected to an information display device, and the information display device is used to display the classification result with display parameters and attributes set by a user or through an artificial intelligence model.

9. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the image classification method based on latent space purification of the multimodal model CLIP as described in any one of claims 1 to 4.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the image classification method based on latent space purification of the multimodal model CLIP as described in any one of claims 1 to 4 are implemented.

Images