Causal decoupling combined zero-order learning image classification method based on energy model

Abstract

The invention provides a causal decoupling combined zero-order learning image classification method based on an energy model. Firstly, image features and combined label text features are extracted by using a pre-trained visual language model; in the visual branch, an attribute encoder and a target encoder are used to decouple features, and good balance of dependence and decoupling is realized through three matching losses of combination, attribute and target. The core technology is an energy model-based generation and decoupling module and a causal decoupling module. The energy model is used to reconstruct an image through Langevin dynamics and to generate an anti-fact image through an intervention operation. The generated anti-fact image then enters a causal decoupling module, constraint is exerted through causal intervention invariant loss, and it is ensured that visual features of attributes and targets are correctly decoupled from the image. The generated anti-fact image enables the model to obtain experience except training data, and the problems of training deviation and generalization difficulty are relieved. According to the method, the performance of combined zero sample image classification is remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of combined zero-shot image classification, in particular to a causal decoupling combined zero-shot learning image classification method based on an energy model. BACKGROUND

[0002] In recent years, with the continuous development of deep learning and computer vision, the accuracy of visual recognition models in classifying single-label images has reached a very high level. However, visual recognition models are difficult to achieve good results for complex images with combined labels. The compositional zero-shot learning (CZSL) task reflects the ability of human compositional learning. It faces the challenge of recognizing new combinations based on known primitives, where the training set contains images and text descriptions (objects and attributes). Specifically, during training, images such as "white cat" and "wet ground" are included, and during testing, images of "wet cat" that have not been seen during training are required to be recognized.

[0003] In this task, the semantic entanglement of objects and attributes is a very complex obstacle, requiring the model to deal with the diverse appearance of objects combined with different attributes. General image classification methods are difficult to deal with the compositional zero-shot image classification task, and there are mainly the following technical difficulties: First, the space of combined labels expands exponentially relative to the space of single-class labels. That is, when the number of attributes and objects is close to N, the number of combined labels is N 2 Therefore, it is extremely difficult to model only the combination.

[0004] Second, simply using two classifiers to predict attributes and objects respectively is also not feasible, because there is a dependency between the two, and the visual decoupling is difficult. The visual representation of the same attribute on different objects is very different. For example, "wet ground" and "wet cat" are very different, the former is presented in the image as a water puddle, and the latter is presented in the image as the adhesion of fur. Similarly, the visual difference of the same object with different attributes is also very large, for example, "rusty metal" and "molten metal", the former is some brown spots on the surface of the solid metal, and the latter is completely liquid.

[0005] Third, existing models are prone to bias towards combinations seen during training, and have difficulty generalizing to zero-shot combinations. For example, if only the combinations "old dog" and "kitten" are trained, the zero-shot combination "old cat" exists during testing, the model may be able to accurately recognize the "cat" object, but will be confused by the unseen combination "old cat", and will give an incorrect prediction.

[0006] Existing combined zero-shot image classification methods generally only give solutions to the problems of one of the above aspects. Most decoupling methods do not guarantee the causality of the decoupling process from the perspective of image generation, resulting in the inability to achieve a good trade-off between inter-combination dependence and decoupling, and it is also difficult to effectively alleviate the problems of training bias and generalization difficulty. Therefore, there is an urgent need for a method that can simultaneously address the above closely related problems. SUMMARY

[0007] The present application aims to at least solve one of the above technical problems in the prior art.

[0008] To this end, the present application provides a causal decoupling combined zero-shot learning image classification method based on an energy model.

[0009] The causal decoupling combined zero-shot learning image classification method based on an energy model provided by the present application comprises: inputting an input image into a visual encoder and an adapter of a pre-trained visual language model to extract combined visual features; at the same time, based on a description text combined label of the input image, using a text encoder of the pre-trained visual language model and a preset prompt template, respectively extracting a combined text feature, an attribute text feature and a target text feature; inputting the combined visual features into an attribute encoder and a target encoder respectively to decouple to obtain attribute visual features and target visual features; calculating an attribute matching loss between the attribute visual features and the attribute text features, a target matching loss between the target visual features and the target text features, and a combined matching loss between the combined visual features and the combined text features; at the same time, using an attribute classifier and a target classifier to classify the attribute visual features and the target visual features respectively, and calculating a classification loss; constructing an attribute energy model and a target energy model, inputting the attribute visual features and the target visual features as conditions into the corresponding energy models, generating reconstructed images through Langevin dynamics iterative sampling, and calculating a reconstruction loss; then performing a causal intervention operation, keeping the attribute visual features unchanged and replacing the target visual features, generating a first counterfactual image through the energy model; keeping the target visual features unchanged and replacing the attribute visual features, generating a second counterfactual image through the energy model; using a visual branch comprising the visual encoder, the adapter, the attribute encoder, the attribute classifier, the target encoder and the target classifier as a proxy model to process the first counterfactual image and the second counterfactual image; calculating a causal intervention invariant loss, which is used to constrain the decoupling process: wherein the constraint target is: when the first counterfactual image is input, its attribute classification result is consistent with the attribute in the original combined label; when the second counterfactual image is input, its target classification result is consistent with the target in the original combined label; A total loss function is constructed based on the attribute matching loss, the target matching loss, the combined matching loss, the classification loss, the reconstruction loss and the causal intervention invariant loss, and the network parameters are jointly trained; in the test stage, the matching degree score of the visual features of the test image and the candidate label text features is calculated, and the label with the highest score is selected as the classification result.

[0010] According to the energy model-based causal decoupling combined zero-order learning image classification method, the following additional technical features can be further provided: In the above technical solution, the preset prompt template comprises: The first prompt template is used to extract the combined text features containing attribute characteristics and target features; The second prompt template is used to extract attribute text features; The third prompt template is used to extract target text features.

[0011] In the above technical solution, the adapter is a multi-layer perceptron added after the CLIP visual encoder, and the parameters of the adapter are learnable during the training process; the calculation methods of the attribute matching loss, the target matching loss and the combined matching loss are as follows: first, the cosine similarity between the two feature vectors is calculated, and then the cross-entropy loss is calculated.

[0012] In the above technical solution, the attribute classifier and the target classifier are both linear classifiers; and the calculation method of the classification loss is as follows:

[0013] wherein, represents the classification loss; CE(·) represents the cross-entropy loss function; represents the attribute classifier; represents the attribute visual feature corresponding to the attribute a; represents the target classifier; represents the target visual feature corresponding to the target o.

[0014] In the above technical solution, the attribute energy model and the target energy model have the same structure but different parameters; and the reconstruction image generated by the Langevin dynamics iterative sampling comprises: The attribute visual feature and the target visual feature are input into the corresponding energy model, respectively, an initial image is sampled from random noise and input into the two energy models at the same time, and the sum of the energy values output by the two energy models is gradually reduced through iterative optimization; the iterative process is represented as:

[0015] wherein, represents the image generated after t iterations; denotes an image generated after iteration t-1 times, t=0, is an initial image sampled from random noise ; denotes a learnable parameter of the energy model; denotes an attribute energy model; denotes a target energy model; denotes an attribute visual feature corresponding to the attribute a; denotes a target visual feature corresponding to the target o; denotes a gradient operator.

[0016] In the above technical solution, the calculation of the reconstruction loss comprises: calculating a mean square error based on the image generated after iteration t times and the original image in the data set, comprising:

[0017] wherein, denotes a reconstruction loss; denotes an original image in the data set; The reconstruction loss is used to train the target energy model to enable it to generate an image reflecting the target visual feature denoted by the target, and train the attribute energy model to enable it to generate an image reflecting the attribute visual feature denoted by the attribute.

[0018] In the above technical solution, the operation of replacing the target visual feature or replacing the attribute visual feature specifically includes two ways: using a random vector to replace the original feature, or randomly selecting image features with different targets or different attributes from the data set to replace.

[0019] In the above technical solution, the proxy model comprises an attribute discrimination branch and a target discrimination branch; the first counterfactual image is input into the attribute discrimination branch comprising a visual encoder, an adapter, an attribute encoder and an attribute classifier; the second counterfactual image is input into the attribute discrimination branch comprising a visual encoder, an adapter, a target encoder and a target classifier; In the calculation of the causal intervention invariant loss, in the proxy model, the parameters of the visual encoder, the adapter, the attribute classifier and the target classifier are frozen, while the parameters of the attribute encoder and the target encoder are learnable, so that the model obtains experience beyond the training data; The calculation method of the causal intervention invariant loss comprises:

[0020] wherein, denotes a causal intervention invariant loss; CE(·) denotes a cross-entropy loss function; all parameter sets representing the attribute discrimination branch; all parameter sets representing the target discrimination branch; a first counterfactual image corresponding to the attribute a; a second counterfactual image corresponding to the target o.

[0021] In the above technical solution, the total loss function constructed based on the attribute matching loss, the target matching loss, the combined matching loss, the classification loss, the reconstruction loss and the causal intervention invariant loss is represented as:

[0022] wherein, the attribute matching loss is represented as: the target matching loss is represented as: the combined matching loss is represented as: the classification loss is represented as: the reconstruction loss is represented as: the causal intervention invariant loss is represented as.

[0023] In the above technical solution, in the test phase, for a test image, after obtaining the visual representation through the visual branch and decoupling, for all candidate combined labels, the corresponding text features are obtained through the prompt template and the text encoder, the matching degree between the corresponding representations is calculated, and the label with the highest score is selected as the predicted combined label.

[0024] In summary, due to the adoption of the above technical features, the beneficial effects of the present application are: The causal decoupling combined zero-shot image classification method based on the energy model effectively improves the performance of the combined zero-shot image classification task. By introducing three matching losses of combination, attribute and target, the present application solves the trade-off problem of combination dependency and decoupling in the combined zero-shot image classification. The causal decoupling module in the present application imposes constraints through the causal intervention invariant loss, solving the problem of lack of causal guarantee in the visual decoupling process. In addition, the generation and decoupling module based on the energy model can generate counterfactual images, so that the model obtains experience beyond the training data, effectively alleviating the training bias and generalization difficulty problem. The method proposed in the present application can not only be applied to combined zero-shot image classification, but also can be extended to any form of visual decoupling and combination generalization task.

[0025] Additional aspects and advantages of the application will become apparent from the following description with reference to specific embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0026] The above and / or additional aspects and advantages of the present application will become apparent and more readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings, in which: Figure 1 is a schematic diagram of the overall architecture of an energy model based causally decoupled combined zeroth order learning image classification method according to an embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0028] In the following description, a large number of specific details are set forth in order to facilitate a thorough understanding of the present application, however, the present application can also be implemented in other different ways from those described herein, and therefore, the protection scope of the present application is not limited by the specific embodiments disclosed below.

[0029] The energy model based causally decoupled combined zeroth order learning image classification method according to some embodiments of the present application will be described below with reference to Figure 1

[0030] Some embodiments of the present application provide an energy model based causally decoupled combined zeroth order learning image classification method.

[0031] As Figure 1 ​As shown, the first embodiment of the present application proposes a causal decoupling combined zero-order learning image classification method based on an energy model. The method first extracts visual features and combined label text features using a pre-trained visual language model CLIP. Then, according to the difficulty of different data sets, an optional adapter is used to migrate the knowledge of the pre-trained visual language model in the visual branch part. The subsequent target encoder, attribute encoder, target classifier, attribute classification, target energy model, and attribute energy model together constitute the generation and decoupling module based on the energy model, which will promote the decoupling of visual features through reconstruction loss and classification loss. The combined matching loss, attribute matching loss, and target matching loss achieve a good trade-off between the dependence and decoupling of the model between combinations. The counterfactual image generated by the generation and decoupling module based on the energy model will enter the causal decoupling module, and the causal intervention invariance loss will further impose causal constraints on the entire decoupling process. The principle comes from the decoupled causal generation process. In the specific example in the present application, if the features of the attribute and the target are decoupled, then when one of the features is changed through an intervention operation, the counterfactual image generated by the energy model will only have the corresponding part of the visual performance changed, while the visual performance of other parts is invariant to intervention. However, we cannot directly measure the visual performance change of the counterfactual image. Naturally, we choose to reuse the model structure of the front part of the visual branch, i.e., the CLIP visual encoder, the adapter, the attribute encoder, the attribute classifier, the target encoder, and the target classifier as a proxy, and finally constrain it through the causal intervention invariance loss. Specifically, it ensures that the counterfactual image generated by intervening in the attribute has the same target classification result as the target in the initial input combined label, and the counterfactual image generated by intervening in the target has the same attribute classification result as the attribute in the initial input combined label. Most of the parameters of the reused part are frozen, but the parameters of the attribute encoder and the target encoder are learnable, so that the model can obtain experience beyond the training data through the generated counterfactual image, and alleviate the problems of training bias and generalization difficulty.

[0032] It should be noted that the images of the data set involved in the method proposed by the present disclosure are all real natural images, and each image contains two text labels, one corresponding to the category of the main target in the image, and the other being the attribute category of the target. The images in the training set are represented as , the combined labels are represented as , the attribute labels are represented as , and the target labels are represented as . Continue to refer to Figure 1The method of the present disclosure adopts a double-branch structure as a whole and is mainly composed of three modules, including a feature extraction module, an energy model-based generation and decoupling module, and a causal decoupling module. The feature extraction module contains a visual branch and a text branch, while the energy model-based generation and decoupling module and the causal decoupling module are only for the visual branch. The visual branch is more complex than the text branch because the text is easier to decouple from the language perspective, while the pixel semantics of the image are more entangled.

[0033] Specifically, the method proposed by the present disclosure includes the following steps S1 to S6.

[0034] S1, input an input image into a visual encoder and an adapter of a pre-trained visual language model to extract combined visual features; at the same time, based on a description text combined label of the input image, use a text encoder of the pre-trained visual language model and a preset prompt template to extract combined text features, attribute text features, and target text features, respectively.

[0035] Specifically, the pre-trained visual language model used by the present disclosure is an existing CLIP (Contrastive Language-Image Pre-training) model. Of course, those skilled in the art can also replace it with other pre-trained visual language models of the same size or above and with roughly the same usage as CLIP.

[0036] In some embodiments, the preset prompt template includes a first prompt template, a second prompt template, and a third prompt template. The first prompt template is used to extract combined text features containing attribute features and target features; the second prompt template is used to extract attribute text features; and the third prompt template is used to extract target text features.

[0037] In Figure 1 In the specific embodiment shown, before extracting the text features, three different prompt templates “a photo of [attribute] [object] (first prompt template)”, “a photo of [attribute] object (second prompt template)”, and “a photo of [object] (third prompt template)” are used to extract the combined, attribute, and target text features, respectively, where “[]” is filled with the corresponding label text, and the corresponding word embedding features are learnable parameters.

[0038] Figure 1The description text combination label "wet dog" shown in the middle gets three different prompts, "a photo of [wet][dog]", "a photo of [wet] object", "a photo of [dog]", through the prompt template, which respectively represent the combination "wet dog", the attribute "wet" and the target "dog".

[0039] Subsequently, as shown in Figure 1 , the text prompt extracts text features using the frozen CLIP text encoder, and the obtained combination text features, attribute text features and target text features are represented by , and respectively.

[0040] In the visual branch, the features are first extracted using the frozen CLIP visual encoder, and in particular, a multi-layer perceptron is additionally added as an adapter after the CLIP visual encoder in order to transfer the knowledge obtained by the CLIP through sufficient pre-training to the combined zero-shot image classification task, and the parameters of the adapter are learnable at this time. After passing through the adapter, the overall image features, i.e., the combination visual features, are obtained, which are represented by .

[0041] S2, input the combination visual features into the attribute encoder and the target encoder respectively, and decouple to obtain the attribute visual features and the target visual features; calculate the attribute matching loss between the attribute visual features and the attribute text features, the target matching loss between the target visual features and the target text features, and the combination matching loss between the combination visual features and the combination text features; meanwhile, use the attribute classifier and the target classifier to classify the attribute visual features and the target visual features respectively, and calculate the classification loss.

[0042] The combination features will be decoupled from the combination visual features by two different multi-layer perceptrons, i.e., the attribute encoder and the target encoder, to obtain the attribute visual features and the target visual features , and the parameters of the encoders are learnable.

[0043] In some embodiments, the attribute matching loss, the target matching loss and the combination matching loss are calculated in the following manner: first, calculate the cosine similarity between the two feature vectors, and then calculate the cross-entropy loss.

[0044] The calculation method of the combination matching loss is as follows: First, calculate the cosine similarity between the two feature vectors: and .

[0045] The form of the combination matching loss is cross-entropy loss:

[0046] where i is the image number; j is the description text combination label number, which corresponds to the attribute text label number and the target text label number.

[0047] Similarly, the calculation method of the attribute matching loss is as follows: First, calculate and the cosine similarity between the two feature vectors

[0048] The form of the attribute matching loss is also cross-entropy loss: .

[0049] The calculation method of the target matching loss is as follows: First, calculate and the cosine similarity between the two feature vectors

[0050] The target matching loss is cross-entropy loss: .

[0051] In step S2, for and two simple linear classifiers will also be used to calculate the classification loss to further help the encoder to achieve decoupling, both of which are learnable at this time, in addition, the linear classifiers here will also play a role in the subsequent calculation of the causal intervention invariant loss.

[0052] In one specific embodiment, the attribute classifier and the target classifier are both linear classifiers; the calculation method of the classification loss is as follows:

[0053] where, represents the classification loss; CE(·) represents the cross-entropy loss function; represents the attribute classifier; represents the attribute visual feature corresponding to the attribute a; represents the target classifier; represents the target visual feature corresponding to the target o.

[0054] S3, construct attribute energy model and target energy model, input the attribute visual feature and the target visual feature as conditions to the corresponding energy model, generate a reconstructed image by iterative sampling of the Langevin dynamics, and calculate a reconstruction loss; then perform a causal intervention operation, keep the attribute visual feature unchanged and replace the target visual feature, generate a first counterfactual image through the energy model; keep the target visual feature unchanged and replace the attribute visual feature, generate a second counterfactual image through the energy model.

[0055] Specifically, step S3 is to further impose a decoupling constraint on the visual branch and generate a counterfactual image through a generation and decoupling module based on an energy model. As shown in Figure 1 The reconstructed image generation process is as follows: Input the attribute visual feature and the target visual feature into the corresponding energy model, sample an initial image from random noise, and input it into the two energy models at the same time, and gradually reduce the sum of the energy values output by the two energy models through iterative optimization; the iterative process is represented as:

[0056] wherein, represents the image generated after iteration t times; represents the image generated after iteration t-1 times, t=0, is the initial image sampled from random noise ; represents the learnable parameters of the energy model; represents the attribute energy model; represents the target energy model; represents the attribute visual feature corresponding to the attribute a; represents the target visual feature corresponding to the target o; represents the gradient operator.

[0057] The number of iterations t should be selected according to the complexity of the images in the data set.

[0058] In some embodiments, the calculation of the reconstruction loss comprises: calculating the mean square error based on the image generated after iteration t times and the original image in the data set, comprising:

[0059] wherein, represents the reconstruction loss; represents the original image in the data set; The reconstruction loss is used to train the target energy model to enable it to generate a counterfactual image reflecting the target visual feature The image representing the target, in the attached figure, is the image of the "dog"; and the attribute energy model is trained to generate images that reflect the visual features of the attributes. The image representing the attribute, which is the "wet" image in the attached diagram, is the combined image of the two. The image representing the combined label is the "wet dog" image in the attached figure. Next, the energy model trained with reconstruction loss will be frozen to generate the counterfactual image after intervention. The energy models with the same name in the figure are essentially identical; they are only used for illustration to show different inputs and combinations.

[0060] Intervention Operation In, random vectors are used Replace original target features Alternatively, images that differ from the target and the original image X can be randomly selected from the dataset and processed with... The same processing results in .Will Input the target energy model for freezing, and... By inputting a frozen attribute energy model and undergoing an iterative generation process similar to the previous one, we can obtain a first counterfactual image where the target changes while the attributes remain unchanged. The iterative process is represented as:

[0061] in It is also sampled from random noise at t=0.

[0062] Intervention Operation In, random vectors are used Replace original attribute features Alternatively, images with attributes different from the original image X can be randomly selected from the dataset and processed with... The same processing results in .Will Input the frozen property energy model, and Inputting a frozen target energy model, and through an iterative generation process similar to the previous one, yields a second counterfactual image where properties change while the target remains unchanged. The iterative process is represented as:

[0063] in It is also sampled from random noise at t=0.

[0064] S4, processing the first counterfactual image and the second counterfactual image by using the visual branch comprising the visual encoder, the adapter, the attribute encoder, the attribute classifier, the target encoder and the target classifier as a proxy model; calculating a causal intervention invariance loss, the causal intervention invariance loss being used to constrain the decoupling process: wherein the constraint target is that when the first counterfactual image is input, the attribute classification result is consistent with the attribute in the original combined label; and when the second counterfactual image is input, the target classification result is consistent with the target in the original combined label.

[0065] Specifically, step S4 is to input the generated counterfactual images into the causal decoupling module, and reuse the model structure of the front part of the visual branch, i.e., the CLIP visual encoder, the adapter, the attribute encoder, the attribute classifier, the target encoder and the target classifier as a proxy model of the causal decoupling module, and use the causal intervention invariance loss for constraint. Specifically, it is to ensure that the target classification result of the counterfactual image generated by intervention of the attribute is consistent with the target in the original combined label, and the attribute classification result of the counterfactual image generated by intervention of the target is consistent with the attribute in the original combined label. and

[0066] wherein the proxy model comprises an attribute discrimination branch and a target discrimination branch; inputting the first counterfactual image into the attribute discrimination branch comprising the CLIP visual encoder, the adapter, the attribute encoder and the attribute classifier; and inputting the second counterfactual image into the attribute discrimination branch comprising the CLIP visual encoder, the adapter, the target encoder and the target classifier. In the calculation of the causal intervention invariance loss, the parameters of the visual encoder, the adapter, the attribute classifier and the target classifier in the proxy model are frozen, and the parameters of the attribute encoder and the target encoder are learnable, so that the model obtains experience beyond the training data, and the problems of training bias and generalization difficulty are alleviated.

[0067] For simplicity, all parameters of the attribute discrimination branch of the causal decoupling module, i.e., the CLIP visual encoder, the adapter, the attribute encoder and the attribute classifier, are denoted as ; and all parameters of the target discrimination branch, i.e., the CLIP visual encoder, the adapter, the target encoder and the target classifier, are denoted as .

[0068] The calculation method of the causal intervention invariance loss comprises:

[0069] wherein, denotes the causal intervention invariance loss; and CE(·) denotes a cross-entropy loss function. ​all parameter sets representing the attribute discrimination branch; all parameter sets representing the target discrimination branch; a first counterfactual image corresponding to the attribute a; a second counterfactual image corresponding to the target o.

[0070] Further, in Figure 1 In the illustrated embodiment, a represents the attribute label in the text combination label, i.e., "wet", and o represents the target label in the text combination label, i.e., "dog".

[0071] S5, based on the attribute matching loss, target matching loss, combination matching loss, classification loss, reconstruction loss and causal intervention invariant loss, constructing a total loss function, jointly training network parameters; in the test stage, calculating the matching degree score of the visual feature of the test image and the candidate label text feature, and selecting the label with the highest score as the classification result.

[0072] Specifically, the total loss function constructed based on the attribute matching loss, target matching loss, combination matching loss, classification loss, reconstruction loss and causal intervention invariant loss is represented as:

[0073] wherein, represents the attribute matching loss; represents the target matching loss; represents the combination matching loss; represents the classification loss; represents the reconstruction loss; represents the causal intervention invariant loss.

[0074] The total loss function is used for end-to-end joint training, and all learnable parameters in the framework are adjusted through the back propagation algorithm to minimize the total loss function to optimize the network parameters until the function value no longer decreases.

[0075] In some embodiments, in the test stage, for a test image, after obtaining the visual representation through the visual branch and decoupling, for all candidate combination labels, the corresponding text features are obtained through the prompt template and the text encoder, the matching degree between the corresponding representations is calculated, and the label with the highest score is selected as the predicted combination label.

[0076] Specifically, during testing, a test image is input, and the visual representation is obtained through the CLIP visual encoder and the adapter of the visual branch, and further and , and then for all candidate combination labels, the corresponding text features are obtained by prompting the template and the CLIP text encoder 、 and , and as in the training process, the degree of matching between the corresponding representations is computed 、 and , and the combination label selection is predicted as the label with the highest score, i.e. .

[0077] In this specification, illustrative statements such as "one embodiment" or "an embodiment" do not necessarily refer to the same embodiment or example, although they can. Likewise, the described specific characteristics, structures, materials, or features can be combined in any suitable manner in one or more embodiments or examples.

[0078] Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. An energy model-based causally decoupled combinatorial zeroth-order learning graph image classification method, characterized in that, The method comprises the following steps: inputting an input image into a visual encoder and an adapter of a pre-trained visual language model to extract a combined visual feature; at the same time, based on a description text of the input image, a text encoder of the pre-trained visual language model and a preset prompt template, a combined text feature, an attribute text feature and a target text feature are extracted respectively; the combined visual feature is inputted into an attribute encoder and a target encoder to decouple an attribute visual feature and a target visual feature; an attribute matching loss between the attribute visual feature and the attribute text feature, a target matching loss between the target visual feature and the target text feature, and a combined matching loss between the combined visual feature and the combined text feature are calculated; at the same time, an attribute classifier and a target classifier are used to classify the attribute visual feature and the target visual feature respectively, and a classification loss is calculated; an attribute energy model and a target energy model are constructed, the attribute visual feature and the target visual feature are inputted into the corresponding energy model as a condition, a reconstructed image is generated by iterative sampling of the Langevin dynamics, and a reconstruction loss is calculated; then, a causal intervention operation is performed, the attribute visual feature is kept unchanged and the target visual feature is replaced to generate a first counterfactual image by the energy model; the target visual feature is kept unchanged and the attribute visual feature is replaced to generate a second counterfactual image by the energy model; a visual branch comprising the visual encoder, the adapter, the attribute encoder, the attribute classifier, the target encoder and the target classifier is used as an agent model to process the first counterfactual image and the second counterfactual image; a causal intervention invariant loss is calculated, and the causal intervention invariant loss is used to constrain a decoupling process, wherein the constraint target is that when the first counterfactual image is inputted, the attribute classification result is consistent with the attribute in the original combined label; when the second counterfactual image is inputted, the target classification result is consistent with the target in the original combined label; a total loss function is constructed based on the attribute matching loss, the target matching loss, the combined matching loss, the classification loss, the reconstruction loss and the causal intervention invariant loss, and network parameters are trained jointly; in the test stage, a matching degree score of a visual feature of a test image and a candidate label text feature is calculated, and the label with the highest score is selected as a classification result.

2. The energy model based causally decoupled combined zeroth order learning graph image classification method according to claim 1, characterized in that, The preset prompt template comprises: a first prompt template for extracting a combined text feature comprising an attribute characteristic and a target feature; a second prompt template for extracting an attribute text feature; a third prompt template for extracting a target text feature. 3.The energy model based causally decoupled combined zeroth order learning graph image classification method of claim 1, wherein, The adapter is a multi-layer perceptron added after the CLIP visual encoder, and the parameters of the adapter are learnable in the training process; the calculation methods of the attribute matching loss, the target matching loss and the combined matching loss are that the cosine similarity between two feature vectors is calculated first, and then the cross-entropy loss is calculated. 4.The energy model based causally decoupled combined zeroth order learning graph image classification method of claim 1, wherein, The attribute classifier and the target classifier are both linear classifiers; the calculation method of the classification loss is: wherein, denotes a classification loss; CE(·) denotes a cross-entropy loss function; denotes an attribute classifier; denotes an attribute visual feature corresponding to attribute a; denotes a target classifier; denotes a target visual feature corresponding to target o.

5. The energy model based causally decoupled combined zeroth order learning graph image classification method according to claim 1, characterized in that, The attribute energy model and the target energy model have the same structure but different parameters; The reconstructed image generated by the iterative sampling of the Langevin dynamics comprises: The attribute visual feature and the target visual feature are input into corresponding energy models respectively, an initial image is sampled from random noise and input into the two energy models, and the sum of energy values output by the two energy models is gradually reduced through iterative optimization; the iterative process is represented as: wherein, denotes the image generated after t iterations; denotes the image generated after t-1 iterations, t = 0, is an initial image sampled from random noise ; denotes the learnable parameters of the energy model; denotes the attribute energy model; denotes the target energy model; denotes the attribute visual feature corresponding to attribute a; denotes the target visual feature corresponding to target o; denotes the gradient operator.

6. The energy model based causally decoupled combined zeroth order learning graph image classification method according to claim 5, characterized in that, The calculation of the reconstruction loss comprises: The mean square error is calculated based on the generated image after iteration t times and the original image in the data set, comprising: wherein, denotes the reconstruction loss; denotes the original image in the dataset; The reconstruction loss is used to train the target energy model to generate images reflecting target visual features The images representing the target, and the training of the attribute energy model to generate images reflecting attribute visual features The images representing the attribute.

7. The energy model based causally decoupled combined zeroth order learning graph image classification method according to claim 1, characterized in that, The operation of replacing the target visual feature or the attribute visual feature specifically comprises two ways: using a random vector to replace the original feature, or randomly selecting an image feature with different targets or different attributes from the data set to replace.

8. The energy model based causally decoupled combined zeroth order learning graph image classification method according to claim 1, characterized in that, The proxy model comprises an attribute discrimination branch and a target discrimination branch; The first counterfactual image is input into the attribute discrimination branch comprising a visual encoder, an adapter, an attribute encoder and an attribute classifier; The second counterfactual image is input into the attribute discrimination branch comprising a visual encoder, an adapter, a target encoder and a target classifier; In the calculation of the causal intervention invariant loss, the parameters of the visual encoder, the adapter, the attribute classifier and the target classifier in the proxy model are frozen, while the parameters of the attribute encoder and the target encoder are learnable, so that the model obtains experience beyond the training data; The calculation method of the causal intervention invariant loss comprises: wherein, represents the causal intervention invariant loss; CE(·) represents a cross-entropy loss function; represents the set of all parameters of the attribute discrimination branch; represents the set of all parameters of the target discrimination branch; represents the first counterfactual image corresponding to the attribute a; represents the second counterfactual image corresponding to the target o. 9.The energy model based causally decoupled combined zeroth order learning graph image classification method of claim 1, wherein, The total loss function constructed based on the attribute matching loss, the target matching loss, the combined matching loss, the classification loss, the reconstruction loss and the causal intervention invariant loss is represented as: wherein, represents an attribute matching loss; represents a target matching loss; represents a combined matching loss; represents a classification loss; represents a reconstruction loss; represents a causal intervention invariance loss. 10.The energy model based causally decoupled combinatorial zeroth order learning graph image classification method of claim 1, wherein, In the test stage, for a test image, after obtaining the visual representation through the visual branch and decoupling, for all candidate combined labels, the corresponding text features are obtained through the prompt template and the text encoder, the matching degree between the corresponding representations is calculated, and the label with the highest score is selected as the predicted combined label.

Images