Concept bottleneck model construction method and device based on variational information bottleneck guiding

Abstract

The application discloses a concept bottleneck model construction method based on a variational information bottleneck guide, which utilizes a visual language model to generate a basic concept pool directly related to the content of an input image, utilizes a large language model to generate a descriptive supplementary concept pool covering diversified attribute features of a target category; based on a variational information bottleneck principle, a concept importance score is calculated through cross-modal alignment of an image-text embedding space to screen high-correlation concepts; a double-branch concept bottleneck network is formed by a basic concept classifier and a supplementary concept classifier, and the prediction results of the two are fused through a classification fusioner; an explainable efficiency index is used to optimize the model to balance the classification accuracy and the efficiency of concept explanation. The application utilizes the variational information bottleneck to deeply align the CLIP encoded image-text embedding, and utilizes the supplementary concept bottleneck to further improve the classification accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep learning explainability, and in particular to a concept bottleneck model construction method and device based on a variational information bottleneck guide. BACKGROUND

[0002] At present, with the wide application of deep neural networks in various practical applications, the demand for understanding the decision-making process of these "black box" models is increasing. To improve the explainability of the model, the concept bottleneck model provides an effective solution to this challenge. The concept bottleneck model introduces a concept bottleneck layer, taking human understandable concepts as intermediate representation, and then makes final decision prediction based on these concepts. However, this method relies on high-density artificial concept labeling, which not only has high cost, but also has great limitations in practical applications, especially in some fields, the acquisition or definition of concept labeling may be difficult and ambiguous. Therefore, the concept bottleneck model is initially limited to small-scale data sets with concept labeling (such as CUB data set), and has been difficult to generalize to larger-scale data sets for a long time.

[0003] In recent years, with the development of large language models and visual language models, the research of concept bottleneck model has been significantly promoted. Some studies attempt to use large language models to replace artificial labeling, automatically generate concept bottlenecks, and thus improve the efficiency of model training. These methods usually generate a concept pool through a large language model, and then combine a visual language model or a neuron-level explanation tool for post-processing training. This progress not only eliminates the high cost of artificial labeling, but also enables the concept bottleneck model to be extended to large-scale data sets such as ImageNet. However, existing methods still face challenges in solving the concept prediction and information leakage problems.

[0004] Firstly, the concept prediction problem refers to the fact that the concept bottleneck model relies on intermediate layer concept prediction to generate the final decision result, so the accuracy of concept prediction directly affects the credibility of model explanation. However, the concept labels automatically generated by large language models may not accurately match the true content of the input image, and in the concept screening process, the visual concepts in the image may not strictly align with the generated text concepts, resulting in deviation in the intermediate concept layer of the model. This deviation may lead the model to give misleading information when explaining the decision, thereby affecting the authenticity of the explanation result. Therefore, how to extract concepts with strong semantic consistency from the automatically generated concept pool has become an important topic in current research.

[0005] Secondly, the information leakage problem indicates that the prediction value of the concept layer may contain additional information unrelated to the input image features. Ideally, concept prediction should only be related to specific features of the input image and should not contain irrelevant information. However, studies have shown that even with a randomly generated set of concept labels, the model can still achieve high classification accuracy, indicating that the model inadvertently encodes noise information unrelated to the task in the concept layer, reducing interpretability and making the reasoning process of the model difficult for humans to understand. To solve this problem, the construction and screening process of the concept pool needs to be optimized to ensure that the concept layer only transmits core information related to the task and avoids noise interference. SUMMARY

[0006] The present application provides a concept bottleneck model construction method and device based on variational information bottleneck guidance to solve the technical problems existing in the prior art.

[0007] The technical solution adopted by the present application to solve the technical problems existing in the prior art is:

[0008] A concept bottleneck model construction method based on variational information bottleneck guidance, comprising the following method steps:

[0009] Step 1: Build a concept bottleneck model according to the following structure: make the concept bottleneck model include a visual language model, a large language model, a variational information bottleneck model A, a variational information bottleneck model B, a text encoder A of CLIP, a text encoder B of CLIP, an image encoder of CLIP, a dot multiplier A, a dot multiplier B, a basic concept classifier, a supplementary concept classifier, and a classification fusion device; make the visual language model, the variational information bottleneck model A, and the text encoder A of CLIP connected in sequence; make the large language model, the variational information bottleneck model B, and the text encoder B of CLIP connected in sequence, connect the output ends of the text encoder A of CLIP and the image encoder of CLIP to the two input ends of the dot multiplier A; connect the output ends of the text encoder B of CLIP and the image encoder of CLIP to the two input ends of the dot multiplier B; connect the output end of the dot multiplier A to the input end of the basic concept classifier, connect the output end of the dot multiplier B to the input end of the supplementary concept classifier, and connect the output end of the basic concept classifier and the output end of the supplementary concept classifier to the input end of the classification fusion device;

[0010] Step 2: input image data into the visual language model, generate a basic concept pool directly related to the content of the input image using the visual language model; input prompt information into the large language model, generate a descriptive supplementary concept pool using the large language model, covering the diverse attribute features of the target category;

[0011] Step 3, filtering out core base concepts with higher relevance from the base concept pool through the variational information bottleneck model A, and filtering out core supplementary concepts with higher relevance from the supplementary concept pool through the variational information bottleneck model B, the core base concepts and the core supplementary concepts being used to construct the concept bottleneck;

[0012] Step 4, the core base concepts and the core supplementary concepts being respectively processed through the text encoders A and B of CLIP to obtain base concept embeddings and supplementary concept embeddings respectively; the image data being input into the image encoder of CLIP to obtain an image embedding through the processing of the image encoder of CLIP; the base concept embedding being dot-multiplied with the image embedding through a dot multiplier A, and the dot multiplication result being input into a base concept classifier; the supplementary concept embedding being dot-multiplied with the image embedding through a dot multiplier B, and the dot multiplication result being input into a supplementary concept classifier; the outputs of the base concept classifier and the supplementary concept classifier being fused through a classification fuser to obtain a comprehensive classification prediction result;

[0013] Step 5, using an explainability efficiency index to evaluate the concept bottleneck model to balance the classification accuracy and the efficiency of concept explanation.

[0014] Further, in step 2, a thinking chain prompt template is set, and a visual language model adopts an Instruct BLIP model; the Instruct BLIP model extracts multiple visual features from the image; the thinking chain prompt template is used to guide the Instruct BLIP model to generate concepts related to the image from multiple angles and multiple levels to finally form a fine-grained base concept pool.

[0015] A large language model adopts a GPT-3.5Turbo model; the GPT-3.5Turbo model generates diversified text descriptions according to the prompts given by the thinking chain prompt template to ensure comprehensive attribute features of the target category.

[0016] Further, in step 3, the variational information bottleneck models A and B are used to align the visual concepts in the text description with the image regions without relying on concept labels; a feature attribution analysis is performed using the information bottleneck of the visual language variant to calculate the importance scores of each concept in the text description and the image regions, and then the comprehensive scores of each concept are obtained by accumulating the frequencies of the concepts and the professional degrees of the concepts; the core base concepts are selected from the base concept pool in the order of the comprehensive scores from high to low; and the core supplementary concepts are constructed from the first V concepts in the supplementary concept pool.

[0017] Further, in step 4, let b (b) is the output of the base concept classifier, and c(c) is to supplement the output of the concept classifier; the classification fuser fuses the outputs of the base concept classifier and the supplementary concept classifier according to the following formula:

[0018]

[0019] In the formula, pred represents a prediction result;

[0020] pred represents a prediction result;

[0021] α represents a weight coefficient;

[0022] b represents a base concept bottleneck;

[0023] c represents a supplementary concept bottleneck;

[0024] f represents a visual feature obtained by encoding using a CLIP image encoder;

[0025] W represents a base concept text feature obtained by using a CLIP text encoder;

[0026] U represents a supplementary concept text feature obtained by using a CLIP text encoder;

[0027] Ψ b () represents a base concept linear classifier function;

[0028] Ψ c () represents a supplementary concept linear classifier function.

[0029] Further, when training the base concept classifier and the supplementary concept classifier, forward and backward propagations are performed twice in each batch to update parameters; in the first propagation, only the base concept classifier is optimized; in the second propagation, the parameters of the base concept classifier are fixed, and the supplementary concept classifier is optimized; in order to avoid the influence of extreme values of the supplementary concept on the result, the cosine similarity is used instead of the projection distance as the concept bottleneck, and the activation value of each concept is standardized to have a mean of 0 and a standard deviation of 1, so as to accelerate the convergence.

[0030] Further, in step 5, the calculation formula of the explainability efficiency index is as follows:

[0031]

[0032] In the formula, IE represents explainability efficiency;

[0033] IE represents explainability efficiency;

[0034] Acc represents classification accuracy;

[0035] i represents a concept serial number;

[0036] j represents a category serial number;

[0037] W F represents a linear layer for mapping concepts to categories;

[0038] (W F ) ij represents the weight in the linear layer corresponding to the i-th concept and the j-th category;

[0039] O{(W F ) ij ≠0} represents the number of non-zero weights in the linear layer;

[0040] R represents the number of concepts;

[0041] S represents the number of categories;

[0042] represents the average concept length.

[0043] Further, in step 3, for the variational information bottleneck models A, B, the objective functions of the two are optimized using the variational approximation method, and the KL divergence between the image and the text is used to adjust the parameters of the two, and the text encoder A, B of the pre-trained CLIP is combined to assign importance scores to the image regions and the text concepts.

[0044] Further, for the variational information bottleneck models A, B, the following objective function is set:

[0045]

[0046] In the formula:

[0047] m represents the image or text modality;

[0048] m' represents the complement of the modality m;

[0049] θ m represents the parameter set of modality m;

[0050] represents the optimization objective under the parameter set of modality m;

[0051] Z m represents the output of the information bottleneck to modality m;

[0052] E m′ represents the embedding of modality m';

[0053] X m represents the input of the information bottleneck to modality m;

[0054] β represents a scaling factor;

[0055] I(Z m , Em′ ;θ m ) represents the embedding E of mode m′ given the parameter set of mode m. m′ The output Z of mode m is related to the information bottleneck. m Mutual information between them;

[0056] I(Z m ,X m ;θ m The information bottleneck for mode m is defined by the parameter set of mode m and the input X. m The output Z of mode m is related to the information bottleneck. m Mutual information between them;

[0057] e m′ The embedding distribution variable represents the mode m′;

[0058] z m Represents the output variable of mode m;

[0059] x m The input variables representing mode m;

[0060] p(e m′ ,z m ;θ m ) represents e under the parameter set of mode m. m′ and z m The joint probability distribution of ;

[0061] p(e m′ ) represents e m′ The probability distribution;

[0062] p(z m ) represents z m The probability distribution;

[0063] p(e m′ ∣z m ) indicates that in z m Under the conditions, e m′ The probability distribution;

[0064] p(x m ,z m ;θ m ) represents x under the parameter set of mode m. m and z m The joint probability distribution of ;

[0065] p(x m ) represents x m The probability distribution;

[0066] p(x m ∣z m ) indicates that in zm the probability distribution of x m under the condition of z

[0067] Further, in practical applications, the objective function is converted into the following empirical objective function through the empirical data distribution:

[0068]

[0069] wherein:

[0070] denotes the empirical data distribution variational optimization objective function under the parameter set of modal m;

[0071] N denotes the total number of input variables;

[0072] n denotes the input variable serial number;

[0073] denotes the nth input variable of modal m;

[0074] denotes the probability distribution of the output variable z under the condition of the nth input variable of modal m m ;

[0075] q(e m′ | z m ) denotes the probability distribution of the embedding distribution variable e m′ of modal m' under the condition of the output variable z m ;

[0076] denotes the KL divergence function of the above two distributions;

[0077] β m denotes the compression factor of modal m;

[0078] g m is defined as the mapping function after the bottleneck layer of the visual language pre-training model for modal m, and the final embedding of each modal is normalized in the embedding dimension for each evaluation point; for the normalized g m (z m ) and e m′ , the logarithmic form of the Gaussian probability density q(f m′ (x m′ ) | g m (z m )) is simplified to be proportional to the cosine similarity between f m′ (x m′ ) and g m (z m ), so as to obtain the following final optimization objective:

[0079]

[0080] In the formula:

[0081] This represents the variational final optimization objective function for the empirical data distribution under the parameter set of mode m;

[0082] S cosine (·,·) denotes the cosine similarity function;

[0083] S cosine (e m′ ,g m (z m )) represents the embedding of mode m′. m′ With mapping output g m (z m The cosine similarity of ).

[0084] The present invention also provides an apparatus for constructing a conceptual bottleneck model based on variational information bottleneck guidance, comprising a memory and a processor, wherein the memory is used to store a computer program; and the processor is used to execute the computer program and, when executing the computer program, implement the steps of the conceptual bottleneck model construction method based on variational information bottleneck guidance as described above.

[0085] The advantages and positive effects of this invention are:

[0086] This invention proposes a concept bottleneck model construction method based on variational information bottleneck guidance. By introducing the thinking chain technique, it prompts the visual language model and the large language model to generate a concept pool with stronger relevance and richer visual attribute information. The variational information bottleneck is used to perform in-depth alignment of CLIP-encoded image text embeddings, and the supplementary concept bottleneck is used to further improve the accuracy of classification.

[0087] This invention applies information bottleneck theory to the concept selection process, essentially reducing irrelevant and redundant information in textual descriptions and retaining only key visual concepts, thereby further improving the interpretability of the concept bottleneck model. Through information compression, the model retains concepts useful for image classification while removing irrelevant parts of the textual description, thus preserving a more concise and effective concept representation.

[0088] The present application proposes a multi-modal information bottleneck optimization target, by introducing embedding similarity into the information bottleneck framework, the present application can effectively establish semantic alignment in image-text pairs, and further improve the attribution and interpretability of the model. This optimization target combines the advantages of variational inference and self-supervised learning, so that the model can learn interpretable latent representations without task-specific labels, improving the flexibility and robustness of multi-modal attribution analysis.

[0089] In addition, the present application proposes a new index for evaluating the interpretability efficiency of the concept bottleneck model, which is tested in various experimental settings. Compared with the most popular concept bottleneck model method, the model of the present application provides better interpretability while achieving similar or even better classification accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0090] Fig. 1 A concept bottleneck model structure diagram based on variational information bottleneck guidance according to the present application;

[0091] Fig. 2 An Instruct BLIP prompt template based on the thinking chain technology mentioned in embodiment 1 of the present application;

[0092] Fig. 3 A GPT-3.5Turbo prompt template based on the thinking chain technology mentioned in embodiment 1 of the present application.

[0093] In the figure:

[0094] f represents the visual features obtained by encoding using the CLIP image encoder.

[0095] α represents a weight coefficient; it corresponds to the output fusion weight of the basic concept classifier.

[0096] b represents the basic concept bottleneck.

[0097] c represents the supplementary concept bottleneck.

[0098] Φ I represents the CLIP image encoder.

[0099] Φ T represents the CLIP text encoder. DETAILED DESCRIPTION

[0100] The present application will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments, it should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application.

[0101] The Chinese interpretation of the following English words, phrases and abbreviations is as follows:

[0102] Instruct BLIP: A multimodal pre-training model that combines visual and language understanding capabilities based on instructions.

[0103] GPT-3.5Turbo: A large language model for text generation, dialogue interaction, and multi-task processing.

[0104] CLIP: A contrastive language-image pre-training model that enables cross-modal understanding through image-text contrastive learning.

[0105] CBM: A concept bottleneck model that improves model interpretability by explicitly learning a "concept layer".

[0106] ImageNet: A large-scale image classification dataset containing 14 million labeled images covering over 20,000 classes. CIFAR-10: A small image classification benchmark dataset containing 10 classes of objects.

[0107] CIFAR-100: A small image classification benchmark dataset containing 100 classes of objects.

[0108] CUB: California Institute of Technology-California University of San Diego bird dataset for fine-grained image classification.

[0109] Food-101: Food classification dataset (101 classes) focusing on specific domain image recognition tasks.

[0110] Flower-102: Flower classification dataset (102 classes) focusing on specific domain image recognition tasks.

[0111] ImageNet-100: A subset of ImageNet, selecting 100 classes for lightweight experiments or resource-constrained scenarios. PCBM: A probabilistic concept bottleneck model that introduces probabilistic concept representations.

[0112] LF-CBM: A concept bottleneck model that does not require human-annotated concept labels.

[0113] LaBo: A concept bottleneck model based on large language model-generated concept and sub-module function screening.

[0114] LM4CV: A concept bottleneck model that learns a small number of visual attributes for image classification.

[0115] ConceptNet: A common-sense knowledge graph containing a large number of concepts.

[0116] Please refer to Figs. 1 to 3 , a concept bottleneck model construction method based on variational information bottleneck guidance, comprising the following method steps:

[0117] Step 1, build a concept bottleneck model according to the following structure: make the concept bottleneck model include a visual language model, a large language model, a variational information bottleneck model A, a variational information bottleneck model B, a text encoder A of CLIP, a text encoder B of CLIP, an image encoder of CLIP, a point multiplier A, a point multiplier B, a basic concept classifier, a supplementary concept classifier, and a classification fusion device; make the visual language model, the variational information bottleneck model A, and the text encoder A of CLIP connected in sequence; make the large language model, the variational information bottleneck model B, and the text encoder B of CLIP connected in sequence, and connect the output ends of the text encoder A of CLIP and the image encoder of CLIP to the two input ends of the point multiplier A; connect the output ends of the text encoder B of CLIP and the image encoder of CLIP to the two input ends of the point multiplier B; connect the output end of the point multiplier A to the input end of the basic concept classifier, connect the output end of the point multiplier B to the input end of the supplementary concept classifier, and connect the output ends of the basic concept classifier and the supplementary concept classifier to the input end of the classification fusion device;

[0118] Step 2, input image data into the visual language model, and use the visual language model to generate a basic concept pool directly related to the content of the input image; input prompt information into the large language model, and use the large language model to generate a descriptive supplementary concept pool covering the diversified attribute features of the target category;

[0119] Step 3, filter out core basic concepts with higher relevance from the basic concept pool through the variational information bottleneck model A, and filter out core supplementary concepts with higher relevance from the supplementary concept pool through the variational information bottleneck model B, and the core basic concepts and the core supplementary concepts are used to construct a concept bottleneck;

[0120] Step 4, process the core basic concepts and the core supplementary concepts through the text encoder A and B of CLIP respectively to obtain basic concept embeddings and supplementary concept embeddings respectively; input the image data into the image encoder of CLIP, process the image data through the image encoder of CLIP to obtain image embeddings; perform point multiplication on the basic concept embeddings and the image embeddings through the point multiplier A, and input the point multiplication result into the basic concept classifier; perform point multiplication on the supplementary concept embeddings and the image embeddings through the point multiplier B, and input the point multiplication result into the supplementary concept classifier; fuse the outputs of the basic concept classifier and the supplementary concept classifier through the classification fusion device to obtain a comprehensive classification prediction result;

[0121] Step 5, evaluate the concept bottleneck model using an explainability efficiency index to balance the classification accuracy and the efficiency of concept explanation.

[0122] Preferably, in step 2, the visual language model can adopt the Instruct BLIP model; the Instruct BLIP model extracts multiple visual features from the image; a thought chain prompt template can be set to guide the Instruct BLIP model to generate concepts related to the image from multiple angles and multiple levels, and finally form a fine-grained basic concept pool.

[0123] The visual language model Instruct BLIP is used to perform a zero-shot image description generation task to generate a basic concept pool based on an input image, wherein the generated concepts can accurately reflect the image content and provide fine-grained visual attribute information related to the image content; a guide template is designed using the thought chain technology to refine the generation process of Instruct BLIP, thereby improving the accuracy and diversity of the generated concept descriptions, especially in fine-grained image classification tasks.

[0124] Instruct BLIP can extract rich visual features from images by combining visual information and language information, and generate related concepts through mutual generation with natural language. These generated concepts can be basic visual attributes that describe images, such as color, shape, texture, etc., or high-level semantic concepts related to image content, such as object categories, scene descriptions, etc.

[0125] To improve the fine-grained description of visual attributes, the thought chain technology is introduced. The thought chain technology guides the model to reason step by step, thereby generating more refined concepts. When generating the prompt template, considering the complexity and diversity of the input image, the thought chain guides the Instruct BLIP model to generate concepts related to the image from multiple angles and multiple levels, and finally forms a fine-grained concept pool.

[0126] In this way, a concept pool closely related to the input image can be obtained, which is refined to each visual attribute, thereby providing more accurate and comprehensive basis for subsequent concept prediction and model explanation.

[0127] Preferably, in step 2, the large language model can adopt the GPT-3.5Turbo model; a thought chain prompt template can be set, and the GPT-3.5Turbo model generates diverse text descriptions according to the prompts given by the thought chain prompt template to ensure comprehensive attribute features of the target category.

[0128] GPT-3.5Turbo has strong text generation capabilities and can generate diverse textual descriptions based on given prompts. To cover the diverse attribute features of the target category, the supplementary concept pool generated by GPT-3.5Turbo will contain multiple descriptive concepts related to the target category, which can involve various attributes of the target object such as structural positioning, local contrast, quantitative description, etc.

[0129] By designing a prompt template based on the thought chain technology, GPT-3.5Turbo can generate diverse supplementary concepts for each category, thereby enriching the final concept pool and ensuring comprehensive attribute features of the target category. This generation method of the supplementary concept pool not only improves the interpretability of the model, but also ensures that the model can fully capture the diverse features of the target category, thereby improving the accuracy and credibility of concept prediction.

[0130] Using the large language model GPT-3.5Turbo to generate a supplementary concept pool that includes diverse descriptive concepts of image categories, the designed thought chain prompt template ensures that the generated concepts accurately describe the image categories and avoid generating noise information unrelated to visual attributes.

[0131] Preferably, in step 3, the variational information bottleneck models A and B can align the visual concepts in the text description with the image regions, and do not rely on concept labels; using the information bottleneck of the visual language variant for feature attribution analysis, the importance scores of each concept in the text description and the image region are calculated, and then the comprehensive scores of each concept can be obtained by accumulating the frequency of concept occurrence and the professional degree of concept; the top K concepts in the basic concept pool can be selected according to the order of comprehensive scores to construct the core basic concept; and the top V concepts of the supplementary concept pool are used to construct the core supplementary concept.

[0132] The present application uses the proposed concept screening method based on variational information bottleneck to extract the top K highest scoring text concepts from the description generated by Instruct BLIP, which are called core basic concepts. For the mapping of core basic concepts to image categories, image embeddings and concept embeddings are extracted using the image encoder and text encoder of CLIP, respectively, and the dot product is calculated to obtain a K-dimensional concept bottleneck. Then, the category scores are obtained by learning the basic concept classifier.

[0133] At the same time, the generation ability of the large language model is utilized, and more abundant, detailed and diverse text descriptions are utilized as supplementary information to construct a supplementary concept bottleneck. By using the same concept screening method, the top V highest scoring text concepts are selected as the core supplementary concepts, and V vectors are obtained by encoding the text encoder of CLIP. The dimensions of these vectors are consistent with the existing concept embedding, and then the visual representation is mapped to these vectors to obtain the supplementary concepts. The prediction results of the original concepts are supplemented by using these concepts.

[0134] Preferably, in step 4, Ψ b (b) is the output of the base concept classifier, Ψ c (c) is the output of the supplementary concept classifier; the classification fusioner can fuse the outputs of the base concept classifier and the supplementary concept classifier according to the following formula:

[0135]

[0136] In the formula:

[0137] represents the prediction result;

[0138] α represents a weight coefficient;

[0139] b represents the base concept bottleneck;

[0140] c represents the supplementary concept bottleneck;

[0141] f represents the visual feature obtained by encoding the CLIP image encoder;

[0142] W represents the base concept text feature obtained by the CLIP text encoder;

[0143] U represents the supplementary concept text feature obtained by the CLIP text encoder;

[0144] Ψ b () represents the base concept linear classifier function;

[0145] Ψ c () represents the supplementary concept linear classifier function.

[0146] Preferably, a double-branch concept bottleneck network is constructed by the base concept classifier and the supplementary concept classifier. In order to maximize the accuracy when only using the base concept, the optimization objective of the base concept classifier can be constructed as follows:

[0147]

[0148] On the other hand, the present application aims to improve the performance of image classification by introducing a complementary concept, which can be solved by the branch of the complementary concept, and the optimization objective of the complementary concept classifier can be constructed as follows:

[0149]

[0150] In the formula:

[0151] Φ I () represents the image encoder function of CLIP;

[0152] t represents the input image;

[0153] y represents the class label;

[0154] L() represents the cross-entropy loss function;

[0155] Ω() represents the complexity measure function of the regularization model;

[0156] λ represents the regularization strength;

[0157] represents the expectation of the expression;

[0158] Ψ b represents the base concept linear classifier for classification using the base concept;

[0159] Ψ c represents the complementary concept linear classifier for classification using the complementary concept;

[0160] When training the base concept classifier and the complementary concept classifier, forward and backward propagation are performed twice in each batch to update the parameters; in the first propagation, only the base concept classifier is optimized; in the second propagation, the parameters of the base concept classifier are fixed, and the complementary concept classifier is optimized; in order to avoid the influence of extreme values of the complementary concept on the results, the cosine similarity is used instead of the projection distance as the concept bottleneck, and the activation value of each concept is standardized to have a mean of 0 and a standard deviation of 1, so as to accelerate the convergence.

[0161] Preferably, in step 5, the calculation formula of the interpretability efficiency index can be as follows:

[0162]

[0163] In the formula:

[0164] IE represents interpretability efficiency;

[0165] Acc represents classification accuracy;

[0166] i represents the concept number;

[0167] j represents the category number;

[0168] W F represents a linear layer for mapping concepts to categories;

[0169] (W F ) ij represents the weight of the i-th concept and the j-th category in the linear layer;

[0170] O{(W F ) ij ≠0} represents the number of non-zero in the linear layer;

[0171] R represents the number of concepts;

[0172] S represents the number of categories;

[0173] represents the average concept length.

[0174] Preferably, in step 3, for the variational information bottleneck models A, B, a variational approximation method can be used to optimize the objective functions of both, and the KL divergence between the image and the text can be used to adjust the parameters of both, combined with the pre-trained text encoder A, B of CLIP, to assign importance scores to image regions and text concepts.

[0175] The present application applies the information bottleneck theory to the process of concept filtering, which essentially reduces irrelevant and redundant information in the text description and only retains key visual concepts, thereby further improving the explainability of the concept bottleneck model. Through information compression, the model removes irrelevant parts in the text description while retaining concepts useful for image classification, thereby retaining more concise and effective concept representations.

[0176] In the process of concept filtering, the original formula of the information bottleneck theory is not directly used, and the present application only uses the generated text and the given image to learn the latent concept representation without relying on concept labels. This method extracts more accurate and concise concepts through the information bottleneck theory and uses them for the construction of the concept bottleneck. In order to construct a variational information bottleneck model for a multi-modal pre-training model, a loss function similar to the optimization objective of the self-supervised method is constructed, which only relies on the given image and the description of the concepts contained in the image to perform representation learning between image regions and text concepts.

[0177] In multi-modal data, there is a reasonable prior for information relevance: if the image and text inputs are related to each other (e.g., the text describes the image content), then a better image encoding should contain information about the text, and vice versa. Based on this intuitive understanding, preferably, for the variational information bottleneck models A, B, the following objective function can be set:

[0178]

[0179] where:

[0180] m denotes an image or text modality;

[0181] m' denotes the complement of modality m;

[0182] Θ m denotes a set of parameters for modality m;

[0183] denotes an optimization objective under the set of parameters for modality m;

[0184] Z m denotes the output of the information bottleneck to modality m;

[0185] E m′ denotes the embedding of modality m';

[0186] X m denotes the input of the information bottleneck to modality m;

[0187] β denotes a scaling factor;

[0188] I(Z m ,E m′ ; Θ m ) denotes the mutual information between the embedding E m′ of modality m' and the output Z m of the information bottleneck to modality m under the set of parameters for modality m;

[0189] I(Z m ,X m ; Θ m ) denotes the mutual information between the input X m of the information bottleneck to modality m and the output Z m of the information bottleneck to modality m under the set of parameters for modality m;

[0190] e m′ denotes an embedding distribution variable for modality m';

[0191] z m denotes an output variable for modality m;

[0192] x m denotes an input variable for modality m;

[0193] p(e m′ ,z m ; Θ m ) denotes the joint probability distribution of e m′ and z m under the set of parameters for modality m;

[0194] p(e m′ ) represents the probability distribution of e m′ ;

[0195] p(z m ) represents the probability distribution of z m ;

[0196] p(e m′ ∣z m ) represents the probability distribution of e m′ under the condition of z m ;

[0197] p(x m ,z m ;θ m ) represents the joint probability distribution of x m and z m under the parameter set of the modal m;

[0198] p(x m ) represents the probability distribution of x m ;

[0199] p(x m ∣z m ) represents the probability distribution of x m under the condition of z m .

[0200] The present application will use the information bottleneck principle of this visual language variant for attribution analysis. Through this optimization goal, the information association between different modalities is constructed, ensuring that the model can capture the image content while also conveying the semantic connection with the text modal, providing a solid foundation for the subsequent construction of the concept bottleneck.

[0201] Preferably, in practical applications, the objective function can be converted into the following empirical objective function through the empirical data distribution:

[0202]

[0203] In the formula:

[0204] p(x m ) represents the probability distribution of x m ;

[0205] N represents the total number of input variables;

[0206] n represents the input variable sequence number;

[0207] x m represents the nth input variable of the modal m;

[0208] denotes the probability distribution of the output variable z under the condition that the n-th input variable of modality m m ;

[0209] q(e m′ | z m ) denotes the probability distribution of the embedding distribution variable e m′ of modality m' under the condition of the output variable z m ;

[0210] denotes the KL divergence function of the above two distributions;

[0211] β m denotes the compression factor of modality m;

[0212] g m may be defined as the mapping function after the bottleneck layer of the visual language pre-training model for modality m, and the final embedding of each modality is normalized in the embedding dimension for each evaluation point; for the normalized g m (z m ) and e m′ , the logarithmic form of the Gaussian probability density q(f m′ (x m′ ) | g m (z m )) is simplified to be proportional to the cosine similarity between f m′ (x m′ ) and g m (z m ), so that the following final optimization objective can be obtained:

[0213]

[0214] In the formula:

[0215] denotes the empirical data distribution variational final optimization objective function under the parameter set of modality m;

[0216] S cosine (·,·) denotes a cosine similarity function;

[0217] S cosine (e m′ , g m (z m )) denotes the cosine similarity of the embedding e m′ of modality m' and the mapping output g m (z m ).

[0218] Under the above optimization goal, by introducing the embedding similarity into the information bottleneck framework, the present application can effectively establish semantic alignment in the image-text pair, and further improve the attribution and interpretability of the model. This optimization goal combines the advantages of variational inference and self-supervised learning, so that the model can learn a latent representation with interpretability without task-specific labels, improving the flexibility and robustness of multi-modal attribution analysis.

[0219] The present application also provides a device for constructing a concept bottleneck model based on variational information bottleneck guidance, comprising a memory and a processor, the memory being used to store a computer program; the processor being used to execute the computer program and realize the steps of the method for constructing a concept bottleneck model based on variational information bottleneck guidance as described above when executing the computer program.

[0220] The working process and working principle of the present application will be further described below with several preferred embodiments of the present application:

[0221] The present application prompts the visual language model and the large language model to generate a concept pool with stronger relevance and richer visual attribute information by introducing the thinking chain technology, aligns the CLIP encoded image-text embedding in depth by using the variational information bottleneck, and further improves the classification accuracy by using the supplementary concept bottleneck.

[0222] In this embodiment, six public data sets, CIFAR-10, CIFAR-100, CUB, Food-101, Flower-102 and ImageNet-100, are selected as the reference basis for image classification accuracy. Image preprocessing includes scaling the input image to a maximum side length of 256 pixels, randomly cropping to 224x224 pixels, normalizing based on the mean and standard deviation of ImageNet, and applying data enhancement methods such as random horizontal flipping, and the division of training set and test set is based on the official split file given by the data set.

[0223] Please refer to Fig. 1 The present application proposes a method for constructing a concept bottleneck model based on variational information bottleneck guidance, comprising the following method steps:

[0224] Step 1, build the concept bottleneck model according to the following structure: make the concept bottleneck model include a visual language model, a large language model, a variational information bottleneck model A, a variational information bottleneck model B, a text encoder A of CLIP, a text encoder B of CLIP, an image encoder of CLIP, a point multiplier A, a point multiplier B, a basic concept classifier, a supplementary concept classifier, and a classification fusion device; make the visual language model, the variational information bottleneck model A, and the text encoder A of CLIP be connected in sequence; make the large language model, the variational information bottleneck model B, and the text encoder B of CLIP be connected in sequence, and connect the output ends of the text encoder A of CLIP and the image encoder of CLIP to the two input ends of the point multiplier A; connect the output ends of the text encoder B of CLIP and the image encoder of CLIP to the two input ends of the point multiplier B; connect the output end of the point multiplier A to the input end of the basic concept classifier, connect the output end of the point multiplier B to the input end of the supplementary concept classifier, and connect the output ends of the basic concept classifier and the supplementary concept classifier to the input end of the classification fusion device; input image data into the visual language model and the image encoder of CLIP, and output a predicted classification result from the classification fusion device.

[0225] Step 2, use the visual language model Instruct BLIP to generate a basic concept pool directly related to the input image content, and optimize the generation of a prompt template through a thinking chain technology to enhance the granularity of visual attribute description:

[0226] For the construction of the basic concept pool, the present application hopes to use Instruct BLIP to generate text concepts with obvious correspondence to image regions, and at the same time, in order to obtain more rich visual attributes, the present application designs a guided improvement of detail richness. At the same time, in order to guide the visual language model to discover visual attributes different from other similar categories in a fine-grained dataset, a reasoning process of local contrast with similar categories is further added. For specific Instruct BLIP prompt templates based on the thinking chain technology, please refer to Fig. 2 .

[0227] Use the large language model GPT-3.5Turbo to generate a descriptive supplementary concept pool covering the diversified attribute features of the target category:

[0228] For the construction of the supplementary concept pool, the present application uses GPT-3.5Turbo to generate more rich descriptive sentences for specific image categories, while limiting the appearance of texts unrelated to visual attributes, such as specific bird calls, behavioral characteristics, etc., to prevent the introduction of noise into the concept bottleneck. For specific GPT-3.5Turbo prompt templates based on the thinking chain technology, please refer to Fig. 3 .

[0229] Step 3, based on the variational information bottleneck principle, calculate the concept importance score through cross-modal aligned image-text embedding space, and filter high-relevance concepts.

[0230] The information bottleneck principle provides a framework for finding compressed representations of neural network models. To obtain a latent representation that can reflect the most relevant information of the input data, the information bottleneck principle aims to express the input source X through a stochastic latent representation Z defined by a parameterized encoder p Z|X (z | x; θ) that can maximize the information about the target Y while limiting the mutual information between the latent representation Z and the input X.

[0231] This principle can be expressed as the following optimization problem:

[0232]

[0233] In the optimization problem expression:

[0234] X represents the input source;

[0235] Y represents the target;

[0236] Z represents the latent representation;

[0237] θ represents the specific parameter representation under the optimization problem;

[0238] I(·,·; θ) represents the mutual information function;

[0239] I(Z,Y; θ) represents the mutual information between the latent representation Z and the target Y under the parameter θ;

[0240] I(Z,X; θ) represents the mutual information between the latent representation Z and the input source X under the parameter θ;

[0241] represents a compression constraint for controlling the upper limit of the mutual information between Z and X.

[0242] where I(·,·; θ) represents the mutual information function, and is a compression constraint for controlling the upper limit of the mutual information between Z and X to ensure that the model only retains the key information in the input.

[0243] This optimization problem can be equivalently transformed into maximizing the following objective function:

[0244]

[0245] In the objective function expression:

[0246] θ represents the specific parameter representation under the optimization problem;

[0247] represents the optimal objective at parameter θ;

[0248] β represents a scaling factor to trade-off between maximizing the information of Z about the target Y and maximizing the compressibility of Z about the input X, when β is larger, the optimization process will pay more attention to information compression; while when β is smaller, the model pays more attention to the relevance of the latent representation to the target Y;

[0249] I(·,·;θ) represents the mutual information function;

[0250] I(Z,Y;θ) represents the mutual information of the latent representation Z and the target Y at parameter θ;

[0251] I(Z,X;θ) represents the mutual information of the latent representation Z and the input source X at parameter θ.

[0252] The present application applies the information bottleneck theory to the process of concept filtering, which essentially reduces irrelevant and redundant information in the text description and only retains key visual concepts, thereby further improving the explainability of the concept bottleneck model. Through information compression, the model removes irrelevant parts in the text description while retaining concepts useful for image classification, thereby retaining more concise and effective concept representations.

[0253] In order to construct a variational information bottleneck principle for a multimodal pre-training model like CLIP, a loss function more similar to the self-supervised method optimization objective needs to be designed, which only relies on a given image and a description of the concept contained in the image to perform representation learning between image regions and text concepts. This learning method is essentially different from the supervised learning paradigm in single-modal task settings, for example, for an image of a dog X dog and the corresponding label Y dog ="dog", in a single-modal classification task, the present application can simply maximize I(Y dog ,Z dog ;θ)-βI(X dog ,Z dog ;θ), where Z dog represents the latent representation of X dog . In image-text representation learning, there is usually a text description L dog such as "this is a photo of a dog running in the grass, the dog has four legs, soft fur and a wagging tail", rather than a definite and single label. In this setting, X dog and L dog are inputs, and there is no predefined label.

[0254] To obtain a task-agnostic image-text representation, the present application wishes to simultaneously use both image and text input modalities, and define a variational information bottleneck principle, while the output is tightly dependent on the specific downstream task. This requires redefining the "fitting term" I(Z, Y; θ) in the classical information bottleneck objective to adapt to this image-text representation learning scenario. In multi-modal data, there is a reasonable prior on information relevance: if image and text inputs are related to each other (e.g. the text describes the image content), then a good image encoding should contain information about the text, and vice versa. Based on this intuitive understanding, an information bottleneck objective for visual language can be expressed as:

[0255]

[0256] In the information bottleneck objective expression for visual language:

[0257] m denotes the image or text modality;

[0258] m' denotes the complement of modality m;

[0259] θ m denotes the parameter set for modality m;

[0260] denotes the optimization objective under the parameter set for modality m;

[0261] Z m denotes the output of the information bottleneck for modality m;

[0262] E m′ denotes the embedding of modality m';

[0263] X m denotes the input of the information bottleneck for modality m;

[0264] β denotes a scaling factor;

[0265] I(Z m ,E m′ ; θ m ) denotes the mutual information between the embedding E m′ of modality m' and the output Z m of the information bottleneck for modality m under the parameter set for modality m;

[0266] I(Z m ,X m ; θ m ) denotes the mutual information between the input X m of the information bottleneck for modality m and the output Z m of the information bottleneck for modality m under the parameter set for modality m.

[0267] Through theoretical derivation of the variational information bottleneck feature attribution method, the final optimization objective expression is:

[0268]

[0269] In the expression of the final optimization objective:

[0270] m represents an image or text modality;

[0271] m' represents the complement of modality m;

[0272] θ m represents a set of parameters of modality m;

[0273] represents an empirical data distribution variational optimization objective function under the set of parameters of modality m;

[0274] N represents the total number of input variables;

[0275] n represents the input variable serial number;

[0276] z m represents the output variable of modality m;

[0277] x m represents the input variable of modality m;

[0278] represents the nth input variable of modality m;

[0279] e m′ represents the embedding distribution variable of modality m';

[0280] represents the probability distribution of the output variable z m of modality m under the condition that the nth input variable of modality m is under the set of parameters of modality m;

[0281] S cosine (·,·) represents a cosine similarity function;

[0282] S cosine (e m′ ,g m (z m )) represents the cosine similarity of the embedding e m′ of modality m' and the mapping output g m (z m );

[0283] represents the KL divergence function of the above two distributions;

[0284] β m represents the compression factor of modality m.

[0285] Under this multi-modal information bottleneck optimization goal, by introducing embedding similarity into the information bottleneck framework, the application can effectively establish semantic alignment in image-text pairs, thereby improving the attribution and interpretability of the model. This optimization goal combines the advantages of variational inference and self-supervised learning, enabling the model to learn interpretable representations without task-specific labels, improving the flexibility and robustness of multi-modal attribution analysis.

[0286] Step 4: A double-branch concept bottleneck network is formed by the basic concept classifier and the supplementary concept classifier, and the prediction results of the two are fused by a classification fusioner.

[0287] First, the proposed concept screening method based on variational information bottleneck is used to extract the top K scored text concepts in the description generated by Instruct BLIP, which is called "basic concept". For the mapping of basic concepts to image categories, a method consistent with previous work is used, which extracts image embeddings and concept embeddings using the image encoder and text encoder of CLIP, respectively, calculates the dot product to obtain a K-dimensional concept bottleneck, and then learns a basic concept classifier to obtain the score of the category.

[0288] At the same time, the generation ability of large language models is used to use more rich, detailed and diverse text descriptions as supplementary information to construct a supplementary concept bottleneck. Using the same concept screening method, the top V scored text concepts are selected, and V vectors are obtained by encoding them using the text encoder of CLIP. The dimensions of these vectors are consistent with existing concept embeddings, and then the visual representation is mapped to these vectors to obtain supplementary concepts. The prediction results of the original concepts are supplemented by these concepts as follows:

[0289]

[0290] In the prediction result expression:

[0291] represents the prediction result;

[0292] α represents the weight coefficient;

[0293] b represents the basic concept bottleneck;

[0294] c represents the supplementary concept bottleneck;

[0295] f represents the visual features obtained by encoding using the CLIP image encoder;

[0296] W represents the basic concept text features obtained by using the CLIP text encoder;

[0297] U represents a supplementary concept text feature obtained by using a CLIP text encoder.

[0298] Step 5: balance the classification accuracy and the efficiency of concept explanation by using the interpretability efficiency index optimization model.

[0299] The concept bottleneck model cannot be simply considered from the aspect of classification accuracy. As an interpretable image classification method, the present application also considers the feasibility and effectiveness of explanation. In addition to using classification accuracy as a basic measurement index of model performance, the present application proposes an index for measuring the interpretability efficiency of the concept bottleneck model, which focuses on how many concepts are really involved in the final classification decision and the average number of words of the text concept used for decision making. The calculation formula of the interpretability efficiency index is:

[0300]

[0301] In the formula:

[0302] IE represents interpretability efficiency;

[0303] Acc represents classification accuracy;

[0304] i represents the concept number;

[0305] j represents the class number;

[0306] W F represents a linear layer for mapping concepts to classes;

[0307] (W F ) ij represents the weight of the i-th concept and the j-th class in the linear layer;

[0308] O{(W F ) ij ≠0} represents the number of non-zero values in the linear layer;

[0309] R represents the number of concepts;

[0310] S represents the number of classes;

[0311] represents the average concept length.

[0312] Example 2:

[0313] Based on example 1 but with the difference that the advanced comparison method selected includes PCBM, LF-CBM, LaBo and LM4CV. Among them, PCBM uses the ConceptNet knowledge graph dataset to construct a concept library, which regards the category as a node and includes the surrounding neighbor nodes in the concept library. By expanding multiple levels outward, more concepts can be included, and the visual embedding is mapped into the category through the residual connection; LF-CBM and LaBo use a large language model to generate concepts, and use different concept screening methods to construct a concept bottleneck; LM4CV uses a learning search method to mine a more concise set of concepts in the dataset, greatly reducing the size of the concept bottleneck, making the explanation result more easily understood. The present application trains and tests the above-mentioned methods on the same dataset, and the results are plotted as Table 1.

[0314] As can be seen from Table 1, the concept bottleneck model based on the variational information bottleneck guide proposed by the present application performs well on the image classification task, and shows similar or even better classification accuracy than other methods.

[0315] As can be seen from Table 2, the experimental results of each comparison method in terms of explainability.

[0316] The explainability efficiency evaluation index proposed by the present application is used to measure the conciseness of the explanation output of the concept bottleneck model method. Experiments show that the explainability efficiency of the present application on six datasets reaches the highest level.

[0317] Table 1: Classification accuracy experimental results of each comparison method

[0318]

[0319] Table 2: Explainability efficiency experimental results of each comparison method

[0320]

[0321]

[0322] Table 3 shows the influence of different concept screening methods on classification accuracy, and the experimental results of all sample numbers are the average of the classification accuracy of six datasets. Among them:

[0323] The randomly selected screening method is to randomly select words from the concept pool to construct the concept bottleneck.

[0324] The similarity sorting screening method is to calculate the similarity between the concepts in the concept pool and the category name respectively, and sort them according to the average similarity size, and select the top K concepts to construct the concept bottleneck.

[0325] Submodular function is a method for selecting the best subset in high-dimensional space, and LaBo applies it to the process of concept filtering.

[0326] And the variational information bottleneck is the concept filtering method proposed in the present application, which uses the cross-modal ability of CLIP to strictly align the concept part in image and text description, and suppresses the influence of noise area. It can be seen that the concept filtering method based on variational information bottleneck proposed in the present application has the highest classification accuracy under all settings.

[0327] Table 3: Classification accuracy based on different concept filtering methods

[0328]

[0329] The above visual language model, large language model, variational information bottleneck model A, variational information bottleneck model B, text encoder A of CLIP, text encoder B of CLIP, image encoder of CLIP, dot multiplier A, dot multiplier B, basic concept classifier, supplementary concept classifier, classification fusioner, variational approximation method, KL divergence, GPT-3.5Turbo model, InstructBLIP model, thinking chain prompt template and other functional modules, systems and algorithms can adopt the applicable functional modules, systems and algorithms in the prior art, or adopt the applicable functional modules, systems and algorithms in the prior art and adopt conventional technical means to construct.

[0330] It should be noted that the above description is only a preferred embodiment of the present application, and does not constitute a limitation on the protection scope of the present application. Any skilled person in the art can make equivalent replacements or changes within the scope of the present application according to the technical solutions and inventive concepts of the present application, which should be considered as included in the protection scope of the present application.

Claims

1. A method for constructing a concept bottleneck model based on variational information bottleneck guiding, characterized in that, The method comprises the following steps: Step 1: build a concept bottleneck model according to the following structure: make the concept bottleneck model comprise a visual language model, a large language model, a variational information bottleneck model A, a variational information bottleneck model B, a text encoder A of CLIP, a text encoder B of CLIP, an image encoder of CLIP, a point multiplier A, a point multiplier B, a basic concept classifier, a supplementary concept classifier and a classification fusion device; make the visual language model, the variational information bottleneck model A and the text encoder A of CLIP be connected in sequence; make the large language model, the variational information bottleneck model B and the text encoder B of CLIP be connected in sequence, and make the output ends of the text encoder A of CLIP and the image encoder of CLIP be connected to the two input ends of the point multiplier A in correspondence; make the output ends of the text encoder B of CLIP and the image encoder of CLIP be connected to the two input ends of the point multiplier B in correspondence; make the output end of the point multiplier A be connected to the input end of the basic concept classifier, the output end of the point multiplier B be connected to the input end of the supplementary concept classifier, and the output ends of the basic concept classifier and the supplementary concept classifier be connected to the input end of the classification fusion device; Step 2: input image data into the visual language model, and generate a basic concept pool directly related to the content of the input image by using the visual language model; make the large language model input prompt information, and generate a descriptive supplementary concept pool covering the diversified attribute features of the target category by using the large language model; Step 3: filter out core basic concepts with higher relevance from the basic concept pool through the variational information bottleneck model A, and filter out core supplementary concepts with higher relevance from the supplementary concept pool through the variational information bottleneck model B, and the core basic concepts and the core supplementary concepts are used to construct a concept bottleneck; Step 4: the core basic concepts and the core supplementary concepts are respectively processed through the text encoder A and the text encoder B of CLIP to obtain corresponding basic concept embeddings and supplementary concept embeddings; the image data is input into the image encoder of CLIP, and the image embeddings are obtained by processing the image data through the image encoder of CLIP; the basic concept embeddings and the image embeddings are multiplied by the point multiplier A, and the multiplication results are input into the basic concept classifier; the supplementary concept embeddings and the image embeddings are multiplied by the point multiplier B, and the multiplication results are input into the supplementary concept classifier; the outputs of the basic concept classifier and the supplementary concept classifier are fused by the classification fusion device to obtain a comprehensive classification prediction result; Step 5: the concept bottleneck model is evaluated by using an explainable efficiency index to balance the classification accuracy and the efficiency of concept explanation. 2.The method of claim 1, wherein, In step 2, a thinking chain prompt template is set, and the visual language model adopts an Instruct BLIP model; a plurality of visual features are extracted from the image by the Instruct BLIP model; the thinking chain prompt template is used to guide the Instruct BLIP model to generate concepts related to the image from multiple angles and multiple levels to finally form a fine-grained basic concept pool. The large language model adopts a GPT-3.5Turbo model; the GPT-3.5Turbo model generates diversified text descriptions according to the prompts given by the thinking chain prompt template, and ensures comprehensive attribute features of the target category. 3.The method of claim 1, wherein, In step 3, the visual concepts in the text description are aligned with the image regions by using the variational information bottleneck models A and B, and are independent of the concept labels; the feature attribution analysis is performed by using the visual language variant information bottleneck, the importance scores of the concepts in the text description and the image regions are calculated, the comprehensive scores of the concepts are obtained according to the frequency of the concept and the professional degree of the concept, and the first K concepts in the basic concept pool are screened according to the order of the comprehensive scores to construct the core basic concept; the first V concepts in the supplementary concept pool are used to construct the core supplementary concept. 4.The method of claim 1, wherein, In step 4, let Ψ b (b) is the output of the base concept classifier, Ψ c (c) is the output of the complementary concept classifier; the classification fuser fuses the outputs of both the base concept classifier and the complementary concept classifier according to the following formula: In the formula: represents the prediction result; Alpha represents a weight coefficient; B represents a basic concept bottleneck; C represents a supplementary concept bottleneck; F represents a visual feature obtained by using a CLIP image encoder; W represents a basic concept text feature obtained by using a CLIP text encoder; U represents a supplementary concept text feature obtained by using a CLIP text encoder; Ψ b () denotes the base concept linear classifier function; Ψ c () denotes the supplementary concept linear classifier function. 5.The method of constructing a conceptual bottleneck model based on variational information bottleneck guidance according to claim 1, wherein, When training the basic concept classifier and the supplementary concept classifier, forward and backward propagation are performed twice in each batch to update the parameters; in the first propagation, only the basic concept classifier is optimized; in the second propagation, the parameters of the basic concept classifier are fixed, and the supplementary concept classifier is optimized; in order to avoid the influence of extreme values of the supplementary concept on the result, the cosine similarity is used instead of the projection distance as the concept bottleneck, and the activation value of each concept is standardized to have a mean value of 0 and a standard deviation of 1, so as to accelerate the convergence. 6.The method of constructing a conceptual bottleneck model based on variational information bottleneck guidance according to claim 1, wherein, In step 5, the calculation formula of the interpretability efficiency index is as follows: In the formula: IE represents interpretability efficiency; Acc represents classification accuracy; I represents a concept serial number; J represents a category serial number; W F denotes a linear layer for mapping the concepts to categories; (W F ) ij represents the weight of the i-th concept and the j-th category in the linear layer; O{(W F ) ij ≠0} represents the number of non-zero in the linear layer; R represents the number of concepts; S represents the number of categories; represents the average concept length.

7. The method of claim 1, wherein the method is based on a variational information bottleneck guiding a concept bottleneck model construction. In step 3, for the variational information bottleneck models A and B, the variational approximation method is used to optimize the objective functions of the two models, the KL divergence between the image and the text is used to adjust the parameters of the two models, and the pre-trained text encoders A and B of the CLIP are combined to assign importance scores to the image regions and the text concepts.

8. The method of claim 7, wherein the method further comprises: For the variational information bottleneck models A and B, the following objective function is set: In the formula: M represents an image or a text mode; M' represents a complement of the mode m; θ m denotes a parameter set of the modality m; represents the optimization objective under the set of parameters of modality m; Z m represents an information bottleneck on the output of modality m; E m′ represents the embedding of the modality m'; X m denotes the information bottleneck on the input of modality m; Beta represents a scaling factor; I(Z m ,E m′ ;θ m ) denotes the mutual information between the embedding E m′ of the modality m' and the output Z m of the modality m to the information bottleneck with respect to the parameter set of the modality m. I(Z m ,X m ;θ m The information bottleneck for mode m is defined by the parameter set of mode m and the input X. m The output Z of mode m is related to the information bottleneck. m Mutual information between them; e m′ embbeded distribution variable representing modality m'; z m output variable representing the modality m; x m input variable representing the modality m; p(e m′ ,z m ; θ m ) denotes the joint probability distribution of e m′ and z m under the parameter set of the modality m; p(e m′ ) denotes the probability distribution of e m′ ; p(z m ) denotes the probability distribution of z m ; p(e m′ | z m ) represents the probability distribution of e m under the condition of z m′ . p(x m ,z m ; θ m ) represents the joint probability distribution of x m and z m under the parameter set of modality m; p(x m ) denotes the probability distribution of x m ; p(x m ∣z m ) represents the probability distribution of x m under the condition of z m .

9. The method of claim 8, wherein the method further comprises: In actual application, the objective function is converted into the following empirical objective function through empirical data distribution: In the formula: represents an empirical data distribution variational optimization objective function under a parameter set of modality m; N represents the total number of input variables; N represents the serial number of input variables; denotes the n-th input variable of the modality m; denotes the probability distribution of the output variable z under the condition that the n-th input variable of the modality m m under the parameter set of the modality m; q(e m′ |z m ) denotes the probability distribution of the embedding distribution variable e m of the modal m' under the condition that the output variable z m′ is observed. KL divergence function representing the above two distributions; β m denotes the compression factor of the modality m; Definition g m is the mapping function for the bottleneck layer of the vision language pre-trained model for modality m, and for each evaluation point, the final embeddings of each modality are normalized on the embedding dimension; for the normalized g m (z m ) and e m′ , the log-form of the Gaussian probability density q(f m′ (x m′ ) | g m (z m )) is simplified to be proportional to the cosine similarity between f m′ (x m′ ) and g m (z m ), resulting in the following final optimization objective: In the formula: represents the empirical data distribution variational final optimization objective function under the parameter set of modality m; S cosine (·,·) denotes the cosine similarity function; S cosine (e m′ ,g m (z m )) represents the cosine similarity of the embedding e m′ and the mapping output g m (z m ).

10. A device for constructing a concept bottleneck model based on variational information bottleneck guiding, comprising a memory and a processor, characterized in that, The memory is used to store a computer program; the processor is used to execute the computer program and implement the concept bottleneck model construction method based on the variational information bottleneck guidance according to any one of claims 1 to 9 when the computer program is executed.

Images