Image generation quality evaluation method of semantic evidence learning

Abstract

The application belongs to the technical field of image information processing, and discloses a semantic evidence learning image generation quality evaluation method. A semantic correlation extraction network is constructed, a CLIP pre-training model is used to extract local features of images and texts, and a semantic correlation matrix is constructed according to the local features to quantify the correlation strength between all local feature pairs. The matrix is aggregated into a semantic strength vector through a pooling operation. A semantic evidence learning network is constructed, the semantic strength vector is mapped into an evidence vector, and the correlation strength distribution is modeled as a probability distribution on a predefined quality level according to the parameterized Dirichlet distribution. The image quality score is directly derived by calculating the expected value of the distribution. The method can not only quantize the fine-grained correlation of the embedded features of images and texts and reveal the strength distribution, but also probabilistically model and fuse the bidirectional evaluation evidence through the Dirichlet distribution, which significantly improves the interpretability, robustness and discriminant accuracy of the quality evaluation.

Description

Technical Field

[0001] This invention belongs to the field of image information processing technology and relates to an image generation quality assessment method based on semantic evidence learning. Background Technology

[0002] Currently, the technologies related to this invention include two aspects: the first is an image quality assessment method; the second is a quality assessment method for cross-modal semantic association.

[0003] First, traditional image quality assessment methods primarily model pixel-level or structural-level distortions (such as noise, blur, and blockiness) that occur in natural images during acquisition, compression, or transmission. However, for images generated by text-to-image models, the essence of their quality defects lies in semantic inconsistencies (e.g., key objects in the text description are not generated, or the image contains illusory content not described in the text). The underlying signal distortion models upon which traditional methods rely are fundamentally different from the characteristics of such semantic-level distortions, making them difficult to apply directly.

[0004] With the maturation of pre-trained visual-language models, quality assessment methods based on cross-modal semantic association have become the mainstream paradigm. For example, some methods directly utilize pre-trained models (such as CLIP) to calculate the cosine similarity between the global embeddings of images and text (such as the CLIP Score) as a quality score. However, these methods have a fundamental limitation: they compress the rich semantic association information between images and text into a single global scalar score. This process loses fine-grained distributional information about "where the image matches which concept in the text," resulting in evaluation results that fail to reflect the specific patterns and intensity differences in the matches. Therefore, such methods suffer from significant deficiencies in evaluation granularity, interpretability, and reliability when evaluating semantically complex or subtly different generated images.

[0005] In practical applications, the aforementioned global scoring-based methods suffer from coarse-grained evaluation results and limited reliability due to their inability to model the strength distribution of semantic associations. To overcome this fundamental limitation, the core of this invention lies in proposing a semantic evidence learning-based image generation quality assessment framework, aiming to quantify and model the distribution of semantic associations. This framework is mainly divided into two stages: In the first stage, a semantic association extraction module is designed to extract local fine-grained features of images and text using a pre-trained model, and to construct a semantic association matrix to quantify their association strength; in the second stage, a semantic evidence learning module is constructed to transform the association strength into evidence vectors, which are then modeled as a Dirichlet distribution, thereby deriving a continuous and interpretable image quality score. Summary of the Invention

[0006] To address the insufficient modeling of semantic association strength distribution in image generation quality assessment, a semantic evidence learning-based image generation quality assessment framework is proposed. The core of this framework is to capture the association strength between cross-modal features by constructing a semantic association matrix and modeling it as a probability distribution using evidence learning, thereby deriving a quality score with a clear probabilistic interpretation. Specifically, the method includes the following two stages: First, a semantic association extraction network is constructed, using a CLIP pre-trained model to extract local features of images and text, and based on this, a semantic association matrix is ​​constructed to quantify the association strength between all local feature pairs; then, pooling operations are used to aggregate this matrix into a semantic strength vector. Second, a semantic evidence learning network is constructed, mapping the semantic strength vector to an evidence vector, and based on this, a Dirichlet distribution is parameterized, thereby modeling the distribution of association strength as a probability distribution over a predefined quality level; finally, by calculating the expectation value of this distribution, a continuous and interpretable image quality score is directly derived.

[0007] The technical solution of the present invention:

[0008] A method for evaluating the quality of image generation based on semantic evidence learning, comprising the following steps:

[0009] (1) Semantic association extraction network;

[0010] To construct the semantic association matrix, we first obtain the feature representations of the input image and input text in a unified semantic space; then, we use the image encoder and text encoder of the CLIP pre-trained model based on the ViT architecture to extract image features and text features respectively.

[0011] The input image is divided into a series of non-overlapping image patches. Each patch is transformed into a visual label sequence after linear projection, and a learnable [CLS] label is inserted at the beginning of the visual label sequence to form a complete visual sequence. This complete visual sequence is then encoded by a visual Transformer to output a feature set { , , , , },in As a global semantic embedding of the input image, { , , , The local semantic embedding matrix that constitutes different regions of the input image is denoted as . ,in The number of local semantic embedding matrices. The feature dimension of each local semantic embedding;

[0012] The input text is first converted into a word sequence through lexicalization, and [SOS] and [EOS] markers are added to the start and end positions of the word sequence, respectively, to form a complete text sequence. This complete text sequence is then encoded using a Text Transformer to output a feature set { , , , , },in As a global semantic embedding describing the input text, { , , , The local semantic embedding matrix of different words in the input text is denoted as} ,in This represents the number of original lexical units obtained after lexicalization of the input text. The feature dimension of each local semantic embedding;

[0013] To quantify the fine-grained semantic association strength between the input image and the input text, local semantic embedding matrices of different regions of the input image are used. Local semantic embedding matrix of different words from the input text Normalization is performed, and the cosine similarity between all normalized feature pairs is calculated to construct a complete semantic association matrix. ;make and These represent the local semantic embedding matrices of different regions of the input image. Local semantic embedding matrix of different words from the input text The row vectors in the array, where, , Then the semantic association matrix Each element is defined as the dot product of the corresponding normalized vectors, and their cosine similarity is:

[0014]

[0015] in, Characterizing the first The image embedding and the first The semantic association strength between text embeddings, whose value is located in The larger the value in the interval, the stronger the semantic consistency; semantic association matrix It fully encodes all fine-grained semantic correspondences between the input image and the input text, laying the foundation for subsequent extraction of association strength distribution features.

[0016] To obtain from a two-dimensional semantic association matrix More representative intensity information is extracted and aggregated to obtain a low-dimensional semantic intensity vector; a max-pooling strategy is employed to extract the intensity along the semantic association matrix. The rows (image embedding dimension) and columns (text embedding dimension) are aggregated. This operation evaluates the strongest semantic support that each image embedding can obtain from all text embeddings, and the most explicit visual representation that each text embedding can obtain from all image embeddings. The aggregation generates two vectors: an image orientation semantic strength vector. semantic strength vector of text direction The elements of the two are as follows:

[0017]

[0018]

[0019] The resulting image orientation semantic intensity vector semantic strength vector of text direction The key association strength information based on bidirectional embedding evaluation is integrated and will be used as input to the subsequent semantic evidence learning network.

[0020] (2) Semantic evidence learning network;

[0021] Semantic evidence learning networks process the semantic intensity vectors of image orientations. semantic strength vector of text direction The polymerization strength information is processed and transformed into a probability distribution for quality assessment.

[0022] First, to achieve a probabilistic representation of continuous quality scores, the rating scale is discretized into... A predefined quality level, whose corresponding score values ​​form an ordered vector. For example, if the range of the score is 0.5-5, then we can set... Subsequently, the image orientation semantic intensity vector semantic intensity vector in text direction Two independent multilayer perceptrons (MLPs) are input separately and mapped to non-negative evidence vectors using the Softplus activation function:

[0023]

[0024]

[0025] in, Each element of the evidence vector quantifies the evidence collected from either the image or text perspective that supports the belonging to the first... Quality levels The strength of evidence, among which ;

[0026] Next, the Dirichlet distribution is parameterized using the evidence vector. The evidence vector is incremented by 1 and used as the concentration parameter of the Dirichlet distribution.

[0027]

[0028]

[0029] in, use Represents; each component Considered to support the The amount of valid evidence for each quality level, among which ;

[0030] For a Vidilike distribution Its probability density function is:

[0031]

[0032] in, Let be a point on the probability simplex. It is a multivariate Beta function; the shape of the Dirichlet distribution is determined by the concentration parameter. The sum of all concentration parameters is determined. Total concentration; Total density divided into image viewpoints Total Concentration of Textual Perspective ;

[0033] Based on the properties of the Dirichlet distribution, its expected probability vector is equal to the normalized concentration parameter; therefore, the optimal probability estimates of the quality level from the image perspective and the text perspective are respectively:

[0034]

[0035]

[0036] This is achieved by calculating an ordered vector composed of the expected probability vector and the corresponding scores of a predefined quality level. The dot product of the two values ​​yields continuous quality scores for both the image and text perspectives:

[0037]

[0038]

[0039] This continuous quality score is essentially the mathematical expectation of the quality grade under its probability distribution;

[0040] A comprehensive quality score is output by arithmetically averaging the continuous quality scores from both the image and text perspectives.

[0041]

[0042] This design explicitly models the strength of evidence in two-way evaluation using the Dirichlet distribution, giving the output score clear probabilistic interpretability. At the same time, the average fusion strategy effectively integrates semantic evidence from the two dimensions of "image interpretability" and "textual realizability", thus obtaining more robust and reliable evaluation results.

[0043] To train the semantic evidence learning network and improve the accuracy and robustness of quality assessment, a composite loss function is used for optimization. This composite loss function consists of two parts:

[0044] The first term is the regression loss, which uses the mean squared error loss and directly constrains the final predicted quality score. Approximating its true labeled score For a training batch containing A sample, defined as:

[0045]

[0046] The second term is the evidence regularization loss, which applies to the evidence vectors of the two independent paths: image and text. and By minimizing the Dirichlet distribution With a uniform prior distribution Between Divergence is used for regularization; for a single sample, the regularization terms for its image path and text path are as follows:

[0047]

[0048]

[0049] in, and Concentration parameters after removing non-misleading evidence.

[0050] The overall training objective is the weighted sum of the two losses mentioned above. For a batch... For each sample, the total loss is defined as:

[0051]

[0052] in, To balance the hyperparameters of the two contributions.

[0053] The beneficial effects of this invention are as follows: By constructing a semantic association extraction network and a semantic evidence learning network, this invention effectively solves the problem of insufficient modeling of semantic association strength distribution in image generation quality assessment. This method not only performs fine-grained association quantification of the embedded features of images and text, revealing their strength distribution, but also uses Dirichlet distribution to probabilistically model and fuse bidirectional assessment evidence, thereby significantly improving the interpretability, robustness, and discriminative accuracy of the quality assessment. Attached Figure Description

[0054] Figure 1 This is a schematic diagram of the semantic association extraction network structure.

[0055] Figure 2 This is a schematic diagram of the semantic evidence learning network structure. Detailed Implementation

[0056] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings and technical solutions.

[0057] Figure 1 This is a schematic diagram of the semantic association extraction network structure. First, using the image encoder and text encoder of the CLIP pre-trained model, features are extracted from the input image and input text respectively to obtain the local semantic embedding matrix. and local semantic embedding matrix Based on this, a complete semantic association matrix is ​​constructed by calculating the cosine similarity between all normalized local embedding pairs. This is used to quantify the strength of fine-grained semantic associations between images and text. Subsequently, the semantic association matrix is ​​analyzed. Max pooling is performed along both the row and column directions to aggregate the semantic intensity vectors of the image directions. semantic intensity vector in text direction This allows us to gather the strongest semantic support for each local feature of an image or text obtained from another modality.

[0058] Figure 2 This is a schematic diagram of the semantic evidence learning network structure. First, the semantic association is extracted from the image orientation semantic intensity vector output by the network. semantic intensity vector in text direction Each evidence vector is mapped to a non-negative evidence vector using independent MLP and Softplus activation functions. and Subsequently, the evidence vector is incremented by 1 and used as the concentration parameter, thus obtaining the concentration parameter of the image viewpoint. Concentration parameters from the text perspective Next, the Dirichlet distribution is parameterized. and Based on the properties of this distribution, the optimal probability estimate of the quality level is calculated under both image and text perspectives. and This is then compared with the preset quality level score vector. By performing a dot product, the continuous quality score of the image viewpoint is derived. Continuous quality score from the text perspective Finally, the final comprehensive quality score is output by taking the arithmetic mean of the scores from the two perspectives. The entire network, during training, experiences a total loss... End-to-end optimization is performed, and the loss incorporates the mean squared error regression term and the evidence regularization term, enabling the model to more accurately and robustly evaluate the quality of generated images.

Claims

1. A method for evaluating the quality of image generation through semantic evidence learning, characterized in that, The steps are as follows: Step 1: Construct a semantic association extraction network and obtain the image orientation semantic intensity vector based on the semantic association extraction network. semantic strength vector of text direction ; Step 2: Construct a semantic evidence learning network; Semantic evidence learning networks process the semantic intensity vectors of image orientations. semantic strength vector of text direction The polymerization strength information is processed and transformed into a probability distribution for quality assessment. First, to achieve a probabilistic representation of continuous quality scores, the rating scale is discretized into... A predefined quality level, whose corresponding score values ​​form an ordered vector. ; Subsequently, the image orientation semantic intensity vector semantic strength vector of text direction Two independent multilayer perceptrons (MLPs) are input separately and mapped to non-negative evidence vectors using the Softplus activation function: in, Each element of the evidence vector quantifies the evidence collected from either the image or text perspective that supports the belonging to the first... Quality levels The strength of evidence, among which ; Next, the Dirichlet distribution is parameterized using the evidence vector. The evidence vector is incremented by 1 and used as the concentration parameter of the Dirichlet distribution. in, use Represents; each component Considered to support the The amount of valid evidence for each quality level, among which ; For a Vidilike distribution Its probability density function is: in, Let be a point on the probability simplex. It is a multivariate Beta function; the shape of the Dirichlet distribution is determined by the concentration parameter. The sum of all concentration parameters is determined. Total concentration; Total density divided into image viewpoints Total Concentration of Textual Perspective ; Based on the properties of the Dirichlet distribution, its expected probability vector is equal to the normalized concentration parameter; therefore, the optimal probability estimates of the quality level from the image perspective and the text perspective are respectively: This is achieved by calculating an ordered vector composed of the expected probability vector and the corresponding scores of a predefined quality level. The dot product of the two values ​​yields continuous quality scores for both the image and text perspectives: This continuous quality score is essentially the mathematical expectation of the quality grade under its probability distribution; A comprehensive quality score is output by arithmetically averaging the continuous quality scores from both the image and text perspectives. To train the semantic evidence learning network and improve the accuracy and robustness of quality assessment, a composite loss function is used for optimization. This composite loss function consists of two parts: The first term is the regression loss, which uses the mean squared error loss and directly constrains the final predicted quality score. Approximating its true labeled score For a training batch containing A sample, defined as: The second term is the evidence regularization loss, which applies to the evidence vectors of the two independent paths: image and text. and By minimizing the Dirichlet distribution With a uniform prior distribution Between Divergence is used for regularization; for a single sample, the regularization terms for its image path and text path are as follows: in, and Concentration parameters after removing non-misleading evidence; For a batch For each sample, the total loss is defined as: in, To balance the hyperparameters of the two contributions.

2. The image generation quality assessment method for semantic evidence learning according to claim 1, characterized in that, The specific implementation process of step 1 is as follows: To construct the semantic association matrix, we first obtain the feature representations of the input image and input text in a unified semantic space; then, we use the image encoder and text encoder of the CLIP pre-trained model based on the ViT architecture to extract image features and text features respectively. To quantify the fine-grained semantic association strength between the input image and the input text, local semantic embedding matrices of different regions of the input image are used. Local semantic embedding matrix of different words from the input text Normalization is performed, and the cosine similarity between all normalized feature pairs is calculated to construct a complete semantic association matrix. ;make and These represent the local semantic embedding matrices of different regions of the input image. Local semantic embedding matrix of different words from the input text The row vectors in the array, where, , Then the semantic association matrix Each element is defined as the dot product of the corresponding normalized vectors, and their cosine similarity is: in, Characterizing the first The image embedding and the first The semantic association strength between text embeddings, whose value is located in interval; To obtain from a two-dimensional semantic association matrix More representative intensity information is extracted and aggregated to obtain a low-dimensional semantic intensity vector; a max-pooling strategy is employed to extract the intensity along the semantic association matrix. The rows and columns are aggregated to generate two vectors: an image orientation semantic intensity vector. semantic strength vector of text direction The elements of the two are as follows: 。 3. The image generation quality assessment method for semantic evidence learning according to claim 2, characterized in that, The input image is divided into a series of non-overlapping image patches. Each image patch is transformed into a visual label sequence after linear projection. A learnable [CLS] label is inserted at the beginning of the visual label sequence to form a complete visual sequence. After the complete visual sequence is encoded by a visual Transformer, the output feature set is { , , , , },in As a global semantic embedding of the input image, { , , , The local semantic embedding matrix that constitutes different regions of the input image is denoted as . ,in The number of local semantic embedding matrices. The feature dimension of each local semantic embedding.

4. The image generation quality assessment method for semantic evidence learning according to claim 3, characterized in that, The input text is first converted into a word sequence through lexicalization, and [SOS] and [EOS] markers are added to the start and end positions of the word sequence, respectively, to form a complete text sequence. This complete text sequence is then encoded using a Text Transformer to output a feature set { , , , , },in As a global semantic embedding describing the input text, { , , , The local semantic embedding matrix of different words in the input text is denoted as} ,in This represents the number of original lexical units obtained after lexicalization of the input text. The feature dimension of each local semantic embedding.

Images