A method and system for generating a quality assessment of an image

Abstract

The application provides a quality evaluation method and system of generated images, the method comprises constructing a cross-modal generated image quality evaluation model, the generated image quality evaluation model comprises a visual quality branch and a semantic consistency branch; based on a text prompt word, a training sample set containing generated images and their corresponding subjective quality scores is constructed; the generated image quality evaluation model is trained using the training sample set to obtain a generated image quality scoring model; and the generated image to be evaluated is input into the generated image quality scoring model to output the quality score of the generated image to be evaluated. The application can comprehensively evaluate the generated image from the two dimensions of perceptual fidelity and conditional consistency, improve the accuracy and robustness of the evaluation result, and can be used in application scenarios such as generated model screening, generated result sorting and quality control.

Description

Technical Field

[0001] This application relates to the field of image quality assessment technology, and more specifically, to a method and system for assessing the quality of generated images. Background Technology

[0002] In recent years, text-generated images and graph-generated images based on generative models such as diffusion models and generative adversarial networks have been widely used. However, generated images may still suffer from artifacts, missing details, structural errors, and semantic deviations in terms of realism, sharpness, structural consistency, and semantic consistency with the input conditions (text prompts or reference images). Therefore, reliable quality evaluation of generated images is needed to support model selection, parameter tuning, online screening, and result ranking.

[0003] Existing image quality assessment methods are mostly geared towards natural or distorted image scenarios, typically focusing only on visual distortions such as image sharpness, noise, and blockiness, failing to adequately reflect the semantic alignment issues unique to generated images. Furthermore, some methods rely on reference images or specific distortion types, resulting in insufficient generalization ability. On the other hand, training generated image quality assessment models also depends on high-quality labeled data; a lack of datasets covering multiple generation models, quality levels, and task types will limit the accuracy and robustness of the assessment models. Generated image quality assessment methods can be broadly categorized into subjective and objective assessments. Subjective assessment relies on human observer perception, evaluating quality by scoring the visual effect, semantic rationality, and matching degree with the text description of the generated image. Objective assessment, on the other hand, uses algorithms to simulate the human visual system, quantitatively assessing image quality from the perspectives of image-text alignment, semantic consistency, and aesthetics through automated metrics. CLIP (Contrastive Language-Image Pretraining), proposed by OpenAI, is a multimodal pretraining technique that maps images and text into a shared representation space through training on large-scale image-text datasets, thereby achieving powerful cross-modal understanding and generalization capabilities. CLIP, leveraging a contrastive learning framework, successfully constructs a unified cross-modal representation by maximizing the similarity of matching image-text pairs while minimizing the similarity of non-matching image-text pairs. It employs a dual-encoder architecture, including a visual encoder and a text encoder, which work together to generate embedded representations of images and text in the same space. CLIP is applied to analyze images and videos, evaluating the semantic consistency and quality features between content and text descriptions through its cross-modal alignment capabilities. Subsequent work has shifted attention to the similarity between generated image and text cues. Yuan et al., in "Jiquan Yuan, Xinyan Cao, Linjing Cao, JinlongLin, Xixin Cao: PSCR: Patches Sampling-based Contrastive Regression for AIGCImage Quality Assessment. CoRR abs / 2312.05897 (2023)", proposed a patch-sampling-based contrastive regression method, PSCR, which effectively captures the correlation between local visual features and global semantic information by randomly sampling local regions in images and combining it with a contrastive learning framework. This approach not only improves sensitivity to local quality defects but also better adapts to the diversity and complexity present in generated images. Furthermore, PSCR introduces a contrastive regression mechanism that models the quality differences between the generated and reference images at the image patch level, thereby achieving accurate image quality assessment.However, existing CLIP-based methods lack explicit aesthetic and perceptual quality assessments, focusing primarily on text-image alignment rather than overall human perception. Secondly, the effectiveness of CLIP-based metrics largely depends on the domain coverage of the pre-trained model, which can lead to biased evaluations of certain AI-generated images.

[0004] A search revealed that Chinese invention patent application CN121010997A, entitled "Method and Apparatus for Quality Evaluation of Generative Face Images," first acquires input information, including the generative face image to be evaluated, a reference image, and text prompts. This input information is then fed into a pre-trained target quality evaluation model. A text processing module converts the text prompts into text embedding vectors. An image feature extraction module extracts image features to obtain image feature data. A projector module maps the image feature data to a unified language space to obtain target image feature data. Finally, a large language model module outputs the image quality evaluation result based on the text embedding vectors and the target image feature data. While this approach can comprehensively evaluate generative face images at the semantic level and output evaluation conclusions, its evaluation process relies on large language model inference and cross-modal projection mapping, making it susceptible to factors such as model size, inference overhead, and prompt word settings. This results in high deployment costs. Furthermore, this approach is primarily aimed at generative face image quality evaluation, limiting its applicability and making it difficult to directly extend to general generative image quality evaluation scenarios outside of face categories. Therefore, it lacks versatility and cross-category generalization. Chinese invention patent application CN121147719A, entitled "Image Quality Evaluation Method, Apparatus, Electronic Device, and Storage Medium," proposes an image quality evaluation scheme: receiving an image to be evaluated, extracting multimodal feature vectors from the image, inputting the multimodal feature vectors into a quality evaluation network, outputting quality prediction results to characterize the naturalness and / or rationality of the image to be evaluated, and providing a final quality evaluation accordingly. This scheme is applicable to any image to be evaluated and, without relying on the original prompt, achieves a comprehensive score for the semantic consistency, structural integrity, and visual naturalness of the generated image through reverse description of image content and alignment of image and text features, thereby improving review efficiency and stability. However, this scheme uses "image-to-semantic inference" as the basis for semantic consistency evaluation. When the generated image has semantic drift or local errors, the reverse description may be biased, resulting in insufficient alignment with the user's original intent. Furthermore, its output indicators are macroscopic, and its stable quantification ability for fine-grained low-level defects (artifacts, unnatural textures, missing details, etc.) may be limited, thus affecting the evaluation accuracy and generalization ability in complex scenarios. Chinese invention patent application CN113284100A, entitled "Image Quality Evaluation Method Based on Hybrid Domain Attention Mechanism for Restored Image", mainly addresses the quality evaluation problem of distorted images. First, it reconstructs the distorted image by constructing an image restoration network to obtain the corresponding restored image. Then, it introduces a hybrid domain attention mechanism at the dual feature levels of spatial and frequency domains to jointly model and weightedly fuse the difference features between the original distorted image and the restored image, extracting multi-scale and multi-dimensional perceptual feature information. Finally, it completes the image quality prediction output based on the extracted features.This scheme improves the ability to perceive complex distortion types by introducing image restoration contrast and a hybrid domain attention mechanism. However, its overall process relies on an additional image restoration network and a multi-branch feature extraction structure, resulting in a relatively complex model structure and high computational overhead during training and inference. Furthermore, the core design of the method still revolves around modeling the differences between distorted and restored images, making it primarily suitable for traditional distorted image quality assessment scenarios. Its scalability for quality assessment of no-reference generated images or cross-category general generated content is relatively limited. Chinese invention patent application CN121459144A, entitled "A Nighttime No-Reference Image Quality Assessment Method Based on Controlled Distortion Synthesis," extracts features from the input image using a convolutional neural network, enhances the expressive power of key regions through multi-level feature aggregation or attention mechanisms, and outputs image quality scores using regression or classification modules. This type of solution can achieve automated quality prediction under an end-to-end framework and achieve good fitting results on specific datasets. However, it relies on large-scale labeled data for supervised training, and the model's generalization ability is greatly affected by the distribution of training data. At the same time, its quality judgment logic is mainly based on implicit feature mapping relationships obtained through statistical learning, lacking clear semantic interpretation capabilities. When facing cross-scene, multi-category, or novel generated image content, it suffers from insufficient adaptability and stability fluctuations, and still has certain limitations in terms of universality and scalability.

[0005] Therefore, there is an urgent need for a quality assessment method and system that can comprehensively evaluate generated images from two dimensions: perceptual quality and semantic consistency, while avoiding recognition omissions caused by semantic drift and other problems. Summary of the Invention

[0006] In view of the deficiencies in the prior art, the purpose of this application is to provide a method and system for quality assessment of generated images.

[0007] According to a first aspect of this application, a method for quality assessment of generated images is provided, comprising: A cross-modal generated image quality assessment model is constructed, which includes a visual quality branch and a semantic consistency branch. The visual quality branch is used to extract and fuse multi-level visual features of the generated image to obtain perceptual quality features. The semantic consistency branch is used to align the generated image with the text prompts in its input conditions across modalities to obtain semantic consistency features. Based on text prompts, a training sample set containing generated images and their corresponding subjective quality scores is constructed; The generated image quality evaluation model is trained using the training sample set to obtain a generated image quality scoring model. The generated image to be evaluated is input into the generated image quality scoring model, and the quality score of the generated image to be evaluated is output.

[0008] Optionally, the cross-modal generated image quality assessment model further includes a quality regression network, which is used to fuse the perceived quality features and the semantic consistency features, and output a quality score corresponding to the generated image.

[0009] Optionally, the visual quality branch is used to extract and fuse multi-level visual features of the generated image to obtain perceptual quality features, including: For the generated image, a backbone network is used to extract feature maps of at least two different scales; Feature maps of different scales are aligned and hierarchically fused to obtain fused features; The fused features are aggregated to obtain the perceived quality features.

[0010] Optionally, the semantic consistency branch is used to perform cross-modal alignment between the generated image and the text prompts in its input conditions to obtain semantic consistency features, including: The generated image is encoded using an image encoder to obtain the image embedding; The text embedding is obtained by encoding the text prompts in the input conditions using a text encoder. Cross-modal similarity is obtained based on the image embedding and the text embedding, and is used as the semantic consistency feature.

[0011] Optionally, the step of constructing a training sample set containing the generated images and their corresponding subjective quality scores based on text prompts includes: Collect multiple text prompts for the text-to-image task, input each text prompt into multiple texts into the image generation model, and generate the corresponding generated image; Multiple reference images are acquired for the image-to-image task, and corresponding text descriptions are generated for the reference images. Each reference image and its corresponding text description are input into the image generation model to generate the corresponding generated image. The generation hyperparameters of the text-to-image generation model and / or the image-to-image generation model are dynamically adjusted, and the above image generation steps are repeated to generate generated images covering different quality levels. Obtain subjective quality scores for each generated image; Construct a training sample set, which includes generated images and their corresponding input conditions and subjective quality scores.

[0012] Optionally, obtaining the subjective quality score for each generated image includes: Based on the preset scoring granularity and quality scoring range, fine-grained quality scoring is performed on all generated images to form corresponding quality scores; The quality score is then subjected to consistency processing and outlier removal to obtain a calibrated quality score, which is used as the subjective quality score.

[0013] Optionally, the preset scoring granularity includes image clarity, realism, and semantic consistency.

[0014] According to a second aspect of this application, a quality assessment system for generated images is provided, comprising: The model building module is used to construct a cross-modal generated image quality evaluation model, which includes a visual quality branch and a semantic consistency branch. The visual quality branch is used to extract and fuse multi-level visual features of the generated image to obtain perceptual quality features. The semantic consistency branch is used to align the generated image with the text prompts in its input conditions across modalities to obtain semantic consistency features. The sample set construction module is used to construct a training sample set containing generated images and their corresponding quality scores based on text prompts. The model training module is used to train the generated image quality evaluation model using the training sample set to obtain the generated image quality scoring model. The quality scoring module is used to input the generated image to be evaluated into the generated image quality scoring model and output the quality score of the generated image to be evaluated.

[0015] According to a third aspect of this application, a non-transitory computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of a quality assessment method for generating an image provided in the first aspect of this application.

[0016] According to a fourth aspect of this application, an electronic device is provided, comprising: At least one memory for storing program instructions; At least one processor is configured to invoke program instructions stored in the memory and execute the steps of the image quality assessment method provided in the first aspect of this application according to the obtained program instructions.

[0017] This application provides a method for quality assessment of generated images. Through a dual-branch structure of visual quality and semantic consistency, it can comprehensively evaluate generated images from both dimensions, resulting in more comprehensive evaluation results that better meet the requirements of the generation task. A training sample set is constructed using text prompts, covering generated images and corresponding quality scores. The generated image quality assessment model aligns the text prompts with the generated images to improve the ability to identify issues such as semantic drift and prompt mismatch, thereby enhancing the accuracy of the quality assessment results. This method can directly output the quality score of the generated images for quantitative detection of their quality, reducing manual review costs and improving the efficiency of automated management of generated images.

[0018] Other technical effects resulting from the additional features will be further illustrated in the corresponding embodiments. Attached Figure Description

[0019] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a flowchart of an image quality assessment method in one embodiment of this application; Figure 2 This is a schematic diagram of the structure of an image quality assessment model generated in one embodiment of this application; Figure 3 This is a schematic diagram of multi-level feature extraction and fusion of the visual quality branch in one embodiment of this application; Figure 4 This is a schematic diagram of cross-modal alignment of semantic consistency branches in one embodiment of this application; Figure 5 This is a schematic diagram illustrating the process of constructing a training sample set / generated image quality database in one embodiment of this application; Figure 6 This is a schematic diagram of the structure of a quality assessment system for generated images in one embodiment of this application. Detailed Implementation

[0020] The present application will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present application, and these all fall within the protection scope of the present application. Parts not described in detail in the following embodiments can be implemented using existing technology.

[0021] It should be noted that all information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with relevant regulations.

[0022] In recent years, text-generated images and graph-generated images based on generative models such as diffusion models and generative adversarial networks have been widely used. However, generated images may still suffer from artifacts, missing details, structural errors, and semantic deviations in terms of realism, clarity, structural consistency, and semantic consistency with input conditions (text prompts or reference images). Therefore, reliable quality evaluation of generated images is needed to support model selection, parameter tuning, online screening, and result ranking. CLIP is a multimodal pre-training technique that maps images and text to a shared representation space by training on large-scale image-text datasets, thereby achieving powerful cross-modal understanding and generalization capabilities. However, existing CLIP-based methods lack explicit aesthetic and perceptual quality evaluation; they mainly focus on the alignment of text and images rather than overall human perception. Based on the above problems, this application provides a quality evaluation method for generated images to address the aforementioned issues.

[0023] Reference Figure 1 As shown in the embodiment of this application, a method for quality assessment of generated images is provided, including: S1. Construct a cross-modal generated image quality assessment model, which includes a visual quality branch and a semantic consistency branch. The visual quality branch is used to extract and fuse multi-level visual features of the generated image to obtain perceptual quality features. The semantic consistency branch is used to align the generated image with the text prompts in its input conditions across modalities to obtain semantic consistency features. S2. Based on the text prompts, construct a training sample set containing the generated images and their corresponding subjective quality scores; S3. Using the training sample set, train the generated image quality evaluation model to obtain the generated image quality scoring model; S4. Input the generated image to be evaluated into the generated image quality scoring model, and output the quality score of the generated image to be evaluated.

[0024] The embodiments described above, through a dual-branch structure of visual quality and semantic consistency, enable comprehensive evaluation of generated images from both visual quality and semantic consistency dimensions, resulting in more comprehensive evaluation results that better meet the requirements of the generation task. By utilizing text prompts to construct a training sample set that covers generated images and corresponding quality scores, the generated image quality evaluation model can align text prompts with generated images, improving its ability to identify issues such as semantic drift and prompt mismatch, thereby enhancing the accuracy of the quality evaluation results. The method of this application can directly output the quality score of the generated image for quantitative detection of its quality, reducing manual review costs and improving the efficiency of automated management of generated images.

[0025] In some specific embodiments of this application, the cross-modal generated image quality assessment model further includes a quality regression network, which is used to fuse perceptual quality features and semantic consistency features, and output a quality score corresponding to the generated image.

[0026] For example, in the generated image quality assessment model, the visual quality branch is used to obtain perceptual quality features to characterize sharpness, texture detail, structural rationality, and artifact level (i.e., visual naturalness); the semantic consistency branch is used to obtain semantic consistency features to characterize the degree of matching between the generated image and the conditional constraint information (i.e., conditional consistency). For the text-based image quality assessment task, the input conditions are text prompts (conditional constraint information) and the corresponding generated image, and the text prompt words in the input conditions are text prompts; for the image-based image quality assessment task, the input conditions are a reference image, the corresponding text description (conditional constraint information), and the image generated from the reference image and the text description, and the text prompt words in the input conditions are text descriptions.

[0027] The text prompts in the input conditions include text prompts (for text-to-image tasks) and text descriptions (for image-to-image tasks).

[0028] For example, a perceived quality score can be generated based on perceived quality features, and a semantic consistency score can be generated based on semantic consistency features; the perceived quality score and the semantic consistency score can be fused and regressed through a quality regression network to obtain the final quality score.

[0029] In one specific application instance, the final quality score is a continuous value; in another specific application instance, the final quality score can also be mapped to a quality level for screening and threshold control.

[0030] The embodiments described above in this application, through a dual-branch structure of visual quality and semantic consistency, can simultaneously and comprehensively evaluate the generated image from two dimensions: visual naturalness and input condition consistency. This results in a more comprehensive evaluation that better meets the requirements of the generation task. Furthermore, this application obtains semantic consistency features through cross-modal alignment, enhancing the ability to identify issues such as semantic drift and mismatched prompts in the generated image, thereby improving the accuracy and robustness of the quality evaluation.

[0031] In some specific embodiments of this application, the visual quality branch is used to extract and fuse multi-level visual features of the generated image to obtain perceptual quality features, and may further include: S111. For the generated image, a backbone network is used to extract feature maps of at least two different scales; S112. Align and hierarchically fuse feature maps of different scales to obtain fused features; S113. Perform feature aggregation on the fused features to obtain the perceived quality features.

[0032] For example, multi-level visual feature extraction can extract four layers of feature maps at different scales to simultaneously represent local texture details and global structural information. The visual quality branch (i.e., the perceptual quality branch) can align, hierarchically fuse, and aggregate multi-scale features to enhance the ability to represent quality defects such as artifacts, unnatural textures, and missing details.

[0033] In some specific embodiments of this application, the semantic consistency branch is used to perform cross-modal alignment between the generated image and the text prompts in its input conditions to obtain semantic consistency features, which may further include: S121. Encode the generated image using an image encoder to obtain the image embedding; S122. Use a text encoder to encode the text prompts in the input conditions to obtain the text embedding; S123. Obtain cross-modal similarity based on image embedding and text embedding of generated images, and use it as a semantic consistency feature.

[0034] For example, the text encoder and image encoder can adopt the encoding structure corresponding to the cross-modal pre-trained model. During training, the image encoder is fine-tuned and the text encoder is frozen. The semantic consistency branch can obtain semantic consistency features by calculating the cross-modal similarity between the text embedding vector and the image embedding vector; the similarity can be cosine similarity, dot product similarity, or attention alignment score.

[0035] In some implementations, for the image-to-image quality assessment task, this application, in addition to encoding the generated image, also uses an image encoder to encode the reference image to obtain an image embedding representation, and obtains the similarity between the image embedding of the generated image and the image embedding of the reference image as a supplement to semantic consistency, and incorporates it into the semantic consistency features of the image embedding of the generated image and the text embedding to form the final semantic consistency features. The final semantic consistency features are then fused with the perceptual quality features.

[0036] In some specific embodiments of this application, constructing a training sample set containing generated images and their corresponding subjective quality scores based on text prompts may further include: S21. Collect multiple text prompts for the text-to-image task, input each text prompt into multiple texts into the image generation model, and generate the corresponding generated image; S22. Collect multiple reference images for the image generation task and generate corresponding text descriptions for the reference images. Input each reference image and its corresponding text description into the image generation model to generate the corresponding generated image. S23. Dynamically adjust the generation hyperparameters of the text-to-image generation model and / or the image-to-image generation model, and repeat the above image generation steps to generate generated images covering different quality levels. S24. Obtain the subjective quality score for each generated image; S25. Construct a training sample set, which includes generated images and their corresponding input conditions and subjective quality scores.

[0037] For example, text-to-image generation models can use Stable Diffusion, PixArt, etc., while image-to-image generation models can use FLUX, MidJourney, etc. Taking the Stable Diffusion text-to-image generation model as an example, its generation hyperparameters mainly include sampling method, number of generation steps, cue word guidance coefficient, and random seed. Taking MidJourney as an example, its generation hyperparameters include style intensity, detail preservation weight, and image weight. When generating high-quality images, the default Eulerian sampling method can be used, with the number of iterations set to 25 and the cue word guidance coefficient set to 5. When generating low-quality images, the number of generation steps is adjusted to between 5 and 10, and the cue word guidance coefficient is set to between 3 and 4; the parameters will vary depending on the model configuration.

[0038] In addition, the subjective quality score is a quantitative score given by relevant personnel based on visual perception of the generated image in terms of clarity, realism and semantic consistency. This quantitative score and the generated image form a generated image quality database.

[0039] The embodiments described above in this application construct a generated image quality database covering multiple generation models, multiple task types, and multiple quality levels. By combining consistency processing and anomaly removal based on subjective ratings from multiple individuals, the diversity of training samples and the reliability of labels are improved, thereby enhancing the generalization ability of the evaluation model, reducing the impact of subjective evaluation noise on model training, and thus improving the credibility of the evaluation results.

[0040] In some specific embodiments of this application, obtaining the subjective quality score of each generated image may further include: S241. Based on the preset scoring granularity and quality scoring range, perform fine-grained quality scoring on all generated images to form corresponding quality scores. S242. Perform consistency processing and outlier removal on the quality score to obtain the calibrated quality score as the subjective quality score.

[0041] In some specific embodiments of this application, the preset scoring granularity includes image clarity, realism, and semantic consistency.

[0042] For example, the scoring granularity includes scoring dimensions such as sharpness, realism, and semantic consistency. All generated images are comprehensively evaluated according to the three scoring dimensions to obtain the final comprehensive quality score.

[0043] Consistency processing is used to ensure the comparability of scores across different dimensions and avoid distortion caused by inconsistent scoring scales. After collecting all subjective quality scores, the data is statistically processed according to the ITU-R BT.500-13 standard. First, the mean and standard deviation of the subjective quality score for each generated image are calculated, where the mean represents the average perceived quality level of the image, and the standard deviation measures the dispersion of scores among different evaluators. Further, a 95% confidence interval for each image score is calculated; scores exceeding this interval are identified as outliers and removed. Finally, the average subjective opinion score for each image is recalculated based only on the retained valid scores (i.e., the calibrated quality score is used as the subjective quality score), which serves as the final quality labeling result for the generated image.

[0044] In some specific embodiments of this application, step S4 involves inputting the generated image to be evaluated into the generated image quality scoring model and outputting a quality score for the generated image to be evaluated, including: S41. Obtain the generated image to be evaluated and its corresponding text prompts; S42. Input the generated image to be evaluated and the corresponding text prompts into the pre-trained generated image quality scoring model; S43. Generate the image quality scoring model and output the final score.

[0045] In step S41 of this embodiment, obtaining the generated image to be evaluated and the corresponding text prompt can include the following steps: S411. Obtain the generated image to be evaluated; S412. Obtain the text prompt words corresponding to the generated image to be evaluated; S413. The generated image to be evaluated and the text prompts are used as input information, formatted, and then input into the subsequent feature extraction process.

[0046] In a specific application example, the generated image to be evaluated can be a text-generated image or an image-generated image; the text prompt can be one or more natural language descriptions.

[0047] In another specific application example, the input information may further include a reference image and its text description, used to evaluate the consistency of the image-generated image results.

[0048] This application fully considers that the quality of generated images is affected by both visual perception quality and semantic consistency. On the one hand, generated images may have low-level quality problems such as artifacts, unnatural textures, missing details, and structural anomalies; on the other hand, they may also have consistency problems such as mismatch with text prompts and semantic drift. Therefore, this application constructs a generated image quality evaluation model that includes visual quality and semantic consistency branches, models perceptual quality and semantic consistency features respectively, and fuses the two types of features through a quality regression network to output a final quality score, thereby achieving a comprehensive evaluation of the quality of generated images. This application is applicable to the quality evaluation, ranking, and quality control of generated results for tasks such as text-to-image and image-to-image generation. The quality score output by the embodiments of this application can be directly used in application scenarios such as automatic filtering, ranking, threshold control, and quality monitoring of generated results, reducing manual review costs, improving business processing efficiency and consistency, and improving the efficiency of automated management of generated images.

[0049] The present application will be further described below with reference to specific embodiments in order to better understand the above technical solutions of the present application. It should be understood that the following are only some examples and are not intended to limit the present application.

[0050] Example 1: This application provides a method for quality assessment of generated images. The method includes first constructing a generated image quality dataset, then constructing a generated image quality evaluation model, training the generated image quality evaluation model based on the dataset, and finally outputting a quality score for the generated image to be evaluated, and performing a consistency evaluation on the dataset.

[0051] Step 1, Generate Image Quality Dataset Construction: (e.g.) Figure 5As shown, this embodiment first constructs a generated image quality dataset, which is used to train and evaluate the generated image quality assessment model. The dataset construction process includes obtaining the input conditions for the generation task and generating generated images. The input conditions include at least text prompts, and in the image-to-image task, reference images and their text descriptions may also be included. Generated images are generated based on at least one text-to-image generation model and / or at least one image-to-image generation model, and the generation hyperparameters are adjusted to make the generated results cover different quality levels. Multiple annotators are organized to subjectively evaluate the generated images and give subjective quality scores. The scoring results are processed for consistency and outliers are removed to obtain the final subjective quality score label for each generated image. The generated images, corresponding input conditions, and quality score labels are combined into sample entries and divided into training set, validation set, and test set to form the generated image quality dataset.

[0052] Step two: Construct the image quality assessment model. For example... Figure 2 As shown, this embodiment constructs a generated image quality evaluation model. This model is an end-to-end learning structure, which includes at least a perceptual quality branch, a semantic consistency branch, and a quality regression network, used to extract the perceptual quality features and semantic consistency features of the generated image and fuse them to output a quality score.

[0053] Step 3: Feature extraction and alignment of the three-branch structure. For example... Figure 3 As shown, the perceptual quality branch is used to extract and fuse multi-level visual features of the generated image to characterize the sharpness, texture naturalness, structural rationality, and artifact degree of the generated image, and outputs perceptual quality features. Figure 4 As shown, the semantic consistency branch is used to perform cross-modal alignment between the text prompts in the input conditions and the generated image. The text encoder encodes the text prompts to obtain text embedding vectors, and the image encoder encodes the generated image to obtain image embedding vectors. The semantic consistency features are obtained by calculating the cosine similarity between the text embedding vectors and the image embedding vectors, where the cosine similarity characterizes the degree of matching between the generated image and the text prompts. During the training phase, the image encoder is preferably fine-tuned while the text encoder is frozen to improve the adaptability to the distribution of generated images and maintain text space stability. The quality regression network receives the perceptual quality features and semantic consistency features, performs fusion regression, and outputs a quality prediction value.

[0054] Step four, model training and quality score output. Based on the generated image quality dataset constructed in Step one, the generated image quality evaluation model constructed in Step two is trained to ensure that the model's output quality prediction values ​​are consistent with the subjective quality score labels, thus obtaining the trained generated image quality evaluation model as the generated image quality scoring model. During the inference phase, the generated image to be evaluated and its corresponding input conditions are input into the generated image quality scoring model. The perceptual quality branch outputs perceptual quality features, the semantic consistency branch outputs semantic consistency features, and the quality regression network fuses these features to output the final quality score. The final quality score can be further mapped to a quality level for use in result filtering, ranking, and quality control.

[0055] Step 5, Testing and Performance Evaluation. The generated image quality scoring model is evaluated using the test set defined in Step 1. The generated images from the test set samples and their corresponding input conditions are input into the model to obtain predicted scores, which are then compared with the subjective quality score labels of the test set to measure the consistency and stability of the model's predictions with the subjective scores. Evaluation can be performed on different generation models or test subsets with different prompt word categories to verify the model's generalization ability.

[0056] In a preferred embodiment, the quality of the generated image depends not only on its perceived realism but also on its semantic consistency with the input text prompt. Therefore, this embodiment employs a cross-modal alignment model to obtain robust semantic similarity features, preferably using the CLIP framework. Let the input image be I and the corresponding text prompt be T. The image and text are projected into a shared embedding space using the image encoder and text encoder of the cross-modal alignment model, resulting in image embedding vectors and text embedding vectors, expressed as: in, For image embedding vectors; For text embedding vectors; For image encoders, is the text encoder, and d is the dimension of the shared embedding space.

[0057] To enhance the model's sensitivity to artifacts specific to generated images, this embodiment modifies the image encoder during the training phase. Fine-tuning, text encoder To maintain semantic consistency priors, a frozen state is maintained; a lightweight adaptive projection layer is added on top of the image encoder to align the backbone output dimension with the feature dimension of the perceptual quality branch, thus facilitating subsequent feature fusion. Image-text similarity features are calculated using cosine similarity. This is used as a semantic consistency feature, and its definition is as follows: In a specific implementation, semantic consistency features It can be used in conjunction with image quality features extracted from the perceptual quality branch for quality prediction to enhance deep image representation.

[0058] In a preferred embodiment, a quality regression network is configured to map the fused features to quality scores. Let the quality features output by the perceptual quality branch be... The similarity features output by the semantic consistency branch are The two are then connected and fused to obtain the input features. After feature fusion, it is preferable to first aggregate the spatial dimensions using a global average pooling layer to obtain the channel-level response. The formula is as follows: Where H represents height; W represents width; C represents number of channels; k represents index of height; j represents index of width; and c' represents index of number of channels. This represents the input feature located at the position of the c'th channel, the kth row in the height direction, and the jth column in the width direction.

[0059] The obtained channel-level responses As feature vectors All feature vectors The flattened data is then fed into a two-layer fully connected regression network. The first layer optimizes the features through nonlinear transformation, and the second layer outputs the predicted quality score. in, These represent the weights and biases of the first fully connected regression network; , , respectively, represent the weights and biases of the second fully connected regression network; ReLU represents the activation function used for nonlinear transformation; h is the output of the first fully connected regression network; The output of the second fully connected regression network During the training phase, the quality regression network is optimized using the mean squared error loss function, which is: in, For loss; For batch size, For the first The predicted quality score for each input. This corresponds to the actual score. The generated image quality scoring model is obtained through the above training process. During the inference phase, inputting the generated image to be evaluated and its text prompt will output the corresponding quality score.

[0060] The following evaluation uses the GIQA-4K-1.2K database created in this application, and combines it with two publicly available AIGI quality assessment databases, AIGCIQA2023 and PKU-AIGIQA-4K, to assess the generalization ability and advancement of the generated image quality assessment method proposed in this application. For comprehensive model training and evaluation, this embodiment trains and tests on the above three databases respectively, and compares and analyzes the results.

[0061] This experiment uses three commonly used evaluation metrics to quantitatively evaluate model performance: Spearman's rank correlation coefficient (SRCC), Pearson's linear correlation coefficient (PLCC), and Kendall's rank correlation coefficient (KRCC). These metrics measure the consistency and correlation between the model's predicted quality score and the subjective quality score; a higher metric value indicates a greater consistency between the model's prediction and the subjective score, and a better evaluation result.

[0062] This experiment was conducted on a single NVIDIA RTX 3090 GPU using PyTorch 2.0.0 and CUDA 11.8. The training and validation sets for each database were randomly split in a 4:1 ratio, and stratified according to the generated model type to reduce splitting bias. The Adam optimizer was used for optimization, with an initial learning rate of 5e-5, a batch size of 16, and a total of 100 epochs. To enhance the reliability of the experimental results, the training and evaluation processes were repeated multiple times using different random partitions, and the final reported metric is the average performance across multiple trials. Twelve models were selected for comparison, including the handcrafted feature-based methods CEIQ and NIQE, and deep learning-based methods TOPIQ, ResNet50, VGG16, VGG19, CNNIQA, CLIPIQA, HyperIQA, StairIQA, HPSv2, and IP-IQA. HPSv2 and IP-IQA are quality assessment methods oriented towards perceptual alignment.

[0063] Table 1 Table 1 presents the performance comparison results of the proposed method SA-GIQA with twelve other methods on three databases: GIQA-4K-1.2K, AIGCIQA2023, and PKU-AIGIQA-4K. As shown in Table 1, the proposed method outperforms existing methods on all these databases, achieving better results in relevance and consistency evaluation metrics. This indicates that the proposed method can more accurately predict the subjective quality score of generated images, possesses good robustness and generalization ability, and is applicable to the image quality evaluation needs of different generation models and different generation task scenarios.

[0064] This application jointly models the quality of generated images based on a perceptual quality branch and a semantic consistency branch, and outputs a quality score by a quality regression network. The semantic consistency branch uses a cross-modal alignment model to obtain semantic similarity features between the image and text prompts. During training, the image encoder is fine-tuned and the text encoder is frozen to maintain semantic prior stability, thereby improving the ability to recognize AIGI-specific artifacts and semantic deviations. Simultaneously, the perceptual quality branch enhances the representation of texture details, structural anomalies, and artifacts through multi-level visual feature extraction and fusion. Experimental results on databases such as GIQA-4K-1.2K, AIGCIQA2023, and PKU-AIGIQA-4K demonstrate that this application exhibits high robustness and accuracy in objective consistency metrics, and is applicable to the quality evaluation of generated images under different generation models and conditions.

[0065] This application can comprehensively evaluate generated images from two dimensions: perceptual fidelity and conditional consistency, thereby improving the accuracy and robustness of the evaluation results. It can also be used in application scenarios such as generative model screening, generative result ranking, and quality control.

[0066] Based on the same concept, another embodiment of this application provides a quality assessment system for generated images, referring to... Figure 6 As shown, the quality assessment system 100 includes: The model building module 110 is used to build a cross-modal generated image quality evaluation model. The generated image quality evaluation model includes a visual quality branch and a semantic consistency branch. The visual quality branch is used to extract and fuse multi-level visual features of the generated image to obtain perceptual quality features. The semantic consistency branch is used to align the generated image with the text prompts in its input conditions across modalities to obtain semantic consistency features. The sample set construction module 120 is used to construct a training sample set containing generated images and their corresponding quality scores based on text prompts. The model training module 130 is used to train the generated image quality evaluation model using the training sample set to obtain the generated image quality scoring model. The quality scoring module 140 is used to input the generated image to be evaluated into the generated image quality scoring model and output the quality score of the generated image to be evaluated.

[0067] For example, the generated image quality assessment model adopts a three-branch structure. The visual quality branch extracts multi-level visual quality features of the generated image to characterize sharpness, texture naturalness, structural rationality, and artifact severity. The semantic consistency branch projects the generated image and text prompts into a shared embedding space based on a cross-modal alignment model and uses cosine similarity to obtain semantic consistency features. The quality regression network fuses the perceptual quality features and semantic consistency features to output a quality score. The quality regression network consists of multiple fully connected layers and nonlinear activation functions, used to perform nonlinear mapping on the fused perceptual quality features and semantic consistency features, achieving regression prediction from a high-dimensional feature space to a continuous quality score space. During the training phase, the image encoder is fine-tuned and the text encoder is frozen to enhance sensitivity to artifacts specific to the generated image and maintain the stability of the semantic consistency prior, thereby improving the consistency between quality prediction and subjective scoring.

[0068] In the text-to-image task, generated images are generated based on multiple text prompts and various text-to-image generation models; in the image-to-image task, generated images are generated based on multiple reference images, corresponding text descriptions, and various image-to-image generation models; and generated images of different quality levels are obtained by adjusting the generation hyperparameters to form a training sample set covering the quality distribution.

[0069] It should be noted that the modules in the image quality assessment system provided in the above embodiments of this application correspond to the steps of the image quality assessment method in any of the above embodiments. Those skilled in the art can refer to the step features of the image quality assessment method to implement the corresponding modules in the image quality assessment system, which will not be elaborated here.

[0070] In another embodiment of this application, a non-transitory computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the above-described image quality assessment method.

[0071] In another embodiment of this application, an electronic device is also provided, including a memory and a processor; the memory is used to store program instructions; the processor is used to call the program instructions stored in the memory and execute the steps of the above-described image quality assessment method according to the obtained program instructions.

[0072] Optionally, the memory is used to store programs; the memory may include volatile memory, such as random-access memory (RAM), such as static random-access memory (SRAM), double data rate synchronous dynamic random-access memory (DDR SDRAM), etc.; the memory may also include non-volatile memory, such as flash memory. The memory is used to store computer programs (such as application programs and functional modules that implement the above methods), computer instructions, etc., and the aforementioned computer programs and computer instructions can be partitioned and stored in one or more memories. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by the processor.

[0073] The aforementioned computer programs, computer instructions, etc., can be stored in partitions within one or more memory locations. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by a processor.

[0074] A processor is used to execute a computer program stored in memory to implement the various steps of the methods involved in the above embodiments. For details, please refer to the relevant descriptions in the preceding method embodiments.

[0075] The processor and memory can be separate structures or integrated structures. When the processor and memory are separate structures, they can be coupled together via a bus.

[0076] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0077] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0078] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0079] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0080] The preferred features in the above embodiments can be used individually in any embodiment, or in any combination thereof, provided they do not conflict with each other. Furthermore, parts not described in detail in the embodiments can be implemented using existing technologies.

[0081] The foregoing has described some specific embodiments of this application. It should be understood that this application is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the substantive content of this application. The above-described preferred features can be used in any combination without conflict.

Claims

1. A method for quality assessment of generated images, characterized in that, include: A cross-modal generated image quality assessment model is constructed, which includes a visual quality branch and a semantic consistency branch. The visual quality branch is used to extract and fuse multi-level visual features of the generated image to obtain perceptual quality features. The semantic consistency branch is used to align the generated image with the text prompts in its input conditions across modalities to obtain semantic consistency features; Based on text prompts, a training sample set containing generated images and their corresponding subjective quality scores is constructed; The generated image quality evaluation model is trained using the training sample set to obtain a generated image quality scoring model. The generated image to be evaluated is input into the generated image quality scoring model, and the quality score of the generated image to be evaluated is output.

2. The method for quality assessment of generated images according to claim 1, characterized in that, The cross-modal generated image quality assessment model also includes a quality regression network, which is used to fuse the perceived quality features and the semantic consistency features, and output a quality score corresponding to the generated image.

3. The method for quality assessment of generated images according to claim 2, characterized in that, The visual quality branch is used to extract and fuse multi-level visual features of the generated image to obtain perceptual quality features, including: For the generated image, a backbone network is used to extract feature maps of at least two different scales; Feature maps of different scales are aligned and hierarchically fused to obtain fused features; The fused features are aggregated to obtain the perceived quality features.

4. The method for quality assessment of generated images according to claim 2, characterized in that, The semantic consistency branch is used to perform cross-modal alignment between the generated image and the text prompts in its input conditions to obtain semantic consistency features, including: The generated image is encoded using an image encoder to obtain the image embedding; The text embedding is obtained by encoding the text prompts in the input conditions using a text encoder. Cross-modal similarity is obtained based on the image embedding and the text embedding, and is used as the semantic consistency feature.

5. The method for quality assessment of generated images according to claim 1, characterized in that, The step of constructing a training sample set containing generated images and their corresponding subjective quality scores based on text prompts includes: Collect multiple text prompts for the text-to-image task, input each text prompt into multiple texts into the image generation model, and generate the corresponding generated image; Multiple reference images are acquired for the image-to-image task, and corresponding text descriptions are generated for the reference images. Each reference image and its corresponding text description are input into the image generation model to generate the corresponding generated image. The generation hyperparameters of the text-to-image generation model and / or the image-to-image generation model are dynamically adjusted, and the above image generation steps are repeated to generate generated images covering different quality levels. Obtain subjective quality scores for each generated image; Construct a training sample set, which includes generated images and their corresponding input conditions and subjective quality scores.

6. The method for quality assessment of generated images according to claim 1, characterized in that, The process of obtaining subjective quality scores for each generated image includes: Based on the preset scoring granularity and quality scoring range, fine-grained quality scoring is performed on all generated images to form corresponding quality scores; The quality score is processed for consistency and outlier removal to obtain a calibrated quality score as the subjective quality score.

7. The method for quality assessment of generated images according to claim 6, characterized in that, The preset scoring granularity includes image clarity, realism, and semantic consistency.

8. A quality assessment system for generated images, characterized in that, include: The model building module is used to construct a cross-modal generated image quality assessment model, which includes a visual quality branch and a semantic consistency branch. The visual quality branch is used to extract and fuse multi-level visual features of the generated image to obtain perceptual quality features. The semantic consistency branch is used to align the generated image with the text prompts in its input conditions across modalities to obtain semantic consistency features; The sample set construction module is used to construct a training sample set containing generated images and their corresponding quality scores based on text prompts. The model training module is used to train the generated image quality evaluation model using the training sample set to obtain the generated image quality scoring model. The quality scoring module is used to input the generated image to be evaluated into the generated image quality scoring model and output the quality score of the generated image to be evaluated.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method as described in any one of claims 1-7.

10. An electronic device, characterized in that, include: At least one memory for storing program instructions; At least one processor is configured to invoke program instructions stored in the memory and execute the steps of the method as described in any one of claims 1-7 according to the obtained program instructions.

Images