Text-to-image generation model evaluation method and system based on multi-modal large model

Abstract

The invention discloses a text-to-image generation model evaluation method and system based on a multi-modal large model. The method comprises multiple steps of multi-level image information extraction and image feature matrix construction, layered text prompt generation and optimization, generated image batch generation, interactive image comparison and scoring, and multi-dimensional feedback and prompt revision, and a generation strategy is dynamically adjusted by repeatedly executing the steps until a preset stop condition is met. And finally, comprehensively analyzing the performance of the target generation model according to the multi-dimensional visual evaluation results of the reference image and the optimal generation image. According to the evaluation method, the generative model can be accurately generated based on the content and style of the reference image through the image regeneration task, the defects of an existing evaluation method in the aspect of cross-modal information alignment are overcome, and the evaluation method has high application value.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence, and particularly relates to a method and system for evaluating a text-to-image generation model based on a multimodal large model. Background Art

[0002] In the current digital age, generative artificial intelligence, especially text-to-image (T2I) models, is developing rapidly and showing broad application prospects in many fields such as artistic creation, visual design, and automated media generation. Such models can generate corresponding images according to text descriptions, providing users with diverse visual generated content. However, with the wide application of T2I models in various fields, the accurate evaluation of the generation capabilities of these models has gradually become an urgent need. Traditional evaluation methods for generation models usually rely on cross-modal matching between text inputs and image outputs, mainly calculating semantic similarity through models such as CLIP, or generating questions and answers through automatic question generation (QG&QA) to test the rationality of the generated content. However, these evaluation methods can only measure single attributes or overall semantic consistency, and it is difficult to comprehensively and accurately measure the performance of generation models at a fine-grained level. In practical applications, generation models need to have the ability to handle complex scenarios, details, and multi-level styles. Due to the problem of cross-modal information asymmetry, traditional evaluation methods are difficult to accurately measure the grasp of details and multi-dimensional visual information by generation models. At the same time, the quality of generated images may vary significantly in different contents and styles, which requires a comprehensive evaluation system that can evaluate at multiple levels such as content, style, and details to effectively reflect the actual generation capabilities of the models. Summary of the Invention

[0003] The purpose of the present invention is to overcome the existing deficiencies and provide a method and system for evaluating a text-to-image generation model based on a multimodal large model.

[0004] To achieve the purpose of the present invention, the following technical solutions are provided:

[0005] In the first aspect, the present invention provides a method for evaluating a text-to-image generation model based on a multimodal large model, which includes:

[0006] S1. Use a multi-modal large language model (MLLM) to perform multi-level analysis on the reference image. First, extract the overall macroscopic global features in the reference image, then screen out the key objects in the reference image according to the global features, identify the interaction relationships between pairwise key objects and perform importance scoring to screen out important interaction relationships, generate interaction relationship descriptions for the two key objects corresponding to each important interaction relationship, and finally further extract the fine-grained feature information displayed by each key object in the reference image to construct a key object information table, and merge the interaction relationship descriptions into the key object information table. Use the global features and the key object information table as the image feature matrix information output by the multi-level analysis finally;

[0007] S2. Generate an initial text prompt based on the image feature matrix information, and perform synonym expansion and semantic enhancement on the initial text prompt to generate a diverse text prompt set;

[0008] S3. Use the diverse text prompt set to drive the text-to-image generation model to be evaluated for image generation to obtain a set of generated images;

[0009] S4. Calculate the semantic similarity, visual similarity, and structural similarity between each generated image in the set of generated images and the reference image, and evaluate the quality of each generated image based on the comprehensive score combining the three similarities, and select the optimal generated image with the highest comprehensive score and the corresponding optimal text prompt;

[0010] S5. Use the multi-modal large language model to analyze the differences between the optimal generated image and the reference image in terms of semantics, vision, and structure, and feedback and improve the optimal text prompt based on the difference analysis results and use it as a new initial text prompt;

[0011] S6. Continuously iterate and execute S2~S5 until a preset stop condition is met to obtain the final multi-dimensional visual evaluation and quantitative analysis results.

[0012] As a preference of the first aspect above, the specific implementation method of S1 is as follows:

[0013] S11. Use a multi-modal large language model (MLLM) to extract the overall macroscopic global features from the input reference image, including the image theme, image style and tone, and background environment, and record the global features in dictionary form;

[0014] S12. Using a multimodal large language model (MLLM), first extract all objects from the input reference image, then screen the key objects among all objects from the visual effect level by combining the extracted global features. After orthogonal pairing of all key objects, identify the interaction relationships between key object pairs and give their importance scores. Screen out the interaction relationships with importance scores exceeding the score threshold as important interaction relationships. Finally, extract the object categories of the two key objects corresponding to each important interaction relationship, and generate an interaction relationship description of each key object relative to the other key object. The interaction relationship description includes a natural language description of the action relationship and a natural language description of the spatial relationship;

[0015] S13. For each identified key object, use a multimodal large language model (MLLM) to extract fine-grained feature information from the reference image and record it in the attribute fields of the key object information table. At the same time, record the interaction relationship description of each key object relative to other key objects in the attribute fields of the key object information table;

[0016] S14. Structurally store the global features and the key object information table obtained for the reference image to form the finally output image feature matrix information.

[0017] As a preference of the above first aspect, in S12, after extracting all objects from the input reference image and screening out the key objects, write all other objects that are not key objects into the global features in the form of auxiliary objects.

[0018] As a preference of the above first aspect, the specific implementation method of S2 is as follows:

[0019] S21: Based on the image feature matrix (ITM) information, extract the global features, the fine-grained feature information of each key object, and the interaction relationship description therefrom, and fill them into a preset hierarchical text prompt template to form an initial text prompt;

[0020] S22. Expand the initial text prompt through synonym expansion and semantic replacement to form a diverse text prompt set; wherein, the synonym expansion needs to replace the keywords in the initial text prompt with synonyms, and the semantic replacement needs to replace the original language expression in the initial text prompt with other language expressions with the same meaning.

[0021] As a preference of the above first aspect, in S3, input the text prompts in the diverse text prompt set into the text-to-image generation model to be evaluated one by one, and combine the random adjustment of the model hyperparameters to make each text prompt correspond to a batch of generated images. The generated images obtained from all text prompts constitute a generated image set.

[0022] Preferably, for the first aspect described above, the specific implementation method of S4 is as follows:

[0023] S41. For each generated image in the generated image set, calculate the similarity quantification between it and the corresponding reference image from three dimensions: semantics, vision, and structure. Among them, the semantic similarity uses the CLIP scores of the two images, the visual similarity uses the DINOv2 model to calculate the similarity scores of the two images in the visual feature space, and the structural similarity uses a multimodal large language model to evaluate the consistency scores of the two images in terms of image structure;

[0024] S42: For each generated image in the generated image set, calculate the weighted similarity of the semantic similarity, visual similarity, and structural similarity as the comprehensive score of this generated image. Select the generated image with the highest comprehensive score as the optimal generated image, and at the same time use the text prompt that generates this optimal generated image in the diverse text prompt set as the optimal text prompt.

[0025] Preferably, for the first aspect described above, the specific implementation method of S5 is as follows:

[0026] S51. Use a multimodal large language model to perform multi-dimensional comparisons of the optimal generated image and the reference image at the semantic level, visual level, and detail level to obtain the differential analysis results for each dimension; among them, at the semantic level, it is necessary to analyze whether the generated image accurately expresses the theme and main objects in the reference image, at the visual level, it is necessary to compare the differences in visual effects between the two images, and at the detail level, it is necessary to check whether the fine-grained feature information of each key object and the interaction relationship between key objects are accurate;

[0027] S52. Based on the differential analysis results obtained from the multi-dimensional comparison, generate improvement feedback suggestions for the text prompt, and let the large language model optimize the optimal text prompt based on the generated improvement feedback suggestions to generate an optimized text prompt and use it as the initial text prompt for the next round of iteration.

[0028] In the second aspect, the present invention provides a text-to-image generation model evaluation system based on a multimodal large model, which includes:

[0029] An input module for allowing a user to input a reference image set required for evaluation and specifying a text-to-image generation model to be evaluated;

[0030] An evaluation module for obtaining the multi-dimensional visual evaluation and quantification analysis results of each reference image in the reference image set according to the text-to-image generation model evaluation method based on a multimodal large model described in any one of the above first aspect solutions.

[0031] In a third aspect, the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it can implement the method for evaluating a text-to-image generation model based on a multi-modal large model according to any one of the above-mentioned first aspects.

[0032] In a fourth aspect, the present invention provides a computer electronic device, which includes a memory and a processor;

[0033] The memory is used to store a computer program;

[0034] The processor is used to implement the method for evaluating a text-to-image generation model based on a multi-modal large model according to any one of the above-mentioned first aspects when executing the computer program.

[0035] The present invention provides a method for evaluating a text-to-image generation model based on a multi-modal large model. This method can not only comprehensively evaluate the quality of the text-to-image generation model through the image regeneration task, but also improve the quality and consistency of the generation results through a multi-modal evaluation mechanism and dynamic feedback optimization. Specifically, this method comprehensively uses various evaluation models such as CLIP, DINOv2, and GPT-4V to quantitatively analyze the generated images from multiple dimensions of semantic consistency, visual features, and perceptual quality, ensuring that the generated results highly match the reference images at the global feature, object attribute, and detail levels; generating differential feedback based on the evaluation results and dynamically adjusting the prompt content to make the generation process gradually fit the features and styles of the reference images, significantly optimizing the generation quality; through iterative image generation, screening, and feedback revision, gradually improving the performance of the generated images until the set stop condition is met, and finally outputting the best generation result. In addition, this method uses an image feature matrix (ITM) to hierarchically organize image information, providing a clear guiding structure for the generation process to ensure that the model can accurately understand and restore the content and style of the reference images. By constructing a scientific evaluation and optimization system, this method not only achieves accurate evaluation of the generation quality, but also significantly improves the generation effect of the text-to-image generation model, having important application value and promotion potential. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a schematic diagram of the steps of the method for evaluating a text-to-image generation model based on a multi-modal large model;

[0037] Figure 2 It is a schematic diagram of the process for generating an image feature matrix;

[0038] Figure 3 It is a schematic diagram of the process for generating an image feature matrix for an exemplary reference image;

[0039] Figure 4Schematic diagram of the module composition of a text-to-image generation model evaluation system based on a multimodal large model;

[0040] Figure 5 Schematic diagram of the structure of a computer electronic device. Detailed implementation manners

[0041] To make the above objects, features, and advantages of the present invention more obvious and understandable, the following will describe the detailed implementation manners of the present invention with reference to the accompanying drawings. Many specific details are set forth in the following description to fully understand the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without departing from the connotation of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed below. The technical features in each embodiment of the present invention can be combined correspondingly without conflict.

[0042] Before the specific description, several concepts mentioned in the present invention are defined as follows:

[0043] Image regeneration task: A new task for evaluating a text-to-image (T2I) generation model, which requires the generation model to generate a similar image based on a given reference image. The generation ability of the model is evaluated through the visual consistency between the generated image and the reference image. Compared with the simple text prompt input, this method provides a more direct visual evaluation.

[0044] Multimodal large language model (MLLM): A deep learning model that can process multiple modalities of input such as text and images, and is used for comprehensive analysis and understanding of image and text information. The MLLM supports a wide range of applications in image understanding and generation tasks by receiving image input and generating natural language descriptions or guiding prompts, and is particularly suitable for generating semantic analysis and feedback information in image understanding trees and feedback optimization.

[0045] Image feature matrix (ITM): A structured matrix for hierarchical representation of image information, aiming to systematically organize and express the global features, object attributes, and detailed features in an image. The ITM contains three levels of information: global features (such as image theme, style, and background), object attributes (such as position, color, and interaction relationship), and fine-grained features (such as texture, shape, and decorative details). Through this hierarchical representation, the ITM can provide clear guiding information for the text-to-image generation model, ensuring that the generation process more accurately restores the image content and features.

[0046] CLIP: A multimodal model that maps text and images into a common embedding space, making similar text and images adjacent in the embedding space. The CLIP model can measure the semantic consistency between text and images and is used in this invention to generate a semantic similarity assessment of the generated image and the reference image.

[0047] DINOv2: A visual feature model that efficiently represents visual features by analyzing visual information such as color, texture, and structure in images. In this invention, DINOv2 is used to measure the visual feature consistency between the generated image and the reference image, ensuring that the generated image conforms to the features of the reference image in terms of visual elements.

[0048] GPT-4V: A multimodal large language model capable of processing visual and text information, used to generate and understand high-level semantic information. In this invention, GPT-4V is used to evaluate the fine-grained content consistency and perceptual quality of the generated image, thereby measuring the detail restoration degree and overall visual perception effect of the generated image.

[0049] Few-shot learning: A machine learning method that guides the model with a small number of examples to perform better in new tasks. In this invention, it is used to enhance the performance of MLLM in differential analysis and feedback generation, making the generated prompts more accurately improve the generation results.

[0050] This invention constructs an image regeneration task and combines it with a multimodal large language model (MLLM) to achieve multi-dimensional information parsing of the reference image and generate an initial text prompt to guide the generation model. The system uses multimodal evaluation models such as CLIP, DINOv2, and GPT-4V to comprehensively compare and accurately score the generated image from multiple dimensions such as semantic consistency, visual feature similarity, and detail fidelity, ensuring that the evaluation results highly conform to human perception standards. At the same time, by introducing an image feature matrix (ITM) to hierarchically organize image information, it provides clear hierarchical guidance for the generation process. Together with a dynamic feedback optimization mechanism, it realizes the gradual iterative improvement of the generation results and finally obtains high-quality generated images. This invention not only makes up for the deficiencies of existing evaluation methods in fine-grained visual information and semantic consistency evaluation but also significantly improves the accuracy and intuitiveness of the generated image quality evaluation through the introduction of multimodal large models, which has important practical application value for enhancing the performance of text-to-image generation models.

[0051] As Figure 1 shown, in a preferred embodiment of this invention, a method for evaluating a text-to-image generation model based on a multimodal large model is provided, and the steps are as follows:

[0052] S1: Multi-level Image Information Extraction: Use a multi-modal large language model (MLLM) to perform multi-level analysis on the reference image. First, extract the overall macroscopic global features in the reference image. Then, based on the global features, screen out the key objects in the reference image, identify and score the importance of the interaction relationships between pairwise key objects to screen out important interaction relationships, generate interaction relationship descriptions for the two key objects corresponding to each important interaction relationship, and finally further extract the fine-grained feature information displayed by each key object in the reference image to construct a key object information table, and merge the interaction relationship descriptions into the key object information table. Use the global features and the key object information table as the image feature matrix (ITM) information output by the multi-level analysis to provide multi-dimensional support for generating text prompts.

[0053] It should be noted that the form of the multi-modal large language model (MLLM) used in the present invention is not limited. For example, Tongyi Qianwen, GPT-4o, etc. can all be used.

[0054] As a specific implementation manner of the embodiment of the present invention, the ITM (Image Token Matrix) in the above step S1 is constructed by combining hierarchical tree-like thinking and tabular screening and summarization. The process is as Figure 2 shown, and the specific implementation method is as follows:

[0055] S11: Use a multi-modal large language model (MLLM) to extract the overall macroscopic global features from the input reference image, including the image theme, image style and tone, and background environment, and record the global features in dictionary form.

[0056] See Figure 3 shown, which shows an exemplary reference image of the present invention. When extracting the global features of the image, the input image can be analyzed as a whole through the multi-modal large model (MLLM), and its global feature information can be extracted and summarized, including the image theme (text description of the main scene content), image style and tone (overall style features, dominant tone), and background environment (lighting environment), etc., to form a preliminary understanding of the macroscopic semantics of the image and provide a basic reference for subsequent object screening. In this exemplary reference image, the extracted content is "A boy is running on the grass in the public area, and there is a little dog chasing beside him, and the atmosphere is happy", the style feature is "cartoon style, soft lines, medium to high color saturation", the main tone is "mainly green (grass, trees, water), supplemented by blue sky and warm-color human elements", and the lighting environment is "bright sunlight, uniform light".

[0057] S12. Utilize a multimodal large language model (MLLM) to first extract all objects from the input reference image, then screen the key objects among all objects from the visual effect level by combining the extracted global features. After orthogonal pairing of all key objects, identify the interaction relationships between key object pairs and give their importance scores. Screen out the interaction relationships with importance scores exceeding the score threshold as important interaction relationships. Finally, extract the object categories of the two key objects corresponding to each important interaction relationship, and generate an interaction relationship description of each key object relative to the other key object. The interaction relationship description includes a natural language description of the action relationship and a natural language description of the spatial relationship.

[0058] Continue with Figure 3The reference image shown is an example. The process of constructing the key object recognition and interaction relationship matrix is as follows: Based on the MLLM, all objects in the image are traversed and recognized. In this reference image, the list of all objects is: [boy, puppy, duck (by the pond / in the water), child flying a kite, running adult, couple sitting on a bench, flying bird, kite, picnic mat, trees, grass, pond]. Combining the main content description extracted in the previous step, the importance analysis and pre-screening of all objects are carried out, redundant or secondary objects are screened out, and only the set of key objects is retained. The key objects can be screened from three dimensions: narrative centrality, visual salience, and action participation. The list of key objects retained thus is: [boy, puppy, duck], while the remaining objects are all regarded as non-key objects and added to the auxiliary object list: [child flying a kite, running adult, couple sitting on a bench, flying bird, kite, picnic mat, trees, grass, pond]. On this basis, an object relationship matrix is constructed to traverse and analyze the interaction relationships (including action relationships and spatial relationships) between pairwise key objects in the form of an orthogonal table, and an importance score is given to the interaction relationships between two pairs of key objects. By setting a scoring threshold, the important interaction relationships with narrative or visual importance are marked, which serve as an important part of subsequent feature organization. The importance score can be given by the MLLM. In this exemplary reference image, the importance score between the boy and the puppy is 0.8, the importance score between the puppy and the duck is 0.1, and the importance score between the boy and the duck is 0.3, and its actual score is consistent with the actual visual effect in the reference image. Finally, the relationships between the two pairs of key objects, the boy and the puppy, and the boy and the duck, can be selected as important interaction relationships. Then, the object categories of the two key objects corresponding to each important interaction relationship are extracted by the MLLM, and an interaction relationship description of each key object relative to the other key object is generated. It should be noted that since the interaction relationship is relative, for two key objects A and B with an important interaction relationship, it is necessary to generate the action relationship and spatial relationship of the other key object relative to the reference object in the form of natural language description with key object A and key object B as the reference respectively. In this exemplary reference image, taking the boy as the reference, the generated action relationship description is "chasing the puppy", and the spatial relationship of the boy in the image is located in the "center of the grassland". Similarly, taking the puppy as the reference, the generated action relationship description is "following the boy", and the spatial relationship of the puppy in the image is located in the "right of the center of the grassland"; there is no clear action relationship between the duck and the other two key objects, so only the spatial relationship located in the "pond side" needs to be generated.

[0059] S13. For each identified key object, extract fine-grained feature information from the reference image using a multimodal large language model (MLLM), record it in the attribute fields of the key object information table, and at the same time record the description of the interaction relationship of each key object relative to other key objects in the attribute fields of the key object information table.

[0060] In the above S12 step, it is necessary to extract and summarize the fine-grained features of the key objects, and this process is also executed by the MLLM. The MLLM can further extract the fine-grained attribute information of each selected key object, including features such as color, pose, expression, material texture, accessory details, etc. The fine-grained attribute information of all objects is structurally summarized in a unified tabular form, and at the same time the important interaction relationships marked in the second step are integrated into the object feature table to form a unified information matrix, ensuring the integrated expression of object attributes and object relationships.

[0061] It should be noted that the specific attribute fields of the above key object information table can be designed according to actual needs. Continuing with Figure 3 the reference image shown as an example, the specific attribute fields of this key object information table include object ID, category, main appearance (color and clothing, etc.), pose, expression, accessories, action relationship, spatial relationship, etc.

[0062] In addition, in the embodiments of the present invention, after all objects are extracted from the input reference image and the key objects are screened out, other objects that do not belong to the key objects can be further written into the global features in the form of auxiliary objects. Continuing with Figure 3 the reference image shown as an example, the auxiliary object list [the child flying a kite, the running adult, the couple sitting on the bench, the flying bird, the kite, the picnic mat, the trees, the grassland, the pond] can be added to the global features.

[0063] S14. Structurally store the global features and the key object information table obtained for the reference image to form the finally output image feature matrix information.

[0064] In the embodiments of the present invention, the above global features and key object information table can be standardized and integrated in the form of JSOM, and finally the information is encapsulated in a unified JSON structure to realize the generation and storage of ITM. This structure not only retains the hierarchical logic in tree-like thinking but also has the advantages of tabular screening and efficient organization, and is suitable for subsequent image generation, understanding, and control tasks.

[0065] S2: Hierarchical Text Prompt Generation and Optimization: Generate an initial text prompt based on the image feature matrix information, and perform synonym expansion and semantic enhancement on the initial text prompt to generate a diverse set of text prompts, reducing the sensitivity of the generation model to a single prompt and ensuring the robustness of the results.

[0066] As a specific implementation manner of this embodiment, the specific implementation method of the above step S2 is as follows:

[0067] S21: Based on the image feature matrix (ITM) information, extract the global features, the fine-grained feature information of each key object, and the interaction relationship description therefrom, and fill them into a preset hierarchical text prompt template to form an initial text prompt.

[0068] S22. Expand the initial text prompt through synonym expansion and semantic substitution to form a diverse set of text prompts; wherein, the synonym expansion needs to replace the keywords in the initial text prompt with synonyms, and the semantic substitution needs to replace the original language expression in the initial text prompt with other language expressions with the same meaning.

[0069] For example, for a street scene image, the initial text prompt may be described as "On a busy city street, there are multiple cars of different colors, and the background is a city skyline with many high-rise buildings." Next, expand and optimize the generated text prompt, including:

[0070] Synonym expansion: Generate diverse prompt versions by replacing multiple expressions of keywords (such as replacing "busy street" with "crowded road");

[0071] Semantic optimization: Enhance specific descriptions according to the key information of the image features (such as emphasizing the architectural style or the color combination of the cars).

[0072] Finally, generate a set of diverse and hierarchical text prompts to provide input for the subsequent batch image generation step.

[0073] In the embodiment of the present invention, based on the aforementioned obtained image feature matrix (ITM) information, with "{}" as the starting identifier, the prompt word template for forming the initial text prompt is as follows:

[0074] {Global features of the image = <Main scene content description>, <Overall style feature>, <Dominant color tone>, <Lighting environment>

[0075] Descriptions of key objects in the scene = [Description of key object 1], [Description of key object 2], ……, [Description of key object n]

[0076] Descriptions of auxiliary objects in the scene = [Significant auxiliary objects in the background (optional)]}

[0077] In the above prompt template, the description of each key object = [category] + [main appearance] + [posture] + [expression] + [accessories] + [action relationship] + [spatial relationship]).

[0078] It should be noted that the specific form of this prompt template can be appropriately adjusted, and some description fields can be deleted or replaced accordingly if they do not exist.

[0079] Continuing with Figure 3 the reference image shown as an example, based on the information of the image feature matrix (ITM) obtained above, with "{}" as the starting identifier, the initial text prompt formed is as follows:

[0080] {In a sunny park, a boy and a dog are playing happily. The boy is wearing a red T-shirt and blue shorts, running in the middle of the grass, swinging his arms, with a bright smile on his face, chasing the little dog.

[0081] The little yellow-haired dog is wearing a collar and running lively on the right side of the boy, wagging its tail and sticking out its tongue excitedly.

[0082] Near a nearby pond, a yellow-feathered duck is walking quietly.

[0083] The overall is in a cartoon style, with the picture mainly in green, dotted with warm colors, the sunlight is bright and uniform, and the atmosphere is lively and cheerful.}

[0084] The present invention generates diverse prompts through semantic substitution and synonym expansion, reducing the model's dependence on a single language expression. This is mainly considering the objective situation that the actual generation performance of the text-to-image (T2I) generation model is easily affected by the form of the prompt. Because in actual applications, even different prompts expressing the same meaning may generate images of different qualities. Therefore, simply using a single prompt to generate images and then evaluating the model performance will result in inaccurate evaluation results. Moreover, in the subsequent iterative process of the present invention, a prompt optimization strategy is also introduced to adjust the prompt content according to the feedback of the generation results of different prompts, improving the semantic accuracy and generation effect of the prompt.

[0085] S3: Generating images in multiple batches: Using the diverse text prompt set to drive the text-to-image generation model to be evaluated for image generation, obtaining a set of generated images, and improving the diversity and stability of the generation results.

[0086] As a specific implementation manner of this embodiment, the specific implementation method of the above step S3 is as follows:

[0087] Input each text prompt in the diverse text prompt set generated in step S2 into a text-to-image (T2I) generation model. Each text prompt, combined with random adjustment of model hyperparameters (such as seeds or diffusion steps), correspondingly obtains a batch of generated images.

[0088] Thus, through the multi-batch generation mechanism, the diverse text prompt set uses different prompt semantic structures and model hyperparameters to generate diverse image results. The system can fully explore the performance capabilities of the generation model under different prompt conditions, providing rich candidate results for subsequent image selection. Additionally, in this step, all the generated images can be preliminarily screened manually or by invoking the MLLM to eliminate samples that significantly deviate from the prompt description, providing a high-quality candidate image set for the subsequent evaluation stage.

[0089] S4: Interactive Image Comparison and Scoring: By calculating the semantic similarity, visual similarity, and structural similarity between each generated image in the generated image set and the reference image, the quality of each generated image is evaluated based on the comprehensive score combining the three similarities, and the optimal generated image with the highest comprehensive score and the corresponding optimal text prompt are selected.

[0090] As a specific implementation manner of this embodiment, in the above step S4, by calculating the semantic, visual, and structural similarities between the generated image and the reference image, and comprehensively using multi-modal metrics such as CLIP scores, DINOv2, and GPT-4V, the quality of the generated image is evaluated and the optimal result and the corresponding prompt are selected. The specific implementation method is as follows:

[0091] S41: Multi-Dimensional Similarity Calculation: Use a variety of evaluation metrics to compare the similarity between the generated image and the reference image, and conduct a comprehensive evaluation from three dimensions: semantic, visual, and structural:

[0092] Semantic Similarity: Evaluate the semantic matching degree between the image and the reference text based on the CLIP score;

[0093] Visual Similarity: Use the DINOv2 model to calculate the similarity score between the generated image and the reference image in the visual feature space;

[0094] Structural Similarity: Evaluate the structural consistency score between the generated image and the reference image in terms of image structure such as scene layout and object relationships through a multi-modal model (such as GPT-4V).

[0095] The above semantic similarity can ensure the semantic matching between the generated image and the text prompt, and indirectly reflect the semantic consistency between the image and the reference image. The visual similarity can quantify the similarity degree between two images in terms of overall layout, color distribution, etc. The structural similarity can ensure that the candidate image can fully restore the detailed information of the reference image.

[0096] S42: Comprehensive Analysis of Scoring Results: Combine multi-dimensional similarity scores, assign weights to different scoring metrics, and generate a comprehensive score. Specifically, calculate the weighted similarity of semantic similarity, visual similarity, and structural similarity as the comprehensive score for each generated image. Thus, select the generated image with the highest comprehensive score as the optimal generated image, and at the same time use the text prompt that generates this optimal generated image in the diverse text prompt set as the optimal text prompt for subsequent optimization steps.

[0097] S43: User Interactive Adjustment (Optional): According to actual needs, provide a human-computer interaction interface that allows users to participate in the adjustment of scoring results to ensure that the final selection result meets the specific requirements of the target application scenario.

[0098] S5: Multi-dimensional Feedback and Prompt Revision: Use a multi-modal large language model to analyze the differences between the optimal generated image and the reference image in terms of semantics, vision, and structure. Based on the results of the difference analysis, feedback and improve the optimal text prompt and use it as the new initial text prompt.

[0099] As a specific implementation of the embodiment of the present invention, in the above step S5, it is necessary to generate targeted optimization feedback based on the differences between the generation result and the reference image, revise specific dimensions of the text prompt (such as position, color, or style details), and generate optimized prompt text. The specific implementation method is as follows:

[0100] S51: Analysis of Generation Result Differences: Use a multi-modal large language model to conduct multi-dimensional comparisons of the optimal generated image and the reference image at the semantic level, visual level, and detail level, obtain the difference analysis results for each dimension, and identify the deficiencies and improvement directions of the generated image:

[0101] Semantic level: Analyze whether the optimal generated image accurately expresses the theme and main objects in the reference image;

[0102] Visual level: Compare the differences between the optimal generated image and the reference image in terms of visual effects (color, lighting, texture, etc.), and locate the deviations of significant features;

[0103] Detail level: Check the accuracy of the fine-grained feature information of key objects (object shapes, detail decorations, etc.) and the interaction relationships between key objects, and identify the fine-grained features that need further optimization.

[0104] S52: Feedback Generation: Based on the difference analysis results obtained from the multi-dimensional comparison, generate improvement feedback suggestions for the text prompt.

[0105] The process of generating improvement feedback based on the results of difference analysis can be performed by a large language model. The core is to locate the key dimensions of the problem (such as color, position, or semantic content), and at the same time, relevant descriptions in the prompt need to be added or adjusted as needed, such as enhancing the emphasis on certain features.

[0106] S53: Prompt Revision: The large language model optimizes the optimal text prompt based on the generated improvement feedback suggestions. The specific optimization operations can be modifying the prompt structure or details to more accurately express the target image features, or introducing a dynamic weight mechanism in the prompt to highlight the description of important features. Thus, the improvement feedback suggestions are transformed into an optimized text prompt, generating a new prompt version, and the optimized text prompt is generated and used as the initial text prompt in S2 for the next iteration.

[0107] This step for the feedback generation and prompt revision process of the text prompt will be continuously iterated. According to the results of each iteration, the text prompt is further optimized until the generated image and the reference image meet the preset similarity requirements. Thus, the influence of the prompt form on the accuracy of model evaluation can be eliminated, and the performance of the text-to-image generation model can be evaluated more objectively. It can be seen that the multi-dimensional feedback and prompt revision operations of the present invention conduct detailed analysis and dynamic optimization by comparing the differences between the optimal generated image and the reference image. In the analysis stage, the module first comprehensively evaluates the performance of the generated image at the semantic, visual, and detail levels. At the semantic level, the evaluation ensures that the generated image accurately reflects the theme and objects of the reference image. At the visual level, the global layout, color distribution, and style consistency are compared to discover potential differences. At the detail level, the texture, shape, and mutual relationship of the objects are concerned to extract specific features that need to be optimized. Then, feedback information is generated based on the analysis results, including clear improvement suggestions and specific guidance for adjusting the prompt content. These feedback information are not limited to the adjustment of a single feature, but also cover the optimization of the language expression method to ensure that the generation model can more accurately express the details and style of the target image. The dynamic revision of the prompt content in the present invention is a key link, which makes the generation process gradually approach or exceed the quality level of the reference image through continuous optimization. This systematic method not only improves the accuracy of the performance evaluation of the generation model, but also provides reliable technical support and guarantee for high-quality image generation in practical application scenarios.

[0108] S6: Dynamic Iteration and Termination: Continuously iterate and execute S2~S5, repeat the processes of text prompt generation, image generation, image comparison and scoring, and feedback revision until the preset stop condition is met. Finally, through the optimal generated image and the reference image obtained at the end, multi-dimensional visual evaluation and quantitative analysis are carried out to evaluate the overall performance of the generation model.

[0109] As a specific implementation manner of the embodiment of the present invention, the specific implementation method of the above step S6 is as follows:

[0110] S61: Dynamic iteration start: Based on the initial image generation and prompt revision, dynamically optimize the generation result through an iterative loop, specifically including:

[0111] Generate a new prompt version using the best image generated in the previous round and its corresponding optimization prompt;

[0112] Re-enter the optimized prompt into the text-to-image (T2I) generation model to generate a new batch of candidate images.

[0113] S62: Multi-round optimization execution: In each iteration, optimize the generation effect through the following steps:

[0114] Difference feedback analysis: Perform multi-dimensional difference comparison between the image generated in the current iteration and the reference image to extract the improvement direction;

[0115] Prompt adjustment and enhancement: Optimize the prompt text according to the feedback to make it more accurately express the target image features;

[0116] Image generation and screening: Input the optimized prompt to generate a new image and screen out the optimal result.

[0117] S63: Termination condition judgment: During the iteration process, real-time evaluate the quality of the generation result, and judge whether to terminate the iteration according to the following two conditions 1) and 2). Meeting any one of the conditions can be regarded as meeting the termination condition:

[0118] 1) Reach the similarity threshold: The generated image and the reference image reach the preset threshold in the comprehensive score (such as semantic, visual, and structural similarity);

[0119] 2) Reach the iteration limit: Reach the preset maximum number of iterations to prevent ineffective repeated optimization.

[0120] S64: Final evaluation and output: After meeting the termination condition, select the optimal generated image and the corresponding optimal text prompt obtained in the last iteration as the final result, conduct a comprehensive evaluation of vision and quantification, obtain the multi-dimensional vision evaluation and quantification analysis result, and store the evaluation data to support subsequent analysis.

[0121] In the optimization process of the above S6 step, each iteration is accompanied by difference analysis and feedback generation. According to the aforementioned difference analysis results, the text prompt is revised, and it is ensured that the generated image gradually approaches the features and style of the reference image. By adjusting the parameters of the generation model, the prompt content, and the generation strategy, the system can accurately adjust the output at different iteration stages, further optimizing the generation result. When the similarity between the generated image and the reference image reaches a preset threshold, or after a certain number of iterations, the system will automatically judge whether the stop condition is met. The termination condition not only includes the quantitative standard of similarity but also takes into account the stability and diversity of the generation quality to ensure that the final image meets the high-quality generation standard. If the stop condition is satisfied, the system will output the best generated image and complete the entire evaluation process. Through this dynamic iteration and termination mechanism, the present invention can achieve continuous improvement of the generation quality in multiple rounds of generation and optimization, and while ensuring efficient generation, avoid unnecessary computational waste, and improve the overall efficiency and accuracy of evaluation and generation.

[0122] It should be noted that since model evaluation often requires a large number of reference samples, all reference samples in the reference sample dataset can be obtained corresponding multi-dimensional visual evaluation and quantitative analysis results according to the above S1~S6, so as to objectively evaluate the performance of the text-to-image generation model.

[0123] In summary, the present invention provides a text-to-image generation model evaluation system based on a multimodal large model, aiming to achieve high-precision and multi-dimensional evaluation of the generation model through the introduction of a multimodal model and a feedback optimization mechanism. In practical applications, the evaluation system of the present invention first comprehensively analyzes the reference image by using a multimodal large language model (MLLM) through a multi-level image information extraction module, constructs an image feature matrix (ITM), and extracts the global features, object attributes, and fine-grained features of the image to generate an initial text prompt for guiding the text-to-image generation model. Subsequently, in the hierarchical text prompt generation and optimization module, based on the initial text prompt or the feedback-optimized prompt set, diverse text descriptions are extended and generated to enhance the adaptability of the generation model to different prompts. In the multi-batch image generation module, a batch of candidate images are generated by inputting the optimized prompt set to ensure the diversity and stability of the generation results. Then, in the interactive image comparison and scoring module, multimodal evaluation models such as CLIP, DINOv2, and GPT-4V are used to calculate the similarity between the candidate images and the reference image from dimensions such as semantic consistency, visual features, and detail restoration, and the generation result that best matches the features of the reference image is selected, while the corresponding prompt text is recorded. Through the multi-dimensional feedback and prompt revision module, the differences between the optimal generated image and the reference image are deeply analyzed, targeted feedback information is generated, and the prompt content is adjusted to gradually optimize the generation model and gradually approach the features and styles of the reference image. The entire process can be dynamically iterated multiple times to continuously optimize the text prompt and the generation result until the preset stop condition is met. The present invention not only achieves high-precision and multi-dimensional evaluation, but also effectively improves the performance evaluation and optimization ability of the text-to-image generation model by providing an automated process for gradually improving the generation quality, and has important practical application value.

[0124] It should be emphasized that in the text-to-image generation model evaluation system based on multi-modal information extraction provided by the present invention, the design and implementation of each step are of great significance. Among them, the multi-level image information extraction step uses a multi-modal large language model (MLLM) to comprehensively analyze the reference image and construct an image feature matrix (ITM) to hierarchically organize the global features, object attributes, and fine-grained information of the image, thereby generating a highly instructive initial text prompt. The hierarchical text prompt generation and optimization step ensure the stability of the generation model under diverse prompt expressions and the diversity of generation results through synonymous expansion and semantic optimization of the text prompt. In the multi-batch image generation step, a batch of candidate images are generated based on the optimized prompt set to fully explore the model's response ability to different prompts. The interactive image comparison and scoring step combines multi-modal evaluation models such as CLIP, DINOv2, and GPT-4V to comprehensively compare the generated images from multiple dimensions such as semantic consistency, visual feature similarity, and detail restoration degree, and screen out the best generation result that is closest to the reference image. The multi-dimensional feedback and prompt revision step generates targeted feedback information and adjusts the prompt content by analyzing the differences between the optimal generated image and the reference image, so that the generation process gradually approaches the features and styles of the reference image. Finally, through the dynamic iteration and termination step, in the cycle of multi-round generation, screening, and feedback optimization, it is ensured that the quality of the generated images is continuously improved until the preset stop condition is reached. Each step in the system of the present invention plays a key role in improving the evaluation accuracy of the generation model, optimizing the generation quality, and enhancing the practical application value.

[0125] In addition, it should be noted that the steps of the present invention can be adjusted and flexibly combined according to actual needs. For example, in the multi-level image information extraction step, different multi-modal large language models (MLLM) or construction methods of the image feature matrix (ITM) can be adopted to generate text prompts suitable for diverse generation requirements. In the hierarchical text prompt generation and optimization step, various synonymous expansion and semantic enhancement strategies can be applied to improve the stability of the generation model under different prompt expressions and the diversity of generation results. In the multi-batch image generation step, different prompt expansion methods can be combined to further optimize the diversity of candidate images. In the interactive image comparison and scoring step, the evaluation model can be selected or replaced according to actual application needs, such as CLIP, DINOv2, or GPT-4V, to improve the accuracy of similarity calculation and the comprehensiveness of evaluation. In the multi-dimensional feedback and prompt revision step, different feedback templates can be selected based on the difference analysis results, or the accuracy and adaptability of feedback generation can be enhanced through few-shot learning techniques.

[0126] It should be noted that the method steps shown in S1~S6 above can essentially be implemented in the form of a computer program.

[0127] Thus, based on the same inventive concept, as Figure 4 shown, the present invention also provides a text-to-image generation model evaluation system based on a multi-modal large model, which includes:

[0128] An input module for allowing a user to input a set of reference images required for evaluation and specifying a text-to-image generation model to be evaluated;

[0129] An evaluation module for obtaining the multi-dimensional visual evaluation and quantitative analysis results of each reference image in the set of reference images according to the text-to-image generation model evaluation method based on the multi-modal large model described in the foregoing embodiments.

[0130] In addition, based on the same inventive concept, as Figure 5 shown, the present invention also provides a computer electronic device corresponding to the text-to-image generation model evaluation method based on a multi-modal large model provided in the above embodiments, which includes a memory and a processor;

[0131] The memory is used for storing a computer program;

[0132] The processor is used for implementing the text-to-image generation model evaluation method as described above when executing the computer program.

[0133] Furthermore, when the logical instructions in the above-mentioned memory are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0134] Thus, based on the same inventive concept, the present invention provides a computer-readable storage medium corresponding to the text-to-image generation model evaluation method based on a multi-modal large model. A computer program is stored on the storage medium, and when the computer program is executed by a processor, it can implement the text-to-image generation model evaluation method as described above.

[0135] Thus, based on the same inventive concept, the present invention provides a computer program product, including a computer program / instructions, which can implement the text-to-image generation model evaluation method as described above when the computer program / instructions are executed by a processor.

[0136] Specifically, in the computer-readable storage media of the above three embodiments, the stored computer program is executed by a processor, and the steps of S1 to S6 can be executed.

[0137] It can be understood that the above storage media may include a Random Access Memory (RAM), and may also include a Non-Volatile Memory (NVM), such as at least one disk memory. At the same time, the storage media may also be various media such as a USB flash drive, a mobile hard disk, a magnetic disk, or an optical disc that can store program codes.

[0138] It can be understood that the above-mentioned processor may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0139] In addition, it should be noted that those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the above-described system can refer to the corresponding process in the foregoing method embodiments, and will not be repeated here. In each embodiment provided in the present application, the division of steps or modules in the system and method is only a logical function division, and there may be other division methods in actual implementation. For example, multiple modules or steps may be combined or integrated together, and a module or step may also be split.

[0140] Next, based on the method shown in S1 to S6 in the above embodiments, it is applied to a specific example to demonstrate its effect. The specific process is as described above and will not be repeated. The following mainly shows its specific parameter settings and implementation effects.

[0141] Embodiment

[0142] In this embodiment, 50 complex reference pictures (the number of objects is greater than or equal to 5) are used as the reference image data set, and the model evaluation is carried out according to the text-to-image generation model evaluation method based on the multi-modal large model shown in the foregoing S1 to S6 steps. The foregoing S1 to S6 steps will not be repeated here, and the specific data set, some specific parameter settings, and implementation results of this embodiment will be mainly shown.

[0143] Meanwhile, to better demonstrate the advantages of the Image Feature Matrix (ITM) adopted in step S1 of the present invention, this embodiment also designs a comparative experiment. In the comparative experiment, after using a Multimodal Large Language Model (MLLM) to perform multi-level analysis on a reference image in step S1, an Image Understanding Tree (IUT) is constructed to record the multi-level information in the image, which is used to generate an initial text prompt in step S2. The remaining steps S2 to S6 of the comparative experiment are the same as those of the present invention.

[0144] In the IUT structure, image information unfolds in a tree-like hierarchical manner, which can comprehensively capture global features, objects, and fine-grained attributes, but there are problems such as excessive redundant information, unclear primary and secondary information, and unstable generation. On the basis of retaining the hierarchical understanding logic, the ITM proposed in the present invention actively screens the main objects and key relationships, standardizes the information into a matrix form, and organizes it in order of importance. Compared with the IUT, the ITM significantly improves the generation friendliness, length controllability, and main line clarity, and at the same time shows better adaptability and local control ability in complex scenarios, making it suitable for efficient and high-quality image generation tasks. The comparison between the two is shown in Table 1 below:

[0145] Table 1

[0146]

[0147] After evaluating 50 complex pictures in the reference image set, the construction speed (using GPT4o as the MLLM), prompt length, and prompt key information retention rate of the IUT and ITM are shown in Table 2:

[0148] Table 2

[0149]

[0150] Based on the above comparison results, it can be seen that the IUT organizes the global features, object entities, and their fine-grained attributes in the image through a tree-like hierarchical manner, which can better reflect the hierarchical structure and detail distribution inside the image, and is helpful for comprehensively analyzing the image content, especially suitable for tasks that require fine-grained reasoning and full-scene analysis. However, when actually applied to complex image generation tasks, especially when the goal is to generate high-quality and stable output, the IUT has certain limitations. The main problems include: excessive information redundancy, unclear primary and secondary information, the prompt length is easily exceeded and important information is easily lost during truncation, and the processing and scheduling cost is relatively high in complex scenarios with multiple objects.

[0151] Based on this, the present invention proposes the Image Token Matrix (ITM) as an optimized version of the IUT structure. While retaining the concept of "hierarchical understanding" of the IUT, ITM further introduces mechanisms for active screening, standardized structural reorganization, and importance ranking. Specifically, ITM first scores the importance of all objects through comprehensive indicators such as narrative centrality, visual salience, and action engagement, and screens out key objects; subsequently, only the action-based or narrative-driven relationships directly associated with these main objects are retained; finally, the global features, basic object information, and object fine-grained features are organized into a unified and standardized matrix-style three-layer structure. After such processing, ITM can not only effectively compress irrelevant information but also ensure that the main content is preferentially expressed in the prompt, significantly improving the stability and controllability of the generation task.

[0152] Compared with the original IUT, ITM exhibits the following advantages.

[0153] First, in terms of the structure organization method, ITM changes from a nested tree structure to a flattened matrix structure, greatly reducing the complexity of information access and parsing, and being more suitable for text generation tasks.

[0154] Second, in terms of highlighting the primary and secondary information, through the importance ranking mechanism, ITM makes the generated prompt naturally centered around key objects and key relationships, avoiding the dilution of the main content by redundant details.

[0155] Third, in terms of prompt length control and stability, ITM can dynamically adjust the amount of information according to the model token limit, ensuring that key information is prioritized and not truncated during generation.

[0156] Fourth, in terms of local control and flexible scalability, the matrix structure makes local modifications to a single object or a single attribute efficient and simple, facilitating subsequent expansion in applications such as fine-grained editing, region-based generation, and instance condition control. Finally, in terms of adaptability to complex scenarios, the information scale of ITM grows linearly, enabling it to better handle complex images with multiple objects and multiple relationships without a significant decline in system performance due to information explosion.

[0157] It should be noted that in the process of simplification and focusing, ITM does discard information about secondary objects or background micro-details to a certain extent. Therefore, in inference or traceability analysis tasks that extremely rely on the retention of all details, the original IUT still has its irreplaceable value. However, in application scenarios oriented to efficient image generation and high-quality visual output, ITM demonstrates more superior performance and adaptability compared to IUT in core dimensions such as information compressibility, generation friendliness, and main line clarity.

[0158] In addition, in this embodiment, the practices of the text-to-image generation model evaluation method (I-I eval-ITM in the table) and the comparative experiment (I-I eval-IUT) of the multimodal large model of the present invention are also analyzed for their relevance to human evaluation compared with other existing evaluation methods (GPT4-V, T2I-CompBench, HPSv2, PickScore). The results are shown in Table 3, indicating that the evaluation method of the present invention has achieved better results:

[0159] Table 3

[0160]

[0161] The embodiments described above are only a preferred solution of the present invention, but they are not intended to limit the present invention. Those of ordinary skill in the relevant technical field can still make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, all technical solutions obtained by means of equivalent replacement or equivalent transformation fall within the protection scope of the present invention.

Claims

1. A method for evaluating a text-to-image generation model based on a multimodal large model, characterized in that: include: S1. Use a multimodal large language model to perform multi-level analysis on the reference image. First, extract the overall macroscopic global features in the reference image. Then, screen out the key objects in the reference image based on the global features. Identify and score the importance of the interaction between the two key objects to screen out the important interaction relationships. Generate an interaction relationship description for the two key objects corresponding to each important interaction relationship. Finally, further extract the fine-grained feature information displayed by each key object in the reference image to construct a key object information table, and merge the interaction relationship description into the key object information table. Use the global features and the key object information table as the image feature matrix information finally output by the multi-level analysis. S2, generating an initial text prompt based on the image feature matrix information, and performing synonym expansion and semantic enhancement on the initial text prompt to generate a diversified text prompt set; S3, using the diversified text prompt set to drive the text-to-image generation model to be evaluated to generate images, and obtaining a generated image set; S4, evaluating the quality of each generated image by calculating the semantic similarity, visual similarity and structural similarity between each generated image in the generated image set and the reference image, and selecting the best generated image with the highest comprehensive score and the corresponding best text prompt; S5, using a multimodal large language model to analyze the differences between the optimal generated image and the reference image in terms of semantics, vision, and structure, and providing feedback and improving the optimal text prompt based on the difference analysis results and using it as a new initial text prompt; S6. Continuously iterate and execute S2 to S5 until the preset stop condition is met to obtain the final multi-dimensional visual evaluation and quantitative analysis results.

2. The method for evaluating a text-to-image generation model based on a multimodal large model according to claim 1, characterized in that: The specific implementation method of S1 is as follows: S11. Use a multimodal large language model to extract overall macroscopic global features from the input reference image, including image theme, image style tone and background environment, and record the global features in the form of a dictionary; S12, using a multimodal large language model, first extract all objects from the input reference image, then combine the extracted global features to screen the key objects among all objects from the visual effect level, after orthogonally pairing all the key objects, identify the interaction relationship between the key object pairs and give their importance scores, screen out the interaction relationships whose importance scores exceed the score threshold as important interaction relationships, finally extract the object categories of the two key objects corresponding to each important interaction relationship, and generate an interaction relationship description of each key object relative to another key object, wherein the interaction relationship description includes a natural language description of the action relationship and a natural language description of the spatial relationship; S13, for each key object identified, extract fine-grained feature information from the reference image using a multimodal large language model, and record it in an attribute field of the key object information table, and record the interactive relationship description of each key object relative to other key objects in the attribute field of the key object information table; S14: Structurally store the global features and the key object information table obtained for the reference image to form image feature matrix information that is finally output.

3. The method for evaluating a text-to-image generation model based on a multimodal large model according to claim 2, characterized in that: In S12, after all objects are extracted from the input reference image and key objects are screened out, all other objects that are not key objects are written into the global features in the form of auxiliary objects.

4. The method for evaluating a text-to-image generation model based on a multimodal large model according to claim 1, characterized in that: The specific implementation method of S2 is as follows: S21: based on the image feature matrix information, extracting global features and fine-grained feature information and interactive relationship description of each key object, and filling them into a preset hierarchical text prompt template to form an initial text prompt; S22, expanding the initial text prompt by synonym expansion and semantic replacement to form a diversified text prompt set; wherein, the synonym expansion needs to replace the keywords in the initial text prompt with synonyms, and the semantic replacement needs to replace the original language expression in the initial text prompt with other language expressions with the same meaning.

5. The method for evaluating a text-to-image generation model based on a multimodal large model according to claim 1, characterized in that: In S3, the text prompts in the diversified text prompt set are input into the image generation model one by one along with the text to be evaluated, and combined with the random adjustment of the model hyperparameters, a batch of generated images are obtained for each text prompt, and the generated images obtained from all the text prompts constitute a generated image set.

6. The method for evaluating a text-to-image generation model based on a multimodal large model according to claim 1, characterized in that: The specific implementation method of S4 is as follows: S41. For each generated image in the generated image set, similarity is quantitatively calculated between it and the corresponding reference image from the three dimensions of semantics, vision and structure, wherein the semantic similarity uses the CLIP scores of the two images, the visual similarity uses the DINOv2 model to calculate the similarity scores of the two images in the visual feature space, and the structural similarity uses a multimodal large language model to evaluate the consistency scores of the two images in terms of image structure; S42: For each generated image in the generated image set, the weighted similarity of semantic similarity, visual similarity and structural similarity is calculated as the comprehensive score of this generated image, and the generated image with the highest comprehensive score is selected as the optimal generated image. At the same time, the text prompt of the optimal generated image is generated in the diversified text prompt set as the optimal text prompt.

7. The method for evaluating a text-to-image generation model based on a multimodal large model according to claim 1, characterized in that: The specific implementation method of S5 is as follows: S51. Use a multimodal large language model to perform a multi-dimensional comparison of the optimal generated image with the reference image at the semantic level, visual level, and detail level to obtain a difference analysis result for each dimension; the semantic level needs to analyze whether the generated image accurately expresses the theme and main object in the reference image, the visual level needs to compare the difference in visual effects between the two images, and the detail level needs to check whether the fine-grained feature information of each key object and the interaction relationship between each key object are accurate; S52. Based on the difference analysis results obtained by multi-dimensional comparison, improved feedback suggestions for the text prompt are generated, and the large language model optimizes the optimal text prompt based on the generated improved feedback suggestions, and generates an optimized text prompt as the initial text prompt in the next round of iteration.

8. A text-to-image generation model evaluation system based on a multimodal large model, characterized in that: include: An input module, used for allowing the user to input a reference image set required for evaluation and specifying a text-to-image generation model to be evaluated; An evaluation module is used to obtain a multi-dimensional visual evaluation and quantitative analysis result of each reference image in a reference image set according to the text-to-image generation model evaluation method based on a multimodal large model as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, it can implement the text-to-image generation model evaluation method based on a multimodal large model as described in any one of claims 1 to 7.

10. A computer electronic device, characterized in that: including memory and processor; The memory is used to store computer programs; The processor is used to implement the text-to-image generation model evaluation method based on a multimodal large model as described in any one of claims 1 to 7 when executing the computer program.

Images