Image-text pair quality evaluation method based on enhanced questions and answers

Abstract

The invention discloses an image-text pair quality evaluation method based on enhanced questions and answers. The image-text pair quality evaluation method comprises the following steps: carrying out key semantic element analysis on a cue word text and converting the key semantic element into a structured assertion expression; based on the structured assertion representation, converting the structured assertion representation into a plurality of visual questions and answers by using a large language model; inputting the visual question-answering questions into the visual question-answering model, obtaining the answer prediction confidence of each question, and calculating the matching degree score from the text to the image based on all the answer prediction confidence; performing content analysis on the generated image and converting the generated image into a structured semantic element list; inputting the image assertion representation and the cue word text into a large language model, performing consistency matching analysis, and outputting a matching degree score from the image to the text; and integrating the two scores to calculate and obtain a comprehensive quality evaluation score of the image-text pair. According to the method, the relationship between the text and the image content is modeled bidirectionally, quality evaluation considering global semantic consistency and local detail matching is realized, and the matching quality of the image-text pair is effectively improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of multimodal large models, and particularly relates to a picture-text pair quality evaluation method based on enhanced question answering. BACKGROUND

[0002] In recent years, artificial intelligence (AI) technology has developed rapidly, especially in the field of multimodal large models (MLLM), where many excellent technologies have emerged. The most notable of these are DELL-E, ImageFX, and Midjourney, which are text-to-image models that can generate high-quality, content-rich, and stylistically diverse images based on user input prompts. As these technologies continue to mature, they have shown unprecedented potential in creative design, film production, and game development. However, the images generated by these models do not have specific labels, and how to evaluate whether these images follow the instructions and meet the user's needs has become an important problem that needs to be solved. Unlike traditional image classification or detection tasks, the output of text-to-image models is highly open and diverse, and there is a lack of unified and objective labeling standards. The generated results may have semantic deviation, detail omission, or even semantic inconsistency, and these problems are difficult to discover through image quality evaluation alone. Therefore, how to accurately evaluate whether the generated images accurately meet the content of the prompts has become a major challenge in the field of multimodal research.

[0003] Using human labor to evaluate the generated images and corresponding prompts is the simplest and most effective way, but when faced with a large amount of generated data, the cost of human labor will skyrocket, making it difficult to meet the needs of large-scale evaluation. In addition, the generated images and prompts can be encoded and features extracted using image and text encoders, respectively, and then the cosine similarity between the image features and the text features can be calculated as an evaluation standard. However, even the most advanced vision-language models (VLM) still face a significant challenge in accurately aligning visual-linguistic features.

[0004] Currently, the technical route for evaluating generated images and corresponding prompt words mainly includes two directions: feature embedding-based measurement and image-text content-based measurement. In the embedding-based measurement, the quality of the image-text pair is evaluated based on the visual and text embedding feature representations. Typically, VLM models such as CLIP and BLIP are used to extract image and text features, which are then input into a trained measurement classifier. The quality of the image-text pair is determined based on the classifier score. The content-based measurement analyzes the relationship between the image and the text content, and evaluates the quality of the image-text pair based on this relationship. The prompt word text is usually first decomposed into semantic units, and the objects, attributes, relationships, and scenes contained in the text are parsed into independent sub-components. Then, the presence of these elements in the image and their correct relationships are detected or located to achieve a more fine-grained consistency evaluation of the image and the text.

[0005] Currently, these two types of image-text pair evaluation methods still have many limitations. The embedding-based measurement method mainly relies on the global semantic feature vectors extracted by the pre-trained model on large-scale image-text pair data to calculate the similarity. Although it is efficient, it is difficult to capture the precise correspondence between fine-grained semantic elements in the prompt word and the image content, and it is easy to miss the semantic elements while still getting a high score. On the other hand, the content-based measurement method usually relies on the reasoning ability of large language models to analyze the prompt word and detect the corresponding elements in the image. However, it often only focuses on one-way matching from text to image, ignoring the reverse verification from image to text, resulting in a lack of constraints on redundant or incorrect generated content. Therefore, exploring a quality evaluation method that can model the relationship between text and image content in both directions, while considering both global semantic consistency and local detail matching, has important practical significance for improving the matching quality between generated images and prompt word texts. SUMMARY

[0006] In view of the above, the purpose of this invention is to provide a method for evaluating the quality of image-text pairs based on augmented question answering, and to design a measurement index that can accurately evaluate the consistency between images generated by a multimodal large model and prompt text. This method first uses a Large Language Model (LLM) to augment the prompt text, converting a single prompt text into several fine-grained sub-questions about the generated image. Then, a Visual Question Answering (VQA) query is constructed for each sub-question. The VQA model is used to answer the questions, and the "text-to-image" matching degree is calculated based on the confidence of the correct answer. Next, for the generated image, an image description model is used to convert it into multiple descriptive texts. These descriptive texts and the prompt text are input into the LLM for semantic relevance comparison. The "image-to-text" matching degree is calculated based on the LLM output score, and the final quality evaluation result of the image-text pair is determined by combining the two matching degrees.

[0007] To achieve the above-mentioned objectives, the present invention provides the following technical solution: In a first aspect, the present invention provides a method for evaluating the quality of text-image pairs based on enhanced question answering, comprising the following steps: The key semantic elements of the prompt text are parsed, and the objects, attributes and relationships in the prompt text are extracted and converted into structured assertion representations. Based on structured assertion representation, a large language model is used to convert it into multiple visual question-answering questions, each containing a question and its corresponding standard answer; Input visual question answering questions into the visual question answering model, obtain the answer prediction confidence for each question, and calculate the text-to-image matching score based on the prediction confidence of all answers; The generated image is parsed to extract objects, attributes, and relationships from the image and convert them into a list of structured semantic elements. The image assertion representation and the prompt text are input into a large language model for consistency matching analysis, and the image-to-text matching score is output. By combining the text-to-image matching score and the image-to-text matching score, a comprehensive quality assessment score for the text-image pair is calculated.

[0008] Preferably, the step of parsing key semantic elements of the prompt text, extracting objects, attributes, and relationships from the prompt text, and converting them into structured assertion representations includes: Using a large language model, semantic elements of the prompt text are parsed to identify and extract three types of key textual semantic elements contained in the image generation target defined in the prompt text: objects, attributes, and relationships. Among them, objects refer to entities that can be clearly identified in the image, attributes refer to the features of objects, and relationships refer to the interaction or structural relationships between objects. Based on the extracted text semantic key elements, a structured assertion representation is generated. The assertion representation includes at least: existence assertions generated based on each object, descriptive assertions generated based on each object and its attributes, and relational assertions generated based on the relationships between objects. Existential assertions, descriptive assertions, and relational assertions are merged to form a set of structured assertions for subsequent question-and-answer generation.

[0009] Preferably, the structured assertion representation is converted into multiple visual question-answering questions using a large language model, each question containing a question and a corresponding standard answer, including: Using a large language model, existential and relational assertions are transformed into judgmental questions requiring a yes / no answer, and descriptive assertions are transformed into open-ended questions requiring specific attribute values. For each assertion, at least one corresponding visual question-answering question is generated, and at the same time, the standard answer corresponding to the question is generated, forming multiple visual question-answering questions for subsequent visual question-answering model validation.

[0010] Preferably, the step of inputting visual question-answering questions into a visual question-answering model, obtaining the answer prediction confidence for each question, and calculating the text-to-image matching score based on the prediction confidence of all answers includes: Input all the questions from multiple visual question answering questions into the visual question answering model, and obtain the model's predicted answer for each question and the corresponding predicted probability distribution; For each question, its predicted answer is matched with the corresponding standard answer, and its confidence level is determined according to the question type: for judgment questions, the predicted probability of the affirmative answer is taken as the confidence level; for open-ended questions, the confidence level is calculated based on whether the predicted answer is consistent with the standard answer. The text-to-image matching score is obtained by arithmetically averaging the confidence scores of all questions.

[0011] Preferably, the step of performing content parsing on the generated image, extracting objects, attributes, and relationships from the image, and converting them into a list of structured semantic elements includes: The generated images are parsed using a multimodal large language model to identify and extract three types of semantic key elements: objects, attributes, and relationships. Among them, objects refer to clearly identifiable entities in the image, attributes refer to the visual features of objects, and relationships refer to the interaction or structural relationships between objects. The final result is a structured semantic element list, which includes at least: an object list, recording all the main objects identified in the image; an attribute list, recording each object and its corresponding visual features; and a relationship list, recording the interaction or structural relationship triples between objects.

[0012] Preferably, the step of inputting the image assertion representation and the prompt word text into a large language model, performing consistency matching analysis, and outputting an image-to-text matching score includes: The structured semantic element list is converted into a set of textual representations describing the image content. Each descriptive text in the textual representation set and its corresponding prompt text are input into the large language model. The large language model determines whether each description is reflected in the prompt text based on semantic implication and outputs the corresponding confidence score. The image-to-text matching score is obtained by arithmetically averaging the confidence scores of all descriptions.

[0013] Preferably, the calculation of the comprehensive quality assessment score for the image-text pair by fusing the text-to-image matching score and the image-to-text matching score includes: The text-to-image matching score and the image-to-text matching score are weighted and linearly fused to obtain a comprehensive evaluation score. Based on the comprehensive evaluation score, the quality of the text-image pair is automatically evaluated.

[0014] Secondly, embodiments of the present invention also provide a text-image pair quality assessment device based on enhanced question answering, which is implemented using the above-mentioned text-image pair quality assessment method based on enhanced question answering, including: a text semantic parsing module, a visual question answering generation module, a text-image matching assessment module, an image semantic parsing module, a text-image matching assessment module, and a comprehensive quality assessment module; The text semantic parsing module is used to parse key semantic elements of the prompt word text, extract objects, attributes and relationships in the prompt word text and convert them into structured assertion representations; The visual question-answering generation module is used to convert structured assertion representations into multiple visual question-answering questions using a large language model. Each question contains a question and a corresponding standard answer. The text-image matching evaluation module is used to input visual question answering questions into the visual question answering model, obtain the answer prediction confidence of each question, and calculate the text-to-image matching score based on the prediction confidence of all answers. The image semantic parsing module is used to parse the content of the generated image, extract the objects, attributes and relationships in the image and convert them into a list of structured semantic elements; The image-text matching evaluation module is used to input the image assertion representation and the prompt word text into the large language model, perform consistency matching analysis, and output the image-to-text matching score. The comprehensive quality assessment module is used to integrate the text-to-image matching score and the image-to-text matching score to calculate the comprehensive quality assessment score of the image-text pair.

[0015] Thirdly, embodiments of the present invention also provide an electronic device, including a memory and one or more processors, wherein the memory is used to store a computer program, and the processor is used to implement the above-described enhanced question-answering-based graph-text pair quality assessment method when the computer program is executed.

[0016] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a computer, implements the above-described method for evaluating the quality of text-image pairs based on enhanced question answering.

[0017] Compared with the prior art, the beneficial effects of the present invention include at least the following: (1) The bidirectional question-answering enhanced image-text pair quality assessment method proposed in this invention can effectively improve the assessment accuracy. This method, on the one hand, enhances the prompt text into multiple fine-grained sub-questions using a large language model, and combines this with a VQA model to verify the image content item by item, achieving comprehensive coverage of the semantic elements of the prompt text; on the other hand, it performs structured analysis on the generated image content, and then compares it with the prompt text in reverse, effectively discovering redundant elements in the image that are not mentioned in the text. Compared with existing methods based solely on one-way matching or global embedding similarity, this invention can simultaneously detect both missing and redundant content, significantly improving the consistency between the assessment results and human subjective judgment.

[0018] (2) This invention utilizes a large language model to perform semantic element parsing and assertion construction on the prompt text, decomposing complex prompts into multiple dimensions such as objects, attributes, and relationships, and generating decidable VQA questions based on these elements to achieve fine-grained verification of the generated image. This "enhanced question answering" mechanism breaks through the bottleneck of traditional global embedding-based methods that are difficult to capture local semantic correspondences, and can accurately identify deviations in the generated image at the level of details such as color, action, and spatial relationships, making the evaluation results more robust.

[0019] (3) The evaluation process of this invention adopts a modular design. The steps of text parsing, VQA verification, image parsing and semantic comparison can be replaced or extended independently, which can flexibly adapt to multimodal large models of different sizes and various application scenarios. The system as a whole only relies on the general VQA model and LLM interface, without manual annotation, and supports fully automatic operation on large-scale generated image datasets, which significantly reduces the cost of manual evaluation. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating the image-text pair quality assessment method based on enhanced question answering provided in an embodiment of the present invention. Figure 2 This is a schematic diagram of the framework of the image-text pair quality assessment method based on enhanced question answering provided in the embodiments of the present invention; Figure 3 This is a schematic diagram of the structure of the image-text pair quality assessment device based on enhanced question-and-answer provided in an embodiment of the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.

[0023] The inventive concept of this invention is as follows: Addressing the difficulty in quantifying and assessing the semantic consistency between prompt text and generated images in existing technologies, this invention proposes a text-image pair quality assessment method based on enhanced question answering. The core idea is to parse the prompt text into fine-grained semantic elements, generate multiple check questions and answers for the image, and then combine the image parsing results to reverse-verify text coverage. This achieves bidirectional matching scoring between "text → image" and "image → text," comprehensively measuring the consistency and completeness of the text-image pair.

[0024] To implement this invention, the data and model interface required for the evaluation method are first set up, including but not limited to: the input prompt text. Image generated based on prompt words This involves setting up a VQA image question answering model, an LLM model interface for extracting semantic elements from prompt text, transforming VQA questions, and evaluating the matching score between the image feature list text and the prompt text, and an MLLM for extracting image object attributes and relational features. The prompt text and the generated image constitute the image-text pair to be evaluated. In the configuration of the VQA image question answering model, key parameters such as temperature during LLM inference, sampling probability (Top_P), maximum number of token outputs, inference parallel strategy, and GPU utilization need to be set.

[0025] likeFigure 1 and Figure 2 As shown, the embodiment provides a method for evaluating the quality of text-image pairs based on enhanced question answering, including the following steps: S1 performs key semantic element parsing on the prompt text, extracts the objects, attributes, and relationships in the prompt text, and converts them into structured assertion representations.

[0026] In this embodiment, this step requires semantic element decomposition of the prompt text in the image-text pair to extract the core content that should be included in the image generation target. This invention defines semantic key elements into three categories: Entity: A clearly identifiable entity in an image, such as a person, animal, object, or scene element; Attribute: The characteristics that an object possesses, such as color, shape, size, posture, mood, material, quantity, etc. Relation: The interaction or structural relationship between objects, such as spatial position relationship (above), action relationship (eating), interaction relationship (holding), etc.

[0027] To achieve automated parsing, this invention leverages the powerful natural language understanding and instruction following capabilities of large language models, and designs the following prompt words to perform structured parsing of the prompt word text: Object extraction prompts: You are a text parser. Please extract all objects (entities) that can be clearly identified in the image from the following descriptive text, such as people, animals, objects, scene elements, etc. Input text: "{text}" Output in JSON format: {"objects": ["object1", "object2", "object3",...]} You are a text parser. Please extract the "attributes" related to the objects from the following descriptive text, including color, shape, size, posture, emotion, material, quantity, etc. Input text: "{text}" Object: "{object}" ...]}” ; Attribute extraction words: Output in JSON format: {"attributes": [{"object": "object name", "attribute": "attribute1"},...]} You are a text parser. Please extract the "relationships" between objects from the following descriptive text and object list, including spatial location relationships (such as above, beside), interaction relationships (such as holding, looking at), action relationships (such as eating, running), etc. Input text: "{text}". Object list: {objects} Output in JSON format: {"relations": [{"subject": "object A", "relation": "relationship", "object": "object B"},...]} You are a VQA question generator. Now input is an "object existence" or relationship assertion that describes an entity object or relationship between entity objects that should exist in the image. Please convert the assertion into a clear, specific, and determinable yes / no question, ensuring that the question only targets the existence of the object. Input assertion: "{assertion}" ; Relationship extraction keywords: Output format: { "assertion": "There is a small girl in the picture", "questions": ["Is there a small girl in the picture?"], "answers": ["Yes"]} You are a VQA question generator. Now input is an "attribute" assertion that describes an attribute (color, shape, quantity, size, posture, etc.) of an object in the image. Please convert the assertion into at least one question to ask. The question must clearly mention the object and attribute. Input assertion: "{assertion}" Output format: { "assertion": "The girl's hat is red", "questions": ["What color is the girl's hat?", "answers": ["Red"]} You are an image understanding expert. Now input is given to you an image, please carefully analyze the content contained in it and extract the core elements that can be used for semantic matching. Your task is: 1. Identify all "objects" (such as people, animals, objects, scene elements, etc.) that can be clearly identified in the image; 2. Extract the attributes of each object, including color, shape, size, posture, emotion, material, quantity, etc. ​ .

[0028] The specific execution process is as follows: First, the prompt text is... and Input into LLM and retrieve text The LLM contains a list of objects, objects = {A, B, C, ...}. Then, iterate through the object list, combining each object with the original text and inputting it into the LLM to obtain its corresponding attribute list, attributes = {(A, attribute A), (B, attribute B), (C, attribute C), ...}. Finally, the object list is combined with the prompt text. Inputting LLM together generates a list of object relation triples: relations={(A, relation, B), (A, relation, C), (B, relation, C), ...}.

[0029] After completing the above analysis, based on the three types of elements—objects, attributes, and relationships—LLM is used to uniformly convert them into a set of assertions, including: Set of existence assertions : Generate one entry for each object, such as "There is a {object} in the diagram"; Descriptive assertion set : Generate one record for each attribute, such as "{object}'s {attribute category} is {attribute value}"; Set of relational assertions Each relation generates one entry, such as "{object A}{relational verb}{object B}".

[0030] Ultimately, all types of assertions will be merged into a single assertion set. .

[0031] S2, based on structured assertion representation, uses a large language model to convert it into multiple visual question-answering questions, each containing a question and its corresponding standard answer.

[0032] In this embodiment, the goal of this step is to convert the above set of assertions into question-answer pairs acceptable to the VQA model, so that the VQA model can be used to verify whether the image contains the content described by the prompt text.

[0033] The specific strategies are as follows: For existence and relational assertions, convert them into yes / no (Yes / No) type judgment questions. For example: assertion "There is a cat in the picture" → question "Is there a cat in the picture?" → answer "Yes"; assertion "The cat is sitting on the sofa" → question "Is the cat sitting on the sofa?" → answer "Yes"; For descriptive assertions, convert them into open-ended "what" questions (WH-questions) so that the VQA model outputs specific attribute values ​​as the answer. For example: assertion "The little girl's hat is red" → question "What color is the little girl's hat?" → answer "red".

[0034] To achieve automatic conversion, this invention sets the following prompt word template: Existence / relational assertions → Yes / No questions: ​ ​ ​ ​ ​ ; Descriptive assertion → WH question: ​ ​ ​ ​ ​ .

[0035] Using the aforementioned prompts, this invention can automatically convert structured assertions into question statements and corresponding standard answers that can be directly processed by the VQA model, achieving fully automatic generation from prompt text to VQA verification questions.

[0036] S3 inputs the visual question answering questions into the visual question answering model, obtains the answer prediction confidence for each question, and calculates the text-to-image matching score based on the prediction confidence of all answers.

[0037] In this embodiment, after the assertion is converted into a VQA question, the VQA model is used to reason about the generated question and answer it. The matching score of "text to image" is calculated based on the confidence of the correctness of the answer in order to evaluate whether the generated image accurately contains the semantic elements described by the prompt text.

[0038] The specific process is as follows: First, based on the results in step S2, construct a list of questions. and the corresponding answer list The system then inputs each question from the question list into a pre-defined VQA model for batch inference; subsequently, it calculates the answer matching confidence score for each question. The VQA model will output a predicted answer. and its corresponding predicted probability distribution Compare the model's output answer with the expected answer. Perform matching. For "yes / no" questions, determine whether the output is "yes" and record the "yes" category probability given by the model. As the confidence score "for what type of problem", the precise matching strategy is used to determine whether the predicted value is consistent with the actual attribute, and the confidence score is calculated based on this. The calculation formula is as follows: , in, For a picture and text pair The corresponding set of all questions, A confidence score is assigned to each question. This step yields a quantitative consistency score indicating the semantic matching degree between the prompt text and the generated image. .

[0039] S4 performs content parsing on the generated image, extracting objects, attributes, and relationships from the image and converting them into a structured semantic element list.

[0040] In this embodiment, after obtaining the "text-to-image" matching score, this step aims to perform reverse parsing on the generated image content in order to construct a feature representation for calculating the "image-to-text" matching score.

[0041] Specifically, this step first designs a set of entity-level questions for the generated image, including the main objects present in the image, the salient attributes of these objects, and the spatial relationships between objects, to guide the multimodal model in outputting structured content information. Based on this, the following prompts were designed: ​ ​ ​ "attributes", including color, shape, size, number, pose, emotion, material, etc. "relations", including spatial position relations (such as above, left, front), interaction relations (such as holding, looking at), and action relations (such as eating, running). Input: {image}. Output: {"objects": ["object1", "object2", "object3",...], "attributes": [{"object": "object name", "attribute": "attribute1"}], "relations": [{"subject": "object A", "relation": "relation", "object": "object B"}]} Figure 3 ​ .

[0042] By inputting the image along with the aforementioned prompts into MLLM, the originally unstructured image content can be transformed into a structured semantic element representation, laying the foundation for the next step of calculating the matching degree of "image to text".

[0043] S5 inputs the image assertion representation and the prompt text into the large language model, performs consistency matching analysis, and outputs the image-to-text matching score.

[0044] In this embodiment, after obtaining the generated image content features (object, attribute, relationship) parsed in step S4, this step aims to reverse verify whether the image content is consistent with the semantic requirements of the prompt text, that is, to calculate the "image to text" matching score.

[0045] The specific process is as follows: First, the structured elements (objects, attributes, relationships) parsed from the image are integrated into multiple descriptive text statements. Each statement fully describes the existence state of a certain element or combination of elements in the image. These descriptive texts constitute the textual representation set of the image content. Then, each element in the set is represented by an image and text. Compared with the original prompt text T All of these are input into a pre-configured LLM, which is required to determine whether each textual representation is present in the original prompt word text based on semantic implications. T This is reflected in the data, and the corresponding confidence score is output. Finally, all text is combined to represent the final result. The mean confidence score is used to calculate the image-to-text matching score. The specific calculation formula is as follows: , in, This represents the image-to-text matching score. This indicates the number of image-to-text samples. This step allows for a quantitative measurement of whether the content presented in the generated image fully covers the semantic elements of the prompt text, and detects whether there is any redundant content generated in the image that is not mentioned in the prompt.

[0046] S6. By fusing the text-to-image matching score and the image-to-text matching score, a comprehensive quality assessment score for the text-image pair is calculated.

[0047] In the embodiment, the "text-to-image" matching score is obtained respectively. Image-to-text matching score Then, a weighted linear fusion is performed on the two, and the "text-to-image" matching degree and the "image-to-text" matching degree are weighted by a coefficient. and The weighted sum is used to obtain the overall quality assessment score. : .

[0048] This step yields a final quality score that comprehensively assesses the consistency between text and images, enabling automated scoring of text-to-image results for multimodal large-scale models.

[0049] Based on the same inventive concept, such as ​ As shown, this embodiment of the invention also provides a text-image pair quality assessment device 300 based on enhanced question answering, including: a text semantic parsing module 310, a visual question answering generation module 320, a text-image matching assessment module 330, an image semantic parsing module 340, a text-image matching assessment module 350, and a comprehensive quality assessment module 360.

[0050] The text semantic parsing module 310 is used to parse key semantic elements of the prompt text, extract objects, attributes and relationships in the prompt text and convert them into structured assertion representations.

[0051] The visual question answering generation module 320 is used to convert structured assertion representations into multiple visual question answering questions based on a large language model. Each question contains the question and the corresponding standard answer.

[0052] The text-image matching evaluation module 330 is used to input visual question answering questions into the visual question answering model, obtain the answer prediction confidence of each question, and calculate the text-to-image matching score based on the prediction confidence of all answers.

[0053] The image semantic parsing module 340 is used to parse the content of the generated image, extract the objects, attributes and relationships in the image and convert them into a structured list of semantic elements.

[0054] The image-text matching evaluation module 350 is used to input the image assertion representation and the prompt word text into the large language model, perform consistency matching analysis, and output the image-to-text matching score.

[0055] The comprehensive quality assessment module 360 ​​is used to integrate the text-to-image matching score and the image-to-text matching score to calculate the comprehensive quality assessment score of the image-text pair.

[0056] Based on the same inventive concept, embodiments of the present invention also provide an electronic device, including a memory and one or more processors, wherein the memory is used to store a computer program, and the processor is used to implement the above-described enhanced question-answering-based image-text pair quality assessment method when executing the computer program.

[0057] Based on the same inventive concept, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a computer, implements the above-described enhanced question-answering-based image-text pair quality assessment method.

[0058] It should be noted that the image-text pair quality assessment device, electronic device, and computer-readable storage medium based on enhanced question answering provided in the above embodiments all belong to the same inventive concept as the image-text pair quality assessment method based on enhanced question answering. For details of their specific implementation process, please refer to the embodiments of the image-text pair quality assessment method based on enhanced question answering, which will not be repeated here.

[0059] The specific embodiments described above illustrate the technical solution and beneficial effects of the present invention in detail. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for assessing the quality of text-image pairs based on enhanced question answering, characterized in that, Includes the following steps: The key semantic elements of the prompt text are parsed, and the objects, attributes and relationships in the prompt text are extracted and converted into structured assertion representations. Based on structured assertion representation, a large language model is used to convert it into multiple visual question-answering questions, each containing a question and its corresponding standard answer; Input visual question answering questions into the visual question answering model, obtain the answer prediction confidence for each question, and calculate the text-to-image matching score based on the prediction confidence of all answers; The generated image is parsed to extract objects, attributes, and relationships from the image and convert them into a list of structured semantic elements. The image assertion representation and the prompt text are input into a large language model for consistency matching analysis, and the image-to-text matching score is output. By combining the text-to-image matching score and the image-to-text matching score, a comprehensive quality assessment score for the text-image pair is calculated.

2. The image-text pair quality assessment method based on enhanced question answering according to claim 1, characterized in that, The step of parsing key semantic elements of the prompt text, extracting objects, attributes, and relationships from the prompt text, and converting them into structured assertion representations includes: Using a large language model, semantic elements of the prompt text are parsed to identify and extract three types of key textual semantic elements contained in the image generation target defined in the prompt text: objects, attributes, and relationships. Among them, objects refer to entities that can be clearly identified in the image, attributes refer to the features of objects, and relationships refer to the interaction or structural relationships between objects. Based on the extracted text semantic key elements, a structured assertion representation is generated. The assertion representation includes at least: existence assertions generated based on each object, descriptive assertions generated based on each object and its attributes, and relational assertions generated based on the relationships between objects. Existential assertions, descriptive assertions, and relational assertions are merged to form a set of structured assertions for subsequent question-and-answer generation.

3. The image-text pair quality assessment method based on enhanced question answering according to claim 2, characterized in that, The structured assertion representation is transformed into multiple visual question-answering questions using a large language model. Each question contains a question and a corresponding standard answer, including: Using a large language model, existential and relational assertions are transformed into judgmental questions requiring a yes / no answer, and descriptive assertions are transformed into open-ended questions requiring specific attribute values. For each assertion, at least one corresponding visual question-answering question is generated, and at the same time, the standard answer corresponding to the question is generated, forming multiple visual question-answering questions for subsequent visual question-answering model validation.

4. The image-text pair quality assessment method based on enhanced question answering according to claim 3, characterized in that, The process of inputting visual question-answering questions into a visual question-answering model, obtaining the predicted confidence score of the answer for each question, and calculating the text-to-image matching score based on the predicted confidence scores of all answers includes: Input all the questions from multiple visual question answering questions into the visual question answering model, and obtain the model's predicted answer for each question and the corresponding predicted probability distribution; For each question, its predicted answer is matched with the corresponding standard answer, and its confidence level is determined according to the question type: for judgment questions, the predicted probability of the affirmative answer is taken as the confidence level; for open-ended questions, the confidence level is calculated based on whether the predicted answer is consistent with the standard answer. The text-to-image matching score is obtained by arithmetically averaging the confidence scores of all questions.

5. The image-text pair quality assessment method based on enhanced question answering according to claim 1, characterized in that, The step of parsing the generated image, extracting objects, attributes, and relationships from the image, and converting them into a list of structured semantic elements includes: The generated images are parsed using a multimodal large language model to identify and extract three types of semantic key elements: objects, attributes, and relationships. Among them, objects refer to clearly identifiable entities in the image, attributes refer to the visual features of objects, and relationships refer to the interaction or structural relationships between objects. The final result is a structured semantic element list, which includes at least: an object list, recording all the main objects identified in the image; an attribute list, recording each object and its corresponding visual features; and a relationship list, recording the interaction or structural relationship triples between objects.

6. The image-text pair quality assessment method based on enhanced question answering according to claim 5, characterized in that, The process of inputting the image assertion representation and the prompt word text into a large language model, performing consistency matching analysis, and outputting an image-to-text matching score includes: The structured semantic element list is converted into a set of textual representations describing the image content. Each descriptive text in the textual representation set and its corresponding prompt text are input into the large language model. The large language model determines whether each description is reflected in the prompt text based on semantic implication and outputs the corresponding confidence score. The image-to-text matching score is obtained by arithmetically averaging the confidence scores of all descriptions.

7. The image-text pair quality assessment method based on enhanced question answering according to claim 1, characterized in that, The fusion of text-to-image matching score and image-to-text matching score yields a comprehensive quality assessment score for the image-text pair, including: The text-to-image matching score and the image-to-text matching score are weighted and linearly fused to obtain a comprehensive evaluation score. Based on the comprehensive evaluation score, the quality of the text-image pair is automatically evaluated.

8. A device for evaluating the quality of text-image pairs based on enhanced question answering, implemented using the text-image pair quality evaluation method based on enhanced question answering as described in any one of claims 1 to 7, characterized in that, include: The system includes a text semantic parsing module, a visual question answering generation module, a text-image matching evaluation module, an image semantic parsing module, an image-text matching evaluation module, and a comprehensive quality evaluation module. The text semantic parsing module is used to parse key semantic elements of the prompt word text, extract objects, attributes and relationships in the prompt word text and convert them into structured assertion representations; The visual question-answering generation module is used to convert structured assertion representations into multiple visual question-answering questions using a large language model. Each question contains a question and a corresponding standard answer. The text-image matching evaluation module is used to input visual question answering questions into the visual question answering model, obtain the answer prediction confidence of each question, and calculate the text-to-image matching score based on the prediction confidence of all answers. The image semantic parsing module is used to parse the content of the generated image, extract the objects, attributes and relationships in the image and convert them into a list of structured semantic elements; The image-text matching evaluation module is used to input the image assertion representation and the prompt word text into the large language model, perform consistency matching analysis, and output the image-to-text matching score. The comprehensive quality assessment module is used to integrate the text-to-image matching score and the image-to-text matching score to calculate the comprehensive quality assessment score of the image-text pair.

9. An electronic device comprising a memory and one or more processors, the memory for storing a computer program, characterized in that, The processor is used to implement the enhanced question-answering-based image-text pair quality assessment method according to any one of claims 1 to 7 when executing a computer program.

10. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by a computer, it implements the image-text pair quality assessment method based on enhanced question answering as described in any one of claims 1 to 7.

Images