Evaluation method and device of text graph model, electronic equipment and medium

Abstract

The embodiment of the invention provides an evaluation method and device of a text graph model, electronic equipment and a medium. The method comprises the following steps: inputting a test text into a to-be-evaluated text graph model to obtain a plurality of to-be-evaluated images; obtaining a decision index score according to the to-be-evaluated image; wherein the decision index comprises alignment capability and fidelity; a quality index score is obtained according to the decision index and the to-be-evaluated images, and the quality index at least comprises one of the consistency degree of alignment items of the test text and the to-be-evaluated images, the difference degree between the to-be-evaluated images, the aesthetic degree, the toxicity, the prejudice, the robustness, the generation efficiency and the figure defect degree; and according to the decision index score and the quality index score, determining a total evaluation score of the to-be-evaluated declaration graph model. By determining the index score and the quality index score, the accurate total evaluation score of the to-be-evaluated text graph model can be obtained.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an evaluation method, device, electronic device, and medium for a text graph model. Background Art

[0002] The Wenshengtu model is a cross-modal deep learning model that can parse and understand input text descriptions and generate corresponding images by learning from large-scale image and text data. This technology combines the two major fields of natural language processing and computer vision, demonstrating the great potential and broad prospects of artificial intelligence in content generation.

[0003] The existing technology evaluates the Wensheng graph model by designing certain evaluation items. On the one hand, the evaluation tasks are not comprehensive enough and cannot adapt to all Wensheng graph models on the market; on the other hand, all evaluation tasks are treated equally, lacking a three-dimensional and hierarchical evaluation system. Moreover, the scoring method is not unified, including both graded scoring and percentage scoring. Therefore, for large Wensheng graph models, the existing technology has the problem of inaccurate and incomplete evaluation. Summary of the invention

[0004] The embodiments of the present application provide a method, device, electronic device, and medium for evaluating a large-scale Wensheng graph model, so as to achieve an effect of accurately and comprehensively evaluating a large-scale Wensheng graph model.

[0005] In a first aspect, an embodiment of the present application provides a method for evaluating a text graph model, comprising:

[0006] Input the test text into the to-be-evaluated text-image model to obtain multiple to-be-evaluated images;

[0007] According to the image to be evaluated, a decision index score is obtained; wherein the decision index includes alignment ability and fidelity; the alignment ability represents the degree of consistency between the image to be evaluated and the test text elements; the fidelity represents the degree of difference between the image to be evaluated and the standard image set;

[0008] According to the decision indicator and the image to be evaluated, a quality indicator score is obtained, wherein the quality indicator includes at least one of the consistency of the alignment items of the test text and the image to be evaluated, the difference between the images to be evaluated, the aesthetics, the toxicity, the bias, the robustness, the generation efficiency, and the character defect degree;

[0009] The total evaluation score of the document graph model to be evaluated is determined based on the decision indicator scores and quality indicator scores.

[0010] In a possible implementation, obtaining a quality index score according to the decision index and the image to be evaluated includes:

[0011] Perform weighted summation on the scores of various decision indicators to obtain the total decision indicator score;

[0012] The quality index score is obtained according to the preset threshold, the total decision index score and the image to be evaluated.

[0013] In a possible implementation, obtaining a quality index score according to the decision index and the image to be evaluated includes:

[0014] If the total decision indicator score is less than the preset threshold, the total evaluation score of the document graph model to be evaluated is obtained according to the decision indicator score;

[0015] If the total decision indicator score is greater than or equal to the preset threshold, the total evaluation score of the document graph model to be evaluated is obtained according to the decision indicator score and the quality indicator score.

[0016] In a possible implementation, determining the total evaluation score of the document graph model to be evaluated according to the decision indicator score and the quality indicator score includes:

[0017] Obtain the weight of the first indicator, the weight of the second indicator, the weight of each decision indicator, and the weight of each quality indicator;

[0018] According to the decision indicator score, the first indicator weight, the quality indicator score, the second indicator weight, the weights of each decision indicator, and the weights of each quality indicator, the total evaluation score of the Wensheng graph model to be evaluated is obtained; wherein, the first indicator weight represents the influence of the decision indicator on the total evaluation score of the Wensheng graph model to be evaluated, and the second indicator weight represents the influence of the quality indicator on the total evaluation score of the Wensheng graph model to be evaluated; the weights of each decision indicator represent the influence of each decision indicator on the decision indicator, and the weights of each quality indicator represent the influence of each quality indicator on the quality indicator.

[0019] In a possible implementation, the total evaluation score of the document graph model to be evaluated is obtained according to the decision indicator score, the first indicator weight, the quality indicator score, the second indicator weight, the weights of various decision indicators, and the weights of various quality indicators, including:

[0020] According to the weight of each decision indicator, the decision indicator scores are weighted and summed to obtain the total score of the decision indicator;

[0021] According to the weight of each quality indicator, the quality indicator scores are weighted and summed to obtain the total quality indicator score;

[0022] According to the total score of the decision indicator, the weight of the first indicator, the total score of the quality indicator, and the weight of the second indicator, the total score of the document graph model to be evaluated is obtained.

[0023] In a possible implementation, the total evaluation score of the document graph model to be evaluated is obtained according to the decision indicator score and the quality indicator score, including:

[0024] Obtaining a personalized index score according to the image to be evaluated, wherein the personalized index includes at least one of the image style, text embedded in the image, controllability evaluation, re-creation evaluation, and robustness;

[0025] According to the decision index score, quality index score and personalized index score, the total evaluation score of the document graph model to be evaluated is obtained.

[0026] In a possible implementation, the total evaluation score of the document graph model to be evaluated is obtained according to the decision index score, the quality index score and the personalized index score, including:

[0027] Determine the correlation indicators among the decision indicators, the quality indicators, and the personalized indicators; wherein the correlation indicators include the first correlation indicators and the second correlation indicators;

[0028] Obtaining the indicator difference between the first correlation indicator and the second correlation indicator;

[0029] If the indicator difference exceeds the preset threshold, the target indicator score in the associated indicator is lowered; wherein the target indicator is the largest indicator between the first associated indicator and the second associated indicator;

[0030] If the target indicator is the decision indicator, the total evaluation score of the document graph model to be evaluated is obtained according to the target indicator score and the quality indicator score.

[0031] If the target indicator is a quality indicator, the total evaluation score of the document graph model to be evaluated is obtained according to the target indicator score and the decision indicator score.

[0032] In a second aspect, an embodiment of the present application provides an evaluation device for a text graph model, including:

[0033] A module for inputting a test text into a text-image model to be evaluated to obtain a plurality of images to be evaluated;

[0034] The first acquisition module is used to obtain the decision index score according to the image to be evaluated; wherein the decision index includes alignment ability and fidelity; the alignment ability represents the degree of consistency between the image to be evaluated and the test text element; the fidelity represents the degree of difference between the image to be evaluated and the standard image set;

[0035] A second obtaining module is used to obtain a quality indicator score according to the decision indicator and the image to be evaluated, wherein the quality indicator at least includes one of the consistency of the alignment items of the test text and the image to be evaluated, the difference between the images to be evaluated, the aesthetics, toxicity, bias, robustness, generation efficiency, and the degree of character defects;

[0036] The determination module is used to determine the total evaluation score of the document graph model to be evaluated according to the decision indicator score and the quality indicator score.

[0037] In a third aspect, an embodiment of the present application provides an electronic device, including: a memory, a processor;

[0038] Memory stores computer-executable instructions;

[0039] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the above first aspect and / or various possible implementations of the first aspect.

[0040] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the first aspect above and / or various possible implementations of the first aspect.

[0041] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the above first aspect and / or various possible implementation methods of the first aspect.

[0042] The embodiment of the present application provides a method, device, electronic device, and medium for evaluating a text-based graph model. By inputting a test text into the text-based graph model to be evaluated, a plurality of images to be evaluated are obtained; according to the images to be evaluated, a decision index score is obtained; wherein the decision index includes alignment capability and fidelity; the alignment capability represents the degree of consistency between the image to be evaluated and the test text elements; the fidelity represents the degree of difference between the image to be evaluated and the standard image set; according to the decision index, the basic performance of the image to be evaluated can be accurately evaluated, thereby improving the evaluation accuracy. According to the decision index and the image to be evaluated, a quality index score is obtained, wherein the quality index includes at least one of the degree of consistency of the alignment items of the test text and the image to be evaluated, the degree of difference between each image to be evaluated, the aesthetics, toxicity, bias, robustness, generation efficiency, and the degree of character defects; according to the decision index score and the quality index score, the total evaluation score of the text-based graph model to be evaluated is determined. According to the decision index and the quality index, the image to be evaluated can be comprehensively and accurately evaluated. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0044] Figure 1 A flowchart of a method for evaluating a text graph model provided in this application;

[0045] Figure 2A schematic diagram of a process for obtaining a test image according to a test field provided by the present application;

[0046] Figure 3 A schematic diagram of an alignment capability evaluation process provided for this application;

[0047] Figure 4 A schematic diagram of another evaluation method of the text graph model provided in this application;

[0048] Figure 5 A schematic diagram of the structure of the evaluation of the Wensheng graph model provided in this application;

[0049] Figure 6 A structural schematic diagram of an electronic device is provided for this application.

[0050] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0051] Here, example embodiments are described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following example embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0052] At present, the Wenshengtu large model has demonstrated extensive application value in many key fields, including but not limited to creative design, advertising production, game development, virtual reality, and assisted medical treatment. These models can not only improve production efficiency, but also inspire creators' inspiration and promote the automation and intelligence of visual content generation. However, faced with the dazzling array of Wenshengtu large model products on the market, users often face selection difficulties and urgently need a standardized and comprehensive evaluation system to scientifically evaluate the performance and characteristics of each model, so as to help users accurately match products that meet their needs. At the same time, the evaluation of Wenshengtu large models can also form a positive feedback mechanism for large model product suppliers. Through the evaluation results, the product party can clearly identify the shortcomings of its own model, and then optimize and improve it in a targeted manner, ultimately promoting the continuous improvement of product quality in the entire industry. At the same time, the evaluation of a large number of indicators for all models will cause a lot of waste of computing resources.

[0053] The embodiment of the present application provides a method, device, electronic device and medium for evaluating a Wensheng graph model. By evaluating the decision index of the image to be evaluated, the decision index score of the image to be evaluated is obtained, and then the quality index score is obtained according to the decision index and the image to be evaluated. By setting the decision index, it is determined whether to obtain the quality index score, which can reduce unnecessary steps of the quality index score evaluation of the model, thereby improving efficiency and reducing computing costs. At the same time, by setting the decision index score and the quality index score, the total evaluation score of the Wensheng graph model is obtained, which can evaluate the Wensheng graph model more comprehensively and accurately.

[0054] First, the terms involved in this application are explained:

[0055] Text-to-image model: refers to a generative model that aims to generate corresponding images based on the input text description. This type of model combines natural language processing and computer vision technology to understand and interpret text descriptions, and then generate visual content that matches the description.

[0056] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0057] Figure 1 A flowchart of a method for evaluating a text graph model provided in this application is shown in FIG. Figure 1 As shown, the method includes:

[0058] S101: Input a test text into a text-image model to be evaluated to obtain a plurality of images to be evaluated.

[0059] The test text is the text used to generate the image to be evaluated. The test text can be automatically generated using LLM (Large Language Model). You can preset multiple test fields such as people, objects, backgrounds, colors, locations, character attributes, time, seasons, weather, materials, and quantities. The system automatically combines several test fields to allow LLM to generate test text containing test content. Users can also freely select built-in test fields or add new test fields to generate test text in a specified direction or add test tasks.

[0060] The test text is input into the large model of the text image to be evaluated to generate the image to be evaluated. Each generated image to be evaluated contains multiple images.

[0061] Specifically, Figure 2 A schematic diagram of a process for obtaining a test image according to a test field provided by the present application. Figure 2 As shown, in response to the user randomly selecting the test fields "person", "background", "plant", "position", and "color", a prompt is input to the LLM: an input text is generated for the Wenshengtu model, which contains person, background, plant, position, and color. Then LLM returns the generated test text: In the golden field with the morning light, a girl in a white dress stands in a gorgeous sunflower field with her back to the rising sun. Her smile is set off by the golden sunflowers around her, forming a warm and bright picture. Among them, the person is "a girl", the background is "the golden field with the morning light" and "the rising sun", the plant is "sunflower", the description of the position is "standing in...", "back to", and the words related to color are "gold", "white", and "golden". LLM generates a sentence as the input text for testing the Wenshengtu model based on the input prompt. The sentence is clear and complete, and contains all the specified test fields.

[0062] S102. Obtain a decision index score based on the image to be evaluated; wherein the decision index includes alignment capability and fidelity; the alignment capability represents the degree of consistency between the image to be evaluated and the test text element; and the fidelity represents the degree of difference between the image to be evaluated and the standard image set.

[0063] Among them, the alignment ability reflects whether the text-image model contains the elements described in the test text, and gives a matching degree, that is, a score. The text elements represent the content in the test text, which can be: "character", "background", "plant", "location", "color", etc. The alignment ability is automatically evaluated by using VQA (Visual Question Answering). Figure 3 A schematic diagram of an alignment capability evaluation process provided for this application is shown in FIG. Figure 3 As shown in the figure, the test text is input into the large language model to obtain the question, and then the question and the test image (i.e., the image to be evaluated) are input into the multimodal large model to obtain the answer to the question. According to the answer to the question, the alignment ability score is obtained. For example, the question can be: "Does the image contain..." or "Is... in the image?" The answer to the question can be: "The image contains...", "... is not in the image." According to the percentage of the meaning of "in" in the answer to the question in all questions, the alignment ability score can be obtained.

[0064] Fidelity refers to whether the objects and characters in the image are complete and whether the image as a whole gives people a comfortable feeling. Fidelity is scored by a preset neural network model. The training of this preset neural network model uses an image data set containing positive and negative samples, where positive samples are a set of images with completeness and comfort, and negative samples are a set of images with defects obtained by image deformation, erasing some areas, image shearing, color transformation, and distortion of the face and body of the person. The trained neural network model can give a fidelity score based on the distance between the input image and the positive and negative image sets.

[0065] Optionally, a quality indicator score is obtained according to the decision indicator and the image to be evaluated, including:

[0066] Perform weighted summation on the scores of various decision indicators to obtain the total decision indicator score;

[0067] The quality index score is obtained according to the preset threshold, the total decision index score and the image to be evaluated.

[0068] Among them, different weights are preset for fidelity and alignment ability respectively, and then the fidelity and alignment ability are weighted and summed according to the preset different weights to obtain the total decision indicator score.

[0069] By comparing the preset threshold and the total decision index, it is possible to basically determine whether the image to be measured is a usable image. It is also a direct indicator for determining whether the cultural image model to be evaluated can be put into application.

[0070] The quality index is applicable to all large models of text and image. In addition to the decisive index, it is a detailed and comprehensive evaluation index that reflects the quality of large models of text and image. It is responsible for several special evaluations, such as: evaluation of image aesthetics, evaluation of the matching degree of attribute description, quantity description, and position description in the test text, evaluation of the diversity of generated images, evaluation of the support range of text and symbols, evaluation of the robustness of input text, evaluation of generation efficiency, etc. These evaluation components are also automatically evaluated by combining the evaluation model with LLM, without manual intervention.

[0071] According to the comparison between the preset threshold and the total decision index, when the total decision index is greater than the preset threshold, the quality index score is obtained according to the image to be evaluated.

[0072] The advantage of this setting is that through the weighted summation method, the influence of multiple decisive indicators can be integrated to provide a more comprehensive image quality evaluation. The weight of each indicator can be adjusted according to different application scenarios to adapt to different quality requirements.

[0073] Optionally, a quality index score is obtained according to a preset threshold, a total decision index score and the image to be evaluated, including:

[0074] If the total decision indicator score is less than the preset threshold, the total evaluation score of the document graph model to be evaluated is obtained according to the decision indicator score;

[0075] If the total decision indicator score is greater than or equal to the preset threshold, the total evaluation score of the document graph model to be evaluated is obtained according to the decision indicator score and the quality indicator score.

[0076] Among them, the decision indicator represents the basic usage conditions of the user, which can determine whether it is necessary to obtain the quality indicator score in the later stage. If the Wensheng graph model to be evaluated does not meet the basic usage conditions of the user (that is, the decisive indicator score of the Wensheng graph model to be evaluated is lower than the preset threshold), then there is no need to obtain the quality indicator score. If the Wensheng graph model to be evaluated meets the basic usage conditions of the user (that is, the Wensheng graph model to be evaluated is higher than the preset threshold), it is necessary to obtain the quality indicator score. In this way, a more comprehensive and accurate evaluation result can be obtained.

[0077] If the score of the decisive indicator of the Wensheng graph model to be evaluated is lower than the preset threshold, the total score of the Wensheng graph model to be evaluated will be obtained directly according to the decisive indicator score. Specifically, the decisive indicator score can be substituted into the preset formula, where the score of the quality indicator item is 0. Finally, the evaluation result is generated, which will emphasize that the Wensheng graph model to be evaluated has not passed the decisive indicator test and does not meet the basic user conditions.

[0078] If the large model of the Wensheng graph to be evaluated is higher than the preset threshold, it indicates that the Wensheng graph model to be evaluated has passed the decisive indicator test and meets the basic user usage conditions. Then, based on the image to be evaluated, the quality indicator score of the image to be evaluated is obtained. Then, based on the decisive indicator score and the quality indicator score, the total evaluation score of the Wensheng graph model to be evaluated is obtained.

[0079] The advantage of this setting is that when evaluating the quality of the text graph model, the evaluation process can be flexibly adjusted in combination with specific scenarios by setting the threshold. Specifically, when the user needs to perform personalized evaluation and quality indicator evaluation on all models, the threshold is lowered. When the user does not need to perform personalized evaluation and quality indicator evaluation on all models, and needs to confirm the model with better indicators and improve efficiency, the threshold is raised. This can provide more detailed and accurate feedback, which is suitable for application scenarios that require high precision and flexibility.

[0080] S103. Obtain a quality index score based on the decision index and the image to be evaluated, wherein the quality index includes at least one of the consistency of the alignment items of the test text and the image to be evaluated, the difference between the images to be evaluated, aesthetics, toxicity, bias, robustness, generation efficiency, and the degree of character defects.

[0081] Among them, the consistency evaluation items of the alignment items of the test text and the image to be evaluated include attributes, positions, quantity, color, time, action, etc. Specifically, the consistency score of the alignment items of the test text and the image to be evaluated is obtained by adopting the VQA method, such as questions about attributes such as "Is the girl in the image smiling happily?", questions about positions such as "Is the girl in the image facing away from the sun?", questions about quantity such as "Does the image contain...?", questions about colors such as "Is the skirt worn by the girl in the image white?", questions about time such as "Is the time in the image morning?", questions about actions such as "Is the puppy in the image running?", etc. The result of statistical representation is obtained. According to the result of representation and the total number of questions, the consistency score of the alignment items of the test text and the image to be evaluated can be obtained.

[0082] The degree of difference between the images to be evaluated indicates that the images to be evaluated cannot be completely the same or have a high degree of similarity. Specifically, all the images to be evaluated are input into the image feature extraction module to extract image features, and then the diversity score (based on the degree of difference between the images to be evaluated) is obtained according to the distance between each image feature. The farther the image feature distance is, the higher the diversity of the generated image is (i.e., the greater the difference between the images to be evaluated is), and the closer the feature distance is, the lower the diversity is (i.e., the smaller the difference between the images to be evaluated is).

[0083] Aesthetics is a highly subjective indicator, usually used to evaluate the visual pleasure and attractiveness of an image. The evaluation images are directly scored mainly through a pre-trained neural network model. Specifically, the pre-trained neural network model can be a CNN (Convolutional Neural Network) model.

[0084] Toxicity characterization is to determine whether the image to be evaluated contains insults, discrimination, indecency, and other harmful images. Preset questions, such as "Does the image contain violence or gore?", input the generated image and the preset questions into the pre-trained model for toxicity scoring. Specifically, the results of characterization are counted, and the toxicity score can be obtained based on the results of characterization and the total number of questions.

[0085] Bias is evaluated for gender, age, and ethnic bias. The evaluation text set is also pre-set. First, LLM generates questions related to gender, age, and sex based on the test text, such as "Is the person in the image a woman?", "Is the image a young woman?", "Is the woman in the image an oriental or western image?", etc. Then the multimodal large model answers these questions, and LLM gives a comprehensive score based on the answers.

[0086] Robustness is an evaluation of the input text and symbols that can be recognized or understood by the large text-image model. The test also presets a test text set, which mainly contains a series of different texts with similar descriptions and texts with special symbols. For example: whether rare descriptions can generate the same image, whether synonyms and transliterated words can generate the same image, whether different symbols can generate the same image, and then use the VQA method to evaluate and score the generation results. Specifically, the statistical representation is the result, and the robustness score can be obtained based on the result of the representation and the total number of questions.

[0087] Generation efficiency refers to the computing power and time efficiency required to generate an image. A special test text set is set to ensure that the test texts used by different text-generated image models in this test are consistent. Finally, the efficiency is calculated based on the computing power and time consumed by the generated image, and the score is given based on the generation efficiency and time. The corresponding relationship between the score and time can be preset. The less time is used, the higher the score.

[0088] The degree of character defects is determined by defect detection for the characters in the image, and the preset test text set is used for evaluation. The test text set is also generated by LLM, but it is focused on generating test texts for images with different characteristics, such as close-ups of characters, groups of characters, interactions between characters, and interactions between people and objects. When evaluating, first determine whether the image contains a face and a body, and then input the face defect detection model, body defect detection model, and character defect detection model for detection. The face defect detection model, body defect detection model, and character defect detection model are all pre-trained neural network models, which are trained by a series of positive and negative samples of face images and character images. Negative samples are obtained by applying certain strategies to modify normal images, such as erasing part of the body or part of the face, adding fingers or arms, etc.

[0089] When it is determined that the decisive indicator meets certain conditions, the quality indicator score is obtained according to the image to be evaluated. Specifically, when the decisive indicator score is greater than a preset threshold, the quality indicator score is obtained according to the image to be evaluated.

[0090] S104. Determine the total evaluation score of the document graph model to be evaluated according to the decision indicator score and the quality indicator score.

[0091] Substituting the decision indicator score and the quality indicator score into the preset formula, the total evaluation score of the document graph model to be evaluated can be obtained.

[0092] The present application provides an evaluation method for a text graph model, which obtains a total evaluation score of the text graph model to be evaluated by determining the index score and the quality index score. By comprehensively considering the determination index and the quality index, the evaluation is made more comprehensive and accurate.

[0093] In the above embodiment of the present application, the total evaluation score of the document graph model to be evaluated is determined according to the decision index score and the quality index score, including:

[0094] Obtain the weight of the first indicator, the weight of the second indicator, the weight of each decision indicator, and the weight of each quality indicator;

[0095] According to the decision indicator score, the first indicator weight, the quality indicator score, the second indicator weight, the weights of each decision indicator, and the weights of each quality indicator, the total evaluation score of the Wensheng graph model to be evaluated is obtained; wherein, the first indicator weight represents the influence of the decision indicator on the total evaluation score of the Wensheng graph model to be evaluated, and the second indicator weight represents the influence of the quality indicator on the total evaluation score of the Wensheng graph model to be evaluated; the weights of each decision indicator represent the influence of each decision indicator on the decision indicator, and the weights of each quality indicator represent the influence of each quality indicator on the quality indicator.

[0096] The weight of the first indicator and the weight of the second indicator are preset by the user. They can be adjusted according to the specific scenario. For example, when the decision indicator needs to be considered, the weight of the first indicator is increased. On the contrary, when the quality indicator needs to be considered, the weight of the second indicator is increased. It should be noted that the sum of the weight of the first indicator and the weight of the second indicator must be 1.

[0097] The sum of the weights of each decision indicator is 1, and each decision indicator refers to: fidelity and alignment ability.

[0098] The sum of the weights of various quality indicators is 1. The various quality indicators refer to: the consistency of the alignment items of the test text and the image to be evaluated, the degree of difference between the images to be evaluated, the aesthetics, toxicity, bias, robustness, generation efficiency, and the degree of character defects. The weights of various decision indicators and quality indicators are preset by the user. When a decision indicator needs to be considered, the weight of the decision indicator can be increased, and when a quality indicator needs to be considered, the weight of the quality indicator can be increased.

[0099] The weight of the first indicator, the weight of the second indicator, the weight of each decision indicator, and the weight of each quality indicator are preset respectively, and then the scores of each decision indicator, the weight of the first indicator, the scores of each quality indicator, the weight of the second indicator, the weight of each decision indicator, and the weight of each quality indicator are substituted into the preset formula to obtain the total evaluation score of the document graph model to be evaluated.

[0100] The advantage of this setting is that by introducing the weight of the first indicator and the weight of the second indicator, the influence of the decision indicator and the quality indicator on the total score of the model evaluation can be more accurately reflected. This weight distribution mechanism makes the evaluation results more in line with actual application needs. The setting of the weights of each decision indicator and each quality indicator allows the evaluation system to be adjusted according to different application scenarios or needs, providing a flexible evaluation framework. Through reasonable weight distribution and comprehensive indicator considerations, the accuracy, flexibility and practicality of the Wenshengtu model evaluation have been significantly improved.

[0101] In the above embodiment of the present application, the total evaluation score of the document graph model to be evaluated is obtained according to the decision indicator score, the first indicator weight, the quality indicator score, the second indicator weight, the weight of each decision indicator, and the weight of each quality indicator, including:

[0102] According to the weight of each decision indicator, the decision indicator scores are weighted and summed to obtain the total score of the decision indicator;

[0103] According to the weight of each quality indicator, the quality indicator scores are weighted and summed to obtain the total quality indicator score;

[0104] According to the total score of the decision indicator, the weight of the first indicator, the total score of the quality indicator, and the weight of the second indicator, the total score of the document graph model to be evaluated is obtained.

[0105] The total score of the decision indicator is represented by the weighted sum of the scores of each decision indicator and the preset weights corresponding to each preset decisive indicator.

[0106] The total score of the quality indicator is represented by the weighted sum of the scores of each quality indicator and the preset weights corresponding to each preset quality indicator.

[0107] The total score of the Wensheng graph model to be evaluated is:

[0108] ,

[0109] Among them, F is the total score of the document graph model to be evaluated, For each decision indicator score, is the weight of each decision indicator, is the quality score of each item, is the quality indicator weight, α is the first indicator weight, β is the second indicator weight, To determine the total score of the indicator, is the total score of quality indicators.

[0110] Specifically, the total score A of the decisive indicator is obtained by the alignment ability score × the preset weight of the alignment ability + the fidelity score × the preset weight of the fidelity, and the total score B of the quality indicator is obtained by the consistency degree of the alignment items of the test text and the image to be evaluated × the preset weight of the consistency degree of the alignment items of the test text and the image to be evaluated + the difference degree score between the images to be evaluated × the preset weight of the difference degree between the images to be evaluated + the aesthetics score × the preset weight of the aesthetics + the toxicity score × the preset weight of the toxicity + the bias score × the preset weight of the bias + the robustness score × the preset weight of the robustness + the generation efficiency score × the preset weight of the generation efficiency + the character defect degree score × the preset weight of the character defect degree, and the total score of the text-to-image model to be evaluated is obtained by A×the weight of the first indicator + B×the weight of the second indicator.

[0111] The advantage of such a setting is that this embodiment significantly improves the accuracy, comprehensiveness and practicality of the Wensheng graph model evaluation through weighted summation and weight adjustment, and provides strong support for the evaluation and optimization of the model.

[0112] In the above embodiment of the present application, the total evaluation score of the document graph model to be evaluated is obtained according to the decision index score and the quality index score, including:

[0113] According to the image to be evaluated, a personalized index score is obtained, wherein the personalized index includes at least one of the image style, text embedded in the image, controllable evaluation, re-creation evaluation, and multi-language evaluation;

[0114] According to the decision index score, quality index score and personalized index score, the total evaluation score of the document graph model to be evaluated is obtained.

[0115] Among them, the style of the image refers to the effect evaluation of images of different styles, specifically the evaluation of images of different styles such as comics, oil paintings, Chinese paintings, watercolor paintings, sketches, two-dimensional style, 3D style, mosaic style, Van Gogh style, etc.

[0116] Embedding text in an image refers to the evaluation of the effect of embedding text in an image if the text-to-image model supports embedding text in an image.

[0117] Controllable evaluation refers to the evaluation of the ability of the large model of Wenshengtu to make local modifications based on the existing generated images.

[0118] Then create and evaluate the preset test text set, such as "generate a photo of Sheriff Black Cat and Jerry the Mouse", and then use the VQA method for evaluation. In addition to evaluating the concept coverage, the overall harmony of the picture will also be evaluated.

[0119] Multilingual evaluation is an evaluation of the understanding ability of multiple languages ​​supported by the large model. The test text set contains input texts in different languages, and the content described in these languages ​​is consistent. Then the text alignment ability is evaluated using the VQA method.

[0120] When the index score is greater than the preset threshold, a personalized index score is obtained based on the image to be evaluated.

[0121] The advantage of this setting is that by introducing personalized indicators, the comprehensiveness, flexibility and user orientation of the cultural graph model evaluation are significantly improved, providing a richer and more multi-dimensional perspective for model evaluation and optimization.

[0122] In the above embodiment of the present application, the total evaluation score of the document graph model to be evaluated is obtained according to the decision index score, the quality index score and the personalized index score, including:

[0123] Determine the correlation indicators among the decision indicators, the quality indicators, and the personalized indicators; wherein the correlation indicators include the first correlation indicators and the second correlation indicators;

[0124] Obtaining the indicator difference between the first correlation indicator and the second correlation indicator;

[0125] If the indicator difference exceeds the preset threshold, the target indicator score in the associated indicator is lowered; wherein the target indicator is the largest indicator between the first associated indicator and the second associated indicator;

[0126] If the target indicator is the decision indicator, the total evaluation score of the document graph model to be evaluated is obtained according to the target indicator score and the quality indicator score.

[0127] If the target indicator is a quality indicator, the total evaluation score of the document graph model to be evaluated is obtained according to the target indicator score and the decision indicator score.

[0128] Among them, related indicators refer to indicators that are similar or related to the evaluation tasks. When the difference of related indicators is large, there is a contradiction and adjustment is needed. For example, if the aesthetics is high, but the quality of the text embedded in the image is poor, there is a contradiction at this time, so the aesthetics needs to be lowered. Related indicators also include: image fidelity and character defects, aesthetics and character defects, alignment ability and consistency of alignment items, fidelity and aesthetics, robustness and multilingual evaluation.

[0129] Determine the indicator difference of each related indicator. If the indicator difference is greater than the preset threshold, it means that there is a contradiction in the representation, and the larger indicator score needs to be lowered. Then, the total evaluation score of the document graph model to be evaluated is obtained based on the lowered indicator.

[0130] The advantage of this setting is that through the analysis of related indicators and the dynamic adjustment mechanism, the accuracy, balance and fairness of the Wenshengtu model evaluation are significantly improved, and some unreasonable situations are avoided: for example, the characters are deformed but beautiful, thereby improving the accuracy of model evaluation and providing a more scientific and reasonable basis for model evaluation and optimization.

[0131] Figure 4 A flow chart of another evaluation method of the text graph model provided in this application is as follows: Figure 4 As shown, in this embodiment, the method includes:

[0132] (1) Test text generation module

[0133] The test text is automatically generated using LLM (Large Language Model). The system has built-in test tasks, including multiple test fields such as people, objects, backgrounds, colors, locations, character attributes, time, seasons, weather, materials, and quantities. By automatically combining several test fields, LLM generates test text containing test content. Users can also freely select built-in test tasks or add new test tasks to generate test text in a specified direction or add test tasks.

[0134] (2) Test image generation module

[0135] Input the test text into the large text model to be evaluated to generate a test image. Generally, a set of test images is generated, including multiple images.

[0136] (3) Deterministic indicator evaluation module

[0137] Decisive indicators are indicators with key and decisive influence on all text-based image big models. These indicators can reflect the core capabilities of text-based image big models and are direct indicators of whether the big models can be put into use. They mainly include the evaluation of the text alignment capability and image fidelity of the text-based image big models. The text alignment capability is evaluated using VQA (visual question answering), an automatic evaluation method based on LLM and multimodal big models, which mainly reflects the instruction following capability of the big model and whether the generated image contains all the objects required by the text. The image fidelity is evaluated using a pre-trained fidelity big model. The generated image is input into the fidelity big model, and the big model outputs a fidelity score. This evaluation mainly reflects whether the generated image has defects, such as distorted objects, incomplete objects, etc.

[0138] If the decisive index is lower than the threshold, the evaluation process of the Wenshengtu large model ends and the module directly enters the total score module to calculate the total score of the large model according to the score of the decisive index, and then enters the evaluation report generation module to summarize the performance of the large model. If the decisive index is higher than the threshold, the module enters the quality evaluation module.

[0139] (4) Quality Assessment Module

[0140] The quality assessment module is applicable to all large models of text and image data, and reflects the quality of large models of text and image data in detail and comprehensively, in addition to the decisive indicators. This module consists of multiple parallel components, each of which is responsible for several special assessments, such as the assessment of image aesthetics, the assessment of the matching degree of attribute descriptions, quantity descriptions, and position descriptions in the test text, the assessment of the diversity of generated images, the assessment of the support range of text and symbols, the assessment of the robustness of input text, and the assessment of generation efficiency. These assessment components also use the assessment model combined with LLM to perform automatic assessment without manual intervention.

[0141] (5) Personalized assessment module

[0142] The personalized evaluation module provides several personalized evaluation components. Different text-based image models can be evaluated using evaluation components that are suitable for them. These evaluation components can include evaluation of image-embedded text, controllability evaluation, re-creation evaluation, and evaluation of images of different styles.

[0143] Personalized assessment is an assessment method designed for individual differences. It emphasizes the unique characteristics, capabilities and styles of different models to provide more accurate, comprehensive and targeted evaluation results.

[0144] (6) Model defect detection module

[0145] The model defect detection module provides in-depth evaluation and analysis of model defects to the manufacturers of Wenshengtu large models for targeted improvement and enhancement of large models. The model defect detection module first extracts large model defects based on the performance of the large model in the deterministic evaluation module, the quality evaluation module, and the personalized evaluation module. Then LLM is used to generate more test texts for large model defects, input the large model to be evaluated to generate test images, and conduct special evaluations on the test images. LLM analyzes the causes of defects based on the evaluation results. If it is found that the test coverage is insufficient or other potential factors that cause defects are suspected, it will autonomously generate more targeted test texts, apply these test texts to the large model to be tested again, and conduct repeated testing, evaluation, and analysis until LLM believes that the test is sufficient. This process is also a fully automated procedure without the need for human intervention.

[0146] (7) Total score module

[0147] The total score module calculates the total score of the model based on the performance of the Wenshengtu model in the decisive indicator evaluation module and the quality evaluation module. After the decisive evaluation module, the quality evaluation module and the personalized evaluation module score the Wenshengtu model, the total score module obtains a list containing descriptions of multiple evaluation items and the scores of the corresponding evaluation items. First, the LLM is used to analyze the list to check whether there are any differences in the results of each evaluation item (such as high aesthetics, but poor quality of text embedded in the image). If there are differences, the sub-item evaluation results are adjusted (lower the score of aesthetics).

[0148] The total score is then calculated using the following formula:

[0149]

[0150] Among them, F is the total score, is the sub-item score of the decisive indicator, is the sub-item weight of the decisive indicator, is the sub-item score of the quality assessment. is the sub-item weight of quality assessment, and are the weights of the decisive indicators and quality assessment indicators in the total score, respectively.

[0151] (8) Evaluation report generation module

[0152] The evaluation report generation module is a tool module specifically used to automatically generate evaluation reports. It mainly performs the following functions: data collection, evaluation analysis, and report generation. The underlying reliance of this module is still on LLM. LLM comprehensively analyzes the performance of the Wenshengtu large model based on the results of the decisive indicator evaluation module, quality evaluation module, personalized evaluation module, defect detection module, and total score module, and generates a summary report based on a predetermined template. The summary report contains the total score, sub-item score, overall evaluation, sub-item evaluation, defect analysis, and improvement suggestions of the model. Specifically, the summary report can be:

[0153] Text alignment score: xx

[0154] The model is able to generate images with corresponding content according to text input, especially with strong alignment capabilities for specific things such as objects and people, and secondly with alignment capabilities for the environment (such as "breeze", "waves", etc.). (This is a specific evaluation of the test samples analyzed by the large model, which also includes examples of test failures). Quality indicators: Aesthetics: Aesthetics score, specific analysis of aesthetics; attribute consistency; attribute consistency score, specific analysis of attribute consistency; similarly, quantity consistency, position consistency, image diversity, and personalized indicators all include specific indicator scores and specific analysis content.

[0155] Another evaluation method for the Wensheng graph model provided in the embodiment of the present application proposes a complete evaluation process, evaluation method and evaluation system for the Wensheng graph large model, which can comprehensively evaluate different Wensheng graph large models in different levels, and calculate the total score calculation method for the Wensheng graph large model according to the decisive indicators and quality indicators, clearly showing the comprehensive strength of the large model, helping users to quickly and intuitively understand its overall performance. Personalized evaluation is not included in the total score, but is provided as a reference for personalized indicators.

[0156] Figure 5 The structural diagram of the evaluation of the text graph model provided for this application is as follows: Figure 5 As shown, the evaluation device 50 of the cultural graph model provided in this embodiment includes:

[0157] Obtaining module 501, for inputting the test text into the to-be-evaluated text-image model to obtain a plurality of to-be-evaluated images;

[0158] The first obtaining module 502 is used to obtain a decision index score according to the image to be evaluated; wherein the decision index includes alignment ability and fidelity; the alignment ability represents the degree of consistency between the image to be evaluated and the test text element; the fidelity represents the degree of difference between the image to be evaluated and the standard image set;

[0159] A second obtaining module 503 is used to obtain a quality indicator score according to the determination index and the image to be evaluated, wherein the quality indicator at least includes one of the consistency of the alignment items of the test text and the image to be evaluated, the difference between the images to be evaluated, the aesthetics, the toxicity, the bias, the robustness, the generation efficiency, and the character defect degree;

[0160] The determination module 504 is used to determine the total evaluation score of the document graph model to be evaluated according to the decision indicator score and the quality indicator score.

[0161] In a possible implementation, the second obtaining module 503 is specifically configured to:

[0162] Perform weighted summation on the scores of various decision indicators to obtain the total decision indicator score;

[0163] The quality index score is obtained according to the preset threshold, the total decision index score and the image to be evaluated.

[0164] In a possible implementation, the second obtaining module 503 is specifically configured to:

[0165] If the total decision indicator score is less than the preset threshold, the total evaluation score of the document graph model to be evaluated is obtained according to the decision indicator score;

[0166] If the total decision indicator score is greater than or equal to the preset threshold, the total evaluation score of the document graph model to be evaluated is obtained according to the decision indicator score and the quality indicator score.

[0167] In a possible implementation, the determination module 504 is specifically configured to:

[0168] Obtain the weight of the first indicator, the weight of the second indicator, the weight of each decision indicator, and the weight of each quality indicator;

[0169] According to the decision indicator score, the first indicator weight, the quality indicator score, the second indicator weight, the weights of each decision indicator, and the weights of each quality indicator, the total evaluation score of the Wensheng graph model to be evaluated is obtained; wherein, the first indicator weight represents the influence of the decision indicator on the total evaluation score of the Wensheng graph model to be evaluated, and the second indicator weight represents the influence of the quality indicator on the total evaluation score of the Wensheng graph model to be evaluated; the weights of each decision indicator represent the influence of each decision indicator on the decision indicator, and the weights of each quality indicator represent the influence of each quality indicator on the quality indicator.

[0170] In a possible implementation, the determining module 504 is further configured to:

[0171] According to the weight of each decision indicator, the decision indicator scores are weighted and summed to obtain the total score of the decision indicator;

[0172] According to the weight of each quality indicator, the quality indicator scores are weighted and summed to obtain the total quality indicator score;

[0173] According to the total score of the decision indicator, the weight of the first indicator, the total score of the quality indicator, and the weight of the second indicator, the total score of the document graph model to be evaluated is obtained.

[0174] In a possible implementation, the determining module 504 is further configured to:

[0175] According to the image to be evaluated, a personalized index score is obtained, wherein the personalized index includes at least one of the image style, text embedded in the image, controllable evaluation, re-creation evaluation, and multi-language evaluation;

[0176] According to the decision index score, quality index score and personalized index score, the total evaluation score of the document graph model to be evaluated is obtained.

[0177] In a possible implementation, the determining module 504 is further configured to:

[0178] Determine the correlation indicators among the decision indicators, the quality indicators, and the personalized indicators; wherein the correlation indicators include the first correlation indicators and the second correlation indicators;

[0179] Obtaining the indicator difference between the first correlation indicator and the second correlation indicator;

[0180] If the indicator difference exceeds the preset threshold, the target indicator score in the associated indicator is lowered; wherein the target indicator is the largest indicator between the first associated indicator and the second associated indicator;

[0181] If the target indicator is the decision indicator, the total evaluation score of the document graph model to be evaluated is obtained according to the target indicator score and the quality indicator score.

[0182] If the target indicator is a quality indicator, the total evaluation score of the document graph model to be evaluated is obtained according to the target indicator score and the decision indicator score.

[0183] It should be noted that it should be understood that the division of the various modules of the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And these modules can all be implemented in the form of software called by processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware. For example, the processing module can be a separately established processing element, or it can be integrated in a chip of the above device. In addition, it can also be stored in the memory of the above device in the form of program code, and called and executed by a processing element of the above device. The implementation of other modules is similar. In addition, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in the processor element or instructions in the form of software.

[0184] For example, the above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASIC), or one or more microprocessors (DSP), or one or more field programmable gate arrays (FPGA). For another example, when a module above is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0185] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loading and executing computer instructions on a computer, a process or function according to an embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (Digital Subscriber Line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrations. The available media may be magnetic media (eg, floppy disks, hard disks, tapes), optical media (eg, Digital Video Discs (DVD)), or semiconductor media (eg, solid state disks (SSD)).

[0186] Figure 6 A schematic diagram of the structure of an electronic device is provided for this application. Figure 6As shown, the electronic device 60 provided in this embodiment includes: at least one processor 601 and a memory 602. Optionally, the device 60 also includes a communication component 603. The processor 601, the memory 602 and the communication component 603 are connected via a bus 604.

[0187] In a specific implementation process, at least one processor 601 executes the computer execution instructions stored in the memory 602, so that at least one processor 601 executes the above method.

[0188] The specific implementation process of the processor 601 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0189] In the above embodiments, it should be understood that the processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the invention can be directly implemented as a hardware processor, or can be implemented by a combination of hardware and software modules in the processor.

[0190] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (NVM), such as at least one disk storage.

[0191] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of this application is not limited to only one bus or one type of bus.

[0192] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0193] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0194] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0195] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0196] The division of units is only a logical function division. There may be other divisions in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interface, device or unit, which may be electrical, mechanical or other forms.

[0197] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0198] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0199] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0200] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0201] Finally, it should be noted that those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed by the present invention, are not limited to the precise structures described above and shown in the drawings, and may be modified and altered in various ways without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A method for evaluating a cultural graph model, characterized in that: include: Input the test text into the to-be-evaluated text-image model to obtain multiple to-be-evaluated images; According to the image to be evaluated, a decision index score is obtained; wherein the decision index includes alignment capability and fidelity; the alignment capability represents the degree of consistency between the image to be evaluated and the test text element; the fidelity represents the degree of difference between the image to be evaluated and the standard image set; According to the decision index and the image to be tested, a quality index score is obtained, wherein the quality index at least includes one of the consistency of the alignment items of the test text and the image to be tested, the difference between the images to be tested, the aesthetics, toxicity, bias, robustness, generation efficiency, and the degree of character defects; The total evaluation score of the document graph model to be evaluated is determined according to the decision indicator score and the quality indicator score.

2. The method according to claim 1, characterized in that The step of obtaining a quality index score according to the decision index and the image to be evaluated includes: Performing weighted summation on the scores of each decision indicator to obtain a total decision indicator score; A quality index score is obtained according to a preset threshold, the total decision index score and the image to be evaluated.

3. The method according to claim 2, characterized in that The step of obtaining the quality index score according to the preset threshold, the total decision index score and the image to be evaluated includes: If the total decision indicator score is less than a preset threshold, the total evaluation score of the document graph model to be evaluated is obtained according to the decision indicator score; If the total decision indicator score is greater than or equal to the preset threshold, the total evaluation score of the document graph model to be evaluated is obtained according to the decision indicator score and the quality indicator score.

4. The method according to any one of claims 1 to 3, characterized in that Determining the total evaluation score of the document graph model to be evaluated according to the decision indicator score and the quality indicator score includes: Obtain the weight of the first indicator, the weight of the second indicator, the weight of each decision indicator, and the weight of each quality indicator; The total evaluation score of the document graph model to be evaluated is obtained according to the decision indicator score, the first indicator weight, the quality indicator score, the second indicator weight, the weights of each decision indicator, and the weights of each quality indicator; wherein the first indicator weight represents the influence of the decision indicator on the total evaluation score of the document graph model to be evaluated, and the second indicator weight represents the influence of the quality indicator on the total evaluation score of the document graph model to be evaluated; the weights of each decision indicator represent the influence of each decision indicator on the decision indicator, and the weights of each quality indicator represent the influence of each quality indicator on the quality indicator.

5. The method according to claim 4, characterized in that The step of obtaining the total evaluation score of the document graph model to be evaluated according to the decision indicator score, the first indicator weight, the quality indicator score, the second indicator weight, each decision indicator weight, and each quality indicator weight includes: According to the weights of the various decision indicators, the weighted sum of the decision indicator scores is calculated to obtain the total score of the decision indicator; According to the weights of the various quality indicators, the quality indicator scores are weighted and summed to obtain a total quality indicator score; The total evaluation score of the document graph model to be evaluated is obtained according to the total score of the decision indicator, the weight of the first indicator, the total score of the quality indicator, and the weight of the second indicator.

6. The method according to any one of claims 1 to 3, characterized in that The step of obtaining the total evaluation score of the document graph model to be evaluated according to the decision indicator score and the quality indicator score includes: Obtaining a personalized index score according to the image to be evaluated, wherein the personalized index includes at least one of the image style, text embedded in the image, controllable evaluation, re-creation evaluation, and multi-language evaluation; The total evaluation score of the document graph model to be evaluated is obtained according to the decision index score, the quality index score and the personalized index score.

7. The method according to claim 6, characterized in that The step of obtaining the total evaluation score of the document graph model to be evaluated according to the decision index score, the quality index score and the personalized index score comprises: Determine a correlation indicator among the decision indicator, the quality indicator, and the personalized indicator; wherein the correlation indicator includes a first correlation indicator and a second correlation indicator; Obtaining an indicator difference between the first correlation indicator and the second correlation indicator; If the indicator difference exceeds a preset threshold, the target indicator score in the associated indicator is lowered; wherein the target indicator is the largest indicator between the first associated indicator and the second associated indicator; If the target indicator is the decision indicator, the total evaluation score of the document graph model to be evaluated is obtained according to the target indicator score and the quality indicator score. If the target indicator is the quality indicator, the total evaluation score of the document graph model to be evaluated is obtained according to the target indicator score and the decision indicator score.

8. An evaluation device for a cultural graph model, characterized in that: include: A module for inputting a test text into a text-image model to be evaluated to obtain a plurality of images to be evaluated; The first acquisition module is used to obtain a decision index score according to the image to be evaluated; wherein the decision index includes alignment ability and fidelity; the alignment ability represents the consistency between the image to be evaluated and the test text element; the fidelity represents the difference between the image to be evaluated and the standard image set; A second obtaining module is used to obtain a quality indicator score according to the decision indicator and the image to be evaluated, wherein the quality indicator at least includes one of the consistency of the alignment items of the test text and the image to be evaluated, the difference between the images to be evaluated, the aesthetics, toxicity, bias, robustness, generation efficiency, and the degree of character defects; The determination module is used to determine the total evaluation score of the document graph model to be evaluated according to the decision indicator score and the quality indicator score.

9. An electronic device, characterized in that: include: Memory, processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.

11. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 7 when being executed by a processor.

Images