Evaluation method, device and equipment of text graph model and storage medium

Abstract

The invention provides an assessment method, device and equipment for a text graph model and a storage medium, relates to the technical field of data processing, in particular to the fields of text graph models, model assessment and the like, and can be used for application scenes such as generative search, document intelligent editing, intelligent assistants, virtual assistants, intelligent e-commerce and the like. According to the specific implementation scheme, the method comprises the following steps: evaluating the performance of a text graph model by adopting a machine evaluation mode to obtain a first comprehensive evaluation result; under the condition that the first comprehensive evaluation result meets a preset condition, triggering a target evaluation mode to evaluate the performance of the text graph model to obtain a second comprehensive evaluation result; wherein the evaluation precision of the machine evaluation mode is lower than that of the target evaluation mode. In the embodiment of the invention, through an evaluation mode of combining machine evaluation and manual evaluation, more comprehensive and accurate evaluation of the performance of the text graph model is realized from multiple dimensions.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to the fields of text-graph models and model evaluation, and can be used in application scenarios such as generative search, intelligent document editing, intelligent assistants, virtual assistants, and intelligent e-commerce. Background Art

[0002] AI (Artificial Intelligence) painting, as one of the important application fields of artificial intelligence, has made significant breakthroughs in recent years.

[0003] In related technologies, the text-based graph model can intelligently generate creative images of different scenes and styles according to user needs. Therefore, in the training process of the text-based graph model, it is very important to systematically screen and evaluate the text-based graph model that meets the needs. Summary of the invention

[0004] The present disclosure provides a method, apparatus, device and storage medium for evaluating a text graph model.

[0005] According to one aspect of the present disclosure, a method for evaluating a cultural graph model is provided, comprising:

[0006] The performance of the Wensheng graph model is evaluated by machine evaluation, and the first comprehensive evaluation result is obtained;

[0007] When the first comprehensive evaluation result meets the preset conditions, the target evaluation method is triggered to evaluate the performance of the Wensheng graph model to obtain a second comprehensive evaluation result;

[0008] Among them, the evaluation accuracy of the machine evaluation method is lower than that of the target evaluation method.

[0009] According to another aspect of the present disclosure, there is provided an evaluation device for a culture graph model, comprising:

[0010] A first evaluation module is used to evaluate the performance of the Wensheng graph model by machine evaluation to obtain a first comprehensive evaluation result;

[0011] A second evaluation module is used to trigger a target evaluation method to evaluate the performance of the Wensheng graph model to obtain a second comprehensive evaluation result when the first comprehensive evaluation result meets a preset condition;

[0012] Among them, the evaluation accuracy of the machine evaluation method is lower than that of the target evaluation method.

[0013] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0014] at least one processor; and

[0015] a memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute any method in the embodiments of the present disclosure.

[0017] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any method according to the embodiments of the present disclosure.

[0018] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements any method according to the embodiments of the present disclosure when executed by a processor.

[0019] In the disclosed embodiments, a more comprehensive and accurate evaluation of the performance of the text graph model is achieved from multiple dimensions through a complementary evaluation method of machine evaluation and manual evaluation.

[0020] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0022] Figure 1 is a flow chart of a method for evaluating a text graph model according to an embodiment of the present disclosure;

[0023] Figure 2 is a schematic diagram of a process for obtaining a first evaluation result according to an embodiment of the present disclosure;

[0024] Figure 3 is a flow chart of obtaining a first evaluation result by performing machine evaluation from multiple dimensions according to an embodiment of the present disclosure;

[0025] Figure 4 is a flow chart of a training fitting network model according to an embodiment of the present disclosure;

[0026] Figure 5 is a schematic diagram of a process of obtaining a reference ranking according to an embodiment of the present disclosure;

[0027] Figure 6 It is a schematic diagram of the overall process structure of the evaluation method of the cultural graph model according to an embodiment of the present disclosure;

[0028] Figure 7is a schematic structural diagram of an evaluation device for a cultural graph model according to an embodiment of the present disclosure;

[0029] Figure 8 The block diagram is a block diagram of an electronic device for implementing the evaluation method of the culture graph model according to the embodiment of the present disclosure. DETAILED DESCRIPTION

[0030] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0031] The terms "first", "second", etc. in this disclosure are used to distinguish similar objects, and are not necessarily used to describe a specific order or precedence. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, including a series of steps or units. The method, system, product or device is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0032] In the field of machine learning, the evaluation of the graph model is a very critical step, which can help determine whether the performance of the graph model meets expectations and decide whether the graph model can be deployed in the production environment.

[0033] For the evaluation of the text graph model, after the model is trained, the best model can be selected by testing the machine scoring on the evaluation set, or the best model can be selected based on manual scoring. Among these evaluation methods, machine scoring is efficient, but it may not fully reflect the true performance of the model. Especially when dealing with complex tasks, some misleading scoring results may appear, resulting in sometimes unreliable results. Manual scoring is time-consuming, costly, and difficult to scale to large-scale data sets.

[0034] In view of this, an embodiment of the present disclosure provides a method for evaluating a text graph model. The method evaluates the text graph model from different dimensions in a manner that machine evaluation and manual evaluation complement each other, so as to select a high-quality text graph model.

[0035] like Figure 1 FIG. 1 is a flow chart of an evaluation method of a culture graph model proposed in an embodiment of the present disclosure, including:

[0036] S101, using a machine evaluation method to evaluate the performance of the text graph model to obtain a first comprehensive evaluation result.

[0037] Machine evaluation is an automated evaluation method that quantitatively evaluates the performance of the Wensheng graph model through preset algorithms and indicators. By using machine evaluation to measure and analyze the performance of the Wensheng graph model in multiple aspects, a comprehensive evaluation conclusion (i.e., the first comprehensive evaluation result) is obtained, which can preliminarily screen and sort the Wensheng graph models in a short time.

[0038] S102, when the first comprehensive evaluation result meets the preset conditions, trigger the target evaluation method to evaluate the performance of the cultural graph model to obtain a second comprehensive evaluation result; wherein the evaluation accuracy of the machine evaluation method is lower than that of the target evaluation method.

[0039] Compared with the machine evaluation method, the target evaluation method is usually a more accurate and detailed evaluation method, such as manual evaluation, expert review, etc. The target evaluation method can be an evaluation method that consumes more resources and time than the machine evaluation method, such as large model evaluation. The target evaluation method can ensure the accuracy and reliability of the evaluation results. When the first comprehensive evaluation result obtained by the machine evaluation method meets the preset conditions, the target evaluation method can be used to evaluate the model performance again, which can be more accurate and detailed, so as to obtain more comprehensive and in-depth information about the model performance.

[0040] In the disclosed embodiment, the performance of the Vincent graph model is first evaluated by machine evaluation, and a preliminary first comprehensive evaluation result can be quickly obtained, so that the preliminary evaluation of the Vincent graph model can be quickly completed. When the first comprehensive evaluation result meets the preset conditions, the manual evaluation method with higher evaluation accuracy is triggered to further evaluate the model performance and obtain a more accurate second comprehensive evaluation result. By combining the evaluation method of machine evaluation and manual evaluation, a certain efficiency is guaranteed, and by setting the preset conditions of the machine evaluation results, a more accurate evaluation can be performed when needed, thereby providing a process-based implementation scheme that can systematically evaluate the Vincent graph model, which can achieve a more comprehensive and accurate evaluation of the performance of the Vincent graph model, and can also improve the evaluation efficiency of the Vincent graph model.

[0041] In addition, in the embodiment of the present disclosure, when the first comprehensive evaluation result does not meet the preset conditions, the current round of evaluation is terminated and a prompt message for continuing to iteratively optimize the document graph model is output.

[0042] Therefore, when the first comprehensive evaluation result does not meet the preset conditions, the evaluation is terminated in time so as to iteratively optimize the Wensheng graph model as soon as possible and improve the training efficiency of the model.

[0043] In some embodiments, the preset condition for triggering the target evaluation method may be that the confidence of the machine evaluation result is higher than the preset confidence, and the first comprehensive evaluation result obtained by the machine evaluation method indicates that the model performance of the Vincent diagram meets the preset performance requirements. Thus, the target evaluation method can be sampled at an appropriate time to improve the evaluation efficiency.

[0044] In some other embodiments, the preset condition for triggering the target evaluation method is that the performance of the current version of the Wensheng graph model is significantly better than the performance of the previous version of the Wensheng graph model. During implementation, whether the preset condition is met can be determined based on the following steps:

[0045] Step A1, comparing the first comprehensive evaluation result of the Wensheng graph model with the reference evaluation result to obtain a comparison result; the reference evaluation result is an evaluation result of the reference model obtained by the previous round of iterative optimization of the Wensheng graph model using a machine evaluation method;

[0046] Step A2: when the comparison result indicates that the performance of the document graph model is better than that of the reference model, and the performance difference between the document graph model and the reference model is greater than a preset difference, it is determined that the preset condition is met.

[0047] The first comprehensive evaluation result is compared with the reference evaluation result, and the obtained comparison result can measure the performance difference between the previous and next versions of the Wensheng graph model. When the performance of the current version of the Wensheng graph model is better than that of the previous version of the reference model, and the current version of the Wensheng graph model has a clear advantage, it can be determined that the preset condition is met.

[0048] In the disclosed embodiment, by comparing the first comprehensive evaluation result of the Wensheng graph model with the reference evaluation result, the comparison result can quickly determine whether the current version of the Wensheng graph model meets the requirements for entering the next evaluation stage. The preset conditions can ensure the uniformity and stability of the evaluation standards, and ensure that only Wensheng graph models that meet certain conditions enter a more accurate evaluation stage, thereby improving the efficiency of the overall evaluation process and saving resources consumed in evaluating the Wensheng graph model.

[0049] In the embodiment of the present disclosure, a machine evaluation method can be used to evaluate the performance of the cultural graph model from multiple dimensions to obtain a first comprehensive evaluation result. The specific implementation steps are as follows: Figure 2 As shown, including the following:

[0050] S201, build an evaluation set.

[0051] Constructing a comprehensive evaluation set can comprehensively test the capabilities of the document graph model, and then evaluate the iterative optimization direction of the document graph model. The construction of the evaluation set is crucial for accurately evaluating the performance of the model. Therefore, during implementation, a text description containing at least one of the following elements can be constructed to obtain the evaluation set:

[0052] (1) Characteristic sentences of the language used in the text description.

[0053] As a Chinese text graph model, one point that needs to be paid attention to is the ability to generate Chinese semantics. Therefore, representative sentences can be selected to construct an evaluation set to show the unique style of a specific language. For example, characteristic sentences can contain idioms, colloquialisms, slang, or sentences with unique rhetorical devices. These sentences can not only reflect the richness of the language, but also test the ability of the text graph model to understand complex language structures.

[0054] (2) Entity words of property rights objects.

[0055] As a text graph model, users like to generate some entity concepts. In order to evaluate the model's ability to process entity words, some words with clear meanings and ownership can be selected. These words can be brand names, proper nouns, place names, names of people, names of animals, names of colleges and universities, anime characters, proper nouns in film and television works, etc. Through these entity words, it can be tested whether the model can correctly identify and process these important information.

[0056] (3) Control statements used to evaluate the control capability of the Wensheng graph model;

[0057] Control is also an important part of the text graph model. Control can reflect the text graph model's ability to understand the text, and can also reflect the text graph model's ability to generate detailed content required by users.

[0058] Control statements are the requirements and constraints that users put forward for the generated images by the Vincent graph model. That is, some restrictive statements are used to control the details of the generated images by the Vincent graph model. For example, if you ask "generate a cat", the Vincent graph model may randomly generate a cat of a certain type and appearance. But if you add some control statements, such as "the generated cat must have brown fur, blue eyes, and is catching a ball of yarn", the control statements further restrict the generated content of the Vincent graph model. By designing some control statements, the model can be guided to generate specific types of pictures. Through control statements, the control ability of the Vincent graph model can also be evaluated.

[0059] (4) Text used to evaluate text rendering capabilities.

[0060] Text rendering capability refers to allowing the text-based graph model to add the required text to the generated image, which is also one of the special features of the text-based graph model. For example, the text requires the generation of an image with a specific slogan, such as generating a billboard with a specific advertising slogan written on it.

[0061] In the disclosed embodiment, by constructing an evaluation set, the various capabilities of the text graph model can be comprehensively and comprehensively evaluated. By including language feature sentences, the ability of the text graph model to understand and present different language styles can be evaluated; the entity words of property rights objects help to verify the accuracy of the model's depiction of specific concepts and objects; control statements can evaluate the model's ability to generate images under specific requirements; and text used to evaluate text rendering capabilities can evaluate the rendering effect of the text graph model in converting text into images.

[0062] S202, input the evaluation set into the text-generated graph model to obtain an image set generated by the text-generated graph model for the evaluation set.

[0063] The constructed evaluation set text is input into the text-based graph model, which will try to generate corresponding images for each text description in the evaluation set. These generated images constitute the image set.

[0064] S203, processing the evaluation set and the image set in a machine evaluation manner to obtain a first comprehensive evaluation result.

[0065] The evaluation set and the generated image set are processed and analyzed using a machine evaluation method. The machine evaluation method evaluates the performance of the model based on a preset algorithm and multiple dimensions, and then obtains the first comprehensive evaluation result. For example, the generated image is evaluated from aspects such as relevance, aesthetics, deformity rate, and comprehensive score to reflect the overall performance of the Wenshengtu model.

[0066] In the disclosed embodiment, an evaluation set is constructed and input into the Wensheng graph model to obtain a generated image set, and the model can be tested in a standardized and repeatable manner. The first comprehensive evaluation result is obtained after processing by machine evaluation, which can fully understand the performance of the Wensheng graph model in different aspects, which helps to accurately discover the possible deficiencies of the Wensheng graph model, clarify the direction that needs to be improved and optimized, and then continuously improve the quality and performance of the Wensheng graph model through targeted adjustment and training, so that it can better meet the needs of practical applications.

[0067] In the embodiment of the present disclosure, the specific implementation method for machine evaluation is as follows: Figure 3 As shown, including the following:

[0068] S301, using the evaluation strategies corresponding to the multiple evaluation dimensions in the machine evaluation method to respectively evaluate the performance of the text graph model, and obtaining the evaluation score of each dimension.

[0069] During implementation, the multiple evaluation dimensions in the machine evaluation method usually include the image-text relevance of the generated image, image quality diversity, deformity rate, the effect of generated text, etc. Each dimension has its own specific evaluation strategy. Among them, the evaluation dimensions include at least one of the following:

[0070] (1) The text-text correlation between the text description and the generated image.

[0071] That is, the degree of match between the image generated by the text description and the content of the text description. A highly relevant text-generated graph model can generate a visually corresponding and reasonable image based on a given text description. For example, if the text describes a red apple, then the generated image should be an image of a red apple that matches the description.

[0072] (2) Image quality of the generated image.

[0073] Image quality is a measure of the image generated by the text model from multiple perspectives, including the image's realism, aesthetics, resolution, etc. A high-quality image should have a reasonable range of light intensity and show richer details in highlights and shadows.

[0074] (3) Distortion of the generated image.

[0075] Image distortion refers to the phenomenon of distorted generated images, and mainly focuses on whether the image has abnormal distortion, deformation, or disproportion. For example, the generated characters or other objects may have problems with proportions, such as the head is too large and the body is too small, or one part of an object appears too large relative to the whole.

[0076] In order to accurately identify the distortion in the images generated by the Wensheng graph model, the distorted data with deformed features can be screened out from the images produced by the Wensheng graph model. For example, in the generated images of people, deformed data such as limb deformities, hand deformities, and facial deformities can be selected. The produced distorted data is used as training samples to train a deformity detection model that can identify the distortion of the person images generated by the Wensheng graph model.

[0077] During implementation, the collected distortion data can be professionally annotated by industry professionals to clearly indicate which parts are distorted. The annotation content includes but is not limited to information such as the type, location, and degree of the deformity. After collecting and annotating the data, select a suitable deep learning model as the initial model for training based on the specific needs of the task. The loss function is used to evaluate the difference between the prediction results of the trained deformity detection model and the actual annotations, so as to continuously optimize the deformity detection model parameters during the training process to improve the recognition ability of the deformity of the image generated by the Wensheng graph model.

[0078] In some embodiments, the generated results of the Vincent graph model can be detected by a trained deformity detection model, and the deformity rate of the generated results can be counted to complete the machine evaluation method to evaluate the distortion of the generated image by the Vincent graph model.

[0079] Deformity detection for natural scenery and animals can also be trained with reference to the deformity detection model for human images, so as to facilitate the detection of image distortion of natural risks and animals.

[0080] (4) Generating text and text descriptions contained in images requires the relevance of the text content.

[0081] The generated image may contain text, and the relevance of these texts to the text content required by the text description is evaluated. For example, if the text requires generating an image with a specific slogan, then the text in the generated image should be consistent with the required slogan.

[0082] In the disclosed embodiment, the evaluation of the correlation between the image and text can show the degree of fit between the model's understanding of the text description and its conversion into an image, ensuring that the generated image can accurately reflect the intent of the text; the evaluation of image quality can understand the pros and cons of the model-generated image; the evaluation of distortion can find out whether the model will have unreasonable deformation and other problems in the process of generating images; the evaluation of the correlation between the text contained in the generated image and the text description requires the text content to ensure the accurate representation of the text part. Through these evaluation dimensions, we can fully and deeply understand the capabilities of the text-generated image model in various key aspects, provide specific basis and direction for further optimizing and improving the model, and help improve the overall performance and performance of the model.

[0083] S302, input the evaluation scores of each dimension into the fitting network model to obtain a first comprehensive evaluation result.

[0084] The evaluation scores of each dimension obtained by machine evaluation are input into the fitting network model to obtain the first comprehensive evaluation result. The fitting network model can be a neural network model, such as a deep neural network learning model (DNN). This model takes the evaluation scores of each dimension as input, and then fits to obtain a comprehensive evaluation score, which represents the overall performance of the Wensheng graph model.

[0085] In the disclosed embodiment, the text-to-image model is evaluated through multiple evaluation dimensions, and the performance of the model can be analyzed from different angles. For example, the correlation between text and image can measure the model's understanding of the text and the accuracy of conversion, the image quality can reflect the quality of the generated image, and the distortion can examine the rationality of the model's generated image. Then, the evaluation scores of each dimension are input into the fitting network model to obtain the first comprehensive evaluation result, which can integrate and comprehensively evaluate the information of each dimension, thereby more accurately reflecting the overall performance of the model and providing more comprehensive and targeted data support for further optimizing the model.

[0086] In the embodiment of the present disclosure, the fitting network model used to obtain the first comprehensive evaluation result needs to be trained by sample data obtained according to actual conditions. The specific implementation of training the fitting network model is as follows: Figure 4 As shown, including the following:

[0087] S401, for a plurality of training texts, obtaining evaluation scores of a plurality of evaluation dimensions of an image generation capability of a target version model of a text-to-image model for the plurality of training texts.

[0088] For a set of training texts, the target version of the text-to-image model is used to generate images, and the generated images are evaluated according to the multiple evaluation dimensions described above to obtain the evaluation scores corresponding to each dimension. These scores will serve as the basis for subsequent analysis.

[0089] In the initial training stage of the text graph model, the latest version of the text graph model can be used as the target version model. Alternatively, when the text graph model meets the preset performance requirements after multiple iterations of training, the current optimal text graph model is used as the target version model. For the latter, the fitting network model is optimized after the text graph model is trained to a certain stage, and the fitting network model gives the final first comprehensive evaluation result. Before optimizing the fitting network model, the evaluation scores of multiple dimensions can be weighted and summed to obtain the first comprehensive evaluation result.

[0090] S402, inputting the evaluation scores of multiple evaluation dimensions of each training text into the initial network respectively, and obtaining the prediction generation capability value of the target version model obtained by the initial network for each training text.

[0091] The evaluation scores of multiple evaluation dimensions of each training text are used as features and input into an initial network. This initial network can be any model that can process multi-dimensional input and output a single prediction value. The initial network will output a predicted generation ability value, which represents the prediction of the target version model's image generation ability for the training text.

[0092] S403, sorting the plurality of training texts based on the predicted generation capability values ​​to obtain a predicted sorting.

[0093] All training texts are sorted according to the order of the prediction generation ability values ​​output by the initial network from high to low to obtain the prediction ranking.

[0094] S404, based on the gap between the predicted ranking and the reference ranking of the plurality of training texts, the model parameters of the initial network are adjusted, and a fitting network model is obtained when a convergence condition is met.

[0095] The reference ranking is based on certain standards or prior knowledge and represents the ideal ranking of the training text. By comparing the gap between the predicted ranking and the reference ranking, the loss value of the predicted ranking relative to the reference ranking is determined, and then the model parameters of the initial network are adjusted according to the loss value. The above process is repeated until the convergence condition is met, such as the difference between the predicted ranking and the reference ranking is small enough, or the number of iterations reaches the preset number. At this time, the initial network obtained is the fitted network model.

[0096] In the disclosed embodiment, based on the gap between the predicted ranking and the reference ranking of multiple training texts, the model parameters of the initial network are adjusted, and the obtained fitting network model can learn the complex relationship between different evaluation dimensions. By inputting the evaluation scores of multiple dimensions into the fitting network model to integrate this information, the overall performance of the text graph model can be more comprehensively evaluated. Thus, more accurate evaluation and judgment can be provided. Ultimately, the first comprehensive evaluation result obtained can more comprehensively reflect the performance of the model in various aspects, provide a more valuable reference basis for further optimizing and adjusting the model, and help improve the performance and effect of the model.

[0097] In the embodiment of the present disclosure, regarding the method of determining the reference ranking of multiple training texts, such as Figure 5 As shown, it may include the following:

[0098] S501, for each training text, obtain multiple images generated by the target version model for the training text to obtain an image group corresponding to the training text.

[0099] For each training text, the target version of the text-graph model is used to generate multiple images, which will form an image group corresponding to each training text. These images represent the visual representation generated by the text-graph model based on the input training text.

[0100] S502, obtaining a generation capability value of each image in the image group; the generation capability value expresses the generation capability of the text-generated graph model for the training text. The generation capability value is obtained based on an evaluation method other than a machine evaluation method. For example, it can be determined using a target evaluation method.

[0101] For each image in the image group, its generative ability value is calculated. This generative ability value reflects the generative ability of the text-based graph model for the training text, but does not rely on machine evaluation methods and can be obtained through, for example, manual evaluation, expert review or other appropriate evaluation methods.

[0102] S503: Determine a comprehensive ability value of the text-to-graph model's ability to generate training texts based on the generation ability values ​​of the multiple images.

[0103] Based on the generation ability values ​​of multiple images, the comprehensive ability value of the generation ability of the text graph model for the training text is determined. For example, in a manual evaluation method, the image is comprehensively scored to different degrees based on the degree of correlation between the image and the text description and the beauty of the image. During implementation, a corresponding prompt project can be constructed based on the evaluation set. Each prompt project generates N images based on the target version of the text graph model, and then each image is given to different M evaluators to score according to the above evaluation method. Therefore, each prompt project will produce N*M scores, and the average score is finally taken to obtain the comprehensive ability value of the generation ability of the text graph model for the training text.

[0104] S504, obtaining a reference ranking of the plurality of training texts based on the comprehensive capability values ​​of the plurality of training texts.

[0105] Based on the comprehensive ability values ​​of multiple training texts, a reference ranking of these training texts is obtained. This ranking can reflect the performance of the model on various training texts and provide a basis for further optimization.

[0106] In the disclosed embodiment, multiple images are generated for different training texts, and the generation capability values ​​of these images are obtained, and a reference ranking of multiple training texts is obtained based on the comprehensive capability values ​​of multiple training texts. The reference ranking obtained by this method can not only help to understand the performance of the model more comprehensively, but also guide the optimization and improvement of the model, and ultimately improve the application effect of the model.

[0107] In the disclosed embodiment, as the basic capabilities of the multimodal big model continue to improve, in order to further complete the evaluation optimization of the literary graph model, when the evaluation big model simulating manual evaluation has not been trained to converge, the target evaluation method is the manual evaluation method; when the evaluation big model is trained to converge, the target evaluation method is the evaluation method using the evaluation big model for evaluation.

[0108] In the disclosed embodiment, when the large evaluation model has not been trained to converge, a manual evaluation method is used, which can more accurately and comprehensively evaluate the Wensheng graph model. Because the model is unstable at this time, manual evaluation can rely on human subjective feelings and careful observations to conduct in-depth analysis of the image quality, details, and fit with the description, etc., provide more reliable feedback, and help to find problems in a timely manner and make targeted improvements. When the large evaluation model is trained and converged, the large evaluation model is used for evaluation, and the model's own capabilities can be used for efficient evaluation, improving evaluation efficiency and consistency, while also better verifying the performance and accuracy of the model in practical applications. Flexible selection of evaluation methods based on the training status of the large evaluation model can more scientifically and rationally promote the evaluation optimization of the Wensheng graph model to improve the overall effect and quality of the Wensheng graph model.

[0109] In the embodiment of the present disclosure, the evaluation large model can be trained based on the following steps:

[0110] Step B1, obtaining the labeled data obtained by manual evaluation.

[0111] During implementation, a large amount of labeled data can be accumulated through early manual evaluation. These labeled data are used as annotation data for training and evaluating large models.

[0112] Step B2: training the initial large model based on the labeled data to obtain the evaluation large model.

[0113] After obtaining enough annotated data, we build corresponding prompt projects based on these annotated data and train the evaluation model to predict and score. After multiple iterations and optimizations, we get the final evaluation model.

[0114] In the disclosed embodiment, the labeled data is obtained by manual evaluation to ensure the accuracy and reliability of the data. Training the initial large model based on accurate labeled data helps the model to better learn and understand the patterns and laws of the data, thereby improving the performance and accuracy of the model. It allows the model to make more accurate evaluations and predictions when faced with new data.

[0115] In summary, the evaluation method of the cultural graph model provided in the embodiments of the present disclosure is as follows: Figure 6 As shown in the figure, based on the well-built evaluation set, the text graph model can be quickly and effectively evaluated through machine evaluation. When the machine evaluation results meet the preset conditions, the optimal text graph model is further screened out through manual evaluation. When there are enough samples for manual evaluation, a multimodal large language model is trained to simulate manual evaluation preferences and replace manual evaluation to further complete the evaluation of the text graph model.

[0116] Based on the same technical concept, the embodiment of the present disclosure also provides an evaluation device 700 for a cultural graph model, such as Figure 7 As shown, including:

[0117] A first evaluation module 701 is used to evaluate the performance of the text graph model using a machine evaluation method to obtain a first comprehensive evaluation result;

[0118] The second evaluation module 702 is used to trigger the target evaluation method to evaluate the performance of the Wensheng graph model to obtain a second comprehensive evaluation result when the first comprehensive evaluation result meets the preset condition;

[0119] Among them, the evaluation accuracy of the machine evaluation method is lower than that of the target evaluation method.

[0120] In some embodiments, a comparison module is further included for:

[0121] Compare the first comprehensive evaluation result of the Wensheng graph model with the reference evaluation result to obtain a comparison result; the reference evaluation result is an evaluation result of the reference model obtained by optimizing the Wensheng graph model in the previous round of iterations using a machine evaluation method;

[0122] When the comparison result indicates that the performance of the document graph model is better than the performance of the reference model, and the performance difference between the document graph model and the reference model is greater than a preset difference, it is determined that the preset condition is met.

[0123] In some embodiments, the first evaluation module comprises:

[0124] Construction unit, used to construct the evaluation set;

[0125] An input unit, used for inputting the evaluation set into the Wensheng graph model to obtain an image set generated by the Wensheng graph model for the evaluation set;

[0126] The processing unit is used to process the evaluation set and the image set in a machine evaluation manner to obtain a first comprehensive evaluation result.

[0127] In some embodiments, the construction unit is specifically used for:

[0128] Construct a text description containing at least one of the following elements to obtain an evaluation set:

[0129] Characteristic sentences of the language used in the text description;

[0130] The entity word of the property right object;

[0131] Control statements used to evaluate the control capabilities of the Wensheng graph model;

[0132] Text used to evaluate text rendering capabilities.

[0133] In some embodiments, the first evaluation module is specifically configured to:

[0134] The performance of the Wensheng graph model is evaluated by using the evaluation strategies corresponding to the multiple evaluation dimensions in the machine evaluation method to obtain the evaluation scores of each dimension.

[0135] The evaluation scores of each dimension are input into the fitting network model to obtain the first comprehensive evaluation result.

[0136] In some embodiments, a training module is further included for training the fitting network model based on the following method:

[0137] For multiple training texts, obtaining evaluation scores of multiple evaluation dimensions of the image generation capability of the target version model of the text-to-image model for the multiple training texts;

[0138] The evaluation scores of multiple evaluation dimensions of each training text are respectively input into the initial network to obtain the prediction generation capability value of the target version model obtained by the initial network for each training text;

[0139] Sort multiple training texts based on the predicted generation capability values ​​to obtain a predicted ranking;

[0140] Based on the gap between the predicted ranking and the reference ranking of multiple training texts, the model parameters of the initial network are adjusted, and the fitting network model is obtained when the convergence conditions are met.

[0141] In some embodiments, a determination module is further included, which is used to:

[0142] For each training text, obtain multiple images generated by the target version model for the training text to obtain an image group corresponding to the training text;

[0143] Obtaining a generative capability value of each image in the image group; the generative capability value expresses the generative capability of the text-generated graph model for the training text; the generative capability value is obtained based on an evaluation method other than a machine evaluation method;

[0144] Based on the generation ability values ​​of multiple images, determine the comprehensive ability value of the text-generated graph model for the training text;

[0145] Based on the comprehensive ability values ​​of the multiple training texts, a reference ranking of the multiple training texts is obtained.

[0146] In some embodiments, the evaluation dimensions include at least one of the following:

[0147] The correlation between the text description and the generated image;

[0148] Image quality of generated images;

[0149] The distortion of the generated image;

[0150] Generating text and description contained in images requires the relevance of the text content.

[0151] In some embodiments, when the large evaluation model simulating manual evaluation has not been trained to converge, the target evaluation method is a manual evaluation method;

[0152] When evaluating the convergence of large model training, the target evaluation method is to use the evaluation method of the large model for evaluation.

[0153] In some embodiments, an optimization module is further included, which is used to train and obtain an evaluation model based on the following method:

[0154] Obtaining annotated data obtained through manual evaluation;

[0155] Based on the labeled data, an initial large model is trained to obtain an evaluation large model.

[0156] For the description of specific functions and examples of each module and submodule of the device in the embodiment of the present disclosure, reference can be made to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0157] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0158] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0159] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0160] like Figure 8 As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0161] A number of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0162] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as the evaluation method of the Wensheng graph model of data. For example, in some embodiments, the evaluation method of the Wensheng graph model of data may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the evaluation method of the Wensheng graph model of data described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to execute the evaluation method of the data Wensheng graph model in any other appropriate manner (for example, by means of firmware).

[0163] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0164] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0165] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0166] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0167] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0168] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0169] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0170] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for evaluating a text graph model, comprising: The performance of the Wensheng graph model is evaluated by machine evaluation, and the first comprehensive evaluation result is obtained; When the first comprehensive evaluation result meets a preset condition, triggering a target evaluation method to evaluate the performance of the culture graph model to obtain a second comprehensive evaluation result; Among them, the evaluation accuracy of the machine evaluation method is lower than that of the target evaluation method.

2. The method according to claim 1, further comprising: Comparing the first comprehensive evaluation result of the Wensheng graph model with the reference evaluation result to obtain a comparison result; the reference evaluation result is an evaluation result of the reference model obtained by the previous round of iterative optimization of the Wensheng graph model using the machine evaluation method; When the comparison result indicates that the performance of the Wensheng graph model is better than the performance of the reference model, and the performance difference between the Wensheng graph model and the reference model is greater than a preset difference, it is determined that the preset condition is met.

3. The method according to claim 1, wherein: The performance of the Wensheng graph model is evaluated by a machine evaluation method to obtain a first comprehensive evaluation result, including: Constructing an evaluation set; Inputting the evaluation set into the Wensheng graph model to obtain an image set generated by the Wensheng graph model for the evaluation set; The evaluation set and the image set are processed by the machine evaluation method to obtain the first comprehensive evaluation result.

4. The method according to claim 3, wherein: The construction of the evaluation set includes: Construct a text description containing at least one of the following elements to obtain the evaluation set: Characteristic sentences of the language used in the text description; The entity word of the property right object; Control statements used to evaluate the control capabilities of the Wensheng graph model; Text used to evaluate text rendering capabilities.

5. The method according to any one of claims 1 to 4, wherein: The performance of the Wensheng graph model is evaluated by a machine evaluation method to obtain a first comprehensive evaluation result, including: Using the evaluation strategies corresponding to the multiple evaluation dimensions in the machine evaluation method to respectively evaluate the performance of the text graph model, and obtaining the evaluation score of each dimension; The evaluation scores of the various dimensions are input into the fitting network model to obtain the first comprehensive evaluation result.

6. The method according to claim 5, further comprising training the fitting network model based on the following method: For a plurality of training texts, obtaining evaluation scores of a plurality of evaluation dimensions of the image generation capability of the target version model of the text-to-image model for the plurality of training texts; Inputting the evaluation scores of the multiple evaluation dimensions of each training text into the initial network respectively, and obtaining the prediction generation capability value of the target version model obtained by the initial network for each training text; Sorting the plurality of training texts based on the predicted generation capability values ​​to obtain a predicted sorting; Based on the gap between the predicted order and the reference order of the plurality of training texts, the model parameters of the initial network are adjusted, and the fitting network model is obtained when a convergence condition is met.

7. The method according to claim 6, wherein: Determining a reference ranking of the plurality of training texts comprises: For each training text, obtain multiple images generated by the target version model for the training text to obtain an image group corresponding to the training text; Obtaining a generation capability value of each image in the image group; the generation capability value expresses the generation capability of the text graph model for the training text; Determining a comprehensive ability value of the generation ability of the text graph model for the training text based on the generation ability values ​​of the plurality of images; Based on the comprehensive capability values ​​of the plurality of training texts, a reference ranking of the plurality of training texts is obtained.

8. The method according to claim 5 or 6, wherein the evaluation dimension comprises at least one of the following: The correlation between the text description and the generated image; Image quality of generated images; The distortion of the generated image; Generating text and description contained in images requires the relevance of the text content.

9. The method according to claim 1, wherein: In the case where the large evaluation model simulating manual evaluation has not been trained to converge, the target evaluation method is the manual evaluation method; When the training of the large evaluation model converges, the target evaluation method is an evaluation method that uses the large evaluation model for evaluation.

10. The method according to claim 9 further comprises training the evaluation large model based on the following method: Obtaining annotated data obtained through manual evaluation; Based on the labeled data, an initial large model is trained to obtain the evaluation large model.

11. An evaluation device for a cultural graph model, comprising: A first evaluation module is used to evaluate the performance of the Wensheng graph model by machine evaluation to obtain a first comprehensive evaluation result; A second evaluation module is used to trigger a target evaluation method to evaluate the performance of the culture graph model to obtain a second comprehensive evaluation result when the first comprehensive evaluation result meets a preset condition; Among them, the evaluation accuracy of the machine evaluation method is lower than that of the target evaluation method.

12. The device according to claim 11, further comprising a comparison module, configured to: Comparing the first comprehensive evaluation result of the Wensheng graph model with the reference evaluation result to obtain a comparison result; the reference evaluation result is an evaluation result of the reference model obtained by the previous round of iterative optimization of the Wensheng graph model using the machine evaluation method; When the comparison result indicates that the performance of the Wensheng graph model is better than the performance of the reference model, and the performance difference between the Wensheng graph model and the reference model is greater than a preset difference, it is determined that the preset condition is met.

13. The device according to claim 11, wherein: The first evaluation module comprises: Construction unit, used to construct the evaluation set; An input unit, used for inputting the evaluation set into the Wensheng graph model to obtain an image set generated by the Wensheng graph model for the evaluation set; A processing unit is used to process the evaluation set and the image set by using the machine evaluation method to obtain the first comprehensive evaluation result.

14. The device according to claim 13, wherein: The building block is specifically used for: Construct a text description containing at least one of the following elements to obtain the evaluation set: Characteristic sentences of the language used in the text description; The entity word of the property right object; Control statements used to evaluate the control capabilities of the Wensheng graph model; Text used to evaluate text rendering capabilities.

15. The device according to any one of claims 11 to 14, wherein: The first evaluation module is specifically used for: Using the evaluation strategies corresponding to the multiple evaluation dimensions in the machine evaluation method to respectively evaluate the performance of the text graph model, and obtaining the evaluation score of each dimension; The evaluation scores of the various dimensions are input into the fitting network model to obtain the first comprehensive evaluation result.

16. The apparatus according to claim 15, further comprising a training module, configured to train the fitting network model based on the following method: For a plurality of training texts, obtaining evaluation scores of a plurality of evaluation dimensions of the image generation capability of the target version model of the text-to-image model for the plurality of training texts; Inputting the evaluation scores of the multiple evaluation dimensions of each training text into the initial network respectively, and obtaining the prediction generation capability value of the target version model obtained by the initial network for each training text; Sorting the plurality of training texts based on the predicted generation capability values ​​to obtain a predicted sorting; Based on the gap between the predicted order and the reference order of the plurality of training texts, the model parameters of the initial network are adjusted, and the fitting network model is obtained when a convergence condition is met.

17. The apparatus according to claim 16, further comprising a determining module, configured to: For each training text, obtain multiple images generated by the target version model for the training text to obtain an image group corresponding to the training text; Obtaining a generation capability value of each image in the image group; The generation capability value expresses the generation capability of the text graph model for the training text; the generation capability value is obtained based on an evaluation method other than the machine evaluation method; Determining a comprehensive ability value of the generation ability of the text graph model for the training text based on the generation ability values ​​of the plurality of images; Based on the comprehensive capability values ​​of the plurality of training texts, a reference ranking of the plurality of training texts is obtained.

18. The apparatus according to claim 15 or 16, wherein the evaluation dimension comprises at least one of the following: The correlation between the text description and the generated image; Image quality of generated images; The distortion of the generated image; Generating text and description contained in images requires the relevance of the text content.

19. The device according to claim 11, wherein: In the case where the large evaluation model simulating manual evaluation has not been trained to converge, the target evaluation method is the manual evaluation method; When the training of the large evaluation model converges, the target evaluation method is an evaluation method that uses the large evaluation model for evaluation.

20. The apparatus according to claim 19, further comprising an optimization module, configured to train the evaluation large model based on the following method: Obtaining annotated data obtained through manual evaluation; Based on the labeled data, an initial large model is trained to obtain the evaluation large model.

21. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 10.

22. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-10.

23. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 10.

Images