A text-to-image model dynamic evaluation method and device based on a large language model

Abstract

The application provides a text-to-image model dynamic evaluation method based on a large language model, which comprises the following steps: generating test input text according to a selected topic by an LLM, generating a test picture according to the test input text by a text-to-image model to be evaluated, recording the accuracy rate of the test picture, and determining whether the accuracy rate is lower than a threshold value; if yes, performing a text adjustment step; adjusting the test input text to obtain multiple adjusted input texts, generating adjusted pictures according to the test input text by the text-to-image model, and taking the adjusted input text corresponding to the adjusted picture with an accuracy rate lower than the threshold value as a failure text; analyzing the right and wrong reasons of the text-to-image model according to the accuracy rates of the pictures of all the test input texts and the adjusted input texts under the current topic by the LLM, generating a new topic according to the analysis of the right and wrong reasons and the current topic, performing the text generation step again until the user demand is met, and saving all the failure texts as the evaluation result of the text-to-image model.

Description

Technical Field

[0001] This invention belongs to the technical fields of cross-modal text image generation, image generation evaluation, and large-scale language model application, and specifically relates to a dynamic evaluation method and device for text-generated image models based on large-scale language models. Background Technology

[0002] In recent years, numerous new text-to-image models have emerged, allowing users to directly generate vivid images from text and finding applications across a wide range of fields. Despite their powerful capabilities, these models still exhibit various generalization failures, such as clumsy combinations of limbs and failures in generating facial expressions. Therefore, developers need to conduct comprehensive and reliable evaluations of text-to-image models before deploying them to various applications to identify their shortcomings; unfortunately, this remains a significant challenge.

[0003] Current evaluation methods primarily rely on static evaluation, which directly uses pre-collected data to perform a one-time evaluation of the model. The disadvantages of static evaluation are:

[0004] 1. Fixed coverage. The aspects and granularity covered by the dataset are pre-defined, resulting in insufficient scalability and difficulty in discovering model defects outside the pre-defined scope;

[0005] 2. Difficulty in updating and iterating: Static evaluation is difficult to adapt to continuous model updates and iterations. As model performance improves or application scenarios expand, the original static test set may no longer be able to effectively measure the latest performance of the model, and updating these test sets usually requires a lot of human and material resources;

[0006] 3. There is a risk of data leakage. If the static test set is leaked, the model can be targeted for optimization, leading to distorted model evaluation results. While existing technologies include dynamic methods based on adversarial attacks, they are limited to evaluating the model's robustness to noun variations in white-box models, thus restricting their application. Therefore, an open dynamic evaluation technique for text-based graph models is needed.

[0007] Meanwhile, human-assisted evaluation using large language models (LLMs), i.e., open-ended human-in-loop testing, has been well-established in natural language processing and image-to-text tasks. Large language models (LLMs) refer to language models containing hundreds of billions (or more) parameters trained on massive amounts of text data, such as models GPT-3, PaLM, Galactica, and LLaMA. However, when directly applied to the text-to-image domain, these methods cannot effectively evaluate the quality of image generation. Specifically: due to the complexity and diversity of error types in image generation tasks, human error labeling is highly subjective and costly; LLMs tend to generate excessively long and complete test inputs to discover model errors, which does not effectively help understand the model's limitations or accurately determine the boundaries of its capabilities. Summary of the Invention

[0008] The purpose of this invention is to alleviate the problem of reliable evaluation of open-domain text-to-image generation. It proposes a new framework for dynamic testing of open-domain text-to-image models based on a large-scale language model, which includes an error localization and analysis module and finally outputs an error log report to help users analyze and understand the model under test.

[0009] Specifically, such as Figure 3 As shown, this invention proposes a dynamic evaluation method for text-based graph models based on large-scale language models, including:

[0010] In text generation step S100, the LLM generates test input text based on the selected topic, and the text-to-image model to be evaluated generates test images based on the test input text. The accuracy of the generated test images is recorded, and it is determined whether the accuracy is lower than the threshold. If so, the text adjustment step is executed.

[0011] In text adjustment step S200, the test input text is adjusted to obtain multiple adjusted input texts. The text-generated image model generates an adjusted image based on the test input text, and the adjusted input text corresponding to the adjusted image with an accuracy lower than the threshold is regarded as the failure text.

[0012] In text analysis step S300, the LLM analyzes the reasons for the correctness or incorrectness of the text-to-image model based on the accuracy of all test input texts and the image generation of the adjusted input texts under the current topic. The LLM analyzes the reasons for the correctness or incorrectness and the current topic, generates a new topic, and executes the text generation step again until the user's requirements are met. All the current failed texts are saved as the evaluation results of the text-to-image model.

[0013] The dynamic evaluation method for text-generated graph models based on large-scale language models further includes:

[0014] The training update step uses the failed text in the evaluation results as training text to train the text-to-image model so that the trained text-to-image model outputs the correct image that matches the training text.

[0015] The dynamic evaluation method for text-generated graph models based on large-scale language models includes user requirements such as determining whether the number of failed texts reaches a threshold, whether the number of new topics generated reaches a threshold, or whether the test cycle limit is reached.

[0016] The aforementioned dynamic evaluation method for text-generated graph models based on large-scale language models includes the following text adjustment steps:

[0017] The test input text is converted into graph data with objects, relations, and attributes as nodes. The edges in the graph data represent semantic dependencies between two nodes in the original sentence. The graph is split according to the relation nodes, and the nodes connected to them by edges are assigned to the same subgraph. Split text is generated based on the subgraph. One of the split texts is selected as the base text and the remaining split texts are fine-tuned to obtain the fine-tuned text. The base text and the fine-tuned text are merged as the input text for the adjustment.

[0018] This invention also proposes a dynamic evaluation device for text-based graph models based on large-scale language models, including:

[0019] The text generation module generates test input text based on the selected topic. The text-to-image model to be evaluated generates test images based on the test input text. The accuracy of the generated test images is recorded, and it is determined whether the accuracy is lower than the threshold. If so, the text adjustment module is executed.

[0020] The text adjustment module adjusts the test input text to obtain multiple adjusted input texts. The text-to-image model generates adjusted images based on the test input texts and identifies the adjusted input texts corresponding to adjusted images with an accuracy rate below a threshold as failure texts.

[0021] The text analysis module, based on the accuracy of image generation from all test input texts and adjusted input texts under the current topic, analyzes the reasons for the correctness or incorrectness of the text-to-image model. The LLM analyzes the reasons for the correctness or incorrectness and the current topic, generates a new topic, and executes the text generation module again until the user's requirements are met. All the current failed texts are saved as the evaluation results of the text-to-image model.

[0022] The aforementioned dynamic evaluation device for text-based graph models based on large-scale language models further includes:

[0023] The training update module uses the failed text in the evaluation results as training text to train the text-to-image model so that the trained text-to-image model outputs the correct image that matches the training text.

[0024] The aforementioned dynamic evaluation device for text-to-graph model based on a large language model includes user requirements for determining whether the number of failed texts has reached a threshold, or whether the number of new topics generated has reached a threshold, or whether the test cycle limit has been reached.

[0025] The aforementioned dynamic evaluation device for text-based graph models based on large-scale language models, wherein the text adjustment module includes:

[0026] The test input text is converted into graph data with objects, relations, and attributes as nodes. The edges in the graph data represent semantic dependencies between two nodes in the original sentence. The graph is split according to the relation nodes, and the nodes connected to them by edges are assigned to the same subgraph. Split text is generated based on the subgraph. One of the split texts is selected as the base text and the remaining split texts are fine-tuned to obtain the fine-tuned text. The base text and the fine-tuned text are merged as the input text for the adjustment.

[0027] The present invention also proposes an electronic device, including the aforementioned dynamic evaluation device for text-based graph models based on a large language model.

[0028] The electronic device is connected to an information device, which displays the evaluation results.

[0029] The electronic device wherein the information display device is used to display specified information in the evaluation result using user-set display parameters, attributes, or through an artificial intelligence model.

[0030] The present invention also proposes a storage medium for storing a computer program that executes the dynamic evaluation method for text-based graph models based on large language models.

[0031] As can be seen from the above solutions, the advantages of the present invention are:

[0032] This invention demonstrates the effectiveness of Inf-Eval by setting up initial test topics from four perspectives (object, relation, attribute, and global attribute context) and identifying failures of the tested model across various state-of-the-art text-to-image models. Compared to static testing methods, Inf-Eval allows users to detect a greater number of faults using the same amount of text-image pairs.

[0033] The method of this invention effectively identifies complex and rare faults in the tested model, such as frequent failures in pronoun generation and generation of cultural and abstract relational themes. This provides valuable and interpretable insights for further improvement of the tested model. The tested examples can be used as training data to further optimize the model and improve its image generation capabilities. Attached Figure Description

[0034] Figure 1 This is an example diagram illustrating the execution process of a specific embodiment of the present invention;

[0035] Figure 2 This is a diagram illustrating the technical effects of the present invention;

[0036] Figure 3 This is a flowchart of the method of the present invention;

[0037] Figure 4 This is a block diagram of the device of the present invention;

[0038] Figure 5 This is a schematic diagram of the structure of the first electronic device of the present invention;

[0039] Figure 6 This is a schematic diagram of the application environment structure of the first electronic device of the present invention.

[0040] Figure 7 This is a schematic diagram of the structure of the second electronic device of the present invention;

[0041] Figure 8 This is an example diagram illustrating the execution process of the present invention.

[0042] Figure label:

[0043] S100 - Text generation steps;

[0044] S200 - Text Adjustment Steps;

[0045] S300 - Text Analysis Steps;

[0046] S1 - Text generation module;

[0047] S2 - Text Adjustment Module;

[0048] S3 - Text Analysis Module;

[0049] A - First electronic device;

[0050] B- A dynamic evaluation device for text-based graph models based on large-scale language models;

[0051] C-Data acquisition equipment;

[0052] D-Information display device;

[0053] 1000 - Second electronic device;

[0054] Ⅰ-Computational Unit;

[0055] II-ROM;

[0056] III-RAM;

[0057] N-bus;

[0058] V-Interface;

[0059] VI - Input Unit;

[0060] VII - Output Unit;

[0061] VIII - Storage medium;

[0062] IX - Communication Unit. Detailed Implementation

[0063] It should be noted that the processor described in this invention is the control center of an electronic device. It can be a single processor or a collective term for multiple processing elements. For example, it can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of this invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0064] Alternatively, the processor can perform various functions of the electronic device by running or executing software programs stored in memory and by calling data stored in memory.

[0065] In a specific implementation, as one example, the processor may include one or more CPUs. Each of these processors may be a single-core processor or a multi-core processor. Here, "processor" can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions). Electronic devices may include servers, desktop computers, laptops, smartphones, tablets, embedded computers, etc., where the embedded computer includes vehicles and robots, etc.

[0066] The memory is used to store the software program that executes the solution of the present invention, and the execution is controlled by the processor. For specific implementation methods, please refer to the above method embodiments, which will not be repeated here.

[0067] It should be noted that the structure of the electronic device shown in the accompanying drawings of this invention does not constitute a limitation thereof. The actual knowledge structure recognition device may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0068] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0069] It should also be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0070] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0071] It should also be understood that, in various embodiments of the present invention, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0072] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0073] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0074] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0075] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0076] To address the aforementioned limitations, we introduce an LLM-based infinite evaluation framework for text-to-image models, a novel framework for dynamically testing text-to-image models in an open domain. For both white-box and black-box models with accessible outputs, Inf-Eval can adapt to user-defined criteria such as alignment and fairness, and generate more test inputs hierarchically. Specifically, Inf-Eval comprises a test input generation module, a test topic generation module, and an error reflection module. In the test input and topic generation modules, LLM iteratively refines the test inputs and continuously explores the model boundaries based on test records. To reduce over-probing, the error reflection module includes an automatic input minimization module to isolate error-causing components and a self-reflective module to determine the attributes of the errors. Furthermore, under the LLM-driven iterative system, Inf-Eval progressively adjusts the test inputs based on human feedback, theoretically allowing for an infinite expansion of the test scope and granularity, progressively exploring the ever-growing boundaries of the evolving model.

[0077] To make the above-mentioned features and effects of the present invention clearer and easier to understand, specific embodiments are described below in conjunction with the accompanying drawings. This specification discloses one or more embodiments incorporating the features of the present invention. The disclosed embodiments are merely illustrative. The scope of protection of the present invention is not limited to the disclosed embodiments, but is defined by the appended claims.

[0078] Figure 8 This provides an overview of the Inf-Eval method proposed in this invention. Inf-Eval Overview: After setting the initial topic and model hyperparameters, the user begins the Inf-Eval process. Initially, in the test input generation module, the LLM generates test inputs based on the selected topic and existing test records (Step A). ​​The user then annotates the images generated by the model under test (Step B) and updates the test records (Step C). If some test inputs exhibit low pass rates (Step D), Inf-Eval adjusts the test inputs to find the smallest subset of test inputs causing output failures (Step E). Then, the LLM reflects on all information from the current topic, analyzing the reasons for the model's successes and failures (Step F), and then updates the test records (Step G). Whether the evaluation process ends depends on the user's needs. If not, the LLM suggests a new topic based on the test records (Step H), allowing the user to select one (Step I) and generate new test inputs to continue the loop. If available, the current test record tree is output. If the test input shows a high pass rate (step D), the reflection should focus on analyzing the strengths and weaknesses of the tested model. If all tests have a high pass rate, the reflection should focus on identifying the model's strengths and weaknesses, which require further exploration.

[0079] Step C updates the test input and model record of whether they passed the test in step A; step G updates the reflection content (step F) and the intermediate content of the test input adjustment (step E) into the test record.

[0080] The overall process is as follows Figure 1 As shown, we will now introduce them in modules.

[0081] I. Test Input Generation Module:

[0082] After initialization, LLM will generate user-specified test inputs based on the current test topic. If the test record T contains other information about the current test topic, such as the test records of its parent node or reflection content, then this information will also be used as input conditions to inspire LLM to generate test inputs that can delve into the current test topic and discover new errors in the model under test.

[0083] The tested model will then generate images based on the new test input, and the user will label them as pass / fail according to the evaluation criteria. To reduce the user's annotation workload, we provide CLIP-Score annotations, which the user can then modify.

[0084] II. Error Reflection Module:

[0085] When a user finishes annotating a topic, if the result is not satisfactory, we want to further analyze the result to find the cause of the error. Considering that LLM itself does not have visual analysis capabilities, this invention proposes a minimal input function that operates on the input text and assists in attributing causes on the text with user ratings.

[0086] The minimum input approach is to first split the test input into two halves, A and B, and see if A and B pass the test. If not, continue splitting. If both pass, choose one as the baseline and split the other into A' and B'. Test baseline∪A' and baseline∪B', and repeat this process until it cannot be split anymore.

[0087] To ensure that the decomposed sentences do not become semantically incoherent, we use parsing techniques to convert the text into a scene graph. Nodes are categorized into three types: object, relation, and attribute. Edges represent semantic dependencies between two nodes in the original sentence. When decomposing the scene graph, we first split nodes according to the relation type; nodes connected by edges to these relation nodes are assigned to the same subgraph. Nodes without relations are split into object nodes, and finally, attribute nodes are split.

[0088] To support the generation module, we added a reflection task instruction, which requires LLM to analyze the current test records and summarize the strengths and weaknesses of the tested model.

[0089] Through the "Error Reflection Module," Inf-Eval comprehensively analyzes and synthesizes all test data, using these insights to inform and optimize subsequent testing cycles. This reflective process not only improves the accuracy of the testing methodology but also helps deepen the understanding of model performance and potential areas for improvement.

[0090] III. Test Topic Generation Module:

[0091] After labeling the current test topics and going through a reflection process, the next step is to adaptively generate further exploration topics based on the current test results.

[0092] While annotating text and images is easy for humans, generating new topics is challenging and costly. Therefore, inspired by the fluent language generation capabilities, rich knowledge, and summarizing and analyzing abilities of LLM, we entrusted this creative task to LLM.

[0093] To expand to new topics, LLM generates new topics based on the current nodes: these newly generated topics are automatically created as new nodes in the test record T, and the parent-child relationships are updated. The user will select the most promising test topic to explore and proceed to the next test loop.

[0094] like Figure 1 As shown, following the test input module and error response module of the topic "Dog-Human Relationships", LLM proposes a new subtopic "Testing specific actions and interactions between a dog and its owner."

[0095] Through the topic generation module, Inf-Eval, guided by the ever-changing test log T, drives the generation of test topics and inputs for subsequent rounds until the user manually ends the test or a predefined test cycle limit is reached.

[0096] The topic generation module can suggest related secondary topics based on the current topic, thereby broadening the scope of testing and enabling users to refine the focus of upcoming tests.

[0097] The following are system embodiments corresponding to the above method embodiments. This embodiment can be implemented in conjunction with the above embodiments. The relevant technical details mentioned in the above embodiments are still valid in this embodiment, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above embodiments.

[0098] like Figure 4 As shown, this invention also proposes a dynamic evaluation device B for text-based graph models based on large-scale language models, which includes:

[0099] The text generation module S1 generates test input text based on the selected topic. The text-to-image model to be evaluated generates test images based on the test input text. The accuracy of the generated test images is recorded, and it is determined whether the accuracy is lower than the threshold. If so, the text adjustment module is executed.

[0100] The text adjustment module S2 adjusts the test input text to obtain multiple adjusted input texts. The text-to-image model generates an adjusted image based on the test input text and takes the adjusted input text corresponding to the adjusted image with an accuracy lower than the threshold as the failure text.

[0101] The text analysis module S3, LLM, analyzes the reasons for the correctness or incorrectness of the text-to-image model based on the accuracy of all test input texts and the images generated from the adjusted input texts under the current topic. LLM analyzes the reasons for the correctness or incorrectness and the current topic, generates a new topic, and executes the text generation module again until the user's requirements are met. All the current failed texts are saved as the evaluation results of the text-to-image model.

[0102] The aforementioned dynamic evaluation device B for text-based graph models based on large-scale language models further includes:

[0103] The training update module uses the failed text in the evaluation results as training text to train the text-to-image model so that the trained text-to-image model outputs the correct image that matches the training text.

[0104] The dynamic evaluation device B for text-to-graph model based on a large language model includes user requirements such as determining whether the number of failed texts reaches a threshold, or whether the number of new topics generated reaches a threshold, or whether the test cycle limit is reached.

[0105] The aforementioned dynamic evaluation device B for text-based graph models based on large-scale language models, wherein the text adjustment module includes:

[0106] The test input text is converted into graph data with objects, relations, and attributes as nodes. The edges in the graph data represent semantic dependencies between two nodes in the original sentence. The graph is split according to the relation nodes, and the nodes connected to them by edges are assigned to the same subgraph. Split text is generated based on the subgraph. One of the split texts is selected as the base text and the remaining split texts are fine-tuned to obtain the fine-tuned text. The base text and the fine-tuned text are merged as the input text for the adjustment.

[0107] like Figure 5 As shown, in another embodiment of the present invention, a first electronic device A is also proposed, which includes the aforementioned dynamic evaluation device B for text-based graph models based on a large language model.

[0108] like Figure 6 As shown, the first electronic device A can also be connected to the data acquisition device C and the information display device D. The data acquisition device C is used to acquire the topic or input text, and the information display device D is used to display the evaluation results obtained by the analysis of the present invention, or to display the image output by the text-to-image model corresponding to the topic or input text.

[0109] The information display device D can process and organize the data output by the first electronic device A based on an information display mechanism to improve the readability of the data. This information display mechanism can be manually preset, for example, visualizing the data output by the first electronic device A. It can present users with rich and essential key information, such as news updates and system information, based on user settings, allowing users to access this information more promptly without having to access secondary pages or scroll through pages, thus saving user effort. Alternatively, this information display mechanism can be an artificial intelligence (AI) display model, which can learn the user's key information interests based on previous usage habits, such as viewing time, number of clicks, and number of edits, and then automatically present users with rich and essential key information.

[0110] This invention also proposes a storage medium VIII for storing a computer program that executes the dynamic evaluation method for the text-based graph model based on a large language model. It should be understood that the storage medium in the embodiments of this invention can be volatile memory or non-volatile memory, or may include both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DRRAM).

[0111] Figure 7 A schematic block diagram of a second electronic device 1000 that can be used to implement embodiments of the present invention is shown. The second electronic device 1000 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The second electronic device 1000 can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein. The second electronic device 1000 may be the same as or different from the first electronic device A.

[0112] The second electronic device 1000 includes a computing unit I, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory II (ROM) or a computer program loaded from storage medium VIII into random access memory (RAM) III. The RAM III may also store various programs and data required for the operation of the device 1000. The computing unit I, ROM II, and RAM III are interconnected via bus IV. An input / output (I / O) interface V is also connected to bus IV.

[0113] Multiple components in the second electronic device 1000 are connected to I / O interface V, including: input unit VI, such as a keyboard, mouse, etc.; output unit VII, such as various types of displays, speakers, etc.; storage medium VIII, such as a disk, optical disk, etc.; and communication unit IX, such as a network card, modem, wireless transceiver, etc. Communication unit IX allows the second electronic device 1000 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0114] The computing unit I can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of computing unit I include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit I executes the various methods and processes described above, such as method steps S100-S300. For example, in some embodiments, the method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage medium VIII. In some embodiments, part or all of the computer program may be loaded and / or installed on device 1000 via ROM II and / or communication unit IX. When the computer program is loaded into RAM III and executed by computing unit I, one or more steps of the methods described above may be performed. Alternatively, in other embodiments, computing unit I may be configured to execute methods by any other suitable means (e.g., by means of firmware).

[0115] In summary, this invention presents an open-domain dynamic testing framework based on a large-scale language model, overcoming the shortcomings of existing static evaluation and adversarial attack mechanisms. This invention achieves an open-domain, infinitely granular, and dynamically updated evaluation mechanism, primarily comprising a test input generation module, a test topic generation module, and an error reflection module. Finally, it uses an LLM (Language Modeling) and human feedback to construct a hierarchical test record tree for the model under test. After initialization, the test input module generates test input based on the user-specified test topic. The error reflection module analyzes the test results and provides them to the LLM to generate new test topics for the user to choose from, continuing the testing loop until the user finishes the test.

[0116] This invention is based on a divide-and-conquer error localization and analysis algorithm. Considering that LLMs (Model-Level Models) can easily generate excessively long test inputs to find more errors, but such test inputs are meaningless and fail to reveal the model's capability boundaries, we designed a minimum input module. Based on the divide-and-conquer algorithm, it splits the test input in half until the smallest test input causing the error is found, thus achieving error localization. In addition, we allow the LLM to reflect on and summarize all test records, analyzing error types. This enhances interpretability and understanding the capability boundaries of the model under test, and also helps the LLM generate more challenging test inputs and test topics.

[0117] This invention evaluates the effectiveness of the Inf-Eval framework versus an adaptive testing framework, in which the LLM directly generates the same amount of text input. The evaluation is conducted against three different models to assess the error detection capabilities of each approach. To test each model, we computed the average pass rate for four relatively objective initial topics: "material objects," "explicit relations," "static attributes," and "style."

[0118] like Figure 2 As shown, the results indicate that the Inf-Eval framework consistently discovers more failure cases compared to non-adaptive testing methods, reflected in the lower pass rate of models subjected to Inf-Eval testing. Furthermore, Inf-Eval discovers more discriminative test cases; the difference in pass rates between the three popular open-source models SD1-5, SD2-1, and SDXL is larger under Inf-Eval testing, indicating that Inf-Eval's testing efficiency is higher. This demonstrates that the Inf-Eval framework has higher testing efficiency in discovering model vulnerabilities across different text-to-image models and testing methods.

[0119] Although embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the specification and embodiments. They can be applied to various fields suitable for the present invention. For those skilled in the art, other modifications can be easily made. Therefore, without departing from the general concept defined by the claims and their equivalents, the present invention is not limited to the specific details and illustrations shown and described herein.

Claims

1. A dynamic evaluation method for text-based graph models based on large-scale language models, characterized in that, include: In the text generation step, the LLM generates test input text based on the selected topic, and the text-generated graph model to be evaluated generates test images based on the test input text. The accuracy of the generated test images is recorded, and it is determined whether the accuracy is lower than the threshold. If so, the text adjustment step is performed. The text adjustment step involves adjusting the test input text to obtain multiple adjusted input texts. The text-to-image model generates adjusted images based on the test input texts and identifies the adjusted input texts corresponding to adjusted images with an accuracy rate below a threshold as failure texts. In the text analysis step, the LLM analyzes the reasons for the correctness or incorrectness of the text-to-image model based on the accuracy of all test input texts and adjusted input texts under the current topic. The LLM analyzes the reasons for the correctness or incorrectness and the current topic, generates a new topic, and executes the text generation step again until the user's requirements are met. All the current failed texts are saved as the evaluation results of the text-to-image model. The text adjustment steps include: The test input text is converted into graph data with objects, relations and attributes as nodes. The edges in the graph data represent semantic dependencies between two nodes in the original sentence. The nodes are split according to their relationships, and nodes connected to them by edges are grouped into the same subgraph. Split text is generated based on the subgraph. One of the split texts is selected as the base text, and the remaining split texts are fine-tuned to obtain the fine-tuned text. The base text and the fine-tuned text are then merged as the input text for the adjustment.

2. The dynamic evaluation method for text-based graph models based on large-scale language models as described in claim 1, characterized in that, Also includes: The training update step uses the failed text in the evaluation results as training text to train the text-to-image model so that the trained text-to-image model outputs the correct image that matches the training text.

3. The dynamic evaluation method for text-based graph models based on large-scale language models as described in claim 1, characterized in that, The user requirement includes determining whether the number of failed texts has reached a threshold, or whether the number of new topics generated has reached a threshold, or whether the test cycle limit has been reached.

4. A dynamic evaluation device for text-based graph models based on large-scale language models, characterized in that, include: The text generation module generates test input text based on the selected topic. The text-to-image model to be evaluated generates test images based on the test input text. The accuracy of the generated test images is recorded, and it is determined whether the accuracy is lower than the threshold. If so, the text adjustment module is executed. The text adjustment module adjusts the test input text to obtain multiple adjusted input texts. The text-to-image model generates adjusted images based on the test input texts and identifies the adjusted input texts corresponding to adjusted images with an accuracy rate below a threshold as failure texts. The text analysis module, based on the accuracy of image generation from all test input texts and adjusted input texts under the current topic, analyzes the reasons for the correctness or incorrectness of the text-to-image model. The LLM analyzes the reasons for the correctness or incorrectness and the current topic, generates a new topic, and executes the text generation module again until the user's requirements are met. All the current failed texts are saved as the evaluation results of the text-to-image model. This text adjustment module includes: The test input text is converted into graph data with objects, relations and attributes as nodes. The edges in the graph data represent semantic dependencies between two nodes in the original sentence. The nodes are split according to their relationships, and nodes connected to them by edges are grouped into the same subgraph. Split text is generated based on the subgraph. One of the split texts is selected as the base text, and the remaining split texts are fine-tuned to obtain the fine-tuned text. The base text and the fine-tuned text are then merged as the input text for the adjustment.

5. The dynamic evaluation device for text-based graph models based on large-scale language models as described in claim 4, characterized in that, Also includes: The training update module uses the failed text in the evaluation results as training text to train the text-to-image model so that the trained text-to-image model outputs the correct image that matches the training text.

6. The dynamic evaluation device for text-based graph models based on large-scale language models as described in claim 4, characterized in that, The user requirement includes determining whether the number of failed texts has reached a threshold, or whether the number of new topics generated has reached a threshold, or whether the test cycle limit has been reached.

7. An electronic device, characterized in that, Includes the dynamic evaluation device for text-based graph models based on large-scale language models as described in any one of claims 4-6.

8. The electronic device as claimed in claim 7, characterized in that, The electronic device is connected to an information display device, which is used to display the evaluation results and to display specified information in the evaluation results using user-set display parameters, attributes, or through an artificial intelligence model.

9. A storage medium for storing a computer program that performs the dynamic evaluation method for text-based graph models based on any one of claims 1-3.

Images