Model evaluation method, device, equipment, medium and product

Abstract

This specification provides a method, apparatus, device, medium, and product for evaluating a model. The scheme may include: inputting a visualization page generation instruction and a test sample into a model to be evaluated, and obtaining the output result of the model to be evaluated; performing functional verification on the model to be evaluated based on the output result, and obtaining a first verification result; if the first verification result indicates that the model to be evaluated can generate page code that meets preset requirements, then rendering the generated page code to obtain the visualization page; performing visualization component quality verification on the rendered visualization page, and obtaining a second verification result; performing content quality verification on the rendered visualization page, and obtaining a third verification result; and evaluating the model to be evaluated based on the first verification result, the second verification result, and the third verification result.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, device, medium and product for evaluating a model. Background Technology

[0002] With the rapid development of artificial intelligence technology, large language models have the ability to understand and generate multiple types of information such as text, images, and code. They can generate relevant page code according to user instructions, which is then rendered into a visual page containing visual components such as images and tables and displayed to the user.

[0003] The quality of the output results of a large language model affects the page rendering effect and the accuracy of information expression, which in turn affects the user experience. Therefore, it is necessary to evaluate the large language model to ensure the quality of the model output results. Summary of the Invention

[0004] In view of the above, one or more embodiments of this specification provide a model evaluation method, a model evaluation apparatus, a computing device, a computer-readable storage medium, and a computer program product to improve the accuracy of model evaluation.

[0005] According to a first aspect of one or more embodiments of this specification, a method for evaluating a model is provided, comprising: The visualization page generation instruction and the test sample are input into the model to be evaluated to obtain the output result of the model to be evaluated; the visualization page generation instruction is used to instruct the model to be evaluated to generate a visualization page for the test sample; the visualization page is a page containing visualization components; The model to be evaluated is functionally verified based on the output results to obtain a first verification result; the functional verification is used to verify whether the model to be evaluated can generate page code that meets the preset requirements. If the first verification result indicates that the model to be evaluated can generate page code that meets the preset requirements, then the generated page code is rendered to obtain the visual page; The rendered visualization page is subjected to visualization component quality verification to obtain a second verification result; the visualization component quality verification is used to verify the rendering quality of the visualization components in the visualization page. The rendered visualization page is subjected to content quality verification to obtain a third verification result; the content quality verification is used to verify the content quality of the semantic content in the visualization page. The model to be evaluated is evaluated based on the first verification result, the second verification result, and the third verification result.

[0006] According to a second aspect of one or more embodiments of this specification, a model evaluation apparatus is provided, comprising: The input module is used to input the visualization page generation instruction and the test sample into the model to be evaluated, and obtain the output result of the model to be evaluated; the visualization page generation instruction is used to instruct the model to be evaluated to generate a visualization page for the test sample; the visualization page is a page containing visualization components; The functional verification module is used to perform functional verification on the model to be evaluated based on the output result to obtain a first verification result; the functional verification is used to verify whether the model to be evaluated can generate page code that meets preset requirements; The rendering module is used to render the generated page code to obtain the visualization page if the first verification result indicates that the model to be evaluated can generate page code that meets the preset requirements. The component quality verification module is used to verify the quality of the visual components on the rendered visualization page and obtain a second verification result; the visualization component quality verification is used to verify the rendering quality of the visual components in the visualization page. The content quality verification module is used to perform content quality verification on the rendered visualization page and obtain a third verification result; the content quality verification is used to verify the content quality of the semantic content in the visualization page. The evaluation module is used to evaluate the model to be evaluated based on the first verification result, the second verification result, and the third verification result.

[0007] According to a third aspect of one or more embodiments of this specification, a computing device is provided, including a memory, a processor, and computer instructions stored in the memory and executable on the processor, wherein the processor executes the computer instructions to implement the steps of the method described above.

[0008] According to a fourth aspect of one or more embodiments of this specification, a computer-readable storage medium is provided that stores computer instructions which, when executed by a processor, implement the steps of the method described above.

[0009] According to a fifth aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described method.

[0010] One embodiment of this specification can achieve at least the following beneficial effects: by inputting visualization page generation instructions and test samples into the model to be evaluated, the output result of the model to be evaluated is obtained; based on the output result, the model to be evaluated is functionally verified to obtain a first verification result; the functional verification is used to verify whether the model to be evaluated can generate page code that meets preset requirements; if the model to be evaluated can generate page code that meets preset requirements, the generated page code is rendered to obtain a visualization page; the rendered visualization page is subjected to visualization component quality verification to obtain a second verification result; the visualization component quality verification is used to verify the rendering quality of visualization components in the visualization page; the rendered visualization page is subjected to content quality verification to obtain a third verification result; the content quality verification is used to verify the content quality of semantic content in the visualization page; and then the model to be evaluated can be evaluated based on the first verification result, the second verification result, and the third verification result. The embodiments in this specification perform functional verification on the model to be evaluated based on the output results. This can verify the page code generation capability of the model to be evaluated, the renderability of the page code output by the model to be evaluated can be verified through visualization component quality verification, and the semantic content expression capability of the model to be evaluated can be verified through content quality verification. In this way, by evaluating the model to be evaluated based on its page code generation capability, the renderability of the page code output by the model to be evaluated, and the semantic content expression capability of the model to be evaluated, the model to be evaluated can be evaluated in a multi-dimensional and comprehensive manner to improve the accuracy of model evaluation.

[0011] On the other hand, the embodiments of this specification can first perform functional verification on the model to be evaluated. When the model can generate page code that meets the preset requirements, the page code generated by the model to be evaluated is rendered to obtain a visual page. Then, the visual page is subjected to visualization component quality verification and content quality verification. This verification process can avoid invalid page rendering operations and invalid visualization component quality verification and content quality verification processes, thereby improving the efficiency of model evaluation. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a schematic diagram illustrating an application scenario of a model evaluation method provided in one embodiment of this specification. Figure 2This is a flowchart of a model evaluation method provided in one embodiment of this specification; Figure 3 This is a flowchart of a model evaluation method provided in one embodiment of this specification; Figure 4 This is a schematic diagram of the structure of a model evaluation device provided in one embodiment of this specification; Figure 5 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0014] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0015] This specification uses specific terms to describe embodiments thereof. Terms such as "an embodiment," "one embodiment," and / or "some embodiments" refer to a particular feature, structure, or characteristic associated with at least one embodiment of this specification. Therefore, it should be emphasized and noted that references to "an embodiment," "one embodiment," or "an alternative embodiment" in different locations throughout this specification do not necessarily refer to the same embodiment. Furthermore, those skilled in the art can combine and integrate the different embodiments or examples described herein, as well as the features of those different embodiments or examples, without contradiction.

[0016] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “an,” “an,” “the,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification includes any or all possible combinations of one or more associated listed items.

[0017] The terms “comprising,” “including,” or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, product, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, product, or apparatus. Without further limitation, the presence of additional identical or equivalent elements in the process, method, product, or apparatus that includes said elements is not excluded.

[0018] Although the terms "first," "second," etc., may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, "first" may also be referred to as "second," and similarly, "second" may also be referred to as "first," without departing from the scope of one or more embodiments of this specification. Ordinal numbers such as "first," "second," etc., do not necessarily indicate order; often they are used to facilitate the distinction of objects. For example, "first server" and "second server" usually refer to two servers. To distinguish these two servers, they are described as "first server" and "second server." Of course, sometimes these two servers may be the same server.

[0019] Depending on the context, the word "if" as used here can be interpreted as "when," "when," or "in response to determination."

[0020] In this specification, unless explicitly stated otherwise, "receiving and sending data" does not necessarily mean direct receiving and sending; it can also mean indirect receiving and sending. For example, A receiving data sent by B can be understood as A directly receiving the data sent by B, or it can be understood as A indirectly receiving the data sent by B through other entities such as C. Similarly, B sending data to A can be understood as B sending the data directly to A, or it can be understood as B indirectly sending the data to A through other entities such as C. Here, C can be one entity, or it can be two or more entities.

[0021] In this specification, unless explicitly stated otherwise, the relationships between structures can be direct or indirect. For example, when describing "A is connected to B," unless it is explicitly stated that A and B are directly connected, it should be understood that A can be directly connected to B or indirectly connected to B. Similarly, when describing "A is on top of B," unless it is explicitly stated that A is directly above B (AB is adjacent and A is above B), it should be understood that A can be directly above B or indirectly above B (AB is separated by other elements, and A is above B). And so on.

[0022] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0023] The following is an explanation of the terminology used in the embodiments of this specification.

[0024] With the rapid popularization of large language models, user needs have evolved from "providing text answers" to "presenting statistical charts, tables, and explanatory text in the answers simultaneously" in order to conduct visual analysis and decision-making in data-intensive scenarios such as meteorology, finance, and public health.

[0025] Against this backdrop, models capable of understanding and generating multiple types of information, including text, images, and code, have emerged. These models can generate relevant page code based on user instructions, which can then be rendered into a visual page containing images, tables, and other visual components for display to the user.

[0026] In practical applications, to ensure the quality of model output results, the model is usually evaluated. Related technologies typically evaluate whether the model's page code is realistic, accurate, and free of logical errors. However, in real-world applications, what is displayed to users is a visual page, and evaluating the accuracy of the page code does not reflect the quality of the visual page.

[0027] To address the shortcomings of related technologies, this solution provides the following embodiments: Figure 1 This is a schematic diagram illustrating an application scenario of a model evaluation method provided in one embodiment of this specification. For example... Figure 1 As shown, this application scenario includes test sample 1, visualization page generation instructions 2, model to be evaluated 3, and server 4.

[0028] In this application scenario, it is necessary to evaluate the ability of the evaluation model 3 to generate a visualization page. The evaluation model 3 can be a large language model, such as a multimodal large model. Test sample 1 can refer to the reference data used by the evaluation model 3 to generate the visualization page. Visualization page generation instruction 2 can be used to instruct the evaluation model 3 to output a visualization page for test sample 1.

[0029] Server 4 can be a standalone physical server, a server cluster consisting of multiple physical servers, or a distributed file system. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0030] In one embodiment of this specification, test sample 1 and visualization page generation instruction 2 can be input into the model to be evaluated 3 to obtain the output result of the model to be evaluated 3. The server 4 can then perform functional verification on the model to be evaluated 3 based on the output result, obtaining a first verification result. This functional verification verifies whether the model to be evaluated can generate page code that meets preset requirements. If the first verification result indicates that the model to be evaluated can generate page code that meets preset requirements, the server 4 can render the generated page code to obtain a visualization page. The server 4 can then perform visualization component quality verification on the rendered visualization page to obtain a second verification result. This visualization component quality verification verifies the rendering quality of the visualization components in the visualization page. Finally, the server 4 can perform content quality verification on the rendered visualization page to obtain a third verification result. This content quality verification verifies the content quality of the semantic content in the visualization page. Afterwards, the server 4 can evaluate the model to be evaluated 3 based on the first verification result, the second verification result, and the third verification result.

[0031] The evaluation method for the model provided in the embodiments of this specification will be described below with reference to the accompanying drawings.

[0032] Figure 2 This is a flowchart illustrating a model evaluation method provided in one embodiment of this specification. From a programming perspective, the executor of the process can be a program hosted on a terminal server. From a hardware perspective, the executor of the process can be any device, equipment, platform, or cluster of devices with computing and processing capabilities, or a model evaluation platform. This evaluation method is used to evaluate the ability of a model to generate page code for visualizing pages. Figure 2 As shown, the method may include the following steps.

[0033] Step 202: Input the visualization page generation instruction and the test sample into the model to be evaluated to obtain the output result of the model to be evaluated; the visualization page generation instruction is used to instruct the model to be evaluated to generate a visualization page for the test sample; the visualization page is a page containing visualization components.

[0034] In the embodiments of this specification, the test sample may refer to the reference data on which the model under evaluation generates the visualization page. Here, the reference data refers to the original data on which the model under evaluation generates the visualization page, which provides factual basis for the content of the visualization page. Optionally, the visualization page generation instruction can be used to instruct the model under evaluation to generate a visualization page for the test sample. For example, the test sample may be the revenue data of a merchant over the past year, and the visualization page generation instruction may be an instruction instructing the model to display the revenue growth of the merchant in each month over the past year using a visualization chart.

[0035] Optionally, the model to be evaluated can be a model used to generate page code, which is then rendered into a visual page and displayed to the user. As a specific implementation, the model to be evaluated can be a large language model, such as a multimodal large model. A multimodal large model is an artificial intelligence model that integrates data from multiple modalities, capable of understanding and processing data from different modalities such as text, images, and audio, and outputting the corresponding processing results in natural language.

[0036] In the embodiments of this specification, the visualization page is a page containing visualization components, which may include chart components. Chart components may be graphic components used for data visualization, such as bar charts, scatter plots, radar charts, area charts, funnel charts, dashboards, and other various data statistical graphic components used for data visualization, or they may be composite graphic components formed by combining the above charts.

[0037] As one specific implementation, the visualization page generation instruction may also include visualization requirements for the page. For example, visualization requirements may include the types of visualization components in the generated visualization page, the semantic content of the visualization page, etc.

[0038] In the embodiments described in this specification, the output of the model to be evaluated may include page code, such as HTML code. Optionally, the output of the model to be evaluated may also be without page code.

[0039] Step 204: Perform functional verification on the model to be evaluated based on the output results to obtain a first verification result; the functional verification is used to verify whether the model to be evaluated can generate page code that meets the preset requirements.

[0040] In one specific implementation, functional verification is used to check whether the model under test can generate page code that meets preset requirements. For example, if the output of the model under test does not contain page code, it can be determined that the model under test does not have the ability to generate page code, and thus cannot generate page code that meets preset requirements. If the output of the model under test contains page code, it can be determined that the model under test has the ability to generate page code, and then it can be verified whether the page code generated by the model under test meets preset requirements.

[0041] Optionally, you can verify whether the page code generated by the model under evaluation meets the preset requirements. This could include verifying the structural integrity, renderability, and so on of the page code generated by the model under evaluation.

[0042] As one specific implementation method, a functional evaluation model can be used to perform functional verification on the model to be evaluated. For example, prompts can be constructed based on the output results. These prompts can include the output results and task instruction information instructing the functional evaluation model to verify whether the output results meet preset requirements. The constructed prompts are then input into the functional evaluation model to obtain a first verification result. The functional evaluation model can be a pre-trained model specifically designed for functional verification. Optionally, the functional verification of the model to be evaluated can also be performed manually, such as by programmers or code developers, based on the output results.

[0043] In the embodiments of this specification, the first verification result can reflect the ability of the model under test to generate page code. Optionally, the first verification result can be a result indicating that the model under test cannot generate page code that meets the preset requirements, or it can be a result indicating that the model under test can generate page code that meets the preset requirements.

[0044] Step 206: If the first verification result indicates that the model to be evaluated can generate page code that meets the preset requirements, then the generated page code is rendered to obtain the visualization page.

[0045] In the embodiments of this specification, if the model to be evaluated can generate page code that meets preset requirements, the page code generated by the model to be evaluated can be rendered to obtain a visual page. Optionally, a rendering engine, template engine, etc., can be used to render the page code to obtain a visual page.

[0046] Optionally, if the model under test cannot generate page code that meets the preset requirements, page rendering can be skipped, and the subsequent verification process can end. In this case, the evaluation result of the model under test may be that it cannot generate page code that meets the preset requirements, indicating poor model performance.

[0047] Step 208: Perform visualization component quality verification on the rendered visualization page to obtain a second verification result; the visualization component quality verification is used to verify the rendering quality of the visualization components in the visualization page.

[0048] In the embodiments of this specification, the visualization component quality verification is used to verify the rendering quality of visualization components in the visualization page. For example, it can verify whether the visualization component is missing component elements, whether the size of the visualization component is appropriate, and whether the color contrast of the visualization component is appropriate, etc.

[0049] Optionally, a rendering quality assessment model can be used to verify the quality of visual components on the visualization page. For example, prompts can be constructed, which may include the visualization page and task instructions for the rendering quality assessment model to verify the quality of visual components on the page. These prompts are then input into the rendering quality assessment model to obtain a second verification result. The rendering quality assessment model can be a pre-trained model specifically designed for rendering quality verification. Alternatively, the quality of visual components can also be verified manually.

[0050] In the embodiments of this specification, the second verification result can reflect the rendering quality of the visualization component. For example, the second verification result can indicate that the rendering quality of the visualization component is good, poor, or moderate. As a specific implementation, for example, if the model passes the component element missing verification, visualization component size verification, and visualization component color contrast verification, the second verification result can indicate that the rendering quality of the visualization component is good; if the model fails all of the above verifications, the second verification result can indicate that the rendering quality of the visualization component is poor; if the model passes only one or two of the above verifications, the second verification result can indicate that the rendering quality of the visualization component is moderate.

[0051] Alternatively, the second verification result could indicate that the visualization component quality verification has passed, or it could indicate that the visualization component quality verification has failed.

[0052] Step 210: Perform content quality verification on the rendered visualization page to obtain a third verification result; the content quality verification is used to verify the content quality of the semantic content in the visualization page.

[0053] In the embodiments described in this specification, content quality verification can be performed on the visualization page. For example, it can verify the semantic rationality of the semantic content in the visualization page, and verify the consistency between the semantic content in the visualization page and the content of the test sample, etc.

[0054] Optionally, a content quality assessment model can be used to verify the content quality of the visualized page. For example, prompts can be constructed, which may include the visualized page and task instructions for the content quality assessment model to verify its content quality. The constructed prompts are then input into the content quality assessment model to obtain a third verification result. The content quality assessment model can be a pre-trained model specifically designed for content quality verification.

[0055] Optionally, a rules engine can be used to verify the content quality of the visualization page. Alternatively, the content quality of the visualization page can be verified manually.

[0056] In the embodiments of this specification, the third verification result can reflect the quality of the semantic content of the visualization page. For example, the third verification result can indicate good content quality, poor content quality, or moderate content quality. Furthermore, if all models pass the above semantic rationality and consistency verification, the third verification result can indicate good content quality; if all models fail the above semantic rationality and consistency verification, the third verification result can indicate poor content quality; if all models pass only one of the above semantic rationality and consistency verification, the third verification result can indicate moderate content quality.

[0057] Alternatively, the third verification result could indicate that the content quality verification passed, or it could indicate that the content quality verification failed.

[0058] In related technologies, model evaluation typically focuses on whether the output page code is realistic, accurate, and free of logical errors. However, in practical applications, users are presented with a visual page, and the accuracy of the output page code may not reflect the quality of the displayed visual page. In this embodiment, the visual page obtained by rendering the output page code of the evaluation model can be validated. Specifically, this validates the rendering quality of the visual components and the content quality of the semantic content within the visual page. This allows for a comprehensive evaluation of the model from the user's actual perception perspective, improving the alignment of the evaluation results with real-world application scenarios and ultimately enhancing the accuracy of the evaluation.

[0059] Step 212: Evaluate the model to be evaluated based on the first verification result, the second verification result, and the third verification result.

[0060] In the embodiments of this specification, a score can be determined for each verification result. For example, for functional verification, if the verification result indicates that the model under test cannot generate page code that meets the preset requirements, a lower score can be determined; if the functional verification result indicates that the model under test can generate page code that meets the preset requirements, a higher score can be determined. Similarly, for visualization component quality verification, a corresponding score can be determined based on the rendering quality of the visualization component, and this score can be positively correlated with the rendering quality. For content quality verification, a corresponding score can also be determined based on the content quality, and this score can be positively correlated with the content quality.

[0061] Then, the scores of each verification result are used to evaluate the model to be evaluated. For example, the scores of each verification result can be weighted and summed to obtain the evaluation score of the model to be evaluated.

[0062] In the embodiments of this specification, based on the output results of the model to be evaluated, functional verification, visualization component quality verification, and semantic content quality verification can be performed on the model to be evaluated. Through these verifications, the model's page code generation capability, the renderability of the page code output by the model, and the model's semantic content expression capability can be evaluated. This allows for a multi-dimensional and comprehensive evaluation of the model's actual capabilities in visualization page generation tasks, improving the fit between the evaluation results and actual application scenarios, thereby enhancing the accuracy of the evaluation.

[0063] In the embodiments described in this specification, the model to be evaluated can first undergo functional verification. When the verification results show that the model can generate page code that meets the preset requirements, the page code is then rendered to obtain a visual page, and further verification of the quality of the visual components and content is performed. This avoids subsequent invalid page rendering operations and invalid verification processes of visual components and content quality when the model to be evaluated lacks the basic ability to generate page code that meets the preset requirements, thereby improving the efficiency of model evaluation.

[0064] based on Figure 2 In addition to the method described herein, this specification also provides some implementation methods of the method, which will be described below.

[0065] In the embodiments of this specification, the functional verification of the model to be evaluated based on the output result includes at least one of the following verification methods: The responsiveness of the model under test is verified based on the output results; the responsiveness verification is used to verify whether the model under test can generate page code.

[0066] The instruction compliance verification is performed on the model to be evaluated based on the output results; the instruction compliance verification is used to verify whether the output results contain dialogue text other than page code.

[0067] The structural integrity of the model to be evaluated is verified based on the output results; the structural integrity verification is used to verify the structural integrity of the page code contained in the output results.

[0068] The model to be evaluated is subjected to sandbox rendering verification based on the output results; the sandbox rendering verification is used to verify whether the output results can be rendered into a visual page under a preset viewport.

[0069] Optionally, a responsiveness check can be performed on the model to be evaluated based on the output results to verify whether the model can generate page code. For example, it can be determined whether the output results contain page code. As a specific implementation method, a large language model or manual verification can be used to determine whether the output results contain page code. Then, if the responsiveness check result indicates that the model to be evaluated can generate page code, subsequent instruction compliance checks, structural integrity checks, and sandbox rendering checks can be performed. If the responsiveness check result indicates that the model to be evaluated cannot generate page code, subsequent instruction compliance checks, structural integrity checks, and sandbox rendering checks can be omitted.

[0070] Optionally, to improve the accuracy of responsiveness validation, multiple rounds of testing can be performed. For example, inputting prompts containing visual page generation instructions into the model under evaluation multiple times will yield multiple outputs. Each time the prompts are input, one output will be obtained. If none of these multiple outputs contain page code, it can be determined that the model under evaluation cannot generate page code; otherwise, it can be determined that the model can generate page code.

[0071] In related technologies, the server needs to render the model's output to obtain a visual page. If the output of the model to be evaluated contains dialogue text outside the page code, such as "Okay, according to your requirements, the following page code has been generated," then the server may encounter rendering errors when rendering the output using a rendering engine. For example, the rendering engine may be unable to render "Okay, according to your requirements, the following page code has been generated," resulting in the failure to render the visual page. In the embodiments of this specification, instruction compliance verification can be performed on the model to be evaluated to verify whether the output contains dialogue text outside the page code.

[0072] One specific implementation method is to use rule matching to verify whether the output contains dialogue text outside the page code. For example, the page code in the output can be extracted first using features such as HTML tags, and then a preset dialogue text can be matched using string matching or regular expressions to determine whether the output contains dialogue text outside the page code. If the preset dialogue text is matched in the output, it means that the output contains dialogue text outside the page code; otherwise, the output does not contain dialogue text outside the page code.

[0073] As a specific implementation method, a large language model can also be used to verify whether the output contains dialogue text outside of the page code. For example, the large language model can perform semantic judgment on the output to verify whether the output contains dialogue text outside of the page code.

[0074] In the embodiments of this specification, the structural integrity of the page code contained in the output result can also be verified, thereby ensuring the page quality of the subsequently rendered visualization page. As a specific implementation, code inspection tools such as lxml parsing libraries can be used to verify the structural integrity of the page code. Optionally, verifying the structural integrity of the page code can specifically include verifying whether code tags appear in pairs, whether code tags are closed, whether code tags have nesting errors, and whether the page code contains at least one of the following: In the embodiments of this specification, the performance of the model in a real web application environment can also be evaluated. Specifically, the model to be evaluated can be sandboxed and rendered to verify whether the output result can be rendered into a visual page under a preset viewport. Optionally, the Chromium kernel or Playwright automation tools can be used to render the page code output by the model under a preset viewport width to verify whether the output result can be rendered into a visual page under the preset viewport. Optionally, the preset viewport width can be 800px, 1024px, 1280px, 1440px, 1920px, etc. Optionally, the preset viewport width can be the viewport width of the display device used to display the visual page. For example, if the viewport width of the display device used to display the visual page is 1280px, then the preset viewport width can be 1280px; if the viewport width of the display device is 1440px, then the preset viewport width can be 1440px, and so on.

[0075] In the embodiments of this specification, the visualization component quality verification of the rendered visualization page includes at least one of the following verification methods: Verify whether the size of the visual components in the visualization page is within the preset size range.

[0076] Verify whether the visualization page is missing any component elements.

[0077] Verify whether the visualization page has any occlusion defects.

[0078] Verify whether the visualization page contains overflowing content.

[0079] Verify whether the visualization page has insufficient color contrast.

[0080] In the embodiments of this specification, the size of the visual component in the visualization page can be verified based on preset rules to determine whether it is within a preset size range. Optionally, the preset size can be the size of the visual component specified in the page code. As a specific implementation, the Playwright tool can be used to measure the size of the visual component, and then the measured size can be compared with the specified size of the visual component. If the measured size does not match the specified size of the visual component, it can be considered that the size of the visual component is not within the preset size range. As a specific implementation, the Playwright tool can also be used to measure the height of the visual component. If the measured height is greater than a certain multiple of the viewport height, such as greater than twice the viewport height, it can also be considered that the size of the visual component is not within the preset size range.

[0081] In this embodiment of the specification, the step of verifying whether the visualization page is missing component elements may specifically include: A first prompt word is constructed based on the visualization page and the page code; the first prompt word contains task instruction information for instructing the rendering quality assessment model to use the page code to verify whether the visualization page is missing component elements.

[0082] Input the first prompt word into the rendering quality assessment model to obtain the verification result output by the rendering quality assessment model.

[0083] The determination of whether the visualization page is missing component elements is based on the verification results output by the rendering quality assessment model.

[0084] In the embodiments of this specification, a rendering quality assessment model can be used to verify whether a visualization page is missing component elements. Specifically, prompts can be constructed, which may include page code and the visualization page, and may also include task instruction information instructing the rendering quality assessment model to use the page code to verify whether the visualization page is missing component elements. For example, a specific task instruction message could be "Please determine whether this visualization page is missing component elements based on this page code." Optionally, component elements can be visual interactive elements or display elements such as axis labels, data legends, numerical labels, filter buttons, pagination controls, download / export buttons, and time range selectors.

[0085] In one specific implementation, the prompt may also include a reference example, which can be used to assist the rendering quality assessment model in verifying whether the visualization page is missing component elements. Specifically, the reference example may include a reference visualization page, reference page code, and a reference answer indicating whether the reference visualization page is missing component elements.

[0086] As one specific implementation, the verification result output by the rendering quality assessment model can be either a result indicating that the visualization page has missing component elements or a result indicating that the visualization page does not have missing component elements.

[0087] In the embodiments of this specification, the step of verifying whether the visualization page has occlusion defects may specifically include: The visualization page is segmented based on the title tags in the visualization page to obtain various page modules.

[0088] A second prompt word is constructed based on each page module; the second prompt word contains task indication information used to instruct the rendering quality assessment model to verify whether each page module has occlusion defects.

[0089] Input the second prompt word into the rendering quality assessment model to obtain the verification result output by the rendering quality assessment model.

[0090] The presence of occlusion defects in the visualization page is determined based on the verification results output by the rendering quality assessment model.

[0091] In the embodiments described in this specification, occlusion defect refers to the situation where there is occlusion between two component elements in the visualization page.

[0092] As one specific implementation, a heading tag can refer to a tag in the page code used to identify the page's structural hierarchy. For example, in HTML code... <h1> Label,< / h1> <h2> Tags and< / h2> <h3>Tags, etc. Title tags can be used to delineate the module boundaries of page modules within a visualization page. Optionally, a page module can refer to a page unit within the visualization page, such as a title module, chart module, table module, text module, etc. Optionally, a page module can be a module containing component elements. As a specific implementation, the visualization page can be segmented based on title tags to obtain various page modules, and each segmented page module can contain a title tag.

[0093] In the embodiments of this specification, a second prompt word can also be constructed to instruct the rendering quality assessment model to verify whether each page module has occlusion defects. This second prompt word may include information about each page module and the task instruction information for the rendering quality assessment model to verify whether the page module has occlusion defects. The second prompt word is then input into the rendering quality assessment model to obtain the verification result.

[0094] Optionally, the verification result output by the rendering quality assessment model can be either a result indicating that a page module has an occlusion defect, or a result indicating that a page module does not have an occlusion defect. If the verification result indicates that a page module has an occlusion defect, then it can be determined that the visualization page has an occlusion defect. If the verification result indicates that a page module does not have an occlusion defect, then it can be determined that the visualization page does not have an occlusion defect.

[0095] In practical applications, occlusion defects on a visual page may exist in localized areas, such as overlapping component elements within the same module or between adjacent modules. Directly verifying occlusion defects across the entire visual page makes it difficult to accurately identify these localized occlusion problems. The embodiments in this specification divide the visual page into individual page modules, allowing for verification of each module, thus reducing the difficulty of verification and improving accuracy.

[0096] As one specific implementation, verifying whether the visualization page has content overflow may specifically include: Identify the module boundaries of each page module in the visualization page.

[0097] For any page module, calculate whether the outer edge of the component element in the page module exceeds the module boundary of the page module.

[0098] If the outer edge of a component element in any page module exceeds the module boundary of any page module, then it is determined that the visualization page has content overflow.

[0099] In the embodiments of this specification, the visual page can be segmented according to the title tags in the visual page to obtain various page modules, and then the module boundaries of each page module can be identified.

[0100] Optionally, the module boundaries of a page module can be determined through DOM parsing, specifically by determining the coordinates of the module boundaries. Alternatively, the outer edges of each component element within a page module can also be determined through DOM parsing.

[0101] Furthermore, the coordinates of the component element's outer edge can be compared with the coordinates of the module boundary to calculate whether the component element's outer edge exceeds the module boundary. If the comparison result indicates that the component element's outer edge is outside the module boundary, it can be determined that the visualization page has a content overflow problem; otherwise, it can be determined that the visualization page does not have a content overflow problem.

[0102] In practical applications, some specific component elements exceeding the module boundary may not affect the user's visual experience. For example, invisible component elements, decorative elements, and scrollable containers exceeding the module boundary will not affect the user's visual experience. Optionally, invisible component elements refer to elements that are not displayed, such as advertisements that are not displayed. Decorative elements refer to elements that serve a decorative effect on the visual page, such as background patterns. Scrollable container components refer to container components that hold visual content that exceeds the current visible range and can be scrolled or dragged to display this visual content. For example, text boxes or data lists containing a large amount of visual content. Users can scroll or drag scrollable containers to display the content in these container components. In the embodiments of this specification, when verifying whether there is content overflow on the visual page, it is not necessary to verify these specific elements. Optionally, calculating whether the outer edge of a component element in any page module exceeds the module boundary of any page module may specifically include: calculating whether the outer edge of component elements other than specific component elements in any page module exceeds the module boundary of any page module; the specific component elements include at least one of invisible component elements, decorative elements, and scrollable containers.

[0103] In practical applications, insufficient color contrast on a visualization page can cause reading difficulties for users and negatively impact user experience. This specification's embodiments can verify whether a visualization page has insufficient color contrast, thereby evaluating the model to be evaluated. As a specific implementation, verifying whether the visualization page has insufficient color contrast may specifically include: Determine the background color brightness of the component elements of the visualization components in the visualization page.

[0104] Determine the foreground color brightness of the component element.

[0105] Calculate the contrast between the foreground color brightness and the background color brightness.

[0106] If the contrast ratio is less than or equal to a preset contrast threshold, then the color contrast of the visualized page is determined to be insufficient.

[0107] In the embodiments described in this specification, the background color brightness of a component element can be determined by reading the background color brightness parameter of the component element from the application programming interface of the visualization library of the rendered component element.

[0108] As one specific implementation, the background brightness of a component element can be obtained by recursively calculating the background transparency of its parent node. Optionally, determining the background brightness of a visual component in the visualization page may specifically include: For any of the visualization components, traverse the parent nodes of the component elements of the visualization component level by level to obtain the background color and transparency of each parent node; According to the preset color mixing rules, the background color and transparency of each parent node are synthesized layer by layer to obtain the effective background color of the component element of the visualization component; The background brightness of the visualization component is calculated based on the effective background color.

[0109] In the embodiments of this specification, a parent node refers to the upper-level container node that directly or indirectly carries the current component element. For example, for the component element of a filter button, the parent node of this component element could be the operation bar that wraps the filter button. The parent node of the operation bar could be a chart, and the parent node of the chart could be the page background. Optionally, the background color of the parent node refers to the base color displayed when the node is rendered. The background color of the parent node can be represented by RGB color values, HSL color values, etc. The transparency of the parent node is used to represent the degree of transparency of the parent node. Optionally, the transparency of the parent node can be represented by an alpha value. This alpha value takes a value between 0 and 1, where 0 represents complete transparency and 1 represents complete opacity.

[0110] Optionally, during the traversal of parent nodes, the traversal operation can be stopped if the root node or a completely opaque parent node is reached.

[0111] Optionally, the preset color blending rule refers to the rule used to synthesize multiple layers of transparent background colors into a single equivalent solid color. For example, the preset color blending rule can be implemented based on the alpha synthesis algorithm in computer graphics. As a specific implementation, the background color and transparency of each parent node are synthesized layer by layer, which can be done from the bottom layer to the top layer or from the top layer to the bottom layer. Specifically, the RGB color value and alpha value of each parent node can be mixed and calculated channel by channel to obtain the synthesis result of that parent node. As a specific implementation, if the parent node is a gradient color, the color stopping points of the parent node can be taken to obtain the weighted average of the synthesis result of the parent node. Further, the synthesis result of the previous layer can be used as the calculation input of the next layer, and finally, multiple layers of transparent background colors are synthesized into a single equivalent solid color to obtain the effective background color of the component element of the visualization component. Optionally, the effective background color refers to the actual equivalent background color applied to the corresponding component element.

[0112] As one specific implementation, the effective background color can be converted to a color space and mapped to a linear light space to linearize the color channel values ​​of the effective background color. Then, the color channel values ​​can be weighted and summed based on preset weights to finally convert the effective background color into a normalized brightness value, thus obtaining the background brightness of the visualization component.

[0113] In the embodiments described in this specification, the foreground color brightness parameter of the component element can be read from the application programming interface of the visualization library of the rendered component element to determine the foreground color brightness of the component element. Alternatively, it can be determined whether the component element has transparency. If the component element does not have transparency, its RGB color value can be used as a single equivalent solid color. This single equivalent solid color can be converted to a color space, and then the weighted values ​​of each channel after conversion can be calculated to obtain the foreground color brightness of the component element. If the component element has transparency, its RGB color value and alpha value can be mixed and calculated channel by channel to obtain a single equivalent solid color. This single equivalent solid color can then be converted to a color space, and then the weighted values ​​of each channel after color space conversion can be calculated to obtain the foreground color brightness of the component element.

[0114] One implementation method for calculating the contrast ratio between foreground and background color brightness is to first determine the larger of the two values, and then use the ratio of the larger to the smaller value as the contrast ratio. For example, it can be calculated using the formula: L = L1 / L2, where L1 is the larger of the foreground and background color brightness, and L2 is the smaller. Alternatively, the contrast ratio can be calculated based on a correction factor. Specifically, it can be calculated using the formula: L = (L1 + K) / (L2 + K), where K is the correction factor, which can be 0.05, 0.1, etc.

[0115] In one specific implementation, if the calculated contrast ratio is less than or equal to a preset contrast ratio threshold, it can be determined that the color contrast of the visualized page is insufficient. Optionally, the preset contrast ratio threshold can be determined based on expert experience, such as 1.05, 1.5, etc. Alternatively, the preset contrast ratio threshold can be determined based on user characteristics, where user characteristics can be the user's visual characteristics. Alternatively, the preset contrast ratio threshold can be determined based on the display characteristics of the display device of the visualized page, where display characteristics can be display brightness, display contrast, etc.

[0116] In the embodiments of this specification, by combining rule-based algorithms with modular discrimination, the model illusion problem when using rendering quality evaluation models for evaluation can be effectively reduced, ensuring the objectivity and accuracy of occlusion defect verification and color contrast verification, and improving the reliability of visualization component quality verification.

[0117] In this embodiment of the specification, the content quality verification includes at least one of the following: reasonableness verification, completeness verification, and fidelity verification of the semantic content in the visualization page. The reasonableness verification verifies the semantic reasonableness of the semantic content; the completeness verification verifies the completeness of the semantic content containing preset text information; and the fidelity verification verifies the consistency between the semantic content and the test sample. Optionally, a large language model can be used to perform reasonableness verification, completeness verification, and fidelity verification on the model to be evaluated.

[0118] As a specific implementation method, the semantic content in the visualization page is subjected to reasonableness verification, which may specifically include: Verify that the semantic content does not contain at least one of the following: factual errors, inconsistent context, unreasonable numerical values, inconsistent units, missing necessary elements, or incorrect chart type.

[0119] If the semantic content contains at least one of the following: factual error, inconsistency in context, unreasonable numerical value, inconsistent units, missing necessary elements, or incorrect chart type, then the semantic content in the visualization page is determined to have failed the rationality check.

[0120] In the embodiments described in this specification, the semantic content in the visualization page refers to the content represented by the visualization components and text in the visualization page.

[0121] Optionally, factual errors refer to semantic content that contradicts objective facts or common sense. Inconsistent context refers to semantic content that is contradictory or logically conflicting. Unreasonable numerical values ​​refer to numerical values ​​in the semantic content that exceed a reasonable range. Inconsistent units refer to the use of different units for the same indicator. Missing necessary elements refer to visualization components lacking essential elements, such as legends or axes. Inappropriate chart type refers to a mismatch between the chart type and the data characteristics of the test sample targeted by the visualization page's generation instructions. For example, a line chart is required, but a bar chart is displayed on the visualization page. Furthermore, if any of the above situations exist, the semantic content in the visualization page can be considered to have failed the reasonableness check.

[0122] In the embodiments of this specification, before performing completeness verification on the semantic content of the visualized page, the following may also be included: Generate preset text information; the preset text information is the information that needs to be included in the output result of the model to be evaluated.

[0123] Performing a completeness check on the semantic content of the visualization page may specifically include: Calculate the proportion of the preset text information that is covered by the semantic content.

[0124] If the ratio is less than or equal to a preset ratio threshold, then the semantic content in the visualization page is determined to have failed the completeness check.

[0125] Optionally, the preset text information refers to the key points of the answer that the user has pre-defined. These key points can be specific to the current assessment task. Alternatively, they can be specific to the instructions for generating the visualization page.

[0126] As one specific implementation method, key information can be extracted from the semantic content of the visualization page. Then, the proportion of the extracted key information within the preset text information is calculated to obtain the percentage of the preset text information covered by the semantic content. If this percentage is less than or equal to a preset percentage threshold, it indicates that the model output does not meet the generation requirements of the visualization page generation instructions, and the model fails the completeness check.

[0127] In the embodiments of this specification, the fidelity verification of the semantic content in the visualization page may specifically include: The semantic content is broken down into multiple semantic entries. Each semantic entry is the smallest textual information unit in the semantic content that can independently express a complete semantic meaning.

[0128] The semantic entries are compared with the test sample to determine whether each semantic entry contradicts the test sample.

[0129] If the semantic entry contradicts the test sample, then the semantic content in the visualization page has failed the fidelity check.

[0130] In the embodiments of this specification, a semantic entry can refer to the smallest textual information unit in semantic content that can independently express a complete semantic meaning. This smallest textual information unit can be the smallest factual claim that can be independently verified. For example, a specific smallest textual information unit can be "the average temperature over the past week was 25°C" or "sales revenue increased by 5% compared to last month," etc.

[0131] Optionally, a large language model can be used to decompose the semantic content to obtain the individual semantic entries within the semantic content.

[0132] As one specific implementation method, multiple semantic entries can be compared with the test sample individually. For example, if a semantic entry is "marketing revenue increased by 5% compared to last month," but the test sample shows "marketing revenue decreased by 5% compared to last month," then it can be determined that there is a contradiction between the semantic entry and the test sample. Optionally, if any semantic entry contradicts the test sample, it can be determined that the semantic content in the visualization page has failed the fidelity check.

[0133] In the embodiments of this specification, by decomposing semantic content into semantic entries and then performing fidelity verification, it is possible to locate problems in a fine-grained manner, while ensuring the overall credibility of the evaluation and improving the reliability and interpretability of the model evaluation.

[0134] In the embodiments of this specification, the evaluation of the model to be evaluated can be performed by combining the first verification result, the second verification result, and the third verification result. Optionally, the evaluation of the model to be evaluated based on the first verification result, the second verification result, and the third verification result may specifically include: The functional score of the model to be evaluated is determined based on the first verification result.

[0135] The rendering quality score of the model to be evaluated is determined based on the second verification result.

[0136] The content quality score of the model to be evaluated is determined based on the third verification result.

[0137] The evaluation score of the model to be evaluated is determined based on the functionality score, the rendering quality score, and the content quality score.

[0138] As one specific implementation, the functional score of the model under evaluation can be determined based on the responsiveness verification result in the first verification result. For example, if the responsiveness verification result indicates that the model under evaluation cannot generate page code, the functional score can be 0. As another specific implementation, if the model under evaluation cannot generate page code and the resulting functional score is 0, the evaluation process can end, and the rendering quality score and content quality score will not be determined. If the responsiveness verification result indicates that the model under evaluation can generate page code, the functional score can be a score greater than 0, such as 1 or 2. Furthermore, when the model under evaluation can generate page code, the functional score can be determined based on the instruction compliance verification result, structural integrity verification result, and sandbox rendering verification result. For example, the functional score can be determined based on the number of corresponding verifications passed by the model among these three results. The functional score of the model under evaluation is positively correlated with the number of corresponding verifications passed. For example, if the model under evaluation passes the instruction compliance check but fails the structural integrity check and the sandbox rendering check, its functional score could be 1 point. If the model under evaluation passes the instruction compliance check and the structural integrity check but fails the sandbox rendering check, its functional score could be 2 points, and so on.

[0139] As one specific implementation, the rendering quality score of the model under evaluation can be determined based on the number of visualization component quality checks passed by the model. Optionally, the rendering quality score of the model under evaluation can be positively correlated with the number of visualization component quality checks passed. For example, if the model under evaluation only passes the size check during the visualization component quality check process, the rendering quality score can be 1 point, where size check refers to the check method of verifying whether the size of the visualization components in the visualization page is within a preset size range. If the model under evaluation only passes the size check and the color contrast check, the rendering quality score can be 2 points, and so on, where color contrast check refers to the check method of verifying whether the color contrast of the visualization page is insufficient.

[0140] As one specific implementation method, the content quality score of the model to be evaluated can be determined based on the number of content quality checks passed. Optionally, the content quality score of the model to be evaluated can be positively correlated with the number of content quality checks passed. For example, if the model only passes the reasonableness check but fails the completeness and fidelity checks, the content quality score can be 1 point. If the model only passes the reasonableness and completeness checks but fails the fidelity check, the content quality score can be 2 points. Furthermore, if the model only passes the reasonableness, completeness, and fidelity checks, the content quality score can be 3 points.

[0141] In the embodiments of this specification, the functional score, rendering quality score, and content quality score can be weighted and summed to obtain the evaluation score of the model to be evaluated.

[0142] In practical applications, the model under evaluation may generate simple page code to circumvent defects in both the page code and the visualization page. Here, simple page code refers to the code of a visualization page containing fewer than a preset threshold of visualization components. In the embodiments of this specification, to avoid the model generating simple page code and thus preventing it from receiving inflated evaluation scores, a complexity correction coefficient can be set to obtain the evaluation score. Optionally, the model evaluation method may further include: Calculate the complexity of the visualization components contained in the visualization page.

[0143] Determine the complexity correction coefficient corresponding to the complexity; the complexity correction coefficient is positively correlated with the complexity.

[0144] The step of determining the evaluation score of the model to be evaluated based on the functionality score, the rendering quality score, and the content quality score may specifically include: The evaluation score of the model to be evaluated is determined based on the functionality score, the rendering quality score, the content quality score, and the complexity correction coefficient; the evaluation score is positively correlated with the complexity correction coefficient.

[0145] In the embodiments of this specification, the complexity of the visualization components of the rendered visualization page can be calculated, and this complexity can be positively correlated with the final evaluation score. Optionally, the complexity of the visualization components can be determined based on the number of visualization components on the visualization page. Specifically, the complexity of the visualization components can be positively correlated with the number of visualization components.

[0146] Optionally, different levels of complexity can correspond to different complexity correction coefficients. The complexity correction coefficient can be positively correlated with the complexity. For example, if a visualization page contains two visualization components, the complexity of each component can be 2, and therefore the complexity correction coefficient can be 2. Or, if the visualization page contains four visualization components, the complexity of each component can be 4, and so on.

[0147] One specific implementation method is to determine the initial evaluation score of the model under test using functional score, rendering quality score, and content quality score. For example, the initial evaluation score can be obtained by weighted summing of the functional score, rendering quality score, and content quality score. Then, the final evaluation score of the model under test is obtained based on the initial evaluation score and the complexity correction coefficient. For instance, the final evaluation score can be the product of the initial evaluation score and the complexity correction coefficient.

[0148] In the embodiments described in this specification, the evaluation score can reflect the performance of the model under evaluation. Specifically, the performance of the model under evaluation can be positively correlated with the evaluation score.

[0149] As a specific implementation method, if the evaluation results indicate that the model performs well, such as a score greater than or equal to a certain preset threshold, the model can be deployed online for user use. If the evaluation results indicate that the model performs poorly, such as a score less than a certain preset threshold, the model can be optimized to improve its performance before deployment. Optionally, during model optimization, visual page generation instructions and test samples can be used as training samples to train the model under evaluation, optimize model parameters, and thus improve model performance. In practical applications, the model can also be trained specifically based on functionality scores, rendering quality scores, and content quality scores. For example, if the functionality score is below the threshold, training samples that improve the model's page code generation capabilities can be used to train the model. If the rendering quality score is below the threshold, training samples that improve the rendering standardization, geometric layout rationality, and color presentation rationality of visual components in the generated page code can be used to train the model. If the content quality score is below the threshold, training samples that improve the model's semantic understanding accuracy and content generation reliability can be used to train the model.

[0150] In recent years, with the rapid popularization of Retrieval-Augmented Generation (RAG) technology, user needs have evolved from simply providing textual answers to simultaneously presenting statistical charts, tables, and explanatory text within the answers, enabling visual analysis and decision-making in data-intensive scenarios such as meteorology, finance, and public health. This paradigm, which collaboratively generates interpretable text and visual elements within a unified layout, can be referred to as RAVIG (Retrieval-Augmented Visually-rich Generation) in the embodiments of this specification. Optionally, RAVIG can refer to a technique used to generate multimodal data based on Retrieval-Augmented Generation. Multimodal data can refer to a dataset composed of data combining two or more different modalities. Modality here refers to the form or type of data, such as text, charts, and tables.

[0151] In practical applications, to ensure the quality of model output results, the model is usually evaluated. Related technologies typically evaluate whether the model's page code is realistic, accurate, and free of logical errors. However, in real-world applications, users are presented with a visual page rather than the page code; therefore, evaluating the accuracy of the page code may not reflect the quality of the visual page.

[0152] In order to accurately evaluate the model, the embodiments in this specification provide a model evaluation method. Figure 3 This is a flowchart of a model evaluation method provided in one embodiment of this specification, such as... Figure 3 As shown, the evaluation method includes the following steps.

[0153] In the embodiments of this specification, a visualization page generation instruction can be determined first. This instruction can be a test instruction used to test the model to be evaluated. For example, "display the stock price performance of Company A over the past 5 years," or "display the global population percentage by continent." Optionally, the visualization page generation instruction can be generated manually. Alternatively, historical visualization page generation instructions can be obtained. Furthermore, the visualization page generation instruction and test samples can be input into the model to be evaluated for testing. The test samples can be data samples used to generate the visualization page. Test samples can include real data or noisy data.

[0154] Optionally, the model to be evaluated can be a model that processes multimodal data based on retrieval-enhanced generative techniques. Alternatively, the model to be evaluated can be a model that performs the NL2VIS task. NL2VIS (Natural Language to Visualization) refers to the task of converting natural language into visualized data.

[0155] In this embodiment, the model can be evaluated by performing functional verification, visualization component quality verification, and content quality verification. Optionally, the functional verification can be pass / fail, meaning that all verifications must pass. Specifically, if any one of the responsiveness verification, instruction compliance verification, structural integrity verification, or sandbox rendering verification fails, the model is considered to have failed the functional verification. In practical applications, if the functional verification fails, the evaluation process can be terminated, resulting in a poor performance evaluation of the model under evaluation. Further, if the model passes the functional verification, the visualization component quality can be verified. Optionally, the visualization component quality verification can be pass / fail. If any one of the oversized verification, missing element verification, occlusion verification, overflow verification, or low color contrast verification fails, the visualization component quality verification is considered to have failed. Among them, oversized verification refers to verifying whether the size of the visualization component in the visualization page is within a preset size range. Missing element verification refers to verifying whether the visualization page is missing component elements. Occlusion verification refers to verifying whether the visualization page has occlusion defects. Overflow verification refers to a method of checking whether a visualization page has content overflow. Low Color Contrast verification refers to a method of checking whether a visualization page has insufficient color contrast. Optionally, content quality verification can be scored. In practical applications, the BeautifulSoup library in Python can be used to parse the visualization page, removing the CSS code nodes for styles and the JS code nodes for human-computer interaction. This preserves the core semantic content of the visualization page. Further, the core semantic content can be verified for rationality, completeness, and fidelity. Rationality verification can be a binary judgment. If the core semantic content contains any of the following: factual errors, inconsistencies in context, unreasonable values, inconsistent units, missing necessary elements, or incorrect chart type, the model is considered to have failed the rationality verification. Specifically, rationality verification can be a binary judgment. If it fails the rationality verification, it can be scored 0 points; otherwise, it can be scored 100 points. Optionally, the completeness verification score can be determined by calculating the proportion of content in the core semantic content that includes preset text information, for example, converting this proportion into a percentage score. Furthermore, the core semantic content can be parsed to obtain multiple semantic entries. It can then be determined whether there is a logical contradiction between each semantic entry and the test sample, and the score of each semantic entry can be obtained to determine the fidelity verification score.For example, if a semantic entry has a logical contradiction with the test sample, the score of that semantic entry can be determined to be 0 points; if the semantic entry does not have a logical contradiction with the test sample, the score of that semantic entry can be determined to be 1 point.

[0156] Optionally, the first verification result can be either the model failing the functional verification or the model passing the functional verification. The second verification result can be the model's score for the visualization component quality verification. The third verification result can be the model's score for the content quality verification.

[0157] In the embodiments described in this specification, a complexity correction coefficient can also be set to obtain the final evaluation score. The calculation formula can be: .in For the final evaluation score, The weighted sum of the first, second, and third verification results is the score. This is a complexity correction factor. Optional. ,in, , , Indicates the number of visualization components. Indicate the complexity of the graph. Indicates the complexity of the table.

[0158] This specification proposes a three-stage model evaluation method: the first stage assesses responsiveness, structural integrity, and sandbox rendering feasibility to ensure the visualization page is "easy to present"; the second stage verifies the rendering quality of visualization components to ensure the visualization page is "easy to read"; and the third stage reviews the rationality, completeness, and fidelity of the content, enabling fine-grained problem identification and reflecting overall credibility to ensure the user's "reading quality." Furthermore, to avoid inflated scores caused by models "deliberately generating minimalist pages to circumvent detection," this specification introduces a complexity correction coefficient to adjust the evaluation score, improving its accuracy.

[0159] In the embodiments of this specification, the three-stage verification can ensure that the visualization page is "easy to present", "easy to read", and "reading quality" for users, overcoming the limitations of pure text verification or pure chart verification in related technologies, thereby comprehensively evaluating the model.

[0160] In addition, the evaluation methods in the embodiments of this specification can be automated and have interpretable and verifiable engineering properties, making them particularly suitable for fields with strict regulations and high requirements for visual readability and factual credibility.

[0161] The various technical features in the above embodiments can be combined arbitrarily, as long as there is no conflict or contradiction between the combinations of features. However, due to space limitations, they have not been described one by one. Therefore, the arbitrary combination of various technical features in the above embodiments is also within the scope of this specification.

[0162] Based on the same idea, the embodiments of this specification also provide a model evaluation device corresponding to the above model evaluation method.

[0163] Figure 4 This is a schematic diagram of the structure of a model evaluation device provided in one embodiment of this specification, as shown below. Figure 4 As shown, the model evaluation device may include: The input module 402 is used to input the visualization page generation instruction and the test sample into the model to be evaluated, and obtain the output result of the model to be evaluated; the visualization page generation instruction is used to instruct the model to be evaluated to generate a visualization page for the test sample; the visualization page is a page containing visualization components.

[0164] The functional verification module 404 is used to perform functional verification on the model to be evaluated based on the output result to obtain a first verification result; the functional verification is used to verify whether the model to be evaluated can generate page code that meets preset requirements; The rendering module 406 is used to render the generated page code to obtain the visualization page if the first verification result indicates that the model to be evaluated can generate page code that meets the preset requirements.

[0165] The component quality verification module 408 is used to perform visualization component quality verification on the rendered visualization page to obtain a second verification result; the visualization component quality verification is used to verify the rendering quality of the visualization components in the visualization page.

[0166] The content quality verification module 410 is used to perform content quality verification on the rendered visualization page and obtain a third verification result; the content quality verification is used to verify the content quality of the semantic content in the visualization page.

[0167] The evaluation module 412 is used to evaluate the model to be evaluated based on the first verification result, the second verification result, and the third verification result.

[0168] Optionally, the functionality verification module 404 can be used to perform at least one of the following verifications: The responsiveness of the model under test is verified based on the output results; the responsiveness verification is used to verify whether the model under test can generate page code. The instruction compliance verification is performed on the model to be evaluated based on the output results; the instruction compliance verification is used to verify whether the output results contain dialogue text other than page code; The structural integrity of the model to be evaluated is verified based on the output results; the structural integrity verification is used to verify the structural integrity of the page code contained in the output results. The model to be evaluated is subjected to sandbox rendering verification based on the output results; the sandbox rendering verification is used to verify whether the output results can be rendered into a visual page under a preset viewport.

[0169] Optionally, the component quality verification module 408 can be used to perform at least one of the following verifications: Verify whether the size of the visual components in the visualization page is within the preset size range; Verify whether the visualization page is missing any component elements; Verify that the visualization page has no occlusion defects; Verify whether the visualization page contains overflowing content; Verify whether the visualization page has insufficient color contrast.

[0170] Optionally, the step of verifying whether the visualization page is missing component elements may specifically include: A first prompt word is constructed based on the visualization page and the page code; the first prompt word contains task instruction information for instructing the large language model to use the page code to verify whether the visualization page is missing component elements; Input the first prompt word into the large language model to obtain the verification result output by the large language model; The visualization page is determined to be missing component elements based on the verification results output by the large language model.

[0171] Optionally, verifying whether the visualization page has occlusion defects may specifically include: The visualization page is segmented based on the title tags in the visualization page to obtain various page modules; A second prompt word is constructed based on each page module; the second prompt word contains task instruction information for instructing the large language model to verify whether each page module has occlusion defects; Input the second prompt word into the large language model to obtain the verification result output by the large language model; The presence of occlusion defects on the visualization page is determined based on the verification results output by the large language model.

[0172] Optionally, verifying whether the visualization page has content overflow may specifically include: Identify the module boundaries of each page module in the visualized page; For any page module, calculate whether the outer edge of the component element in the page module exceeds the module boundary of the page module. If the outer edge of a component element in any page module exceeds the module boundary of any page module, then it is determined that the visualization page has content overflow.

[0173] Optionally, verifying whether the visualization page has insufficient color contrast may specifically include: Determine the background color brightness of the component elements of the visualization components in the visualization page; Determine the foreground color brightness of the component element; Calculate the contrast between the foreground color brightness and the background color brightness; If the contrast ratio is less than or equal to a preset contrast threshold, then the color contrast of the visualized page is determined to be insufficient.

[0174] Optionally, the content quality verification includes at least one of the following: reasonableness verification, completeness verification, and fidelity verification of the semantic content in the visualization page; the reasonableness verification is used to verify the semantic reasonableness of the semantic content; the completeness verification is used to verify the completeness of the semantic content containing preset text information; and the fidelity verification is used to verify the consistency between the semantic content and the test sample.

[0175] Optionally, the semantic content in the visualization page may be validated for reasonableness, which may specifically include: Verify that the semantic content does not contain at least one of the following: factual errors, inconsistent context, unreasonable numerical values, inconsistent units, missing necessary elements, or incorrect chart type; If the semantic content contains at least one of the following: factual error, inconsistency in context, unreasonable numerical value, inconsistent units, missing necessary elements, or incorrect chart type, then the semantic content in the visualization page is determined to have failed the rationality check.

[0176] Optionally, the device may further include: An information generation module is used to generate preset text information; the preset text information is the information that needs to be included in the output of the model to be evaluated. Performing a completeness check on the semantic content of the visualization page may specifically include: Calculate the proportion of the preset text information that is covered by the semantic content; If the ratio is less than or equal to a preset ratio threshold, then the semantic content in the visualization page is determined to have failed the completeness check.

[0177] Optionally, fidelity verification can be performed on the semantic content in the visualization page, which may specifically include: The semantic content is decomposed to obtain multiple semantic entries; each semantic entry is the smallest text information unit in the semantic content that can independently express a complete semantic meaning. The semantic entries are compared with the visualization page generation instructions to determine whether each semantic entry contradicts the visualization page generation instructions. If the semantic entry contradicts the visualization page generation instruction, then the semantic content in the visualization page is determined to have failed the fidelity check.

[0178] Optionally, the evaluation module 412 can be specifically used for: The functional score of the model to be evaluated is determined based on the first verification result; The rendering quality score of the model to be evaluated is determined based on the second verification result; The content quality score of the model to be evaluated is determined based on the third verification result. The evaluation score of the model to be evaluated is determined based on the functionality score, the rendering quality score, and the content quality score.

[0179] Optionally, the device may further include: A complexity calculation module is used to calculate the complexity of the visualization components contained in the visualization page; A coefficient determination module is used to determine the complexity correction coefficient corresponding to the complexity; the complexity correction coefficient is positively correlated with the complexity; The step of determining the evaluation score of the model to be evaluated based on the functionality score, the rendering quality score, and the content quality score may specifically include: The evaluation score of the model to be evaluated is determined based on the functionality score, the rendering quality score, the content quality score, and the complexity correction coefficient; the evaluation score is positively correlated with the complexity correction coefficient.

[0180] The various technical features in the above embodiments can be combined arbitrarily, as long as there is no conflict or contradiction between the combinations of features. However, due to space limitations, they have not been described one by one. Therefore, the arbitrary combination of various technical features in the above embodiments is also within the scope of this specification.

[0181] Based on the same idea, this specification also provides devices corresponding to the above methods in its embodiments.

[0182] Figure 5 A structural block diagram of a computing device 500 provided according to an embodiment of this specification is shown.

[0183] The computing device 500 includes: Memory 510 and processor 520; The memory 510 is used to store computer programs / instructions, and the processor 520 is used to execute the computer programs / instructions, which, when executed by the processor 520, implement the steps of the above method.

[0184] Specifically, the components of the computing device 500 include, but are not limited to, a memory 510 and a processor 520. The processor 520 is connected to the memory 510 via a bus 530, and the database 550 is used to store data.

[0185] The computing device 500 also includes an access device 540, which enables the computing device 500 to communicate via one or more networks 560. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 540 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and so on.

[0186] In one embodiment of this specification, the aforementioned components of the computing device 500 and Figure 5 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 5 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this application. Those skilled in the art can add or replace other components as needed.

[0187] The computing device 500 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 500 can also be a mobile or stationary server.

[0188] The processor 520 implements the steps of the above method when executing the computer instructions.

[0189] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the evaluation method of the above model belong to the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the above method.

[0190] An embodiment of this specification also provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the steps of the above-described method.

[0191] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium and the technical solution of the evaluation method of the above model belong to the same concept. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the above method.

[0192] An embodiment of this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described method.

[0193] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product and the technical solution of the evaluation method of the above model belong to the same concept. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solution of the above method.

[0194] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for the apparatus and device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiments. The apparatus, device and method provided in the embodiments of this specification are corresponding to each other, and therefore the apparatus and device also have similar beneficial technical effects as the corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the corresponding apparatus and device will not be repeated here.

[0195] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0196] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program a digital system themselves to "integrate" it onto a PLD, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0197] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0198] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0199] For ease of description, the above devices are described separately by function as various units. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware.

[0200] Those skilled in the art will understand that one or more embodiments of this specification can be provided as a method, system, or computer program product. Therefore, the invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0201] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0202] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0203] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0204] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0205] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0206] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital character versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0207] This application can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0208] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.< / h3>

Claims

1. A method for evaluating a model, used to evaluate the model's ability to generate page code for visual pages, comprising: Input the visualization page generation command and test samples into the model to be evaluated, and obtain the output result of the model to be evaluated; The visualization page generation instruction is used to instruct the model to be evaluated to generate a visualization page for the test sample; the visualization page is a page containing visualization components; The model to be evaluated is functionally verified based on the output results to obtain a first verification result; the functional verification is used to verify whether the model to be evaluated can generate page code that meets the preset requirements. If the first verification result indicates that the model to be evaluated can generate page code that meets the preset requirements, then the generated page code is rendered to obtain the visual page; The quality of the visual components of the rendered visual page is verified to obtain a second verification result. The visualization component quality check is used to check the rendering quality of the visualization components in the visualization page; The rendered visualization page is subjected to content quality verification to obtain a third verification result. The content quality verification is used to verify the content quality of the semantic content in the visualization page; The model to be evaluated is evaluated based on the first verification result, the second verification result, and the third verification result.

2. The method as described in claim 1, wherein performing functional verification on the model to be evaluated based on the output result includes at least one of the following verification methods: The responsiveness of the model under test is verified based on the output results; the responsiveness verification is used to verify whether the model under test can generate page code. The instruction compliance verification is performed on the model to be evaluated based on the output results; The instruction follows a check to verify whether the output contains dialogue text outside of the page code; The structural integrity of the model to be evaluated is verified based on the output results; the structural integrity verification is used to verify the structural integrity of the page code contained in the output results. The model to be evaluated is validated using sandbox rendering based on the output results. The sandbox rendering verification is used to verify whether the output result can be rendered into a visual page under a preset viewport.

3. The method as described in claim 1, wherein the visualization component quality verification of the rendered visualization page includes at least one of the following verification methods: Verify whether the size of the visual components in the visualization page is within the preset size range; Verify whether the visualization page is missing any component elements; Verify that the visualization page has no occlusion defects; Verify whether the visualization page contains overflowing content; Verify whether the visualization page has insufficient color contrast.

4. The method as described in claim 3, wherein verifying whether the visualization page is missing component elements specifically includes: The first prompt word is constructed based on the visual page and the page code; The first prompt contains task instruction information that instructs the rendering quality assessment model to use the page code to verify whether the visualization page is missing component elements; Input the first prompt word into the rendering quality assessment model to obtain the verification result output by the rendering quality assessment model; The determination of whether the visualization page is missing component elements is based on the verification results output by the rendering quality assessment model.

5. The method as described in claim 3, wherein verifying whether the visualization page has an occlusion defect specifically includes: The visualization page is segmented based on the title tags in the visualization page to obtain various page modules; A second prompt word is constructed based on the aforementioned page modules; The second prompt contains task instruction information that instructs the rendering quality assessment model to verify whether each page module has occlusion defects; Input the second prompt word into the rendering quality assessment model to obtain the verification result output by the rendering quality assessment model; The presence of occlusion defects in the visualization page is determined based on the verification results output by the rendering quality assessment model.

6. The method as described in claim 3, wherein verifying whether the visualization page has content overflow specifically includes: Identify the module boundaries of each page module in the visualized page; For any page module, calculate whether the outer edge of the component element in the page module exceeds the module boundary of the page module. If the outer edge of a component element in any page module exceeds the module boundary of any page module, then it is determined that the visualization page has content overflow.

7. The method as described in claim 3, wherein verifying whether the visualization page has insufficient color contrast specifically includes: Determine the background color brightness of the component elements of the visualization components in the visualization page; Determine the foreground color brightness of the component element; Calculate the contrast between the foreground color brightness and the background color brightness; If the contrast ratio is less than or equal to a preset contrast threshold, then the color contrast of the visualized page is determined to be insufficient.

8. The method as described in claim 1, wherein the content quality verification includes at least one of the following: reasonableness verification, completeness verification, and fidelity verification of the semantic content in the visualization page; the reasonableness verification is used to verify the semantic reasonableness of the semantic content; the completeness verification is used to verify the completeness of the semantic content containing preset text information; and the fidelity verification is used to verify the consistency between the semantic content and the test sample.

9. The method as described in claim 8, wherein the semantic content in the visualized page is subjected to a reasonableness check, specifically including: Verify that the semantic content does not contain at least one of the following: factual errors, inconsistent context, unreasonable numerical values, inconsistent units, missing necessary elements, or incorrect chart type; If the semantic content contains at least one of the following: factual error, inconsistency in context, unreasonable numerical value, inconsistent units, missing necessary elements, or incorrect chart type, then the semantic content in the visualization page is determined to have failed the rationality check.

10. The method of claim 8, further comprising, before performing a completeness check on the semantic content in the visualized page: Generate preset text information; The preset text information is the information that needs to be included in the output of the model to be evaluated. The semantic content in the visualization page is subjected to completeness verification, specifically including: Calculate the proportion of the preset text information that is covered by the semantic content; If the ratio is less than or equal to a preset ratio threshold, then the semantic content in the visualization page is determined to have failed the completeness check.

11. The method as described in claim 8, wherein performing fidelity verification on the semantic content in the visualized page specifically includes: The semantic content is decomposed to obtain multiple semantic entries; each semantic entry is the smallest text information unit in the semantic content that can independently express a complete semantic meaning. The semantic entries are compared with the test sample to determine whether each semantic entry contradicts the test sample. If the semantic entry contradicts the test sample, then the semantic content in the visualization page has failed the fidelity check.

12. The method as described in claim 1, wherein evaluating the model to be evaluated based on the first verification result, the second verification result, and the third verification result specifically includes: The functional score of the model to be evaluated is determined based on the first verification result; The rendering quality score of the model to be evaluated is determined based on the second verification result; The content quality score of the model to be evaluated is determined based on the third verification result. The evaluation score of the model to be evaluated is determined based on the functionality score, the rendering quality score, and the content quality score.

13. The method of claim 12, further comprising: Calculate the complexity of the visualization components contained in the visualization page; Determine the complexity correction coefficient corresponding to the aforementioned complexity; The complexity correction coefficient is positively correlated with the complexity. The step of determining the evaluation score of the model to be evaluated based on the functionality score, the rendering quality score, and the content quality score specifically includes: The evaluation score of the model to be evaluated is determined based on the functionality score, the rendering quality score, the content quality score, and the complexity correction coefficient. The evaluation score is positively correlated with the complexity correction coefficient.

14. A model evaluation apparatus for evaluating the model's ability to generate page code for visual pages, comprising: The input module is used to input the visualization page generation instructions and test samples into the model to be evaluated, and obtain the output results of the model to be evaluated; The visualization page generation instruction is used to instruct the model to be evaluated to generate a visualization page for the test sample; the visualization page is a page containing visualization components; The functional verification module is used to perform functional verification on the model to be evaluated based on the output result to obtain a first verification result; the functional verification is used to verify whether the model to be evaluated can generate page code that meets preset requirements; The rendering module is used to render the generated page code to obtain the visualization page if the first verification result indicates that the model to be evaluated can generate page code that meets the preset requirements. The component quality verification module is used to verify the quality of the rendered visual components on the visual page and obtain a second verification result. The visualization component quality check is used to check the rendering quality of the visualization components in the visualization page; The content quality verification module is used to verify the content quality of the rendered visualization page and obtain a third verification result. The content quality verification is used to verify the content quality of the semantic content in the visualization page; The evaluation module is used to evaluate the model to be evaluated based on the first verification result, the second verification result, and the third verification result.

15. A computing device, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 13.

16. A computer-readable storage medium storing computer instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 13.

17. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 13.

Images