Image-based visual analytics interface automatic construction evaluation method and system

Abstract

This disclosure provides an automatic construction and evaluation method and system for image-based visual analytics interfaces, relating to the field of data visualization and analysis technology. The method involves acquiring and parsing a visual analytics workbook file; constructing a first intermediate representation representing the author's semantics in the workbook based on the parsing results; generating a second intermediate representation based on the first intermediate representation; generating candidate reference code and iteratively repairing it by incorporating feedback information during the candidate reference code generation process; verifying the iteratively repaired candidate reference code to form a basic benchmark sample; amplifying the basic benchmark sample through interactive recombination to obtain an amplified benchmark sample; generating a visual analytics interface based on the basic benchmark sample and the amplified benchmark sample; comparing the visual analytics interface with a reference executable interface; and outputting the evaluation results. This disclosure addresses the problems of existing visual analytics interfaces lacking executable, verifiable benchmark samples with data semantics and interactive annotations, as well as the lack of a unified evaluation method.

Description

Technical Field

[0001] This disclosure relates to the field of data visualization and analysis technology, specifically to a method and system for automatically constructing and evaluating image-based visual analysis interfaces. Background Technology

[0002] The statements in this section are merely background information relating to this disclosure and do not necessarily constitute prior art.

[0003] Data visualization, visual analytics, and benchmark dataset construction are important research directions in the current field of information processing and intelligent analysis. Data visualization expresses data characteristics graphically and is an important means of data understanding and analysis; visual analytics further emphasizes the combination of multi-view organization, interactive operation, and analytical reasoning to support exploratory analysis in complex data scenarios; benchmark dataset construction provides fundamental support for the unified evaluation, comparison, and optimization of related methods. With the development of multimodal large models and code generation technologies, the automatic construction and evaluation of visual analytics interfaces has gradually become a research hotspot in the interdisciplinary field of these technologies.

[0004] In visual analytics scenarios, compared to single charts, visual analytics interfaces typically involve multi-view organization, data binding relationships, and cross-view interaction logic. Their construction not only involves the reproduction of visual layouts but also the expression of data semantics and interactive behaviors, making their technical complexity significantly higher than that of generating ordinary static web pages or single-chart interfaces. While existing research and applications contain resources such as dashboards, workbooks, or interface screenshots, their specific solutions still have the following limitations: (1) Its layout information, worksheet definition, data connection and interactive actions are often stored in a scattered manner, and the associated data resources often have problems such as non-standard format, inconsistent encoding or missing fields. These resources are difficult to use directly as standardized benchmark samples, and are also difficult to execute, verify and reuse directly.

[0005] (2) Existing public resources usually lack a unified automatic construction process, making it difficult to reliably extract complete benchmark samples that simultaneously include executable code, screenshots, underlying data, and interactive annotations.

[0006] (3) Existing evaluation schemes mostly focus on the similarity of image appearance, lacking a unified evaluation mechanism for executability, data binding correctness and interaction integrity, making it difficult to accurately reflect the model's true capabilities in the visual analysis interface construction task. Summary of the Invention

[0007] To address the aforementioned issues, this disclosure proposes an image-based method and system for automatically constructing and evaluating visual analytics interfaces. It constructs a first intermediate representation and a second intermediate representation of semantic representation by parsing a workbook file, generates candidate reference codes based on these representations, iterates and repairs the code using feedback information during the generation process, and then automatically constructs and evaluates benchmark samples for visual analytics interfaces through verification, sample expansion, and unified evaluation.

[0008] According to some embodiments, the present disclosure adopts the following technical solutions: Image-based visual analytics interfaces automatically construct evaluation methods, including: Acquire and parse the visual analytics workbook file and its associated data resources; A first intermediate representation of the author's semantics in the workbook is constructed based on the parsing results, and a second intermediate representation is generated based on the first intermediate representation. Candidate reference codes are generated based on the second intermediate representation, the first intermediate representation, and the parsed data samples, and feedback information is incorporated during the candidate reference code generation process for iterative repair. Verification is performed on the candidate reference code after iterative repair to form a basic benchmark sample. The basic benchmark sample is then amplified by recombination amplification through interactive preservation to obtain an amplified benchmark sample. A visual analysis interface is generated based on the basic benchmark samples and the expanded benchmark samples. The visual analysis interface is compared with the reference executable interface, and the evaluation results are output.

[0009] Furthermore, the step of acquiring and parsing the visual analytics workbook file and its associated data resources includes: Obtain the visual analytics workbook file to be processed and its associated data resources; parse the worksheet definitions, dashboard layouts, interactive actions, and data connection information in the visual analytics workbook file; restore and clean the parsed data resources to obtain a structured data table that meets the field coverage verification conditions; The process of restoring and cleaning the parsed data resources includes: extracting the workbook description file, embedded data files, and external reference information from the packaged workbook file; unpacking, format conversion, and uniform encoding of embedded data sources, compressed data sources, or data files with non-standard row and column formats; and performing field coverage verification on the restored data table based on the referenced fields in the workbook, retaining only the data resources that can support all view encoding and interactive actions.

[0010] Further, the step of constructing a first intermediate representation of the author's semantics in the workbook based on the parsing results, and generating a second intermediate representation based on the first intermediate representation, includes: A first intermediate representation is constructed based on the workbook file. The first intermediate representation records the chart type, field-to-visual channel mapping relationship, local filtering conditions and sorting information at the view level, as well as the layout relationship, cross-view interaction relationship and text annotation information at the dashboard level, in order to characterize the author semantics in the workbook file. A second intermediate representation is generated based on the first intermediate representation. The second intermediate representation includes the binding relationship between fields and coordinate axes, sorting rules, stacking rules, legend configuration, regional geometric information, and interaction source target field triples, which are used to convert the implicit rendering semantics and interaction semantics in the workbook file into explicit execution constraints.

[0011] Furthermore, the step of generating candidate reference code based on the second intermediate representation, the first intermediate representation, and the parsed data samples, and iteratively repairing the candidate reference code by incorporating feedback information during the candidate reference code generation process, includes: Extract sampled data from the structured data table, construct code generation input triples based on the second intermediate representation, the first intermediate representation, and the sampled data; generate candidate reference code corresponding to the workbook file based on the code generation input triples; During the generation process, missing views, unbound fields, missing interactive functions, compilation errors, runtime exceptions, and rendering issues are detected, and the detection results are fed back to the reference code generation process for automatic repair.

[0012] Furthermore, the verification of the iteratively repaired candidate reference code to form a basic benchmark sample includes: Perform source code-level, build-level, and render-level checks on the candidate reference code. Source code-level checks are used to verify whether the reference code correctly references the data fields specified by the second intermediate representation and whether the data file is loaded correctly. Build-level checks are used to verify whether the project installation, static checks, type checks, and build process are passed. Render-level checks are used to verify whether the expected view appears completely and whether the rendering result is not empty. After verifying the candidate reference code, the basic benchmark sample corresponding to the workbook file is obtained.

[0013] Further, the process of obtaining an amplified reference sample from the basic reference sample through cross-preservation recombination amplification includes: Select layout structures, chart views, or data sources from other basic benchmark samples, reorganize the content corresponding to the basic benchmark samples, and establish field mapping relationships based on field type compatibility, semantic role consistency, and name similarity. Rewrite the internal references of filtering actions, highlighting actions, and linked actions according to the interaction preservation rules. After verifying the recombined candidate samples, the verified candidate samples are added to obtain the amplified baseline samples.

[0014] Furthermore, the step of generating a visual analysis interface based on the basic benchmark samples and the amplified benchmark samples, comparing the visual analysis interface with the reference executable interface, and outputting evaluation results includes: Using basic benchmark samples and amplified benchmark samples as evaluation units, the basic benchmark samples and amplified benchmark samples are input into the model to be evaluated to obtain the generated code, and the generated code is executed to obtain a visual analysis interface. The visual analytics interface is compared with the reference executable interface to obtain the evaluation results; The evaluation results include executability evaluation results, static visual consistency evaluation results, data binding correctness evaluation results, interaction integrity evaluation results, and overall evaluation results.

[0015] According to some embodiments, the present disclosure adopts the following technical solutions: An image-based visual analytics interface is used to automatically build an evaluation system, including: The workbook parsing and recovery module is used to acquire and parse visual analysis workbook files and their associated data resources. The first intermediate representation building module is used to construct a first intermediate representation of the author semantics in the workbook based on the parsing results; The second intermediate representation construction module is used to generate a second intermediate representation based on the first intermediate representation; The candidate reference code generation and repair module is used to generate candidate reference codes based on the second intermediate representation, the first intermediate representation, and the parsed data samples, and to iteratively repair the candidate reference codes by incorporating feedback information during the generation process. The verification module is used to verify the candidate reference code after iterative repair, forming a basic benchmark sample; The sample amplification module is used to obtain an amplified baseline sample by cross-maintaining recombination amplification of the baseline baseline sample; The evaluation module is used to generate a visual analysis interface based on the basic benchmark sample and the amplified benchmark sample, compare the visual analysis interface with the reference executable interface, and output the evaluation results.

[0016] According to some embodiments, the present disclosure adopts the following technical solutions: A computer program product includes a computer program that, when executed by a processor, implements the image-based visual analysis interface automatic construction and evaluation method.

[0017] According to some embodiments, the present disclosure adopts the following technical solutions: An electronic device includes a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to implement the image-based visual analysis interface automatic construction and evaluation method.

[0018] Compared with the prior art, the beneficial effects of this disclosure are as follows: The image-based visual analysis interface automatic construction evaluation method disclosed herein separates and models the author semantics in the workbook file from the execution constraints required for generating the reference code by constructing a first intermediate representation and a second intermediate representation. This method can effectively solve the problem of difficulty in automatically constructing benchmark samples caused by semantic dispersion, implicit default behavior and complex data dependencies in workbook specifications.

[0019] The image-based automatic construction and evaluation method for visual analytics interfaces disclosed herein can automatically generate benchmark samples of visual analytics interfaces containing executable code, interface screenshots, underlying data, and interactive annotations through candidate reference code generation, feedback repair, and source code-level, build-level, and rendering-level verification, thereby improving the completeness, executability, consistency, and reproducibility of benchmark samples.

[0020] The image-based visual analysis interface automatic construction evaluation method disclosed herein, through an interactive constraint-maintaining sample augmentation mechanism, can expand the sample size and sample coverage while preserving the validity of data semantics and interactive semantics, thereby improving the diversity and practical value of the benchmark sample set.

[0021] This disclosure presents an automatic evaluation method for image-based visual analysis interfaces. It proposes a unified evaluation mechanism based on basic and expanded benchmark samples, which can quantitatively evaluate the evaluation model from multiple dimensions such as executability, static visual consistency, data binding correctness, and interaction integrity, thereby improving the comprehensiveness and objectivity of the evaluation results. Attached Figure Description

[0022] The accompanying drawings, which form part of this disclosure, are used to provide a further understanding of this disclosure. The illustrative embodiments of this disclosure and their descriptions are used to explain this disclosure and do not constitute an undue limitation of this disclosure.

[0023] Figure 1 This is a flowchart of an image-based visual analysis interface automatic construction and evaluation method according to an embodiment of the present disclosure. Detailed Implementation

[0024] The present disclosure will be further described below with reference to the accompanying drawings and embodiments.

[0025] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of this disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains.

[0026] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this disclosure. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms “comprising” and / or “including” are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0027] Example 1 One embodiment of this disclosure provides a method for automatically constructing and evaluating an image-based visual analysis interface, the method comprising the following steps: Step 1: Obtain the visual analytics workbook file and its associated data resources, and parse them; Step 2: Construct a first intermediate representation of the author's semantics in the workbook based on the parsing results, and generate a second intermediate representation based on the first intermediate representation; Step 3: Generate candidate reference code based on the second intermediate representation, the first intermediate representation, and the parsed data samples, and iterate and repair the code in conjunction with feedback information during the candidate reference code generation process; Step 4: Verify the candidate reference code after iterative repair to form a basic benchmark sample. Amplify the basic benchmark sample by recombination amplification through interactive preservation. Step 5: Generate a visual analysis interface based on the basic benchmark sample and the amplified benchmark sample, compare the visual analysis interface with the reference executable interface, and output the evaluation results.

[0028] As one embodiment, the image-based automatic construction and evaluation method for visual analytics interfaces disclosed herein achieves automatic construction and evaluation of benchmark samples for visual analytics interfaces through workbook parsing, data recovery, two-layer intermediate representation construction, candidate reference code generation and iterative repair, verification, sample expansion, and unified evaluation. The specific implementation process is as follows: Step 1: Obtain the visual analytics workbook file and its associated data resources, and parse them; Specifically, the visual analytics workbook file refers to a project file generated by visual analytics software that describes the composition and configuration of the visual analytics interface. The visual analytics workbook file records data source configuration, worksheet definitions, dashboard layout, field mapping relationships, visual coding rules, interactive actions, and data connection information. For example, the visual analytics workbook file includes a Tableau workbook file or a packaged Tableau workbook file.

[0029] The associated data resources refer to the data resources that correspond to the visual analysis workbook file and are used to support worksheet view rendering, data field mapping, and interactive action execution. These include embedded data files, packaged data files, external reference data files, data extraction files, database connection information, and data element information such as field names, field types, and field value ranges.

[0030] Furthermore, the parsing method is a hierarchical parsing method based on the workbook's structured description file. Specifically, when the visual analytics workbook file is a packaged workbook file, it is unpacked, and the workbook description file, embedded data files, and external reference information are extracted. When the visual analytics workbook file is an unpackaged workbook file, its workbook description file and the data resources it points to are directly read. Subsequently, the workbook description file is parsed in a structured manner, and data connection information, field definitions, worksheet definitions, field-to-visual channel mapping relationships, dashboard layout relationships, and cross-view interactive action information are extracted sequentially according to the hierarchical relationship of data source nodes, worksheet nodes, dashboard nodes, and interactive action nodes.

[0031] The parsing process includes: first, identifying the workbook file type and obtaining the workbook description file; then, parsing the data source node to obtain the data table path, data connection method, field names, and field types; next, parsing the worksheet node to obtain the chart type, field mapping relationship, filtering conditions, sorting rules, and visual coding information; then, parsing the dashboard node to obtain the position, size, and hierarchical relationship of each worksheet in the dashboard; and finally, parsing the interactive action node to obtain the trigger source, target, associated fields, and action type of cross-view interactive actions such as filtering, highlighting, selection, or linkage. Through the above parsing process, the worksheet definitions, dashboard layout, interactive actions, and data connection information in the workbook file are obtained.

[0032] Furthermore, the parsed data resources are restored and cleaned to obtain a structured data table that meets the field coverage verification conditions. The process includes: like Figure 1As shown, in this embodiment, the input single visual analytics workbook file is denoted as W. The system extracts the workbook description file, embedded data files, and external reference information from the packaged workbook file, and performs unpacking, format conversion, and unified encoding processing on embedded data sources, compressed data sources, or data files with non-standard row and column formats. Subsequently, based on the referenced fields in workbook file W, the system performs field coverage verification on the restored data table, retaining only the data resources that can support all view encoding and interactive actions, thereby obtaining a structured data table D that can be used for subsequent construction.

[0033] Step 2: Construct a first intermediate representation of the author's semantics in the workbook based on the parsing results, and generate a second intermediate representation based on the first intermediate representation; Specifically, the first intermediate representation includes at least view-level information and dashboard-level information. In this embodiment, it is based on a workbook file. W Constructing the first intermediate representation TSS The first intermediate representation TSS Records view-level chart types, field-to-visual channel mappings, local filtering conditions, and sorting information, as well as dashboard-level layout relationships, cross-view interaction relationships, and text annotation information, used to characterize workbook files. W The author's semantics in the text.

[0034] Constructing the first intermediate representation TSS The process includes: First, based on the parsed data connection information and field definitions, determining the data source, referenced fields, field types, and field roles used by each worksheet; then, based on the parsed worksheet definitions, extracting the chart type, field-to-visual channel mapping relationship, aggregation method, local filtering conditions, sorting information, and text annotation information corresponding to each worksheet; next, based on the parsed dashboard layout information, extracting the position, size, hierarchy, container relationship, and arrangement of each worksheet in the dashboard; finally, based on the parsed interaction action information, extracting cross-view interaction relationships, including the triggering view, target view, action type, associated fields, and triggering conditions of the interaction action.

[0035] Through the above process, data source semantics, view-level semantics, dashboard-level semantics, and interaction semantics are uniformly organized into a first intermediate representation. TSS The first intermediate representation TSS It should include at least view-level chart types, field-to-visual channel mappings, local filtering conditions, and sorting information, as well as dashboard-level layout relationships, cross-view interaction relationships, and text annotation information to characterize the workbook file. W The author semantics in the text provide a semantic basis for the subsequent generation of the second intermediate representation.

[0036] Furthermore, the second intermediate representation is used to characterize the execution constraints required for reference code generation. In this embodiment, it is based on the first intermediate representation. TSS Generate a second intermediate representation CTS .

[0037] Specifically, the system represents the first intermediate representation. TSS The view-level semantics, dashboard-level semantics, and interaction semantics in the code are transformed into rules, converting the semantic descriptions for author configuration into execution constraints for code generation.

[0038] In generating the second intermediate representation CTS During the process, the system first represents the first intermediate representation. TSS The mapping relationship between fields in the code is transformed to the visual channels, mapping the horizontal axis, vertical axis, color, size, label, and detail visual channels to the coordinate axis binding relationships, tag attribute configurations, and data field reference relationships required for code generation; then, the first intermediate representation is... TSS The filtering conditions, sorting information, and aggregation methods in the first intermediate representation are transformed to generate corresponding data filtering constraints, sorting rules, aggregation calculation rules, and stacking rules; then, the first intermediate representation is processed... TSS The dashboard layout is transformed to generate the geometric information of each view's region in the target interface, including its position, size, arrangement direction, and hierarchy. Then, auxiliary display information such as legends, titles, and text annotations are transformed to generate legend configurations and interface annotation configurations. Finally, the first intermediate representation... TSS The cross-view interaction relationship is transformed to generate an interaction source target field triplet, which represents the interaction trigger view, the interaction target view, and the data fields associated between the two.

[0039] Through the above conversion process, the second intermediate representation CTS It should include at least the field-to-axis binding relationship, sorting rules, stacking rules, legend configuration, region geometry information, and interaction source-target field triples, used to link workbook files. W The implicit rendering and interaction semantics are converted into explicit execution constraints that can be directly used in the candidate reference code generation process.

[0040] Step 3: Generate candidate reference code based on the second intermediate representation, the first intermediate representation, and the parsed data samples, and iterate and repair the code in conjunction with feedback information during the candidate reference code generation process; In this embodiment, from structured data tables D Extracting sampling data And construct code to generate input triples. G Its formula is: in, CTS For the second intermediate representation, provide the execution-level constraints that the reference code must satisfy. This serves as the first intermediate representation, providing a semantic reference to the original author. Provide the field name, field type, and representative value range.

[0041] Input triples generated from code G Generate workbook file W Corresponding candidate reference code C The candidate reference code C This refers to the workbook file W The initial executable web page code project generated from the author semantics, execution constraints, and sampled data to reproduce the original visual analytics interface includes at least data loading code, view component code, dashboard layout code, and interaction processing code.

[0042] Specifically, the code will generate input triples. G The second intermediate representation in CTS First intermediate representation TSS and sampling data The organization provides a description of the code generation task; among which, CTS It provides execution-level constraints such as field binding, coordinate axis configuration, sorting rules, region geometry information, and interaction source-target field triples. TSS It provides semantic references for authors, including chart types, field-to-visual channel mappings, layout relationships, and cross-view interaction relationships. Provide the field name, field type, and representative value range. The system is based on... CTS and TSS Generate view components, dashboard layouts, data field bindings, visual coding configurations, and interaction handling functions, and based on... Generate or correct data loading paths and field referencing logic to obtain candidate reference code. C .

[0043] Furthermore, during the generation process, missing views, unbound fields, missing interaction functions, compilation errors, runtime exceptions, and rendering issues are detected, and the detection results are fed back to the candidate reference code generation process for automatic repair. After one or more generation and repair cycles, iteratively repaired candidate reference code is obtained. C It is used for subsequent source code-level verification, build-level verification, and rendering-level verification.

[0044] Step 4: Validate the candidate reference code after iterative repair to form a basic benchmark sample. Amplify the basic benchmark sample by recombination amplification through interactive preservation. In this embodiment, the candidate reference code CPerform source code-level verification, build-level verification, and render-level verification.

[0045] (1) Source code level verification. The system reads candidate reference code. C Extract the data file path, field name references, view component definitions, and interaction handling functions from the source file, and combine them with the second intermediate representation. CTS The system compares the triplet of data fields, data paths, view definitions, and interaction source target fields recorded in the database. If there are any issues such as missing data file paths, inconsistent field names, missing necessary fields, missing view components, or missing interaction processing functions, the source code-level verification is deemed to have failed, and the corresponding error information is fed back to the candidate reference code generation and repair process.

[0046] (2) Build-level verification. Install candidate reference code in the preset runtime environment. C The system checks for required dependencies and performs static checks, type checks, and build commands. If a dependency installation fails, a static check fails, a type check fails, or the build fails, the build-level verification is deemed to have failed, and the corresponding compilation error, type error, or build log is recorded as feedback information for subsequent repairs.

[0047] (3) Render-level verification. The system starts the built candidate reference code in the preset runtime environment. C The interface image is then rendered by the browser; subsequently, it is based on the second intermediate representation. CTS The system records the expected number of views, view area geometry information, and view identifier information to detect whether the expected views appear completely, whether each view area has valid rendering content, and whether the rendering result is blank. If there are cases where the expected view is missing, the view area is empty, the page runs incorrectly, or the rendering result is blank, the rendering level verification is determined to have failed, and the rendering error information is fed back to the candidate reference code generation and repair process.

[0048] After the above verification, the workbook file is obtained. W The corresponding baseline sample B. In this embodiment, the baseline sample B can be represented as: B=( (D, A, U) in, For reference screenshots, D represents the underlying dataset, A represents the structured interaction specification, and U represents the reference executable interface. The basic benchmark sample also includes executable reference code, corresponding interface images, underlying data, and interaction annotations.

[0049] Furthermore, an expanded benchmark sample is constructed based on the basic benchmark sample B. In this embodiment, the system first parses the layout structure, worksheet view, data source, field specifications, chart type, and interactive actions corresponding to the basic benchmark sample B and other basic benchmark samples, and then filters basic benchmark samples that can be recombined with the basic benchmark sample B based on data pattern similarity. The data pattern similarity is determined based on the intersection and union of the field set of the basic benchmark sample B and the field set of another basic benchmark sample, and its calculation formula is as follows:

[0050] in, This represents the set of data fields corresponding to the baseline sample B. This represents the set of data fields corresponding to the i-th other baseline sample. When the data pattern similarity meets a preset threshold, the i-th... i One baseline sample was used as a candidate recombination object.

[0051] Specifically, the system selects a dashboard layout as a layout template from the base baseline sample B or candidate reorganization objects. Based on the number of view slots in the layout template, it selects one or more remappable worksheet views and chooses a target data source. Then, the system establishes a mapping relationship between the source fields in the worksheet view and the target fields in the target data source based on field type compatibility, field role consistency, semantic role consistency, and field name similarity. Worksheet views for which a valid field mapping relationship cannot be established are not added to the candidate augmentation sample.

[0052] After completing the field mapping, the system rewrites the data source references, field references, visual coding configurations, and interactive action references in the worksheet view based on the field mapping relationship, and places the rewritten worksheet view into the dashboard area corresponding to the layout template to form candidate augmentation samples. For example, a multi-view dashboard layout of the base benchmark sample B can be selected, and the bar chart and line chart views in another base benchmark sample can be selected, and their field references can be remapped to fields in the target data source that are compatible in type and semantics; if there are filtering or highlighting interactions between the original views, the triggering view, target view, and associated fields in the interactive actions are rewritten synchronously.

[0053] After validating the recombined candidate amplification samples, the validated candidate amplification samples are added to the amplification baseline sample set.

[0054] The interaction retention rules must include at least one of the following: (1) Restrict arbitrary remapping of views that carry interactive actions; (2) Copy the filtering action only if a compatible field exists in the target data source; (3) Preserve the linkage between highlighting and brushing when the shared dimensions across views can still be resolved.

[0055] In this embodiment, the above-mentioned interaction preservation rules can ensure that the amplified sample retains effective data semantics and interaction semantics during the recombination process, reducing the risk of interaction failure during sample amplification.

[0056] Step 5: Generate a visual analysis interface based on the basic benchmark sample and the amplified benchmark sample, compare the visual analysis interface with the reference executable interface, and output the evaluation results.

[0057] Specifically, using the basic baseline sample and the amplified baseline sample B=( (D, A, U) are used as evaluation units, where, For reference screenshots, D represents the underlying dataset, A represents the structured interaction specification, and U represents the reference executable interface.

[0058] The structured interaction specification A refers to the interaction description information extracted and standardized from the interaction actions in the visual analytics workbook file, which is used to record the triggering method, target object and expected result of cross-view interaction in the visual analytics interface.

[0059] Specifically, the structured interaction specification A includes at least the interaction action type, trigger view, target view, trigger field, target field, trigger condition, and post-interaction state condition; wherein, the interaction action type includes one or more of filtering, highlighting, selection, and linkage, and the post-interaction state condition includes one or more of filtering result, highlighting result, linkage result, and selection state.

[0060] The model to be evaluated refers to a multimodal large language model or image-to-code generation model used to perform visual analysis interface generation tasks. The model typically includes a visual input encoding part, a text and structured information input encoding part, a cross-modal fusion and inference part, and a code generation output part; wherein, the visual input encoding part is used to receive and encode reference screenshots. The text and structured information input encoding part is used to receive and encode the field information of the underlying dataset D and the structured interaction specification A. The cross-modal fusion and reasoning part is used to fuse interface visual information, data semantic information and interaction constraint information. The code generation output part is used to generate executable interface code.

[0061] Reference screenshots The underlying dataset D and the structured interaction specification A are input into the model to be evaluated, and the generated code is obtained: Execute the generated code Then, a visual analysis interface generated by the model to be evaluated is obtained. The visual analysis interface The evaluation results are obtained by comparing the interface with the reference executable U.

[0062] Furthermore, the evaluation results include executability evaluation results. Static visual consistency evaluation results Data binding correctness evaluation results Interaction integrity evaluation results and overall evaluation results When the visual analytics interface generated by the model to be evaluated... When the build and rendering can be completed without fatal runtime errors in the preset runtime environment, ,otherwise .

[0063] Among them, (1) Static visual consistency evaluation results The formula used to measure the visual consistency between the interface generated by the model under evaluation and the reference executable interface U in its initial state is as follows: in, Represents global visual semantic similarity. Indicates structural similarity. This represents the normalized inverse mean square error. This indicates the similarity based on the tree structure of the Interface Document Object Model.

[0064] (2) Data binding correctness evaluation results The formula used to measure the consistency between rendered data values ​​in the interface generated by the model under evaluation and reference data values ​​is as follows: in, This represents the collection of views in the interface generated by the model to be evaluated. Represents a view The matching ratio between the extracted rendered data values ​​and the reference data values; when the view When the correct data file is not loaded or fake data values ​​are rendered, .

[0065] (3) Results of interaction integrity evaluation The formula used to measure whether the behavior of the interface generated by the model under evaluation after an interaction is triggered meets the reference interaction specifications is as follows: Where T represents the set of interactive playback events compiled according to the structured interaction specification A, and the interactive playback events... It should at least include the triggering action, the affected target view, and the post-interaction state conditions; when the event When all the corresponding post-state conditions are met within the preset time window, ,otherwise The post-state conditions include at least one or more of the following: filtering result, highlighting result, linkage result, and selection state.

[0066] (4) Overall evaluation results The formula used to comprehensively characterize the overall performance of the model under evaluation in terms of executability, static visual consistency, data binding correctness, and interaction integrity is as follows:

[0067] Based on the above evaluation results, the actual capabilities of the evaluation model in the visual analytics interface construction task can be assessed in a unified, quantitative, and repeatable manner.

[0068] Simulation Experiment To verify the effectiveness of the method described in this disclosure, this embodiment evaluates the visual analysis interface generated by the model to be evaluated based on the constructed basic benchmark sample and the expanded benchmark sample. Table 1 shows the evaluation results of different models to be evaluated on the benchmark sample constructed in this disclosure.

[0069] Table 1. Evaluation results of different models under evaluation on the benchmark sample constructed in this disclosure.

[0070] It can be seen that the benchmark samples constructed in this disclosure can support the unified execution, comparison and quantitative evaluation of the output results of the model to be evaluated, and can distinguish the model capabilities from multiple dimensions such as executability, visual consistency, data binding correctness and interaction integrity, thereby verifying the effectiveness of the method of this disclosure in the automatic construction and unified evaluation of benchmark samples for visual analysis interfaces.

[0071] Example 2 One embodiment of this disclosure provides an image-based visual analysis interface for automatically constructing an evaluation system, comprising: The workbook parsing and recovery module is used to acquire and parse visual analysis workbook files and their associated data resources. The first intermediate representation building module is used to construct a first intermediate representation of the author semantics in the workbook based on the parsing results; The second intermediate representation construction module is used to generate a second intermediate representation based on the first intermediate representation; The candidate reference code generation and repair module is used to generate candidate reference codes based on the second intermediate representation, the first intermediate representation, and the parsed data samples, and to iteratively repair the candidate reference codes by incorporating feedback information during the generation process. The verification module is used to verify the candidate reference code after iterative repair, forming a basic benchmark sample; The sample amplification module is used to obtain an amplified baseline sample by cross-maintaining recombination amplification of the baseline baseline sample; The evaluation module is used to generate a visual analysis interface based on the basic benchmark sample and the amplified benchmark sample, compare the visual analysis interface with the reference executable interface, and output the evaluation results.

[0072] Example 3 One embodiment of this disclosure provides a computer program product, including a computer program that, when executed by a processor, implements the image-based visual analysis interface automatic construction and evaluation method.

[0073] Example 4 One embodiment of this disclosure provides an electronic device, including: a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to implement the image-based visual analysis interface automatic construction and evaluation method.

[0074] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0075] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0076] While the specific embodiments of this disclosure have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of this disclosure. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of this disclosure are still within the scope of protection of this disclosure.

Claims

1. A method for automatically constructing and evaluating image-based visual analysis interfaces, characterized in that, include: Acquire and parse the visual analytics workbook file and its associated data resources; A first intermediate representation of the author's semantics in the workbook is constructed based on the parsing results, and a second intermediate representation is generated based on the first intermediate representation. Candidate reference codes are generated based on the second intermediate representation, the first intermediate representation, and the parsed data samples, and feedback information is incorporated during the candidate reference code generation process for iterative repair. Verification is performed on the candidate reference code after iterative repair to form a basic benchmark sample. The basic benchmark sample is then amplified by recombination amplification through interactive preservation to obtain an amplified benchmark sample. A visual analysis interface is generated based on the basic benchmark samples and the expanded benchmark samples. The visual analysis interface is compared with the reference executable interface, and the evaluation results are output.

2. The method for automatically constructing and evaluating image-based visual analysis interfaces as described in claim 1, characterized in that, The process of acquiring and parsing the visual analytics workbook file and its associated data resources includes: Obtain the visual analytics workbook file to be processed and its associated data resources; parse the worksheet definitions, dashboard layouts, interactive actions, and data connection information in the visual analytics workbook file; restore and clean the parsed data resources to obtain a structured data table that meets the field coverage verification conditions; The process of restoring and cleaning the parsed data resources includes: extracting the workbook description file, embedded data files, and external reference information from the packaged workbook file; unpacking, format conversion, and uniform encoding of embedded data sources, compressed data sources, or data files with non-standard row and column formats; and performing field coverage verification on the restored data table based on the referenced fields in the workbook, retaining only the data resources that can support all view encoding and interactive actions.

3. The method for automatically constructing and evaluating an image-based visual analysis interface as described in claim 1, characterized in that, The process of constructing a first intermediate representation of the author's semantics in the workbook based on the parsing results, and generating a second intermediate representation based on the first intermediate representation, includes: A first intermediate representation is constructed based on the workbook file. The first intermediate representation records the chart type, field-to-visual channel mapping relationship, local filtering conditions and sorting information at the view level, as well as the layout relationship, cross-view interaction relationship and text annotation information at the dashboard level, in order to characterize the author semantics in the workbook file. A second intermediate representation is generated based on the first intermediate representation. The second intermediate representation includes the binding relationship between fields and coordinate axes, sorting rules, stacking rules, legend configuration, regional geometric information, and interaction source target field triples, which are used to convert the implicit rendering semantics and interaction semantics in the workbook file into explicit execution constraints.

4. The method for automatically constructing and evaluating an image-based visual analysis interface as described in claim 1, characterized in that, The process of generating candidate reference codes based on the second intermediate representation, the first intermediate representation, and the parsed data samples, and iteratively repairing them by incorporating feedback information during the candidate reference code generation process, includes: Extract sampled data from the structured data table, construct code generation input triples based on the second intermediate representation, the first intermediate representation, and the sampled data; generate candidate reference code corresponding to the workbook file based on the code generation input triples; During the generation process, missing views, unbound fields, missing interactive functions, compilation errors, runtime exceptions, and rendering issues are detected, and the detection results are fed back to the reference code generation process for automatic repair.

5. The method for automatically constructing and evaluating an image-based visual analysis interface as described in claim 1, characterized in that, The process of verifying the candidate reference code after iterative repair to form a basic benchmark sample includes: Perform source code-level, build-level, and render-level checks on the candidate reference code. Source code-level checks are used to verify whether the reference code correctly references the data fields specified by the second intermediate representation and whether the data file is loaded correctly. Build-level checks are used to verify whether the project installation, static checks, type checks, and build process are passed. Render-level checks are used to verify whether the expected view appears completely and whether the rendering result is not empty. After verifying the candidate reference code, the basic benchmark sample corresponding to the workbook file is obtained.

6. The method for automatically constructing and evaluating an image-based visual analysis interface as described in claim 1, characterized in that, The process of obtaining an amplified baseline sample through cross-preservation recombination amplification of the baseline baseline sample includes: Select layout structures, chart views, or data sources from other basic benchmark samples, reorganize the content corresponding to the basic benchmark samples, and establish field mapping relationships based on field type compatibility, semantic role consistency, and name similarity. Rewrite the internal references of filtering actions, highlighting actions, and linked actions according to the interaction preservation rules. After verifying the recombined candidate samples, the verified candidate samples are added to obtain the amplified baseline samples.

7. The method for automatically constructing and evaluating an image-based visual analysis interface as described in claim 1, characterized in that, The process involves generating a visual analysis interface based on the basic benchmark samples and the expanded benchmark samples, comparing the visual analysis interface with a reference executable interface, and outputting evaluation results, including: Using basic benchmark samples and amplified benchmark samples as evaluation units, the basic benchmark samples and amplified benchmark samples are input into the model to be evaluated to obtain the generated code, and the generated code is executed to obtain a visual analysis interface. The visual analytics interface is compared with the reference executable interface to obtain the evaluation results; The evaluation results include executability evaluation results, static visual consistency evaluation results, data binding correctness evaluation results, interaction integrity evaluation results, and overall evaluation results.

8. An image-based visual analysis interface-based automatic evaluation system, characterized in that, include: The workbook parsing and recovery module is used to acquire and parse visual analysis workbook files and their associated data resources. The first intermediate representation building module is used to construct a first intermediate representation of the author semantics in the workbook based on the parsing results; The second intermediate representation construction module is used to generate a second intermediate representation based on the first intermediate representation; The candidate reference code generation and repair module is used to generate candidate reference codes based on the second intermediate representation, the first intermediate representation, and the parsed data samples, and to iteratively repair the candidate reference codes by incorporating feedback information during the generation process. The verification module is used to verify the candidate reference code after iterative repair, forming a basic benchmark sample; The sample amplification module is used to obtain an amplified baseline sample by cross-maintaining recombination amplification of the baseline baseline sample; The evaluation module is used to generate a visual analysis interface based on the basic benchmark sample and the amplified benchmark sample, compare the visual analysis interface with the reference executable interface, and output the evaluation results.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the image-based visual analysis interface automatic construction and evaluation method according to any one of claims 1-7.

10. An electronic device, characterized in that, include: The device includes a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to perform the automatic construction and evaluation method for image-based visual analysis interfaces as described in any one of claims 1-7.

Images