Data quality and preparation cost evaluation device for domain large model

Abstract

The invention discloses a data quality and preparation cost evaluation device for a domain large model, and the device comprises a data quality evaluation system which is used for carrying out the five-dimension evaluation of data size, data inherent quality, data representation quality, data context quality and data redundancy on a data set before training. Therefore, training-free evaluation is carried out on the scale, integrity, consistency, diversity and non-redundancy of the data set, and low-quality data is effectively screened out. The data preparation cost evaluation system is used for carrying out quantitative evaluation on time and resource consumption of each operator in a data preparation process, and recommending an optimal preparation operator workflow through a unit resource consumption quality improvement strategy or a unit time consumption quality improvement strategy under time and resource limitation conditions preset by a user. Therefore, joint optimization of data set quality improvement and cost control is realized. According to the method, the resource and time cost of data preparation can be reduced, and the utilization efficiency of large model training data and the training performance of a final model are improved.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a data quality and preparation cost evaluation device for domain-specific large models, specifically applied to multimodal large model training scenarios based on generating ECharts code from chart images. Background Technology

[0002] With the development of large-scale pre-trained models, multimodal large-scale models for chart code generation are gradually becoming an emerging research direction. After inputting a screenshot of a statistical chart or a hand-drawn sketch, the model can automatically generate the corresponding EChartsJavaScript configuration code, enabling rapid chart reproduction and interaction. This type of task places high demands on the accuracy and diversity of the training data.

[0003] In the application scenarios of large-scale domain models, the evaluation of data quality and preparation costs is a core aspect of ensuring efficient model training and performance optimization. Data quality evaluation involves systematically measuring the data to ensure it meets training requirements, while data preparation cost evaluation requires a comprehensive consideration of various costs throughout the entire data processing workflow.

[0004] In related technologies, data lacks systematic quality checks before entering the training process. Data quality needs to be inferred from the performance results after model training, making the model susceptible to low-quality data and increasing time and resource costs through repeated iterations. Existing quality measurement methods mostly use single statistical indicators, which are difficult to combine with task characteristics such as image-code matching relationships, code syntax executability, and the diversity of chart structures for evaluation. Data preparation cost assessment methods are also difficult to cope with the complex and heterogeneous data sources and query requirements of large-scale domain models.

[0005] Therefore, there is an urgent need for an evaluation system that can be implemented without training, providing a reliable basis for building large models for graph code generation from the perspectives of data quality and preparation cost. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of existing technologies by providing a data quality and preparation cost evaluation device for large domain models.

[0007] The objective of this invention is achieved through the following technical solution: a data quality and preparation cost evaluation device for large domain models, comprising: A data quality assessment system is used to perform multi-dimensional quality assessments on the training data of the model; based on the quality assessment results, appropriate preparation operators are selected, and a set of candidate operators is obtained as input to the data preparation cost assessment system. A data preparation cost assessment system is used to quantitatively evaluate the resource and time consumption during the data preparation process in order to select the optimal operator workflow.

[0008] Furthermore, the data quality assessment system evaluates the quality of the model's training data based on the following five dimensions of indicators: (1) Data volume dimension; (2) Data intrinsic quality dimension; (3) Data representation quality dimensions; (4) Data context quality dimension; (5) Data redundancy dimension.

[0009] Furthermore, the training data is image-code pair data.

[0010] Furthermore, the data volume dimension evaluates the size of the dataset using data volume metrics; The intrinsic quality dimensions of the data include: Syntax check metrics are used to determine whether code can be successfully compiled and run; The structural integrity metric is used to check whether the chart configuration fields are complete; The data represents quality dimensions including: Consistency metrics are used to measure the degree of matching between the original chart image and the code rendering image; The missing rate metric is used to determine whether there are missing paired metadata. The data context quality dimensions include: The type balance index is used to evaluate whether the distribution of different chart types is balanced; The structural diversity index is used to evaluate the differences in the field existence characteristics of the configuration structure, wherein the differences are calculated by a preset diversity measurement method. The data redundancy dimension includes: Code duplication metrics are used to detect redundant code in a dataset; Image redundancy index is used to detect the presence of redundant images in a dataset.

[0011] Furthermore, the input to the data preparation cost evaluation system is a set of candidate operators. Each operator It has the following parameters: processing time for a single record. Number of records to be processed Resource costs Target weights and user-preset time limits and resource cost constraints .

[0012] Furthermore, the time consumption of the operator The calculation method is as follows: Operator resource consumption The calculation method is as follows: when the operator is a large model operator, When the operator is a CPU operator, .

[0013] Furthermore, the data preparation cost evaluation system calculates a comprehensive score for each operator. The overall score is calculated by sorting the data and using the following formula: in, Contribute to quality The normalized value, As a penalty; when a quality improvement strategy based on unit resource consumption is adopted, , This represents the normalized value of the operator's resource cost; when a quality improvement strategy based on unit time consumption is adopted, , Represents the normalized value of the operator's time cost; parameters This is the adjustment coefficient.

[0014] Furthermore, the operator selection satisfies the following constraints: in, This represents the accumulated time consumed. Preset time limits for users For the accumulated resource consumption, Preset resource limits for users This is the preset score threshold.

[0015] When the condition is met, the operator will be... Add to the selected set and update the cumulative time and cumulative resources.

[0016] Furthermore, if an operator is selected in two consecutive rounds but the improvement is less than a preset threshold, the system will automatically remove it from the candidate set; the improvement is determined based on the change in the operator's score on its corresponding data quality indicator.

[0017] Furthermore, the operator execution sequence output by the device is an optimal workflow generated under user-preset time and resource constraints.

[0018] The beneficial effects of this invention are: through the above five-dimensional indicator system, this invention can systematically evaluate the scale, completeness, consistency, diversity and non-redundancy of the dataset without training, thereby effectively filtering out low-quality data, reducing ineffective training overhead, and improving model building efficiency and performance.

[0019] By identifying data targets through data quality assessment, the system can optimize processing steps in a targeted manner, focusing on solving the most challenging data problems, thereby maximizing the overall quality of the dataset.

[0020] The data preparation cost assessment system comprehensively measures the resource and time consumption throughout the entire data preparation process, providing a quantitative basis for the acquisition and management of large-scale model training data. Specifically, relying on data quality assessment results, the system intelligently recommends the optimal data preparation steps and workflow under resource and time constraints based on different quality assessment scores. This effectively improves the quality of the training dataset and reduces resource and time overhead during model training. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a framework diagram of the data quality assessment system of this invention.

[0023] Figure 2 This is a flowchart illustrating the cost assessment system of the present invention. Detailed Implementation

[0024] The present invention will now be described in detail with reference to the accompanying drawings. Unless otherwise specified, the features of the following embodiments and implementations can be combined with each other.

[0025] The present invention provides a data quality and preparation cost assessment device for domain-wide large-scale models, comprising a data quality assessment system and a data preparation cost assessment system; A data quality assessment system is used to perform multi-dimensional quality assessments on the training data of the model; based on the quality assessment results, appropriate preparation operators are selected, and a set of candidate operators is obtained as input to the data preparation cost assessment system. A data preparation cost assessment system is used to quantitatively evaluate the resource and time consumption during the data preparation process in order to select the optimal operator workflow.

[0026] In one embodiment, the training data for the model takes image-code pair data as an example. Image-code pair data refers to paired sample data consisting of images and their corresponding codes, typically also including pair metadata for maintaining the correspondence and key chart configurations. By uniformly modeling and analyzing these four types of data, the data characteristics in the chart code generation task can be more comprehensively characterized.

[0027] In this embodiment, the data quality assessment system evaluates the quality of training data using multi-dimensional metrics. The system synthesizes scores from five different dimensions to help assess whether the image code dataset (hereinafter referred to as the dataset) is suitable for supporting the training needs of large-scale domain models. Specifically, see [link to documentation]. Figure 1 This invention, taking into account the characteristics of chart code generation tasks, proposes a data quality assessment system based on the following five dimensions: 1. Data volume dimension: This dimension establishes a data volume metric to measure whether the dataset size is sufficient to support the training needs of the model.

[0028] 2. Data intrinsic quality dimension: This dimension includes syntax checking metrics and structural integrity metrics, which are used to determine whether the ECharts code can be successfully compiled and run, and to check whether the chart configuration fields are complete, ensuring that the semantics of the generated charts are clear.

[0029] 3. Data Representation Quality Dimension: This dimension includes consistency metrics and missing rate metrics, which are used to measure the degree of matching between the original chart image and the generated code rendering image, and to determine whether there are missing paired metadata.

[0030] 4. Data Context Quality Dimension: This dimension includes type balance indicators and structure diversity indicators, which are used to evaluate whether the distribution of different chart types is balanced and whether the configuration structure has sufficient diversity, respectively.

[0031] 5. Data Redundancy Dimension: This dimension establishes code redundancy indicators and image redundancy indicators to detect whether there are too many redundant samples in the dataset, so as to avoid overfitting or inefficient learning during model training.

[0032] Through the above five-dimensional indicator system, this invention can systematically evaluate the scale, completeness, consistency, diversity, and non-redundancy of datasets without training, thereby effectively filtering out low-quality data, reducing ineffective training overhead, and improving model building efficiency and performance.

[0033] Next, we will explain in detail the indicators and calculation methods of the data quality assessment system in each dimension.

[0034] Data volume dimension: In terms of data volume, the data quality assessment system uses a data volume metric to measure whether the size of the dataset is sufficient to support the training needs of large-scale domain models. The score for this metric is calculated using the following formula: In the above formula, data_size represents the number of samples in the current dataset; min_size represents a preset minimum data size threshold. When the number of samples in the dataset is equal to or less than this threshold, the data size index score is adjusted. Recorded as 0; max_size represents the preset maximum data volume threshold. When the number of samples in the dataset is equal to or greater than this threshold, the data volume index score is... Recorded as 100.

[0035] By introducing a logarithmic function to map the data volume, a smooth transition can be achieved between data samples of different orders of magnitude, avoiding excessive differences in scores between large-scale and small-scale datasets, thereby improving the sensitivity and adaptability of the metric. In practical applications, min_size and max_size can be configured according to the specific training task and the scale requirements of the target model. For example, in chart code generation tasks, an appropriate threshold range can be determined based on the model parameter size and the number of chart categories. Those skilled in the art can adjust these thresholds according to the actual situation to ensure that the scoring results truly reflect the degree to which the data volume supports model training.

[0036] It should be noted that the above calculation method is not limited to using a logarithmic function for mapping; other monotonically increasing normalization functions can also be used to achieve the same purpose. This embodiment does not limit the form of this function.

[0037] Intrinsic quality dimensions of data: In terms of intrinsic data quality, the data quality assessment system includes two important metrics: syntax detection metrics and structural integrity metrics. These two metrics are used to evaluate the legality of code and the completeness of chart configurations within the data.

[0038] Syntax detection metrics: The purpose of these metrics is to ensure that the ECharts code in the training data is syntactically valid, thereby guaranteeing that the code generated after model training is executable.

[0039] Specifically, during execution, the system iterates through all code files in the specified directory. These files typically store chart configuration information in JSON format. Each JSON file is encapsulated into JavaScript variable declarations, forming a JS expression that the compiler can parse. The system then calls the Node.js engine and uses static syntax validation commands to check these JS expressions. For each code file, the system records its compilation result; if the syntax validation passes, it is recorded as a valid file; otherwise, it is recorded as an invalid file.

[0040] In the statistical process, the final score of the grammar detection index Calculate using the following formula: This calculation method yields the overall pass rate of the current dataset in terms of grammatical executableness. Since this metric relies on the actual compilation process, it accurately reflects potential syntax errors and illegal structures in the data, effectively filtering out unqualified samples before the data enters the training phase. Those skilled in the art can choose different versions of the Node.js engine or replace it with other static code analysis tools as needed; this invention does not limit this choice.

[0041] Structural integrity metric: This metric is used to check whether the chart configuration file contains the necessary key fields and their nested sub-items to ensure that the generated chart structure is clear and semantically complete.

[0042] Specifically, the system first extracts the target configuration template, which consists of predefined standard configuration items from a Markdown document and is categorized and stored according to different chart types. Then, the system reads the contents of the generated configuration file, which typically represents the actual code configuration of the ECharts chart in JSON format. The system uses a recursive matching method to compare the generated configuration file with the standard template in terms of structure and key-value correspondence, statistically analyzing whether each configuration item matches the standard template layer by layer. "Match" means that all configuration items marked as required in the template exist in the generated configuration, and the hierarchical path and key name are consistent, and the value type conforms to the template constraints and is not empty. "Mismatch" means any of the following: missing required items, inconsistent hierarchical or key names, incorrect value types, or empty required items. Optional items are not included in this metric.

[0043] During the statistical process, the score of the structural integrity index... Calculate using the following formula: For the entire dataset, the system calculates a completeness score for each configuration file and uses the average of these scores as the final evaluation result. This method effectively identifies missing configuration items or incomplete structural definitions, thus preventing the model from learning from non-standard data samples. Those skilled in the art can adjust the definition of the standard template according to different task requirements, such as introducing finer-grained field classifications or using a hierarchical configuration description language; this invention does not limit this.

[0044] Data represents quality dimensions: In terms of data representation quality, the data quality assessment system includes two important metrics: consistency metrics and missing rate metrics. These two metrics are used to evaluate the degree of matching between images and code in the data, as well as the completeness of paired metadata.

[0045] Consistency Metric: This metric measures the degree of matching between the original chart image and the generated ECharts configuration code rendering. The metric's score is based on the similarity between the images to ensure visual consistency between the chart image and the generated code.

[0046] Specifically, the system first dynamically executes the JSON configuration and generates a rendered image using tools such as Playwright or ECharts. Then, the system uses computer vision techniques (such as the NCC algorithm) to calculate the similarity between the generated image and the original image. NCC (Normalized Cross-Correlation Coefficient) is a commonly used method for measuring image similarity. For each pair of original and rendered images, the system calculates the NCC value to arrive at a score.

[0047] Consistency index score Calculate using the following formula: Where A and B represent the pixel values ​​of the original image and the generated image, respectively. and It is the mean of the two images. and This is the standard deviation of the images. The formula calculates the similarity between images and derives the final score based on the similarity value.

[0048] This calculation method allows the system to evaluate the visual consistency between the generated image and the original image, thereby effectively judging the model's performance in image generation tasks. It should be noted that the similarity calculation method can be flexibly adjusted to adapt to the requirements of different tasks, and this invention does not limit it in this regard.

[0049] Missing data rate metric: This metric aims to assess the integrity of paired metadata in the data, specifically checking for missing data entries. It reflects data integrity issues by statistically analyzing the proportion of missing metadata in a dataset.

[0050] Specifically, the system first reads the paired metadata (such as chart configuration information in a JSON file) from the dataset and checks whether each data entry contains all the necessary fields. If some entries are missing key fields, the system marks them as missing data (otherwise they are recorded as complete data) and calculates the proportion of missing data. For each dataset, a score is awarded for the missing data rate metric. Calculate using the following formula: This calculation formula determines the dataset's completeness score by comparing the number of complete rows to the total number of rows. A higher missing field rate and lower completeness score in a dataset negatively impacts the overall data quality assessment.

[0051] This calculation method enables the system to comprehensively assess the completeness of the dataset and effectively screen out samples with missing information, thus laying the foundation for the data preparation process and helping to effectively filter and optimize data in subsequent processing stages, thereby improving data quality and the reliability of model training. This method can adjust the definition and criteria of missing fields according to the requirements of specific tasks and domains; this invention does not impose any limitations on this.

[0052] Data context quality dimension: In terms of data context quality, the data quality assessment system includes two important metrics: type balance and structural diversity. These two metrics are used to evaluate whether the distribution of chart types in the dataset is balanced and the diversity of configuration items.

[0053] Type balance metric: The purpose of this metric is to evaluate whether the different chart types in the dataset are evenly distributed, so as to ensure that the model can fully learn the characteristics of various chart types during training, thereby improving the model's generalization ability.

[0054] Specifically, the system first extracts all chart types from the dataset and counts the frequency of each chart type. Then, it calculates the relative frequency (i.e., sample proportion) of each chart type. By calculating the Shannon entropy for all chart types, it measures the balance of the chart type distribution. Shannon entropy measures the degree of disorder or diversity in a dataset; a higher entropy value indicates a more even distribution of chart types, while a lower entropy value indicates a higher concentration of a particular chart type.

[0055] The score for the type balance index is calculated according to the following steps: 1. Chart Type Extraction: Extract all chart types from the metadata and count the frequency of occurrence of each type; 2. Entropy Calculation: The Shannon entropy formula is used to measure the distribution of chart types.

[0056] The formula for calculating Shannon entropy is: Where n is the total number of chart types, It is the proportion of the nth chart type.

[0057] 3. Normalized Score Calculation: The type balance score is calculated by normalizing the ratio of the entropy value to the maximum entropy value. The score range is from 0 to 100, and the formula is as follows: Where n is the total number of chart types, and H is the Shannon entropy value calculated using the formula above.

[0058] Using the above calculation method, the system can evaluate the balance of chart types in the dataset. A higher score indicates a balanced distribution of chart types and good diversity in the dataset; a lower score indicates that some chart types dominate the dataset, which may lead to insufficient learning of certain types by the model during training.

[0059] Structural diversity metric: The purpose of this metric is to reflect whether the configuration structure of the generated chart has sufficient diversity, ensuring the differences between different configuration items in the dataset, so as to improve the model's generalization ability in the chart generation task.

[0060] Specifically, the system first extracts field existence features from the chart configuration file to generate a "field existence feature matrix". Each field is represented by "1" if it exists in the configuration file, and "0" if it is missing. Each row of the matrix represents a chart configuration file, and each column represents a field. The element value in the matrix is ​​whether the field exists.

[0061] The system calculates the structural diversity score through the following steps: 1. Field extraction and feature matrix construction: Extract field names from all chart configuration files (JSON format) and construct a "field existence feature matrix" to ensure that the field order is consistent across files.

[0062] 2. Dimensionality Reduction and Visualization: Principal Component Analysis (PCA) is used to reduce the dimensionality of high-dimensional data, mapping field features to a two-dimensional space, which facilitates subsequent clustering and analysis.

[0063] 3. Automatic Clustering (KMeans): The KMeans clustering algorithm is used to perform cluster analysis on the dimensionality-reduced data, automatically determining the optimal number of clusters K and analyzing the differences between different clusters. KMeans clustering helps the system group charts with similar field configurations into one category and identify differences between different configurations.

[0064] 4. Diversity score calculation: The diversity score of the dataset is calculated using the Davies-Bouldin index (DBI) and silhouette score.

[0065] Finally, the system calculates the structural diversity score according to the following formula: Where n is the total number of chart configuration fields in the dataset, DBI is the Davies-Bouldin index, and Silhouette Score is the silhouette coefficient.

[0066] This calculation method allows the system to assess the diversity of chart configuration options. A higher score indicates greater diversity in chart configurations, resulting in more varied chart structures and improving the model's generalization ability.

[0067] Data redundancy dimension: In terms of data redundancy, the data quality assessment system uses code redundancy index and image redundancy index to evaluate whether there are too many redundant samples in the dataset, so as to avoid overfitting or inefficient learning of the model during training.

[0068] Code duplication metric: This metric aims to assess whether duplicate code structures exist in a dataset. By identifying and removing redundant code, it ensures higher diversity in the training data. This metric calculates the code duplication rate by comparing standardized hash values.

[0069] Specifically, the system first standardizes each code configuration in the dataset to ensure structural consistency. Then, it uses the MD5 hash algorithm to hash the standardized JSON configuration files, obtaining a hash value for each file. For files with the same hash value, the system determines whether these files belong to a duplicate code structure.

[0070] The proportion of non-duplicate files is used as the code duplication score, calculated using the following formula: Here, the number of non-duplicate code files represents the number of code files after removing duplicates, and the total number of code files represents the total number of code files in the dataset. A higher score indicates less redundant code in the dataset, allowing the model to learn more diverse features more efficiently during training.

[0071] This method enables the system to effectively eliminate redundant code, improve data quality, and ensure code diversity during the training process.

[0072] Image duplication metric: This metric aims to assess whether duplicate images exist in a dataset, thereby ensuring that the model can learn a sufficiently diverse range of image features. It uses a perceptual hashing algorithm (pHash) to calculate the similarity between images, thus identifying image duplication.

[0073] Specifically, the system uses the ImageHash library to generate a perceptual hash value for each image. Perceptual hash values ​​capture the visual features of an image; even visually similar images with differences in color, brightness, etc., can still generate similar hash values. The system determines whether the hash values ​​of two images are similar; if the similarity is below a preset threshold, the two images are considered duplicates.

[0074] Image repeatability score is calculated using the following formula: Here, the number of non-duplicate images represents the number of images after removing duplicates, and the total number of images represents the total number of images in the dataset. A higher score indicates fewer redundant images and greater image diversity in the dataset.

[0075] This method effectively removes duplicate images, enhances the diversity of the training dataset, and avoids overfitting or learning bias caused by duplicate images.

[0076] Based on each dimension and indicator in the data quality assessment, appropriate preparation operators are selected for processing. These preparation operators employ corresponding data optimization and repair measures to address different types of data quality problems, thereby improving the overall quality of the dataset.

[0077] To address this, the proposed operator library provides a variety of scalable operator types, aiming to cover data quality issues across different dimensions. It should be understood that the operator library shown in the table below is only one specific implementation of this invention. The operator library can be extended according to different application scenarios, such as introducing operators for removing redundant samples or for automatically generating metadata. This invention does not limit the scope of such extensions.

[0078] Table 1: Correspondence between preparation operators and data quality indicators In this embodiment, the data quality assessment system evaluates the data using the following five dimensions of indicators, with the final score for each dimension being the average of the scores of its subordinate indicators.

[0079] Data volume dimension: The score is calculated directly from the data volume metric.

[0080] Intrinsic quality dimension of data: The score is the average of the grammar check and structural integrity indicators.

[0081] Data quality dimension: The score is the average of the consistency and missing rate indicators.

[0082] Data context quality dimension: The score is the average of the type balance and structural diversity indicators.

[0083] Data redundancy dimension: The score is the average of the code redundancy and image redundancy indicators.

[0084] In this embodiment, see Figure 2 The data preparation cost evaluation system is used to quantitatively evaluate the time and resource consumption of candidate preparation operators, and automatically selects the optimal operator workflow under user-preset constraints. The system input is a set of candidate operators. , where each operator It has the following parameters: processing time for a single record. Number of records to be processed Resource costs Target weights and user-preset time limits and resource cost constraints .

[0085] 1. The method for calculating time cost is as follows: in, Operator Total time consumed.

[0086] 2. The method for calculating resource costs is as follows: For large model operators: For CPU operators: The above price parameters can be adjusted according to the specific hardware environment.

[0087] 3. The quality contribution index is defined as follows: in, It is calculated based on the weights of the various data quality indicators affected by the operator, and is used to measure the contribution of the operator to the overall improvement of the dataset quality; the data quality indicators include data volume indicators, syntax detection indicators, structural integrity indicators, consistency indicators, missing rate indicators, type balance indicators, structural diversity indicators, code duplication indicators, and image duplication indicators.

[0088] In one alternative implementation, the data quality indicator weights can be obtained through normalization, for example: in, Indicates the first The score values ​​of each data quality indicator, This represents the sum of all data quality indicator scores.

[0089] The above calculation method ensures that the sum of all weights is 1, thus enabling comparable allocations between different quality indicators.

[0090] In other alternative implementations, the weights can also be obtained through expert experience assignment, task relevance analysis, user-preset parameters, or automatic estimation by machine learning models; this invention does not limit this method. (Comprehensive Score) The calculation method is as follows: First of all , , Normalization: Then, a comprehensive score is calculated based on the chosen strategy: If a quality improvement strategy based on unit resource consumption is adopted, then If a quality improvement strategy based on unit time consumption is adopted, then parameter This is an adjustment factor used to balance the weights of quality contributions and penalties. This parameter is set based on the relative importance of resources / time and quality improvement in the application scenario. For example, if... This indicates a complete focus on quality, while ignoring the consumption of resources / time; if This indicates a complete focus on resource / time consumption without considering the impact on quality improvement. In application, The value can be freely adjusted within the range of 0 to 1 according to requirements to balance the relationship between resources / time and quality.

[0091] Quality improvement strategy per unit of resource consumption: Under this strategy, the system calculates the quality improvement index of each preparation operator per unit of resource consumption. The system prioritizes the operator with the lowest resource consumption to ensure maximum efficiency under resource constraints.

[0092] Quality improvement strategy per unit time consumption: Under this strategy, the system calculates the quality improvement index of each preparation operator per unit time consumption, and prioritizes the operator with the least time consumption in order to achieve the maximum quality contribution within a specified time.

[0093] 4. The operator selection process is as follows: (1) System initialization cumulative time Accumulated resources Selected operator set .

[0094] (2) Calculate for all candidate operators And sort them according to their scores from highest to lowest.

[0095] (3) Check the sorted operators in turn. If they satisfy the condition... Then select the operator and add it to the set. Update cumulative time and cumulative resources: (4) Execute the operator and update the dataset, re-evaluate the quality, obtain new operator parameters, and then proceed to the next round of calculation.

[0096] (5) If an operator is selected in two consecutive rounds but the improvement is less than the set threshold, the operator is marked as invalid and removed from the candidate set. The improvement of the operator refers to the change in the score of the operator on its corresponding indicator, which is calculated as the ratio of the new score to the old score for each indicator. If the operator affects multiple indicators, the average of the improvement of these indicators is taken.

[0097] (6) Repeat the above process until the user-preset time or resource limit is reached, or the candidate operator set is empty.

[0098] The final output operator execution sequence is the optimal workflow that satisfies the constraints.

[0099] Through the above implementation methods, the cost evaluation system can automatically select the optimal preparation operator sequence under resource and time constraints, achieving joint optimization of dataset quality improvement and cost control. The method described in this invention avoids the uncertainty and inefficiency of manual selection, and can obtain maximum quality benefits within a limited cost, thereby providing high-quality datasets for large model training.

[0100] The above embodiments are only used to illustrate the design concept and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, all equivalent changes or modifications made based on the principles and design ideas disclosed in the present invention are within the protection scope of the present invention.

Claims

1. A data quality and preparation cost evaluation device for large domain models, characterized in that, include: A data quality assessment system is used to perform multi-dimensional quality assessments on the training data of a model. Based on the quality assessment results, the appropriate preparation operators are selected, and the candidate operator set is used as the input of the data preparation cost assessment system. A data preparation cost assessment system is used to quantitatively evaluate the resource and time consumption during the data preparation process in order to select the optimal operator workflow.

2. The apparatus according to claim 1, characterized in that, The data quality assessment system evaluates the quality of the model's training data based on the following five dimensions of indicators: (1) Data volume dimension; (2) Data intrinsic quality dimension; (3) Data representation quality dimensions; (4) Data context quality dimension; (5) Data redundancy dimension.

3. The apparatus according to claim 1, characterized in that, The training data consists of image-code pairs.

4. The apparatus according to claim 3, characterized in that, The data volume dimension evaluates the size of the dataset using data volume metrics; The intrinsic quality dimensions of the data include: Syntax check metrics are used to determine whether code can be successfully compiled and run; The structural integrity metric is used to check whether the chart configuration fields are complete; The data represents quality dimensions including: Consistency metrics are used to measure the degree of matching between the original chart image and the code rendering image; The missing rate metric is used to determine whether there are missing paired metadata. The data context quality dimensions include: The type balance index is used to evaluate whether the distribution of different chart types is balanced; The structural diversity index is used to evaluate the differences in the field existence characteristics of the configuration structure, wherein the differences are calculated by a preset diversity measurement method. The data redundancy dimension includes: Code duplication metrics are used to detect redundant code in a dataset; Image redundancy index is used to detect the presence of redundant images in a dataset.

5. The apparatus according to claim 1, characterized in that, The input to the data preparation cost evaluation system is a set of candidate operators. Each operator It has the following parameters: processing time for a single record. Number of records to be processed Resource costs Target weights and user-preset time limits and resource cost constraints .

6. The apparatus according to claim 5, characterized in that, Operator time consumption The calculation method is as follows: ； Operator resource consumption The calculation method is as follows: when the operator is a large model operator, ； When the operator is a CPU operator, 。 7. The apparatus according to claim 6, characterized in that, The data preparation cost evaluation system calculates a comprehensive score for each operator. The overall score is calculated by sorting the data and using the following formula: ； ； ； in, Contribute to quality The normalized value, As a penalty; when a quality improvement strategy based on unit resource consumption is adopted, , This represents the normalized value of the operator's resource cost; when a quality improvement strategy based on unit time consumption is adopted, , Represents the normalized value of the operator's time cost; parameters This is the adjustment coefficient.

8. The apparatus according to claim 7, characterized in that, Operator selection satisfies the following constraints: ； in, This represents the accumulated time consumed. Preset time limits for users For the accumulated resource consumption, Preset resource limits for users The preset score threshold; When the condition is met, the operator will be... Add to the selected set and update the cumulative time and cumulative resources.

9. The apparatus according to claim 8, characterized in that, If an operator is selected in two consecutive rounds but the improvement is less than a preset threshold, the system will automatically remove it from the candidate set; the improvement is determined based on the change in the operator's score on its corresponding data quality indicator.

10. The apparatus according to any one of claims 1-9, characterized in that, The operator execution sequence output by the device is the optimal workflow generated under user-preset time and resource constraints.

Images