Data evaluation methods and apparatus
By adopting a multi-level evaluation framework and adaptive indicator configuration based on industry attribute information in the data evaluation method, the evaluation challenges of multi-industry and multi-modal data are solved, and the reuse and accuracy improvement of the evaluation system across industries are realized.
Patent Information
- Application Number
- CN202611232809.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-13
- Publication Date
- 2026-11-10
AI Technical Summary
In existing technologies, data quality assessment methods are difficult to adapt to multi-industry and multi-modal data, resulting in high system expansion and maintenance costs, low computing resource reuse rates, and difficulty in guaranteeing the accuracy of assessment results.
A multi-level evaluation framework is adopted, which dynamically determines the evaluation indicators and their weights based on industry attribute information. Industry-differentiated indicators and weights are attached to the first-level evaluation dimension to achieve cross-industry reusability and the adaptability of the evaluation system.
It improves the reusability of the assessment system across multiple industry scenarios, reduces assessment bias, lowers system expansion and maintenance costs, and enhances the accuracy and applicability of assessment results.
Smart Images

Figure CN123070792A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of computer technology, and more particularly to data evaluation methods and apparatus. Background Technology
[0002] In related technologies, data quality assessment typically employs a pre-defined fixed indicator system, using general dimensions such as accuracy and consistency as top-level categories and attaching corresponding fine-grained indicators. The system then weights and aggregates the scores of each level of indicators based on fixed weights to generate a comprehensive score to support data governance decisions. This type of method is suitable for scenarios where assessment rules are relatively stable.
[0003] However, the definition of indicators and weight configuration in the above schemes are usually fixed and bound to business scenarios, making it difficult to dynamically adjust with industry attributes. When facing multi-industry and multi-modal data, multiple heterogeneous instances need to be deployed or the core evaluation logic needs to be frequently modified, resulting in high system expansion and maintenance costs, low computing resource reuse rate, and difficulty in guaranteeing the accuracy of evaluation results. Summary of the Invention
[0004] This disclosure provides a data evaluation method, apparatus, computer-readable storage medium, and electronic device to at least partially address the aforementioned problems in the related art.
[0005] According to one aspect of this disclosure, a data evaluation method is provided, comprising: acquiring data to be evaluated and industry attribute information corresponding to the data to be evaluated; determining evaluation indicators at each level and their weights corresponding to the industry attribute information based on a preset multi-level evaluation framework; wherein the multi-level evaluation framework includes at least one primary evaluation dimension, each primary evaluation dimension is configured with at least one secondary evaluation indicator, and each secondary evaluation indicator is associated with the industry attribute information; the primary evaluation dimension includes one or more of the following: data scale dimension, data quality dimension, data representation dimension, data context dimension, and data redundancy dimension; performing evaluation processing on the data to be evaluated according to the evaluation indicators at each level to obtain the corresponding indicator scores; and aggregating the indicator scores based on the weights to obtain the evaluation result.
[0006] According to one aspect of this disclosure, a data evaluation apparatus is provided, comprising: an acquisition module for acquiring data to be evaluated and industry attribute information corresponding to the data to be evaluated; a determination module for determining evaluation indicators at each level and their weights corresponding to the industry attribute information based on a preset multi-level evaluation framework; wherein the multi-level evaluation framework includes at least one primary evaluation dimension, each primary evaluation dimension is configured with at least one secondary evaluation indicator, and each secondary evaluation indicator is associated with the industry attribute information; the primary evaluation dimension includes one or more of the following: data scale dimension, data quality dimension, data representation dimension, data context dimension, and data redundancy dimension; a first evaluation module for performing evaluation processing on the data to be evaluated according to the evaluation indicators at each level to obtain the corresponding indicator scores; and a second evaluation module for aggregating the indicator scores based on the weights to obtain the evaluation result.
[0007] According to one aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements any of the above methods.
[0008] According to one aspect of this disclosure, an electronic device is provided, including a processor and a memory, wherein a computer program is stored in the memory, and the processor executes the computer program to implement any of the above methods.
[0009] One embodiment of this disclosure provides a data evaluation method, comprising: acquiring data to be evaluated and industry attribute information corresponding to the data; determining evaluation indicators at each level and their weights corresponding to the industry attribute information based on a preset multi-level evaluation framework; wherein the multi-level evaluation framework includes at least one primary evaluation dimension, each primary evaluation dimension is configured with at least one secondary evaluation indicator, and each secondary evaluation indicator is associated with the industry attribute information; the primary evaluation dimension includes one or more of the following: data scale dimension, data quality dimension, data representation dimension, data context dimension, and data redundancy dimension; performing evaluation processing on the data to be evaluated according to the evaluation indicators at each level to obtain the corresponding indicator scores; and aggregating the indicator scores based on the weights to obtain the evaluation result. In this way, by attaching industry-differentiated indicators and weights to the primary dimension, the reusability of the evaluation system in multiple industry scenarios is improved, which helps to reduce the evaluation bias caused by fixed weights and reduces the system expansion and maintenance overhead. Attached Figure Description
[0010] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0011] Figure 1 A schematic diagram of a system architecture is shown in one exemplary embodiment of this disclosure. Figure 2 A flowchart illustrating a method in one exemplary embodiment of this disclosure is shown; Figure 3 This diagram illustrates the overall architecture of a multi-industry adaptive data quality assessment system according to one exemplary embodiment of the present disclosure. Figure 4 A schematic diagram of a double-key routing lookup mechanism in one exemplary embodiment of this disclosure is shown; Figure 5 This diagram illustrates an industry dictionary configuration in one exemplary embodiment of the present disclosure; Figure 6 A schematic diagram of the structure of an electronic device is shown in one exemplary embodiment of the present disclosure. Detailed Implementation
[0012] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.
[0013] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.
[0014] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this disclosure, "a plurality of" means two or more, unless otherwise expressly specified.
[0015] The method in one embodiment of this disclosure can run on a terminal device or a server. The terminal device can be a local terminal device. When the method in the embodiment runs on a server, the method can be implemented and executed based on a cloud interaction system, wherein the cloud interaction system includes a server and client devices. Figure 1 The figure shows a cloud interaction system architecture diagram provided in this disclosure. As shown, the cloud interaction system may include: a client device 10 and a server 20, wherein the client device 10 can be connected to the server 20 via a network 30.
[0016] It should be noted that the information (including but not limited to: user input information, such as information entered by the user into an input box), data (including but not limited to data used for analysis, stored data, and displayed data, such as context code, all code of the current project, service pressure corresponding to operations performed on all code of the current project, and code development status of the current project), and signals involved in the embodiments of this disclosure are all authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with relevant laws, regulations, and standards. For example, the context code, operations performed on all code of the current project, and the service pressure and code development status involved in the operations in the embodiments of this disclosure are all obtained under full authorization.
[0017] According to one embodiment of the data evaluation method of this disclosure, such as Figure 2 As shown, the method may include: Step S210: Obtain the data to be evaluated and the industry attribute information corresponding to the data to be evaluated; Step S230: Based on the preset multi-level evaluation framework, determine the evaluation indicators and their weights at each level corresponding to the industry attribute information; wherein, the multi-level evaluation framework includes at least one primary evaluation dimension, each primary evaluation dimension is configured with at least one secondary evaluation indicator, and each secondary evaluation indicator is associated with the industry attribute information; the primary evaluation dimension includes one or more of the following: data scale dimension, data quality dimension, data representation dimension, data context dimension, and data redundancy dimension; Step S250: Based on the evaluation indicators at each level, perform evaluation processing on the data to be evaluated to obtain the corresponding indicator scores; Step S270: Aggregate the index scores based on the weights to obtain the evaluation results.
[0018] According to one embodiment of this disclosure, by attaching industry-differentiated indicators and weights to the first-level dimension, the reusability of the evaluation system in multiple industry scenarios is improved, which helps to reduce the evaluation bias caused by fixed weights and reduce the system expansion and maintenance costs.
[0019] like Figure 3The diagram shows the overall architecture of a multi-industry adaptive data quality assessment system. The following sections will introduce the system from two dimensions: system modules and the assessment process. The overall architecture, from top to bottom, is divided into a data input layer, an evaluator routing layer, an industry evaluator layer, a five-dimensional primary indicator calculation layer, and an assessment result output layer. Each layer has a clear responsibility and is decoupled through a unified interface. The system layered architecture overview diagram is shown. The data input layer (Dataset, Industry identifier, DataType data modality), the evaluator routing layer (EvaluatorMeta registry, double-key route lookup find(data_type,industry)), the industry evaluator layer (PowerTableEvaluator, DigitalGovEvaluator, MedicalEvaluator, EmergencyAudioEvaluator, PublicTransportEvaluator, HealthcareGraphEvaluator, InternetEvaluator), the five-dimensional primary indicator calculation layer (data volume, intrinsic data quality, data representation quality, data context quality, data redundancy), and the assessment result output layer (five-dimensional primary indicator scores + industry-differentiated secondary indicator detailed scores) are displayed, showing a unified two-layer JSON structure. The data input layer receives the dataset along with its industry identifier and data modality; the evaluator routing layer uses the EvaluatorMeta registry and the find(data_type,industry) double-key routing to find matching industry evaluators; the industry evaluator layer hosts instances of various industry evaluators (PowerTableEvaluator, MedicalEvaluator, EmergencyAudioEvaluator, etc.); the five-dimensional primary indicator calculation layer completes the scoring of five dimensions: data volume, intrinsic data quality, data representation quality, data context quality, and data redundancy; the evaluation result output layer outputs a unified two-layer JSON structure. The core features of this architecture are "unchanging skeleton, replaceable dictionary, and pluggable evaluators".
[0020] The embodiments of this disclosure will now be further described.
[0021] In step S210, the data to be evaluated and its corresponding industry attribute information are obtained. This allows the evaluation system to simultaneously understand the data entity and its industry background, providing a prerequisite for subsequent differentiated selection of evaluation indicators based on industry dictionaries, thus ensuring the adaptive configuration capability of the evaluation framework from the outset.
[0022] For example, the system receives a batch of data to be evaluated from the upstream node of the data preparation pipeline. This data is presented as a time-series monitoring table for the power industry, a standard image file for the medical industry, or a raw audio sample stream for emergency management. The request message accompanying this data carries industry attribute information identifying its domain, such as an industry identifier with a field value of "smart-energy" or "healthcare". The system parses this message to synchronously read the data entity and industry background into the evaluation engine. Optionally, the data to be evaluated includes at least one of time series, images, audio, video, text, or knowledge graphs, and its modality type is determined and synchronously reported by the upstream node.
[0023] Optionally, the data to be evaluated can be represented at the physical level in various modalities such as time-series tables, medical images, emergency audio, traffic videos, government documents, and knowledge graphs. For example, in a smart energy evaluation scenario, the data to be evaluated can be a time-series sampling table containing measurement points such as voltage, current, and power factor, with a time granularity covering different scales from minutes to days. In a healthcare evaluation scenario, the data to be evaluated can be raw image files conforming to standard image formats, or structured relational tables generated after feature extraction. Furthermore, in an emergency management scenario, the data to be evaluated can be raw audio sampling streams, or relational data that has been converted into structured event scoring tables after model inference. It should be noted that the above descriptions of the modalities of the data to be evaluated are only some examples. In actual implementation, other modalities can be integrated according to specific industry needs. This disclosure is not intended to limit the type of data to be evaluated.
[0024] Optionally, industry attribute information is used to identify the industry sector to which the data belongs and is mapped to a corresponding differentiated set of secondary indicators and weights according to the industry dictionary configuration. Optionally, industry attribute information can be stored and maintained through configuration files or request message metadata, and its values can include industry identifiers such as smart energy, healthcare, emergency management, public transportation, internet, and digital government. Each industry dictionary is a nested dictionary structure that maps primary dimension names to secondary indicator names and weights, enabling the same evaluation framework to automatically present differentiated sets of secondary indicators and weight coefficients across different industries. To avoid interruptions in the evaluation process due to missing industry identifiers, when acquiring data to be evaluated, the system prioritizes parsing the preset metadata fields in the data file header to extract industry attribute information; if this field is missing, it further queries the industry identifier parameters reported by the data preparation task pipeline. It should be noted that the above description of industry identifiers is only a partial example, and in actual deployment, it can be dynamically expanded according to the number of industries connected to the platform. This disclosure does not intend to limit the specific form and source of industry attribute information.
[0025] In optional implementations, the data to be evaluated includes one or more of the following: time-series data, image data, audio data, video data, text data, and knowledge graph data. By incorporating various physical representations such as time-series tables, medical images, emergency audio, traffic videos, government documents, and entity relationship graphs into the evaluation object, not only are mainstream data formats across multiple industries covered, but the evaluation system's adaptability to training data from different modalities is also effectively expanded, significantly improving the applicability and industry universality of the evaluation method.
[0026] In practical applications, the data modality of the data to be evaluated can be flexibly varied according to the business attributes of the target industry. For example, in the field of smart energy, the data to be evaluated can be a time-series table reflecting the operating status of generator sets, including time-series sampled values such as voltage, current, and power; in the field of healthcare, the data to be evaluated can be medical images, pathological slide images, or medical knowledge graphs; in the field of emergency management, the data to be evaluated can be emergency audio or monitoring video collected on-site; and in the fields of internet and digital government, the data to be evaluated can be user behavior text, online comments, or government documents. The above-mentioned data modalities can originate from multiple acquisition channels within a single industry, or they can be uniformly accessed as heterogeneous data sources within the same data preparation platform.
[0027] Optionally, the data modality of the data to be evaluated may include time series data, image data, audio data, video data, text data, or knowledge graph data, which is determined based on the business attributes of the target industry and the data source.
[0028] Optionally, the aforementioned data modalities are used to characterize the physical representation type of the data to be evaluated. For example, they can be time-series sampled values reflecting the operating status of equipment, such as voltage and current monitoring meters in a power system; image data used for visual analysis, such as medical images or pathological slides; or audio data recording environmental acoustic information, such as voiceprint collection at emergency sites. In addition, video data, text data, and knowledge graph data can also be used as evaluation objects. Video data can be public transportation surveillance footage, text data can be government documents or user comments, and knowledge graph data can be medical entity relationship networks. It should be noted that the above list is not intended to limit the composition of data modalities; in actual implementation, other physical representations such as 3D point clouds and geographic information rasters can also be included. Considering the high degree of differentiation in the physical representation of training data for large domain models, incorporating the above multiple modalities into the evaluation framework can, on the one hand, meet the data source access needs of different industries, and on the other hand, provide the underlying data foundation for subsequent cross-modal continuous evaluation, thereby enabling the five-dimensional primary indicator skeleton to cover more physical forms of input data.
[0029] Optionally, different data modalities can be mapped to industry attributes to form differentiated evaluation objects. As one possible implementation, time-series data in the smart energy industry is typically voltage and current time series collected by sensors, while time-series data in the medical industry might be patient vital sign monitoring records. Similarly, image data in the medical industry is mostly video footage, while image data in the internet industry might be charts and screenshots. It should be understood that the above mapping relationship is not unique; multiple data modalities may exist simultaneously within the same industry. For example, the emergency management industry needs to process both on-site audio and surveillance video. Furthermore, in the data preparation chain, the original modal data can be converted into a structured relational table after operator processing. This conversion only changes the physical storage form of the data without altering its industry semantic attributes, thereby ensuring the consistency of the evaluation logic across the industry dimension and allowing the same evaluator to be reused based on a double-key routing mechanism.
[0030] In step S230, based on a pre-defined multi-level evaluation framework, evaluation indicators at each level and their weights corresponding to industry attribute information are determined. The multi-level evaluation framework includes at least one primary evaluation dimension, with at least one secondary evaluation indicator configured under each primary evaluation dimension, and each secondary evaluation indicator is associated with industry attribute information. The primary evaluation dimension includes one or more of the following: data scale dimension, data quality dimension, data representation dimension, data context dimension, and data redundancy dimension. Thus, by pre-setting a multi-level evaluation framework and adaptively determining the evaluation indicators and their weights at each level in conjunction with industry attribute information, the evaluation system possesses both a unified top-level abstraction across industries and the ability to adapt to industry-specific differences, thereby effectively improving the accuracy and applicability of the evaluation results.
[0031] In one implementation, the evaluator for the smart energy industry, after reading the industry identifier, loads the corresponding configuration information from a multi-level evaluation framework. It determines secondary evaluation indicators such as time-granularity coverage and trend strength, along with their weight coefficients, under the data scale dimension; secondary evaluation indicators such as time-series stationarity, along with their weights, under the data representation dimension; and secondary evaluation indicators such as missing rate, along with their weights, under the data quality dimension. This completes the adaptive assembly of evaluation indicators and weights at each level. Furthermore, the evaluator for the healthcare industry, under the same multi-level evaluation framework, loads secondary evaluation indicators such as sample sufficiency, sample balance, and disease coverage under the data scale dimension and assigns differentiated weights. This allows the same top-level framework to automatically present differentiated indicator sets and aggregation criteria across different industries.
[0032] Optionally, this multi-level evaluation framework is used to unify the abstract data quality evaluation dimensions and is compatible with industry-specific secondary evaluation indicators.
[0033] Optionally, the aforementioned multi-level evaluation framework is designed for evaluating the quality of training data for large-scale domain models. It includes at least five primary evaluation dimensions: data scale, data quality, data representation, data context, and data redundancy. This framework uses top-level abstraction to converge common quality concerns across industries into a unified framework. This allows different industries to integrate without redefining primary dimensions; they only need to attach secondary evaluation indicators tailored to their specific domain characteristics under each primary dimension. For example, in the smart energy industry, secondary indicators such as time-series stationarity can be attached under the data representation dimension; in the healthcare industry, secondary indicators such as sampling rate consistency and image format compliance can be attached under the same dimension. It should be noted that the above division of the five primary dimensions is merely an exemplary top-level abstraction method. In actual implementation, the names, number, or connotations of the primary dimensions can be adjusted according to specific business needs, and this disclosure does not impose any limitations on this. The core purpose of this framework is to provide stable structural support for the flexible attachment of industry-differentiated indicators while ensuring the comparability and interpretability of cross-industry evaluation results, avoiding the repetitive work of rebuilding the entire evaluation system for each new industry.
[0034] Optionally, the primary evaluation dimensions serve as a consistent top-level evaluation abstraction across industries, providing a unified indicator framework and aggregation criteria for various sectors. These primary evaluation dimensions may include one or more of the following: data scale, data quality, data representation, data context, and data redundancy. Specifically, the data scale dimension assesses the total sample size, category coverage, and category distribution balance of the training data; the data quality dimension assesses data integrity, label consistency, and industry-specific intrinsic quality attributes, such as the clinical rationality of medical images and the signal-to-noise ratio rationality of emergency audio; the data representation dimension assesses the standardization of data at the numerical or signal level, including but not limited to temporal stationarity, sampling rate consistency, format compliance, and scale consistency; the data context dimension assesses the completeness of data metadata, semantic richness, and the compliance and recognizability of industry-specific knowledge; and the data redundancy dimension assesses sample deduplication rate, feature independence, and industry-specific diversity levels, such as case diversity or event scenario diversity. Considering the inherent differences in quality concerns across industries, the five dimensions mentioned above together form a top-level container without industry bias. Each industry can selectively configure secondary indicators under the dimensions based on data characteristics, so that the same framework can cover both the stability assessment of power time series data and the semantic richness assessment of medical images, thereby achieving true cross-industry reuse and unified standardization.
[0035] Optionally, secondary evaluation indicators are attached to primary evaluation dimensions, and their names and weights are dynamically configured based on industry attribute information. Alternatively, secondary evaluation indicators and their weights are configured and maintained through an industry dictionary, which records a nested structure mapping from primary dimension names to secondary indicator names and weights. For example, in the smart energy industry, six secondary indicators can be attached to the data scale dimension: time granularity coverage (weight 0.25), trend strength (weight 0.05), and seasonality strength (weight 0.25). In the healthcare industry, the same data scale dimension is replaced by three secondary indicators: sample adequacy (weight 0.35), sample balance (weight 0.30), and disease coverage (weight 0.35). To avoid evaluation distortion due to missing indicators, when performing weighted aggregation, if all secondary indicators are missing under a primary dimension, the system marks that dimension as unevaluable instead of assigning full marks by default, thus ensuring the authenticity and objectivity of the evaluation results. Furthermore, the configuration information of the aforementioned secondary indicators can be stored in a dynamically loadable configuration file. When adding a new industry, only an industry dictionary and the corresponding indicator calculation module need to be added, without modifying the platform-level evaluation process, which greatly improves the system's scalability and maintenance efficiency.
[0036] like Figure 5 The diagram illustrates an industry dictionary configuration, using three industries as examples: The Smart Energy dictionary uses indicators such as time granularity coverage (0.25), trend strength (0.05), seasonality strength (0.25), dominant frequency strength (0.20), sample balance (0.10), and number of entries (0.15) under the "Data Volume" dimension; the Medical and Health dictionary uses indicators such as sample quantity sufficiency (0.35), sample balance (0.30), and disease coverage (0.35) under the "Data Volume" dimension; and the Emergency Management dictionary uses indicators such as sample quantity sufficiency (0.35), event category coverage (0.30), and scenario balance (0.35) under the "Data Volume" dimension. It is evident that under the same "Data Volume" dimension, the names and weights of the secondary indicators vary across different industries, but all converge within a unified primary dimension framework. The five-dimensional primary framework serves as a constant top-level structure across industries, with industry-specific dictionaries attached downwards. Each dictionary, within a unified dimensional framework, is equipped with industry-specific secondary indicator names and weights, automatically presenting differentiated indicator sets across different industries using the same framework. When an industry lacks a secondary indicator, that indicator is not included in the denominator normalization; when a primary dimension lacks all secondary indicators, the system marks that dimension as unevaluable rather than defaulting to full score. This process ensures that indicators can be gradually added when new industries are introduced, while avoiding falsely high scores due to missing indicators, thus ensuring the authenticity of the evaluation results.
[0037] In an optional implementation, industry attribute information is stored and maintained through configuration files. These configuration files contain configuration records of the various levels of evaluation indicators and their weights corresponding to each industry attribute. By externalizing industry evaluation rules for maintenance through configuration files, the evaluation indicators and weights for each industry can be dynamically adjusted without modifying the system source code, thereby significantly improving the system's maintainability and cross-industry scalability.
[0038] In one implementation, the system maintains separate configuration files for the power, healthcare, and emergency response industries. Taking the power industry as an example, its configuration file includes secondary indicators such as "time-granularity coverage" and "trend strength" under the "data scale dimension," with weights assigned to 0.25 and 0.05 respectively. In the healthcare industry's configuration file, different indicators such as "sample quantity sufficiency" and "disease coverage" are attached to the same dimension, each with different weights. When the platform needs to add the public transportation industry, operations personnel only need to add a new configuration file containing the industry's indicators and weights and import it into the system. Without modifying the routing layer or the source code of the evaluation framework, the registration and activation of the new industry's evaluation capabilities can be completed under the five-dimensional primary framework.
[0039] Optionally, the configuration file is used to externally store the evaluation indicators and weights for each industry in a non-coded form, allowing the evaluation framework to perform differentiated scoring. Optionally, the configuration file can use a nested dictionary structure to organize industry evaluation rules, where the outer key is the name of the first-level evaluation dimension, and the inner key is a mapping record. Each mapping record maps the name of a second-level evaluation indicator to its weight coefficient. For example, in the configuration file for the smart energy industry, second-level indicators such as "time series stationarity" and "sampling rate consistency" can be attached under the "data representation dimension," with corresponding weight coefficients configured. In the configuration file for the healthcare industry, second-level indicators more suitable for medical imaging scenarios, such as "format compliance" and "scale consistency," may be attached under the same dimension. This nested dictionary is loaded into memory during system initialization, and the evaluator directly reads the dictionary entry corresponding to that industry when performing weighted aggregation. With this design, the industry differentiation logic is completely isolated at the configuration file level. The platform layer code only needs to parse the dictionary and traverse and calculate by entry, without needing to write hard-coded indicator and weight branches for each industry. This ensures the unified execution of the evaluation logic while providing flexible definition capabilities for different industry evaluation indicator sets.
[0040] Optionally, in addition to being deployed as static files on the local file system, the aforementioned configuration files can also be stored in a remote configuration center or database table, and uniformly distributed and version-controlled by the configuration management service. In one possible implementation, when operations personnel modify the weight coefficient of an industry or add a secondary indicator through the management backend, the configuration management service pushes the changed configuration record to each evaluation node in real time. The evaluator only needs to reload the configuration before the next evaluation task to take effect, without needing to restart the evaluation framework. It should be noted that when adding an industry, if the configuration file does not contain all the secondary indicators under a certain primary evaluation dimension, the system marks that dimension as unevaluable rather than assigning a default value, thus effectively avoiding false high scores due to missing configurations. This mechanism allows the evaluation framework to dynamically respond to the evolution of industry standards and changes in business needs without modifying the core code, simply by adding, deleting, and modifying configuration files, greatly reducing maintenance costs during continuous operation.
[0041] In step S250, the data to be evaluated is processed according to the evaluation indicators at each level to obtain the corresponding indicator scores. In this way, by using industry-adaptive evaluation indicators to specifically measure heterogeneous data, not only are fine-grained indicator scores matching industry characteristics obtained, but also an interpretable data foundation is provided for subsequent quality aggregation and target localization. In one implementation, for time-series table data in the smart energy industry, the system performs standardized measurement and scoring calculations on each indicator based on the six secondary evaluation indicators already attached to the "data volume" dimension—time granularity coverage, trend strength, seasonality strength, dominant frequency strength, sample balance, and number of entries—to obtain indicator scores matching industry characteristics. In another implementation, for medical imaging data in the healthcare industry, the evaluation process is performed based on three secondary indicators under the "data volume" dimension: sample quantity sufficiency, sample balance, and disease coverage, outputting the corresponding indicator scores.
[0042] Optionally, the evaluation process aims to perform multi-dimensional quality measurement and scoring calculations on the data to be evaluated based on the established evaluation indicators at each level, in order to output the corresponding indicator scores. Optionally, the evaluation process aims to transform industry-specific secondary indicators into quantifiable quality measurement results. As one possible implementation, this process is executed by an evaluator instance in the `evaluate` interface. The evaluator parses the column features or signal features of the data to be evaluated in the current modality and calls semantically equivalent measurement algorithms for each secondary indicator. For example, in the audio modality of the emergency management industry, MFCC (Mel-frequency cepstral coefficients) features are extracted for "event identifiability" to calculate the inter-class and intra-class distance ratios; in the structured modality after table conversion, the measurement is performed using the F-ratio (variance ratio) of inter-group variance and intra-group variance, with both outputs being normalized scores. In this way, the evaluation process not only achieves automated quality diagnosis of heterogeneous data but also ensures semantic consistency before and after modality conversion through dual-algorithm diversion. The evaluator mentioned above can be automatically registered to the global registry by the metaclass, or it can be dynamically injected by the routing layer after a double-key lookup using the data modality and industry identifier. This embodiment does not limit this. It should be noted that the evaluation process can be triggered automatically at a specific stage of the data preparation pipeline, or it can be started in response to a front-end evaluation request.
[0043] Optionally, the indicator score can be a percentage score obtained after standardization mapping of each secondary evaluation indicator, used to characterize the specific quality performance of the corresponding dimension of the data under a unified dimension.
[0044] Optionally, the indicator score, as a direct output of the evaluation process, represents the quality quantification result of the data to be evaluated on a specific secondary evaluation indicator. In one implementation, the score is obtained by linearly mapping the original metric value to the [0,100] interval; for example, the missing rate is inversely mapped to the integrity score, and the signal-to-noise ratio is mapped to the signal quality score. It should be noted that the calculation of the indicator score not only depends on the predefined weights and calculation formulas in the industry dictionary, but is also affected by the data modality: the same secondary indicator is calculated using the first metric method in the original data modality, and is calculated using the semantically equivalent second metric method after structured processing, but the output under both modalities is the same indicator score. In this way, in the data preparation chain, the indicator scores of the original data and the processed data can be directly compared horizontally in the same coordinate system, thereby quantifying the improvement effect of the evaluation operator on data quality. Among them, for multiple indicator scores within a single primary dimension, normalized weighted aggregation can be performed according to the industry dictionary weights to obtain the dimension score under that dimension.
[0045] In an optional implementation, the evaluation data is processed according to the evaluation indicators at each level to obtain the corresponding indicator scores. This includes: calling the evaluator corresponding to the industry attribute information; and using the evaluator to perform measurement processing on the data to be evaluated to obtain the indicator scores. This encapsulates the industry-specific measurement logic into an independent evaluator, and implements the main process call through a unified interface. This allows the evaluation processing to dynamically adapt to industry attributes, improving the evaluation accuracy in multi-industry scenarios while avoiding tight coupling between the main process code and industry logic. In one implementation, after determining the evaluation indicators at each level, the main control module queries and calls the evaluator instance matching the industry based on the industry attribute information. This evaluator encapsulates the calculation logic for all secondary indicators corresponding to that industry. The main control module only needs to pass the data to be evaluated into the evaluator through a unified calculation interface. The evaluator then performs item-by-item scoring on the data according to the measurement rules configured in the industry dictionary, and finally outputs the original scores for each secondary indicator. This eliminates the need to write industry-related conditional branches in the main process, and the industry-specificity of the evaluation is completely isolated within the respective evaluator implementations, making it easy to extend the evaluator only when adding new industries.
[0046] Optionally, the evaluator carries industry-specific indicator calculation logic and performs measurement processing on the data to be evaluated through a unified interface when it receives a call request.
[0047] Optionally, the aforementioned evaluator can be a dedicated computational class for specific industries and modalities. For example, in the smart energy time-series data scenario, the evaluator could be a dedicated processor for parsing power grid state time-series tables; in the emergency management audio scenario, the evaluator could be a dedicated processor for extracting Mel-frequency cepstral coefficients and calculating signal-to-noise ratio (SNR) rationality; and in the medical and health imaging scenario, the evaluator could be a dedicated processor for measuring clinical rationality and consistency with imaging scales. It should be noted that the above description of the industries to which the evaluator belongs is only one example, and this disclosure is not intended to limit the industry coverage of the evaluator. In actual implementation, it can be flexibly added or removed according to business expansion needs. To avoid the problem of adding new industries requiring intrusion into the core framework due to manual registry maintenance, the aforementioned evaluator can be automatically registered in the global registry through a metaclass registration mechanism during system deployment. This registration behavior is automatically triggered by the metaclass when the class definition of the evaluator subclass is interpreted and executed, without requiring the developer to explicitly call the registration function. Each evaluator declares at least two methods: an acceptance / determination method and a metric execution method. The acceptance / determination method determines whether it matches the target modality and industry during the routing phase. The metric execution method receives the dataset to be evaluated and the progress callback interface, calculates scores item by item according to the secondary indicators configured in the industry dictionary, and reports the processing progress in real time, so that the upper-layer pipeline does not need to be aware of the specific industry's computational logic differences. One objective of this embodiment is that, through automatic metaclass registration and unified interface encapsulation, it reduces the risk of code intrusion when adding new evaluators and enables the metric logic of different industries to be flexibly scheduled and run in isolation within the same framework.
[0048] Optionally, to accommodate the continuous evaluation needs before and after modality transformation during data preparation, the same evaluator can declare simultaneous support for multiple modalities. For example, an emergency audio evaluator can process both raw audio waveform data and event scoring table data after structured transformation, and a medical image evaluator can simultaneously support both raw image formats and feature-extracted relational table formats. In this case, the evaluator does not need to be internally split into multiple independent subclasses. Instead, it automatically perceives the current modality by examining the structural features of the dataset and calls semantically equivalent but algorithmically different measurement logics for the same secondary indicator. For example, for the audio modality, Mel-frequency cepstral coefficients are extracted to calculate the inter-class distance ratio, while for the relational table modality, the F-ratio (between-group variance to within-group variance) is calculated. Both outputs are normalized scores. If modality identification is unclear, a pre-defined modality rule base can be used for further matching and verification. It should be understood that although the above two measurement algorithms have different calculation paths, their normalized indicator semantics remain consistent, allowing the evaluation results before and after preparation to be directly compared in the same coordinate system, quantifying the actual improvement of data quality by the operators. Based on this, the evaluator and the industry dictionary adopt a loosely coupled design. The evaluator framework is responsible for organizing the calculation process according to the secondary indicator names and weights given by the industry dictionary. The specific indicator meanings, weight coefficients and calculation formulas are all recorded in the industry dictionary. When adding a new industry, you only need to create the corresponding evaluator subclass and configure the industry dictionary. The system will automatically register the new industry without modifying the routing module or pipeline core code.
[0049] In an optional implementation, before invoking the evaluator corresponding to the industry attribute information, the method further includes: obtaining the modality identifier corresponding to the data to be evaluated; and matching the corresponding evaluator from a preset set of evaluators based on the modality identifier and the industry attribute information. In this way, by obtaining the modality identifier and matching the evaluator with the industry attribute information, both the appropriate instance can be accurately located, and mismatches caused by modality changes can be avoided, thereby effectively supporting continuous cross-modal evaluation and comparison of preparation effects.
[0050] like Figure 4The diagram illustrates a two-key routing lookup mechanism. It shows how dataset + industry identifier + data modality are searched using the `find(data_type, industry)` method. Upon a match, the corresponding evaluator is instantiated (e.g., Table × Smart-energy corresponds to PowerTableEvaluator, Image / Table × Healthcare corresponds to MedicalEvaluator, Audio / Table × Emergency corresponds to EmergencyAudioEvaluator), and the evaluator instance is injected into `Pipeline.evaluate`. The dataset, along with its industry identifier and data modality, is sent to the `find(data_type, industry)` two-key routing lookup module. This module iterates through the `EvaluatorMeta` registry (which automatically registers all evaluators by the metaclass), and upon a match with the corresponding industry evaluator, it is instantiated and injected into the `Pipeline.evaluate` stage.
[0051] In one implementation, after receiving a batch of data to be evaluated, the system first parses its metadata to obtain modal identifiers, such as determining that the current data is of the original audio type and belongs to the emergency management industry. Then, the system uses the audio modal identifier and the emergency management industry attribute information together as search criteria, inputting them into a preset set of evaluators for double-key matching. It iterates through the matching interfaces of each candidate evaluator and returns the first successfully judged emergency management audio evaluator to complete instantiation, which is then injected into the pipeline for subsequent measurement processing. This preset set of evaluators is automatically registered by the metaclass when the evaluator subclass is defined. When adding a new industry, only the corresponding evaluator module needs to be imported to complete the registration; no modification to the core routing code is required.
[0052] Optionally, the modality identifier is used to characterize the physical representation type of the data to be evaluated, and its value includes at least one of time series tables, images, audio, video, text, and knowledge graphs.
[0053] Optionally, the aforementioned modality identifier can specifically be a classification label representing the physical representation type of the data, such as an audio label corresponding to the original audio stream, an image label corresponding to a medical image file, or a table label corresponding to a relational table obtained after structured processing of the original signal. In one scenario, when the original audio data is converted into a structured event scoring table through model inference, its modality identifier is correspondingly converted from an audio label to a table label, but the evaluator remains the same industry instance, thereby ensuring that the two evaluations before and after preparation can be directly compared under a semantically consistent coordinate system of the same index. It should be noted that the modality identifier is not limited to reading the file extension or metadata header; it can also be automatically determined by scanning the column name characteristics of the data table. The system can dynamically identify the current modality type based on the matching results of structural features and statistical features with preset modality rules.
[0054] Optionally, considering that data may undergo multiple modality transformations during the data preparation process, to avoid mismatches by the evaluator due to modality recognition errors, the aforementioned modality identification process can also be comprehensively judged in conjunction with the statistical characteristics of the data. For example, the system can simultaneously calculate the entropy distribution, field type proportion, and missing patterns of the data, and compare their similarity with typical feature templates in the preset modality rule base. If the original medical image becomes a relational table after feature extraction, but retains pixel column name features, the system will prioritize identifying it as a table modality based on the high confidence judgment result of the table type in the statistical feature template, and trigger the dual-algorithm splitting logic within the evaluator to ensure that the index calculation is strictly adapted to the current physical representation. In other words, the modality identifier not only carries the data format category information, but also serves as the primary criterion for the evaluator's routing and measurement method selection.
[0055] Optionally, a pre-defined evaluator set aggregates candidate evaluators from various industries. The system performs a matching search within this set based on two key-value pairs: modality identifier and industry attribute information. Alternatively, this pre-defined evaluator set can be a global registry maintained by a metaclass automatic registration mechanism. Whenever a new subclass of an evaluator for a new industry is loaded, the metaclass automatically registers it in this set during the class definition phase, without requiring developers to explicitly call the registration function or modify the registry center code. Each candidate evaluator in this set exposes a matching interface. At runtime, the dual-key routing module inputs both the modality identifier and industry attribute information into the matching interface of each candidate evaluator, taking the first evaluator that returns a successful match as the target instance. For example, when the modality identifier is audio and the industry attribute is emergency management, only the emergency management evaluator, which declares support for both audio and table modalities, will return a successful match, thus being instantiated and injected into the pipeline; while the public transportation evaluator, which only supports the video modality, will return a failure. This not only achieves precise binding between evaluators and industries / modalities but also provides a routing foundation for subsequent cross-modal continuous evaluation.
[0056] Optionally, besides automatic discovery via the global registry and double-key matching, in scenarios with a large evaluator set, an inverted index or hash mapping can be built for candidate evaluators. The target evaluator can be directly located using a composite key generated from the modality identifier and industry attribute information, reducing traversal overhead. It should be understood that the above matching process is applicable not only to zero-intrusion expansion scenarios for new industries but also to complex situations where there are multi-modal adaptation needs within the same industry. For example, in the smart energy industry, time-series table modalities and knowledge graph modalities correspond to different evaluator subclasses, and the system can still accurately route to the corresponding subclass based on the modality identifier of the current input data. Furthermore, in cases where matching fails or no candidate evaluator returns success, the system can mark the request as unevaluable and report it, avoiding the forced invocation of the default evaluator, which could lead to missing industry-specific indicators and distorted scoring.
[0057] In an optional implementation, based on modal identifiers and industry attribute information, a corresponding evaluator is matched from a pre-defined set of evaluators. This includes: inputting the modal identifiers and industry attribute information into the matching interface of each candidate evaluator; and identifying the evaluator that returns a successful match as the evaluator. By performing traversal double-key verification on the matching interfaces provided by each candidate evaluator, hard-coded branch judgments can be avoided, allowing newly added industry evaluators to be automatically included in the candidate range after registration. This significantly reduces the maintenance cost of framework expansion and decouples the evaluators from the routing logic. In one implementation, the system has pre-registered multiple candidate evaluators, such as a smart energy table evaluator, a medical and health image evaluator, and an emergency management audio evaluator, through a metaclass registration mechanism. When a dataset to be evaluated, carrying audio modality identifiers and emergency management industry attribute information, enters the routing layer, the double-key lookup module sequentially calls the matching interface of each candidate evaluator and passes in the two key values mentioned above. When it traverses to the emergency management audio evaluator, its internal acceptance judgment method verifies that it supports audio modality and emergency management industry, and then returns a matching success signal. The double-key lookup module then instantiates the evaluator and determines it as the evaluator used for the current evaluation task.
[0058] Optionally, each candidate evaluator belongs to a preset evaluator set and is used to receive modality identifiers and industry attribute information and return a matching result. Optionally, the aforementioned candidate evaluators can be the various evaluator classes to be matched that are automatically registered by the evaluator metaclass when the corresponding evaluator subclass is imported by an external module. That is, all registered evaluators recorded in the global registry are considered candidate evaluators during the routing phase. When the double-key routing lookup module receives the modality identifier and industry attribute information carried by the data to be evaluated, the module iterates through the aforementioned candidate evaluators in the registry, passing the modality identifier and industry attribute information to the matching interface of each candidate evaluator one by one to inquire whether it supports the current evaluation task. Considering that the number of industries and modal types involved in the training data of the domain large model may continue to expand with the evolution of business, all registered evaluators are traversed and matched as candidates. On the one hand, this ensures that the evaluators of newly added industries automatically enter the candidate sequence immediately after import, without any manual maintenance of the core routing logic. On the other hand, it also makes it easier for the system to flexibly control the matching priority according to the traversal order, and complete the hit when the first candidate evaluator that declares a match is encountered, thus balancing the accuracy of route lookup and execution efficiency.
[0059] Optionally, in a real-world deployment environment, the aforementioned candidate evaluators can not only be arranged in a fixed order according to their registration, but also dynamically sorted based on their verification status or priority weight corresponding to industry attribute information. For example, for an industry evaluator in the verification stage, its position in the candidate queue can be moved to the back, so that it is only attempted to match when none of the mainstream evaluators match; or two candidate evaluators can be configured for the same industry, with a primary and a backup. When the first evaluator that returns a successful match fails to complete instantiation due to a runtime exception, the system automatically traces back and uses the next matching candidate evaluator as the actual execution object. This multi-level fault-tolerant design can effectively reduce the probability of the entire evaluation task failing due to incorrect registration information or service anomalies of a single evaluator, thereby improving the stability and availability of evaluation task triggering in the production environment while ensuring the ease of expansion of the dual-key routing mechanism.
[0060] Optionally, the matching interface is the call entry point exposed by each candidate evaluator, used to receive modality identifiers and industry attribute information and return the hit results.
[0061] Optionally, the matching interface described above can be an acceptance / determination method declared internally by each candidate evaluator. This method receives two input parameters: a data modality and an industry identifier. It compares the passed-in actual parameters with the supported range declared by the evaluator class itself. If the comparison matches, it returns a positive result indicating a successful match; otherwise, it returns a negative result. It should be noted that the specific implementation of the matching interface is not limited to the naming of this method. In other deployment environments, it can also be represented as a call entry point supporting a decision function or equivalent functionality. Its parameter organization can be either two separate independent fields or encapsulated as a composite key or a structure object. Furthermore, since a candidate evaluator can simultaneously declare support for multiple data modalities and multiple industry identifiers, such as the emergency management audio evaluator declaring during registration that it supports both audio modalities and table modalities as well as the emergency management industry, the matching interface can perform multi-condition OR logic checks during the internal comparison process. As long as the input modal identifier and industry attribute information fall within the intersection of its declared supported range, the double-key routing match is considered successful, and the candidate evaluator is then determined as the target evaluator required for the current data to be evaluated. There is no need to maintain independent matching logic or routing branches for different modalities within the same industry.
[0062] In an optional implementation, the evaluator performs measurement processing on the data to be evaluated to obtain indicator scores, including: identifying the current modality type of the data to be evaluated; determining the corresponding measurement method according to the modality type; and calculating the indicator score based on the determined measurement method. In this way, the evaluator can complete the split calculation under the same secondary indicator semantics by internally sensing the data modality and adaptively selecting the measurement method, making the evaluation results before and after preparation directly comparable and effectively quantifying the actual improvement of data quality by the preparation operator. In one implementation, taking the emergency audio preparation chain as an example, after receiving the dataset to be evaluated, the evaluator first extracts structural features such as data column names and identifies the current modality type: if the original audio field is detected, it determines to use the first measurement method of extracting MFCC (Mel frequency cepstral coefficient) features to calculate the inter-class and intra-class distance ratios; if the converted scoring field is detected, it determines to use the second measurement method of calculating the inter-group variance and intra-group variance F-ratio (the ratio of inter-group variance to intra-group variance), and maps the original measurement value to the corresponding indicator score based on the determined measurement method, so that the two evaluation results before and after preparation can be directly compared.
[0063] Optionally, the modality type is used to characterize the current physical representation of the data to be evaluated, and the evaluator identifies and triggers subsequent metric splits accordingly.
[0064] Optionally, the aforementioned modality type is used to characterize the specific physical representation of the data to be evaluated in the data preparation process. This modality type can encompass both the raw data modality obtained from the initial acquisition, such as unprocessed audio waveforms, medical image pixel matrices, video frame sequences, or sets of triples in a knowledge graph, and the data modality after structured processing, such as event scoring tables with the suffix "_score" generated through model inference, numerical relational tables obtained after feature extraction, and bounding box coordinate sequence tables output after target detection. During the identification process, the evaluator can automatically perceive the current modality type by extracting the structural and statistical features of the dataset column names. Taking the emergency audio processing process as an example, if the dataset column names contain fields related to the original signal, such as audio and sampling rate, it is identified as the raw audio modality; if the column names contain fields related to structured scoring, such as event type, "_score," and confidence level, it is identified as the relational table modality after conversion. By incorporating this modality-aware logic, the evaluator can perform accurate identification internally without relying on externally explicitly input modality identifiers, thereby enhancing the automation and fault tolerance of the evaluation process when facing complex data preparation environments.
[0065] Optionally, besides modality recognition via dataset column names, the evaluator can also determine the modality type by analyzing the statistical distribution characteristics of the data content itself. For example, when all columns of the data to be evaluated are floating-point values and their values are concentrated in a specific range, it can be determined as a relational table modality after feature extraction; when a large binary object or a multidimensional array structure in the time-frequency domain is detected in the data, it can be determined as the original signal modality. Furthermore, in one possible implementation, the evaluator can also perform cross-validation by combining the externally input modality identifier with the internally parsed modality type: if they match, the subsequent measurement method is directly determined; if they do not match, an exception handling process is triggered and a prompt message is generated for verification and correction by maintenance personnel or the upper-level orchestration module. This dual verification mechanism not only improves the recognition accuracy but also provides an effective foolproof barrier against potential risks caused by format conversion anomalies or metadata errors in the preparation chain.
[0066] Optionally, the metric is used to map the scores of secondary indicators with the same name to semantically equivalent but differently implemented algorithm branches based on the current modality type.
[0067] Optionally, the above-mentioned measurement methods may include a first measurement method corresponding to the original data modality and a second measurement method corresponding to the data modality after structured processing. In one embodiment, for the secondary indicator of emergency event identifiability in emergency audio data, when the original audio modality is identified, the evaluator adopts the first measurement method: firstly, extract the MFCC (Mel-frequency cepstral coefficients) feature vector from the audio signal, then calculate the ratio of inter-class distance to intra-class distance between different event categories, and map this ratio to the [0,100] interval as the indicator score; when the relational table modality after conversion is identified, the evaluator switches to the second measurement method: directly group the event categories in the structured scoring table and calculate the F-ratio (the ratio of inter-class variance to intra-class variance), and then map the F-ratio to the [0,100] interval as the corresponding score. It should be noted that although the two measurement methods mentioned above have different underlying calculation paths, their output results converge to the same semantic coordinate system. Therefore, the two evaluations before and after preparation can be directly compared on the same coordinate axis, thereby accurately measuring the improvement of data quality by the preparation operator.
[0068] Optionally, the specific implementation of the above-mentioned measurement methods is not limited to mapping through distance ratio or variance ratio. In the healthcare scenario, for the secondary indicator of the clinical rationality of medical images, the first measurement method under the original image modality can be measured by the confidence distribution output by a lesion detection model based on deep learning; while under the relational table modality after structured processing, the second measurement method can be transformed into a rule verification score of the consistency between the lesion annotation field and the clinical diagnosis code in the table. Alternatively, in the smart energy scenario, for the time series stationarity indicator, the first measurement method under the original time series modality can use the coefficient of variation of the time series signal or the time series stationarity test statistic, while the second measurement method after table transformation can directly calculate the standard deviation ratio within the sliding window of the difference feature column in the relational table. It can be seen that the measurement methods of the same secondary indicator under different modalities can be flexibly defined according to the characteristics of the industry domain. As long as the outputs of the two algorithms are reduced to the same numerical range and point to the same quality semantics, continuous and comparable evaluation before and after preparation can be achieved.
[0069] In optional implementations, the modality types include the original data modality and the data modality after structured processing; the measurement methods include a first measurement method corresponding to the original data modality and a second measurement method corresponding to the data modality after structured processing. Thus, by distinguishing between the original data modality and the data modality after structured processing and configuring semantically equivalent measurement methods, the evaluation results before and after preparation can be directly compared in the same coordinate system, thereby quantifying the actual improvement of data quality by the operator. In one implementation, the data to be evaluated is the original audio stream in the emergency response field, whose time-frequency characteristics exist in the form of unstructured waveforms. At this time, the evaluator calls the first measurement method to extract MFCC (Melbourne frequency cepstral coefficients) and calculate the inter-class and intra-class distance ratios; after the same batch of data is transformed into a structured relation table containing event score columns through model inference, the evaluator automatically senses the modality switch and calls the second measurement method to calculate the F-ratio (F ratio) of the inter-group and intra-group variances. The two evaluations output the same index score, both within the range of zero to one hundred points, which can be directly used for quality comparison before and after preparation.
[0070] Optionally, the raw data modality is used to represent the data form without structure transformation, directly carrying the physical representation of the original domain signal. Optionally, the raw data modality can be in the form of audio waveforms, video frame sequences, medical image pixel matrices, or natural language text character streams without any pre-structured processing. Considering that domain-wide large model training data is often directly stored in the database in its raw media form during the early preparation stage, when the above-mentioned first metric method matches this modality, its calculation logic closely revolves around the physical characteristics of the raw signal. For example, in emergency audio scenarios, a feature space is constructed by extracting MFCC (Mel-frequency cepstral coefficients) and calculating the ratio of inter-class distance to intra-class distance, thereby measuring the identifiability of the sample without destroying the original data media form. It should be noted that the above description of the form of the original data modality is only one example. This disclosure is not intended to limit the carrier form of the original data. In actual implementation, other modalities such as time series tables and knowledge graphs can also be used. In addition to checking whether the column names of the dataset contain media type identifiers, the evaluator can also judge the original data modality based on the magic number of the file header or the metadata tags. This disclosure does not limit this aspect.
[0071] Optionally, the structured data modality is used to represent the tabular form of the original data after operator processing, using a relational structure to carry the extracted semantic information. Optionally, the structured data modality is represented as a relational table containing scoring columns, label columns, and feature columns. For example, in an event scoring table obtained after model inference from the original audio, each column may have a specific suffix to indicate that it was generated through structured processing. The reason for using the second metric to evaluate this modality is that the original media signal has been converted into high-dimensional semantic features or probability scores after operator processing. At this point, continuing to use physical layer algorithms oriented towards waveforms or pixels will not accurately reflect the statistical distribution of the data. Therefore, statistical measures such as the F-ratio (F ratio) are needed to measure the separability of different event categories. It should be understood that the structured processing methods may include, but are not limited to, feature extraction, model inference, signal transcription or vectorization encoding, and the generated results may also be in the form of semi-structured documents or attribute graphs; and the evaluator can identify the modality not only by checking whether there is a structured processing identifier column in the dataset, but also by the user explicitly declaring the modality transition state when submitting the data. This disclosure does not limit this aspect.
[0072] Optionally, the first measurement method is used to directly measure the data to be evaluated in the original data modality at the physical level, and output a semantically standardized index score. Optionally, the first measurement method corresponds to the original data modality, and its specific algorithm can be dynamically selected according to the industry to which the data belongs and the original media type. Considering that the data in the original modality exists in the form of physical signals, the measurement process needs to fully preserve the temporal or spatial structural information of the media. For example, in emergency audio scenarios, the first measurement method may include extracting MFCC (Mel frequency cepstral coefficients) and constructing a feature vector space, and then calculating the ratio of the distance between the centers of each category to the scattering radius of the same type of samples, thereby reflecting the identifiability of different event categories in the original audio. In medical imaging scenarios, the first measurement method may also include the tissue contrast coefficient or lesion edge sharpness index calculated based on the pixel intensity distribution histogram, to measure the intrinsic representation quality of the original image; in addition, in order to avoid evaluation bias due to the limitations of a single algorithm, the above measurement function may also introduce multi-scale feature fusion or regularization constraints. This disclosure is not limited to a specific calculation formula.
[0073] Optionally, the second metric is used to perform statistical measurements on the structured data modality, making its output semantically equivalent to the first metric. Optionally, the second metric corresponds to the structured data modality, and its calculation object has been transformed from the original media signal into structured numerical values; therefore, the algorithm design focuses more on measuring the statistical distribution separation. In the scenario of emergency audio transcoding, the second metric may include calculating the F-ratio (F-ratio) of the between-group variance and within-group variance for each event category on the scoring column. This statistic and the distance ratio under the original modality both mathematically reflect the separability of categories; therefore, both output metric scores with the same name and dimensions. In the video transcoding scenario, the second metric may also include the trace ratio of the between-class scatter matrix and the within-class scatter matrix calculated based on the structured feature columns. It should be noted that the specific formula for the second metric should be determined based on the column distribution characteristics of the structured data, and extreme outliers can be truncated or smoothed to ensure that the output is comparable in the same coordinate system as the first metric, thereby providing a direct quantitative basis for the quality improvement before and after preparation.
[0074] In an optional implementation, identifying the current modality type of the data to be evaluated includes: extracting structural and statistical features of the data to be evaluated; and determining the modality type based on the matching results of the structural and statistical features with preset modality rules. This avoids the risk of misjudgment in complex preparation scenarios using a single judgment method, enabling modality recognition to utilize both data structure regularities and numerical distribution regularities, thereby improving the accuracy and robustness of modality perception in cross-modal continuous evaluation.
[0075] For example, in one implementation, when the data to be evaluated is transformed from raw audio into a structured event scoring table through inference, the evaluator first extracts the structural features of the dataset, such as whether there are scoring-related suffix fields in the column name set and the data type distribution of each field; simultaneously, it extracts statistical features, such as the sampling frequency distribution and mean-variance characteristics of numerical fields. Subsequently, the system compares these two types of features with a preset modality rule, which defines that when column names contain specific suffixes and the numerical distribution exhibits discrete categorical statistical features, the current data is determined to be in a relational table modality after structured processing. If the structural features show that the original waveform array field still exists, and the statistical features show a continuous time-domain sampling distribution, then the original audio modality is matched, thereby determining that the corresponding measurement method should be selected to calculate the index score.
[0076] Optionally, the aforementioned structural features may include a set of column names, a distribution of field types, or format header information, which can be used to provide a basis for modality determination at the data structure level.
[0077] Optionally, the aforementioned structural features are primarily used to capture static information about the data at the organizational level. This may include, but is not limited to, the set of column names in a data table, the distribution of data types in the fields, file format header information, and whether modality-related keywords are included. Taking the audio-to-table conversion scenario in cross-modal continuous evaluation as an example, if the dataset still contains fields named "audio path" or "original waveform array," the structural features directly indicate that it may be in the original audio modality. Conversely, if the set of column names contains a large number of scoring fields or category label fields with scoring-related suffixes, it indicates that the data has been converted to a structured relational table. Considering that intermediate files or nested data structures may appear during data preparation, the extraction of structural features can also cover metadata such as nesting depth and primary / foreign key relationship patterns. It should be noted that the column names mentioned above are just one example; in actual implementation, they can be dynamically expanded according to typical industry data table structure conventions, thereby enabling modality recognition to have sufficient coverage and flexibility at the structural level.
[0078] Optionally, the above statistical characteristics may include numerical sampling frequency, field mean and variance, data distribution skewness, or frequency domain energy distribution, which can be used to provide a basis for modal determination from the perspective of numerical patterns.
[0079] Optionally, the aforementioned statistical features are primarily used to characterize the dynamic patterns of data at the numerical distribution level. These features may include, but are not limited to, the sampling frequency distribution, mean and variance, skewness and kurtosis, frequency domain energy distribution, and the category count proportion of categorical fields. Taking audio data as an example, the statistical features of the original audio modality typically exhibit a continuous temporal sampling distribution and a relatively stable energy spectral density; while the relational table modality after structured processing often presents a discrete category count distribution or a highly concentrated range of score values. To avoid statistical distortion due to individual outliers, the system can perform outlier removal or binning smoothing before extracting statistical features. Furthermore, the granularity of statistical feature extraction can be at the global dataset level, or it can be calculated separately by field or by sample subset and then summarized. It should be understood that the aforementioned statistics can be adaptively selected based on the physical characteristics of the industry in which the data is located. For example, traffic video data focuses more on the statistical distribution of inter-frame differences, while government text data emphasizes word frequency statistical features.
[0080] Optionally, the aforementioned preset modality rules may include a feature-modality mapping table, statistical threshold intervals, or multi-condition joint judgment logic to automatically determine the current modality type. Optionally, the aforementioned preset modality rules are the key decision logic connecting feature extraction results and modality type determination, aiming to map the combination pattern of structured features and statistical features into a clear modality category label. As a possible implementation, the rule can take the form of a conditional decision tree. For example, when an image pixel matrix field is detected in the structural features and the statistical features exhibit a two-dimensional spatial frequency distribution, it is mapped to the original image modality; when the structural features mainly contain floating-point numerical columns and the statistical features exhibit a categorical discrete distribution, it is mapped to a relational table modality after structured processing. Considering that a single feature may be ambiguous, multi-condition joint judgment logic can be set in the preset modality rules, requiring both structural features and statistical features to simultaneously meet their respective condition thresholds before triggering modality confirmation. To avoid misjudgment due to blurred feature boundaries, a confidence mechanism can also be set in the rules. When the partial matching degree between a feature and multiple rules is high, the system can trigger secondary sampling verification or output a pending confirmation label, rather than directly forcing classification. One objective of this embodiment is that, through this dual-feature linkage rule design, the accuracy of modality recognition can be improved on the one hand, and the rule maintenance cost when adding new modalities can be reduced on the other hand.
[0081] Optionally, in addition to the modality rules based on conditional decision trees mentioned above, preset modality rules can also be expressed as feature similarity matching tables or lightweight classification discriminators. For example, the system can pre-build a standard modality template library, with each modality corresponding to a set of typical structural feature vectors and statistical feature vectors. During actual evaluation, the cosine similarity or Euclidean distance between the feature vector of the data to be evaluated and each type of standard template is calculated, and the modality type corresponding to the template with the highest similarity is taken as the current judgment result. If the feature vector is very close to the standard features of multiple templates, a manual review node or an auxiliary discriminant model can be introduced for verification. This disclosure does not limit this. It should be noted that the specific implementation of the above rules is an example. In actual deployment, it can be dynamically adjusted according to the frequency of occurrence of industry data and the determinism of modality transformation paths. For industry scenarios with clear and single modality transformation paths, hard-coded precise matching rules can be used to reduce computational overhead; while for complex scenarios with ambiguous modality boundaries or intermediate transition states, fuzzy matching or probabilistic discriminant models can be used to improve the robustness and fault tolerance of the judgment.
[0082] In step S270, the indicator scores are aggregated based on weights to obtain the evaluation results. This hierarchical aggregation of indicator scores allows for the simultaneous output of fine-grained secondary indicator scores and macro-dimensional scores, making the evaluation results both interpretable and comparable. This facilitates downstream modules in accurately identifying quality defects and developing targeted data preparation strategies. In one implementation, after calculating the scores for each secondary indicator, the system enters the aggregation stage. Taking the healthcare industry as an example, assuming that its sample quantity adequacy score under the "data volume" dimension is 88, sample balance score is 72, and disease coverage score is 90, with corresponding weights of 0.35, 0.30, and 0.35, respectively. The system first weights and sums the secondary indicator scores under this dimension, and after normalizing the denominator, obtains the dimension score for the "data volume" primary dimension. This process is repeated until all five primary dimensions have been calculated. Subsequently, the system can further aggregate the scores of each dimension into a comprehensive evaluation score. Since different industries have different numbers of secondary indicators attached to each primary dimension, the system only includes the actual attached indicators in the denominator during the normalization process, thereby ensuring that the scores of the primary dimensions of different industries are comparable horizontally.
[0083] The above aggregation process is used to weight and aggregate the scores of secondary indicators into scores of primary dimensions based on the industry dictionary weights, and further summarize them into a comprehensive evaluation score.
[0084] Optionally, considering the differences in the number and weight of secondary evaluation indicators across the five primary dimensions for different industries, the above aggregation process, when performing normalized weighted aggregation within each primary evaluation dimension, only includes the actually configured secondary evaluation indicators under that dimension in the denominator calculation. For example, the smart energy industry has configured six secondary evaluation indicators under the data volume dimension, while the healthcare industry has only configured three. When calculating the primary dimension scores for these two industries, the system uses the actual number of indicators they are configured as the normalized denominator, thus avoiding incomparability of dimension scores between different industries due to differences in the number of indicators. Furthermore, to prevent falsely high scores due to incomplete industry dictionaries, when all secondary evaluation indicators are missing under a primary evaluation dimension, the system marks that primary evaluation dimension as unevaluable, excluding it from subsequent comprehensive evaluation score calculations. This processing logic maintains the horizontal comparability of evaluation results between industries on the one hand, and forces the gradual improvement of industry dictionaries through the unevaluable marking mechanism on the other, ensuring that the evaluation conclusions truly reflect the data quality level.
[0085] The aforementioned evaluation results include scores for each primary dimension and detailed scores for each secondary indicator, output in a unified two-layer dictionary structure. Optionally, the evaluation results are carried out in a unified two-layer dictionary format at the data structure level. The outer layer stores the corresponding dimension scores using the primary evaluation dimensions as keys, while the inner layer stores the detailed scores obtained after measurement processing using the secondary evaluation indicators as keys. This structured output allows the downstream target discovery module to directly traverse the evaluation results of any industry without being aware of industry differences. The platform layer can also interpret the data quality levels of different industries through a unified graphical user interface. It should be noted that the aforementioned two-layer dictionary structure is only one example. In actual implementation, the evaluation results can also be expanded to include the weight source, quality threshold status, and peer comparison ranking information for each dimension. This disclosure does not intend to limit the data configuration of the evaluation results. This unified output method not only supports cross-industry comparable quality acceptance but also provides interpretable input basis for the operator recommendation module, enabling the quality improvement direction in the data preparation chain to be accurately quantified.
[0086] In an optional implementation, the evaluation results include: the dimension scores corresponding to each primary evaluation dimension; and the correspondence between each secondary evaluation indicator and its corresponding score under each primary evaluation dimension. This allows downstream modules to directly traverse evaluation results from any industry without needing to perceive industry differences. The platform layer can uniformly interpret the data quality of different industries under the same interface and acceptance rules, significantly improving the cross-industry comparability of evaluation results and downstream consumer convenience.
[0087] In one example, after completing a five-dimensional assessment of medical image data, the system outputs the following two-layer dictionary structure: the outer layer records a score of 75 for the "data volume" dimension and 82 for the "data context quality" dimension; the inner layer details the corresponding scores for indicators such as "sample quantity adequacy" (70) and "disease coverage" (85). After parsing this structure, the downstream target discovery module directly identifies the weakness in "sample quantity adequacy," triggering the corresponding sample augmentation operator recommendation without needing to consider the specific industry to which this indicator belongs. Optionally, the dimension scores are weighted aggregates of the secondary assessment indicators under the primary assessment dimension, reflecting the overall data quality level of that dimension.
[0088] Optionally, the dimensional score is calculated by normalizing and weighting the scores of all secondary evaluation indicators under the primary evaluation dimension according to the preset weight coefficients in the industry dictionary. Its value range can be mapped to 0 to 100 to facilitate cross-industry horizontal comparison. It should be noted that the above normalization logic aims to eliminate the incomparability caused by differences in indicator dimensions across different industries, enabling the time-series stability of the smart energy industry and the clinical rationality of the medical industry to be presented as comparable scores on the same coordinate system. The dimensional score is not only directly presented in the display layer but also serves as the primary basis for the target discovery module to determine whether there are quality shortcomings in that dimension. When the dimensional score is lower than a preset threshold, the system automatically marks that dimension as a target to be addressed.
[0089] Optionally, in practical implementation, in addition to using a weighted arithmetic mean, the aggregation calculation of dimensional scores can also use a weighted geometric mean or a dynamic weighting model based on entropy weighting, depending on the statistical characteristics of different industries. This disclosure is not limited to these. Furthermore, to enhance the interpretability of the evaluation results, the dimensional scores can also simultaneously carry the original scores, weight proportions, and calculation process metadata of each secondary evaluation indicator under that dimension when outputting, enabling downstream modules to fully trace the composition details of the scores. This hierarchical disclosure mechanism avoids the problem of a single comprehensive score masking specific quality defects, and also provides more refined decision input for the operator recommendation module, enabling it to accurately recommend data cleaning or sample augmentation operators for secondary indicators with low values, thereby improving the iterative optimization efficiency of the domain data preparation chain.
[0090] Optionally, the correspondence is a mapping structure between each secondary evaluation indicator and its score, used to form inner-layer detailed data in the evaluation results. Optionally, the above correspondence is specifically manifested as a key-value mapping from the name of the secondary evaluation indicator to its measurement score, such as "sample balance" mapping to a score of 85, "number of items" mapping to a score of 72, etc. It should be noted that each key-value pair strictly belongs to a specific primary evaluation dimension, and the key names in the inner dictionary adopt a semantically consistent naming convention to ensure that indicators with the same name maintain semantic alignment at different preparation stages after data modality transformation. This mapping structure can be nested under dimension scores, or it can be encapsulated alongside dimension scores as a unified object; this disclosure does not limit its specific serialization format. In practical applications, this detailed correspondence gives the quality evaluation results good auditability and traceability, allowing downstream modules to quickly locate the root cause of quality defects without parsing the original data.
[0091] Optionally, the output format of the correspondence can be a flat key-value list or a multi-level nested dictionary structure. If only some secondary evaluation indicators exist under a certain evaluation dimension, only the calculated indicators and their corresponding scores will be output. Missing secondary evaluation indicators will not be filled in to prevent false full scores from misleading users. Similarly, if a secondary evaluation indicator is calculated using a substitute algorithm due to modality transformation, its score output unit can also include an algorithm identifier so that downstream modules can identify whether the score was obtained in the original modality using the first metric method or in the transformed modality using the second metric method, thereby ensuring the rigor of the comparison evaluation before and after preparation.
[0092] In an optional implementation, the indicator scores are aggregated based on weights to obtain the evaluation results. This includes: within each primary evaluation dimension, weighted aggregation of indicator scores is performed to obtain the dimension score for the primary evaluation dimension; and the scores for each dimension are aggregated to obtain the comprehensive evaluation score. In this way, by weighted aggregation within dimensions followed by cross-dimensional aggregation to obtain the comprehensive evaluation score, the differentiated weights across industries are realized within a unified framework, and downstream modules can easily identify quality weakness dimensions and trigger targeted preparation strategies based on the hierarchical results.
[0093] For example, in one implementation, when the data volume dimension of the smart energy industry is attached with six secondary indicators, namely time granularity coverage (weight 0.25), trend strength (weight 0.05), seasonality strength (weight 0.25), dominant frequency strength (weight 0.20), sample balance (weight 0.10), and number of entries (weight 0.15), the system first multiplies the indicator scores of each secondary indicator by their corresponding weights and sums them, then divides them by the sum of the weights of all attached indicators under that dimension to complete the normalization, thus obtaining the dimension score under that primary evaluation dimension. Subsequently, the system calculates the dimension scores of the four dimensions of data intrinsic quality, data representation quality, data context quality, and data redundancy in the same way, and then aggregates the five dimension scores again to finally obtain a comprehensive evaluation score covering all dimensions, which is then consumed by the downstream target discovery module.
[0094] Optionally, weighted aggregation normalizes and sums the scores of secondary indicators within a dimension according to industry dictionary weights, generating a base dimension score for cross-dimensional aggregation. Optionally, considering the significant differences in the number and weight distribution of secondary evaluation indicators attached to the same primary evaluation dimension across different industries, the above weighted aggregation employs a normalization strategy. This aims to eliminate the incomparability of scores caused by variations in the number of indicators or the total weights. Specifically, for any primary evaluation dimension, the system first obtains all effective secondary evaluation indicators attached to that dimension and their corresponding weight coefficients. The system multiplies the score of each secondary evaluation indicator by its corresponding weight coefficient and sums the results to obtain a weighted total score. Subsequently, this weighted total score is divided by the sum of the weight coefficients of all effective secondary evaluation indicators under that dimension to complete normalization, ultimately obtaining a dimension score falling within a unified dimension. This approach ensures horizontal comparability of scores across different industries under the same primary evaluation dimension. On the other hand, it eliminates the need to recalibrate all weights when adding or removing secondary evaluation indicators; only local adjustments to the industry dictionary are required for the changes to take effect. This balances the stability of the evaluation results with the flexibility of industry adaptation.
[0095] Optionally, besides normalized linear weighting based on weight coefficients, the above aggregation method can be upgraded to threshold-truncated weighted aggregation or hierarchical weighted aggregation according to actual business needs. As one possible implementation, threshold-truncated weighted aggregation truncates the scores of each secondary evaluation indicator before normalization, for example, by forcibly constraining abnormally high scores exceeding a preset upper limit within the boundary range to avoid excessive influence of a single extreme indicator on the dimension score. As another implementation, hierarchical weighted aggregation further divides the secondary evaluation indicators into core indicator groups and auxiliary indicator groups. Weighted aggregation is first performed within each group to obtain the group score, and then a secondary aggregation is performed on the two group scores using different inter-group weights to obtain the final dimension score. It should be understood that regardless of the weighting method used, the technical purpose is to converge industry-differentiated secondary evaluation indicators into a unified primary evaluation dimension framework, enabling horizontal comparison of data quality evaluation results from different industries within the same coordinate system.
[0096] Optionally, the comprehensive evaluation score aggregates the five-dimensional scores and outputs a single quantitative result, representing the overall quality level and providing a unified consumption option for downstream applications. Optionally, the aforementioned comprehensive evaluation score is not simply an arithmetic average of the scores of the five primary evaluation dimensions, but rather a top-level aggregation based on a unified aggregation rule preset by the platform layer. This aggregation rule can employ an equal-weighted summation strategy, or it can configure differentiated platform-level weights for different dimensions based on the overall orientation of the current evaluation task. For example, while ensuring data scale, priority can be given to intrinsic data quality and data context quality. Alternatively, a strategy of first classifying and then aggregating can be adopted, that is, first dividing the scores of each dimension into high, medium, and low levels according to preset thresholds, then assigning level scores to different levels, and finally summing the level scores to obtain the comprehensive evaluation score. It should be noted that the output form of the comprehensive evaluation score can be a single value normalized to a preset range, or it can be a structured object with details of the contribution ratio of each dimension; this disclosure does not limit its specific encapsulation form. In one implementation, when a primary evaluation dimension is marked as unevaluable due to the lack of all secondary evaluation indicators, that dimension will be set to null or a preset interpolation strategy will be used when participating in the calculation of the comprehensive evaluation score. This prevents the overall evaluation process from being interrupted due to the incomplete configuration of the industry dictionary, ensuring that the platform layer can continuously output comprehensive evaluation scores with reference value, and supporting the closed-loop operation of the downstream target discovery and operator recommendation modules.
[0097] In an optional implementation, when all secondary evaluation indicators are missing under a certain primary evaluation dimension, the primary evaluation dimension is marked as unevaluable and does not participate in the calculation of the comprehensive evaluation score. By explicitly marking a primary dimension lacking all secondary indicators as unevaluable and excluding it from aggregation calculations, the inclusion of blank dimensions with default full scores in the comprehensive evaluation can be effectively prevented, thereby improving the authenticity and horizontal comparability of cross-industry evaluation results. In one implementation, for example, when the public transportation industry dictionary does not have any secondary evaluation indicators attached to the data redundancy dimension, the evaluator detects that the set of secondary indicators under that dimension is empty and marks the data redundancy dimension as unevaluable. In subsequent aggregation of the platform's comprehensive quality score, this dimension neither participates in the normalized denominator calculation within the primary dimension nor enters the weighted summation of the five primary dimensions, thus avoiding false high scores due to missing dimensions and ensuring horizontal comparability of evaluation results between this industry and industries with complete configurations.
[0098] Optionally, this unevaluable state is intended to characterize the fact that the target's first-level dimension cannot generate an effective metric due to the lack of second-level indicators, thereby blocking its aggregation path to the comprehensive evaluation score.
[0099] Optionally, the aforementioned unmeasurable state can take various forms. It can be explicitly setting the corresponding first-level dimension score field to null in the two-layer dictionary of the evaluation results, writing a dedicated status code in the metadata, or displaying the dimension as a gray placeholder or horizontal bar on the front-end page. This disclosure does not intend to limit this presentation format. Considering that some first-level dimensions may not yet have any second-level indicators attached when progressively integrating new industries, directly treating this dimension as full score or zero score would distort the physical meaning of the overall evaluation. Therefore, an unmeasurable state is introduced to explicitly indicate the invalidity of this dimension. It should be noted that in this state, when the system aggregates the second-level indicator scores into the first-level dimension scores, it does not include this dimension in the denominator for normalization and weighting; similarly, when aggregating the scores of each first-level dimension into the platform's overall quality score, this dimension is skipped, and only dimensions with valid second-level indicator scores are aggregated. Furthermore, this status can also be linked with the downstream target discovery module. When an unevaluable dimension is identified, the target discovery module automatically generates indicator completion suggestions and pushes them to the operator recommendation queue, thereby forming a closed loop for automatic repair of evaluation blind spots in the data preparation chain.
[0100] Optionally, besides triggering an unevaluable state when a primary dimension completely lacks secondary indicators, the system can also set more granular grading rules. For example, when a primary dimension has only a single secondary indicator attached, and that indicator cannot be calculated with a valid score due to missing data, the dimension can be downgraded to a low-confidence state instead of an unevaluable state. In this case, the dimension participates in aggregation but is given a lower weight coefficient, or a confidence label is added next to its score. It should be understood that platform operators can flexibly configure the threshold parameters in the industry dictionary between the low-confidence state and the unevaluable state. For example, it can be set to automatically trigger an unevaluable state when the number of valid secondary indicators is less than two or the coverage weight is less than 30%, and trigger a low-confidence state when the number is higher than this threshold but there are missing indicators. One objective of this embodiment is to avoid directly eliminating dimensions with some measurement capability due to excessive conservatism by configuring differentiated status markers for different degrees of missing data. On the other hand, it can provide data preparation personnel with more granular quality blind spot prompts while ensuring the robustness of the comprehensive evaluation score, thereby balancing the rigor of the evaluation results with the friendliness of the incremental indicator completion process.
[0101] According to one embodiment of the present disclosure, the data evaluation apparatus may include: The acquisition module is used to acquire the data to be evaluated and the corresponding industry attribute information of the data to be evaluated. The determination module is used to determine the evaluation indicators and their weights at each level corresponding to industry attribute information based on a preset multi-level evaluation framework. The multi-level evaluation framework includes at least one primary evaluation dimension, and each primary evaluation dimension is configured with at least one secondary evaluation indicator, and each secondary evaluation indicator is associated with industry attribute information. The primary evaluation dimension includes one or more of the following: data scale dimension, data quality dimension, data representation dimension, data context dimension, and data redundancy dimension. The first evaluation module is used to perform evaluation processing on the data to be evaluated based on the evaluation indicators at each level, and obtain the corresponding indicator scores. The second evaluation module is used to aggregate the index scores based on weights to obtain the evaluation results.
[0102] In this way, by attaching industry-specific indicators and weights to the first-level dimension, the reusability of the evaluation system in multiple industry scenarios is improved, which helps to reduce the evaluation bias caused by fixed weights and reduce the system expansion and maintenance costs.
[0103] The specific details of each part of the above-mentioned device have been described in detail in the method section of the implementation plan. For any undisclosed details, please refer to the implementation plan of the method section, and therefore will not be repeated here.
[0104] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to exemplary embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.
[0105] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure.
[0106] The following is a detailed reference. Figure 6 The diagram illustrates a structural schematic suitable for implementing an electronic device according to embodiments of the present disclosure. The electronic device may include a processor (e.g., a central processing unit, graphics processor, etc.) 1201, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 1202 or a program loaded from memory 1208 into random access memory (RAM) 1203. The RAM 1203 also stores various programs and data required for the operation of the electronic device. The processor 1201, ROM 1202, and RAM 1203 are interconnected via a bus 1204. An input / output (I / O) interface 1205 is also connected to the bus 1204.
[0107] Typically, the following devices can be connected to I / O interface 1205: input devices 1206 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 1207 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; memory devices 1208 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1209. Communication device 1209 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 6 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown, and more or fewer devices may be implemented or have instead.
[0108] In particular, according to one embodiment of this disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, one embodiment of this disclosure includes a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via communication device 1209, or installed from memory 1208, or installed from ROM 1202. When the computer program is executed by processor 1201, it performs the functions defined in the methods described above in various embodiments of this disclosure.
[0109] Figure 6 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.
[0110] This disclosure also provides a computer-readable storage medium in which the methods described in this disclosure can be implemented in hardware or firmware, or implemented as recordable on a storage medium, or implemented as computer code originally stored on a remote storage medium or a non-transitory machine-readable storage medium and subsequently stored on a local storage medium after being downloaded over a network. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium may also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods shown in the above embodiments.
[0111] A portion of this disclosure can be applied to computer program products, such as computer program instructions, which, when executed by a computer, can invoke or provide methods and / or technical solutions according to this disclosure through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, and installation package files. Accordingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions; the computer compiling the instructions and then executing the corresponding compiled program; the computer reading and executing the instructions; or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.
[0112] Although embodiments of the present disclosure have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations all fall within the scope defined by the appended claims.
Claims
1. A data evaluation method, characterized in that, include: Obtain the data to be evaluated and the corresponding industry attribute information of the data to be evaluated; Based on a pre-defined multi-level evaluation framework, evaluation indicators and their weights at each level corresponding to the industry attribute information are determined; wherein, the multi-level evaluation framework includes at least one primary evaluation dimension, each primary evaluation dimension is configured with at least one secondary evaluation indicator, and each secondary evaluation indicator is associated with the industry attribute information; the primary evaluation dimension includes one or more of the following: data scale dimension, data quality dimension, data representation dimension, data context dimension, and data redundancy dimension; Based on the evaluation indicators at each level, the data to be evaluated is evaluated to obtain the corresponding indicator scores; The scores of the indicators are aggregated based on the weights to obtain the evaluation results.
2. The method according to claim 1, characterized in that, The step of performing evaluation processing on the data to be evaluated based on the evaluation indicators at each level to obtain the corresponding indicator scores includes: Invoke the evaluator corresponding to the industry attribute information; The evaluator measures and processes the data to be evaluated to obtain the index score.
3. The method according to claim 2, characterized in that, Before invoking the evaluator corresponding to the industry attribute information, the method further includes: Obtain the modal identifier corresponding to the data to be evaluated; Based on the modality identifier and the industry attribute information, the corresponding evaluator is matched from a preset set of evaluators.
4. The method according to claim 3, characterized in that, The step of matching the corresponding evaluator from a preset set of evaluators based on the modality identifier and the industry attribute information includes: Input the modality identifier and the industry attribute information into the matching interface of each candidate evaluator; The evaluator that returns a successful match is identified as the evaluator.
5. The method according to claim 2, characterized in that, The step of measuring and processing the data to be evaluated using the evaluator to obtain the indicator score includes: Identify the current modality type of the data to be evaluated; Based on the modality type, determine the corresponding measurement method; The score of the indicator is calculated based on a defined measurement method.
6. The method according to claim 5, characterized in that, The modality types include raw data modalities and data modalities after structured processing; The measurement method includes a first measurement method corresponding to the original data modality and a second measurement method corresponding to the structured data modality.
7. The method according to claim 5, characterized in that, The process of identifying the current modality type of the data to be evaluated includes: Extract the structural and statistical features of the data to be evaluated; The modality type is determined based on the matching results between the structural features and statistical features and the preset modality rules.
8. The method according to claim 1, characterized in that, The evaluation results include: The corresponding dimension scores for each of the aforementioned primary evaluation dimensions; and The correspondence between each of the secondary evaluation indicators under each primary evaluation dimension and the corresponding indicator scores.
9. The method according to claim 1, characterized in that, The aggregation of the indicator scores based on the weights to obtain the evaluation results includes: Within each of the primary evaluation dimensions, the indicator scores are weighted and aggregated based on the weights to obtain the dimension scores of the primary evaluation dimensions. The scores of each dimension are aggregated to obtain a comprehensive evaluation score.
10. The method according to claim 9, characterized in that, When all the secondary evaluation indicators are missing under a certain primary evaluation dimension, the primary evaluation dimension is marked as unevaluable and will not be included in the calculation of the comprehensive evaluation score.
