Data set classification evaluation method and device, electronic equipment and readable storage medium

Abstract

This invention discloses a dataset classification and evaluation method, apparatus, electronic device, and readable storage medium. The method includes: collecting raw data from the entire process of a target product; performing metadata sorting on the raw dataset to obtain raw data with metadata information; classifying the raw data hierarchically according to a preset three-level classification system of <modality-use-attribute> based on the metadata information; selecting matching evaluation indicators from a preset evaluation indicator library for different modalities based on the hierarchical classification results, adapting to an optional preset evaluation process, and outputting evaluation results; processing the evaluation results in different levels, generating corresponding application guidance for datasets of different levels, generating a classification evaluation report and pushing it to the production stage; collecting feedback on the actual effect of the evaluation results, and dynamically adjusting the configuration of classification categories and evaluation indicators based on the feedback results. This application is suitable for the classification and evaluation of multimodal datasets throughout the entire process of high-end manufacturing.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a dataset classification and evaluation method, apparatus, electronic device, and readable storage medium. Background Technology

[0002] High-end manufacturing encompasses multiple sub-sectors such as automobile manufacturing, aerospace manufacturing, precision machine tool manufacturing, and high-end electronics manufacturing. Its core characteristics are complex production processes, high precision requirements, and closed-loop production processes. Throughout the entire process, massive amounts of multi-type datasets are generated, including product design data, processing technology data, quality inspection data, and equipment operation data. These datasets are the core support for enterprises to optimize production processes, improve product quality, reduce production costs, and achieve early warning of equipment failures.

[0003] Existing technologies for dataset classification and evaluation mainly fall into two categories: First, general-purpose data classification and evaluation methods. These methods categorize data based on its intended use, employing uniform evaluation metrics (such as data completeness and accuracy) and a single evaluation algorithm to provide homogeneous evaluation for all data types. This approach is suitable for scenarios with relatively simple data types, such as general manufacturing and the internet. Second, industry-specific data evaluation methods. These methods address the specific characteristics of certain fields (e.g., literature). Some technical solutions perform simple classifications and use common comparison and statistical methods for evaluation, focusing primarily on data accuracy. Their limitations include: Using a single-dimensional classification and applying the same general evaluation indicators to different modal data leads to large deviations in evaluation results, which cannot meet the evaluation needs of different modal data in high-end manufacturing and cannot reflect the application value of data in actual production. The classification and evaluation processes are disconnected, and the classification results cannot provide targeted support for the evaluation. The evaluation results are also difficult to be directly applied to the production process, and the value of the data cannot be fully realized. Summary of the Invention

[0004] To address the shortcomings of the existing technologies, this invention provides a dataset classification and evaluation method, apparatus, electronic device, and readable storage medium, employing the following technical solution: Firstly, a dataset classification evaluation method is provided, including: Collect raw data from the entire process of the target product, and perform metadata sorting on the raw dataset to obtain raw data with metadata information; the metadata sorting includes metadata completion, data annotation, and metadata verification, wherein the annotation information serves as a supplement to the metadata; The raw data is classified into three levels according to a preset <modality-purpose-attribute> classification system based on metadata information; each data item is labeled with three levels of classification tags: modality, purpose, and attribute. Based on the hierarchical classification results, matching evaluation indicators are selected from the preset evaluation indicator library for different modal data, and the evaluation results are output by adapting to the optional preset evaluation process. The evaluation indicators include single-modality specific indicators and multi-modality fusion indicators. The preset evaluation process includes single-modality evaluation process and multi-modality fusion evaluation process. The evaluation results are graded and processed, and corresponding application guidance is generated for datasets of different grades. A classification evaluation report is generated and pushed to the production stage. Collect feedback on the actual effects of the evaluation results, and dynamically adjust the classification categories and evaluation indicators based on the feedback.

[0005] Secondly, a dataset classification evaluation device is provided, comprising: The data acquisition and metadata processing module is used to collect raw data from the entire process of the target product, process the raw dataset for metadata, and obtain raw data with metadata information. The metadata processing includes metadata completion, data annotation, and metadata verification, where the annotation information serves as a supplement to the metadata. The data classification module is used to classify the raw data according to a preset three-level classification system of <modality-purpose-attribute> based on metadata information; each data item is labeled with three-level classification tags: modality, purpose, and attribute. The data evaluation module is used to select matching evaluation indicators for different modal data from a preset evaluation indicator library based on the hierarchical classification results, adapt to optional preset evaluation processes, and output evaluation results; the evaluation indicators include single-modality specific indicators and multi-modality fusion indicators; the preset evaluation processes include single-modality evaluation processes and multi-modality fusion evaluation processes. The application module is used to classify the evaluation results, generate corresponding application guidance for datasets of different levels, and generate classification evaluation reports to push to the production process. The feedback optimization module is used to collect actual feedback on the evaluation results and dynamically adjust the configuration of classification categories and evaluation indicators based on the feedback results.

[0006] Thirdly, an electronic device is provided, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, performs the steps of the method as described in the first aspect.

[0007] Fourthly, a readable storage medium is provided, on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method as described in the first aspect.

[0008] The dataset classification evaluation method proposed in this application has the following beneficial effects: This application is adapted for classification and evaluation of multimodal datasets covering the entire high-end manufacturing process.

[0009] A scientific and comprehensive classification system has been constructed, taking data modality as the core classification dimension to achieve deep integration of classification and evaluation, ensuring that data of different modalities and uses can be matched with exclusive evaluation methods and indicators; By integrating the design principles of "change and invariance", we designed differentiated evaluation methods and modality-specific indicators, taking into account both the standardization and relevance of the evaluation, and improving the accuracy of the evaluation results. Supplement the evaluation logic of multimodal fusion data to achieve spatiotemporal alignment and two-way verification of multimodal data, covering the data evaluation needs of high-end manufacturing in all scenarios; It improves classification and evaluation efficiency, enables rapid and real-time processing of massive multimodal data, and is suitable for high-end manufacturing massive data scenarios; A closed-loop mechanism of "evaluation-feedback-optimization" has been established to achieve dynamic optimization of evaluation results, which can be directly applied to the production process, giving full play to the value of data and providing accurate data support for production optimization, quality control and equipment monitoring of high-end manufacturing enterprises.

[0010] The apparatus, electronic device, and readable storage medium corresponding to the dataset classification and evaluation method in this application can achieve the same technical effect, and will not be described in detail here to avoid duplication. Attached Figure Description

[0011] Figure 1 A schematic flowchart illustrating a dataset classification and evaluation method provided in an embodiment of this application; Figure 2 A schematic flowchart illustrating another dataset classification and evaluation method provided in this application embodiment; Figure 3 A system architecture diagram provided for an embodiment of this application; Figure 4 A schematic diagram illustrating the principle of a dataset classification and evaluation method provided in an embodiment of this application; Figure 5 This is a system timing diagram provided for an embodiment of this application. Detailed Implementation

[0012] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the technical solutions in the embodiments of this application are clearly described. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art are within the scope of protection of this application.

[0013] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0014] The steps described in this application and the flowcharts in the accompanying drawings are not necessarily strictly executed according to the step numbers; the execution order of the steps can be changed. Furthermore, certain steps can be omitted, multiple steps can be combined into one step, and / or one step can be broken down into multiple steps.

[0015] This specification provides a dataset classification and evaluation method, and also relates to a dataset classification and evaluation device, a computer device, and a computer-readable storage medium. The following describes each of these in detail with reference to the accompanying drawings and preferred embodiments.

[0016] First, the overall architecture of the dataset classification and evaluation method in this application embodiment will be described as follows.

[0017] This invention adopts a "layered architecture + closed-loop design" to construct a six-level overall architecture consisting of "data input layer - preprocessing layer - classification layer - evaluation layer - application layer - feedback optimization layer". It integrates the core idea of ​​"divide and conquer" and the design principle of "change and invariance" to realize the full-process classification and evaluation of high-end manufacturing datasets from input to application. The functions of each layer are interconnected and work together to ensure the accuracy, efficiency and feasibility of classification and evaluation.

[0018] Data Input Layer: Responsible for collecting various raw datasets generated throughout the high-end manufacturing process, including text modalities (process specifications, inspection reports), image modalities (defect detection images, appearance images), numerical modalities (dimensions, process parameters), time-series modalities (equipment operation data), and multi-modal fusion data (image-text dual modalities, etc.). It supports multiple interface accesses and adapts to the data collection needs of different production stages.

[0019] Metadata organization layer: This layer does not perform any preprocessing on the raw input data, keeping the data as it is. Its core responsibility is to organize the metadata of various raw data input from the data input layer, including metadata completion and simple annotation, annotating existing data, and clarifying key information such as the source, format, and related objects of the data. This provides a clear data foundation for subsequent classification and evaluation work, ensuring the objectivity and accuracy of classification and evaluation.

[0020] Classification layer: A three-layer classification system of "modality-use-attribute" is adopted, with data modality as the core classification dimension. It is further subdivided according to use and attribute, and a dynamic classification adjustment mechanism is introduced to achieve accurate data stratification and ensure that the classification results are highly matched with the evaluation requirements.

[0021] Evaluation layer: Construct a unified evaluation indicator library, including single-modality exclusive indicators and multimodal fusion indicators. For different categories of data, select matching indicators from the indicator library, streamline the exclusive evaluation process, conduct differentiated evaluations, and record evaluation data.

[0022] Application layer: The evaluation results are graded, classified evaluation reports are generated, and targeted application guidance is provided for different evaluation levels, so as to achieve deep integration of evaluation results with production processes.

[0023] Feedback Optimization Layer: Establish an evaluation result feedback mechanism, dynamically adjust classification categories and evaluation indicators based on production application effects, realize continuous optimization of classification evaluation methods, and form a complete closed loop of "input-preprocessing-classification-evaluation-application-optimization".

[0024] Please see Figure 1-4 This application provides a dataset classification evaluation method, including: Step S1: Collect raw data of the entire process of the target product, and perform metadata sorting on the raw dataset to obtain raw data with metadata information; the metadata sorting includes metadata completion, data annotation and metadata verification, wherein the annotation information serves as a supplement to the metadata.

[0025] This step involves metadata processing of the raw dataset collected from the data input layer, but without any preprocessing. Raw data from the entire target product process is collected, including at least product design data, manufacturing process data, quality inspection data, and equipment operation data. Metadata processing specifically includes: Metadata completion: Organize the metadata information of various raw data, complete the missing metadata (such as data collection time, collection equipment, collection process, data format, data volume, etc.), establish a metadata ledger, and ensure that the metadata information of each piece of raw data is complete and traceable; Data annotation: The raw data is annotated with simple annotations using a combination of automated annotation and manual verification. The annotations do not change the content of the data itself. The annotation content includes the data source (such as production workshop, testing process), data type (such as process data, quality data), related products / processes (such as automotive engine parts, cutting process), and data modality (text, image, etc.). The annotation information serves as a supplement to the metadata, which facilitates the rapid identification of data attributes in the subsequent classification process. Metadata validation: Validate the completed and annotated metadata to ensure that the metadata information is true, accurate, and matches the original data, and to avoid metadata annotation errors affecting subsequent classification and evaluation work.

[0026] Step S2: Based on the metadata information, the original data is classified into three levels according to the preset <modality-purpose-attribute> classification system; each data item is marked with three-level classification labels: modality, purpose and attribute.

[0027] Furthermore, the hierarchical classification of the original data according to the <modality-use-attribute> three-level classification system based on metadata information includes: The first layer is classified by data modality, which includes at least text modality data, image modality data, numerical modality data, and time series modality data.

[0028] These four priority core modalities cover data across all scenarios in high-end manufacturing. Text modal data includes process specification documents, inspection reports, production instruction texts, and component instruction documents. Its core characteristic is unstructured text, and its main purpose is to provide process guidance and record inspection results. Image modal data includes images of component appearance defects, real-time images of the processing process, and 3D model renderings. Its core characteristic is visualized pixel information, and its main purpose is defect detection and appearance quality control. Numerical modal data includes dimensional parameters, process parameters, material parameters, and cost data. Its core characteristic is structured numerical data, and its main purpose is accuracy control, process optimization, and cost accounting. Time-series modal data includes time-series data of equipment operating parameters, processing time-series data, and supply chain cycle time-series data. Its core characteristic is time-varying sequence data, and its main purpose is equipment status monitoring and process optimization.

[0029] The second layer further categorizes each type of modal data according to its purpose. Specifically, text modal data includes at least: process guidance text, inspection report text, and production instruction text; image modal data includes at least: defect detection images, appearance quality images, and 3D model images; numerical modal data includes at least: design parameter data, process parameter data, quality inspection values, and cost values; and time-series modal data includes at least: equipment operation time-series data, processing time-series data, and supply chain time-series data. The third layer further categorizes the data for each application by data attributes; specifically, the text modality - process guidance text includes at least: parameter description text and operating specification text; the image modality - defect detection images includes at least: surface defect images and internal defect images; the numerical modality - process parameter data includes at least: cutting parameters, welding parameters and assembly parameters; and the time-series modality - equipment operation time-series data includes at least: speed time-series data, load time-series data and temperature time-series data.

[0030] In practice, secondary classification categories can be added or adjusted according to the personalized needs of different high-end manufacturing scenarios (such as automobile manufacturing and aerospace manufacturing); the first-level modality classification can be flexibly expanded according to the newly added data modalities (such as newly added audio modal equipment noise data) without changing the overall classification logic.

[0031] Step S3: Based on the hierarchical classification results, select matching evaluation indicators from the preset evaluation indicator library for different modal data, adapt to the optional preset evaluation process, and output the evaluation results; The evaluation metrics include single-modal specific metrics and multimodal fusion metrics; the preset evaluation process includes a single-modal evaluation process and a multimodal fusion evaluation process.

[0032] This step combines the principle of "change and constancy" and achieves differentiated evaluation through three stages: "establishing an indicator library, selecting matching indicators, and conducting evaluation".

[0033] Furthermore, in the evaluation index library, the evaluation indicators are configured and stored in a structure of <modal classification-evaluation purpose-indicator name-evaluation method-calculation rule> and support dynamic updates.

[0034] By integrating various data modalities and evaluation indicators corresponding to different evaluation purposes, a dedicated evaluation indicator library for high-end manufacturing datasets is constructed. The library is logically categorized and summarized according to "modal classification - evaluation purpose - indicator name - evaluation method - calculation rules," clarifying the applicable scenarios, operation methods, and calculation standards for each indicator. At the same time, dedicated indicators for multimodal fusion data (with image-text dual modality as the core) are added to address the limitations of single-modal evaluation. The indicator library supports dynamic updates, and indicators can be added according to new modalities and evaluation needs.

[0035] The single-mode specific index includes at least: parameter deviation rate, used to match numerical mode-process parameter data; its calculation formula is: In the formula For the i-th group of actual values, This is the standard value.

[0036] Specifically, the parameter deviation rate index is used to verify the accuracy of process parameters. The evaluation method adopts the standard comparison method, for example, selecting a preset standard cutting speed for a certain cutting process. Ten sets of actual cutting speed data were collected, and the deviation of each set of data from the standard value was calculated using the following formula: in Let i be the actual cutting speed of the i-th group. This is the standard cutting speed. The accuracy of the parameters is then quantitatively evaluated using this.

[0037] The multimodal fusion index includes at least: image-text information consistency rate, used to match image-text bimodal defect detection data; its calculation formula is: .

[0038] The consistency rate of image and text information is evaluated to verify the consistency between defect detection images and corresponding inspection report text. For example, the evaluation method uses machine vision recognition + NLP text comparison, that is, machine vision identifies the defect type and location in the defect image, extracts the defect description information in the text report, and compares the consistency between the two, thereby achieving quantitative evaluation of multimodal data. In addition, for multimodal data such as image-text, audio-text, video-text, and radar-video, multimodal fusion indicators can also include spatiotemporal alignment accuracy, image-text information verification rate, and image-text association coverage rate.

[0039] When selecting matching evaluation metrics, based on the output hierarchical classification results, the second-level data purpose and the third-level data attributes are read, and the corresponding evaluation metrics, evaluation methods, and calculation rules are automatically retrieved and matched from the evaluation metric library. When adapting to optional preset evaluation processes, based on the output hierarchical classification results, the first-level data modality classification results are read. If the classification result is a single text modality, image modality, numerical modality, or time-series modality, it is determined to be single-modality data, and the single-modality evaluation process is adapted. If the classification result involves a combination of two or more modalities, it is determined to be multimodal fusion data, and the multimodal fusion evaluation process is adapted. The single-modal evaluation process directly executes the evaluation based on the matched exclusive evaluation indicators and corresponding evaluation methods and calculation rules; The multimodal fusion evaluation process includes: A spatiotemporal alignment algorithm is used to synchronize and align different modal data in multimodal fusion data in both time and space dimensions. Key features are extracted from each aligned modal data. For image modal data, machine vision recognition methods are used to extract defect type, defect location, and defect morphology features. For text modal data, natural language processing methods are used to extract keywords, semantic descriptions, and detection conclusion features. For time-series modal data, time-series feature extraction methods are used to extract trend, periodic, and abrupt change features. The extracted modal features are evaluated according to the multimodal fusion index and the corresponding evaluation method and calculation rules.

[0040] Following the dedicated evaluation process for various modalities and multimodal fusion data, evaluations are conducted in batches, adhering to the evaluation methods and calculation rules in the indicator library throughout the process. A comprehensive evaluation data recording mechanism is established, recording the indicator name, evaluation time, calculation process, and final result. For multimodal data, additional records are made of spatiotemporal alignment deviations and graphic and textual verification details. Abnormal situations during the evaluation process are recorded separately to form an evaluation data ledger.

[0041] Step S4: The evaluation results are graded and processed, and corresponding application guidance is generated for datasets of different grades. A classification evaluation report is generated and pushed to the production stage.

[0042] This step integrates and categorizes the evaluation results of various data, provides targeted application guidance, and achieves a deep integration of evaluation results with the production process.

[0043] Furthermore, the evaluation results are graded and processed, including: For a single classification dataset, weights are assigned according to the importance of each evaluation indicator, and the scores of each indicator are weighted and summed to obtain the individual score of the classification dataset. The individual scores of all classification datasets are weighted according to the application priority of each classification dataset in production, and a second weighted average is performed to obtain the total score of the entire dataset. The grades are determined by the total score.

[0044] In practice, the evaluation data is summarized and analyzed, and the results are divided into three levels: excellent, qualified, and unqualified using a hierarchical weighted average method. This method can take into account the importance of each indicator and meets the specific evaluation needs of high-end manufacturing data types with their complexity. First, for a specific classification dataset, weights are assigned according to the importance of each evaluation indicator (e.g., core indicators have a weight of 0.6, auxiliary indicators have a weight of 0.4). The scores of each indicator in that classification dataset are then weighted and summed according to their weights to obtain the individual score for that classification dataset. Next, for the individual scores of all classification datasets, weights are assigned based on the application priority of each classification dataset in production, and a second weighted average is performed to obtain the total score of the entire high-end manufacturing dataset. Finally, the levels are divided according to the total score: 85-100 points is excellent, 60-85 points is qualified, and below 60 points is unqualified. To clearly illustrate the application logic of this method, specific examples and corresponding formulas are given in conjunction with actual high-end manufacturing scenarios: We selected two typical classification datasets in high-end manufacturing: "numerical modal-process parameter data (cutting process)" and "image modal-defect detection data (part surface defects)". The specific calculation process is as follows: Individual score calculation for the classification dataset (taking "numerical modal-process parameter data" as an example): This classification dataset selects two core indicators (parameter deviation rate and parameter stability) and one auxiliary indicator (parameter integrity), with a weight allocation of 0.3 for each core indicator and 0.4 for the auxiliary indicator; assuming the scores for the three indicators are 8, 9, and 7 points respectively, the formula for calculating the individual score is as follows: in For the i-th category dataset, the score for each item is... Let k be the weight of the k-th metric in the i-th classification dataset. Let be the score of the k-th metric in the i-th classification dataset, and n be the total number of evaluation metrics in the classification dataset; substitute the data to calculate: Total score calculation for the dataset: Assuming the application priority weight of "Numerical Modality - Process Parameter Data" is 0.6 (core production data), and the individual score of "Image Modality - Defect Detection Data" is 8.2 points with an application priority weight of 0.4 (quality control data), the total score is calculated using the following formula: in The total score for the dataset. The application priority weights for the i-th classification dataset, Let m be the score for each item in the i-th category dataset, and m be the total number of categories. Substitute the data to calculate: The corresponding level is qualified.

[0045] After the grading is completed, a classification evaluation report is generated, using visual charts (bar charts, line charts) to present the evaluation status of each type of data, clarifying the advantages, disadvantages, and optimization directions of each type of data, and applying the data at each level accordingly: Excellent-level data: directly applied to core aspects such as production optimization, quality control, and supply chain scheduling; Pass rating data: After simple optimization (such as parameter fine-tuning and data supplementation), it is applied and regularly tracked and evaluated; Non-compliant data: Return to the data collection or preprocessing stage for re-collection or correction until it passes the evaluation before being put into use.

[0046] Step S5: Collect feedback on the actual effects of the evaluation results, and dynamically adjust the configuration of classification categories and evaluation indicators based on the feedback results.

[0047] This step establishes a feedback mechanism for evaluation results to achieve continuous optimization of the classification evaluation method.

[0048] In practice, the first step is to collect the actual effects of the evaluation results in production applications, analyze the matching degree between evaluation indicators, classification standards and production needs, and identify existing problems (such as unreasonable evaluation thresholds, missing indicators, etc.). Maintaining the "divide and conquer" approach, the overall architecture, and unified judgment standards, the evaluation indicators, calculation rules, and thresholds for different modal data are dynamically adjusted based on feedback results. Simultaneously, classification categories can be adjusted to adapt to changes in the production process. The above steps S1-S4 are repeated to achieve a closed-loop iteration of "preprocessing-classification-evaluation-implementation-optimization," continuously improving the accuracy and relevance of classification and evaluation.

[0049] Based on the above technical solution, this application has the following outstanding substantive effects and significant progress: To address the shortcomings of existing technology classification systems, such as incompleteness and disconnect from evaluation, this application innovatively constructs a three-tiered classification system based on "modality, application, and attribute." By using differences in modalities such as text, images, numerical values, and time series as the core classification criteria and data modality as the core classification dimension, this system achieves a deep integration of classification with the essential characteristics of data and evaluation requirements. Combined with a dynamic adjustment mechanism, it can adapt to different high-end manufacturing scenarios and newly added modal data, resulting in more accurate classification and stronger adaptability. This provides targeted support for differentiated evaluation and solves the pain points of redundant classification and inefficient evaluation in existing technologies.

[0050] To address the issue of weak targeting in existing technical evaluation methods and indicators, this application integrates the design principle of "change and invariance," designs exclusive indicators based on the essential characteristics of different modal data, constructs a unified indicator library containing exclusive indicators for single and multimodal data, designs a differentiated evaluation process, and makes the evaluation indicators highly targeted. This enables differentiated evaluation of different types of data, improves the comprehensiveness of evaluation results, and makes the evaluation results more accurate. It can truly reflect the core value of different modal data and meet the precision requirements and production needs of high-end manufacturing.

[0051] To address the shortcomings of existing technologies, such as incompatibility with multimodal fusion data, low evaluation efficiency, and lack of closed-loop mechanisms, this application designs a dedicated evaluation logic for multimodal fusion data, including image-text and numerical-time series data, which are widely used in high-end manufacturing. This logic achieves spatiotemporal alignment and cross-verification of multimodal data, supplements the specialized evaluation logic for multimodal fusion data, provides a complete system device implementation scheme, and establishes an "evaluation-feedback-optimization" closed-loop mechanism. This mechanism covers the multimodal data evaluation needs across all scenarios in high-end manufacturing, enabling rapid classification and real-time evaluation of massive amounts of multimodal data, meeting the real-time data support requirements of the production process. Furthermore, through the evaluation result feedback optimization mechanism, the evaluation results can be continuously optimized and directly applied to the production process, fully leveraging the value of data to help enterprises optimize production processes and improve product quality. Compared to existing technologies, this application demonstrates significant practicality and innovation.

[0052] Corresponding to the above-described dataset classification evaluation method embodiments, this application provides a dataset classification evaluation apparatus, including: The data acquisition and metadata processing module is used to collect raw data from the entire process of the target product, process the raw dataset for metadata, and obtain raw data with metadata information. The metadata processing includes metadata completion, data annotation, and metadata verification, where the annotation information serves as a supplement to the metadata. The data classification module is used to classify the raw data according to a preset three-level classification system of <modality-purpose-attribute> based on metadata information; each data item is labeled with three-level classification tags: modality, purpose, and attribute. The data evaluation module is used to select matching evaluation indicators for different modal data from a preset evaluation indicator library based on the hierarchical classification results, adapt to optional preset evaluation processes, and output evaluation results; the evaluation indicators include single-modality specific indicators and multi-modality fusion indicators; the preset evaluation processes include single-modality evaluation processes and multi-modality fusion evaluation processes. The application module is used to classify the evaluation results, generate corresponding application guidance for datasets of different levels, and generate classification evaluation reports to push to the production process. The feedback optimization module is used to collect actual feedback on the evaluation results and dynamically adjust the configuration of classification categories and evaluation indicators based on the feedback results.

[0053] Furthermore, the hierarchical classification of the original data according to the <modality-use-attribute> three-level classification system based on metadata information includes: The first layer is classified by data modality, which includes at least text modality data, image modality data, numerical modality data, and time series modality data. The second layer: further classifies each type of modal data according to its purpose; The third layer: Further classify the data according to data attributes for each type of use.

[0054] Furthermore, in the evaluation index library, the evaluation indicators are configured and stored in a structure of <modal classification-evaluation purpose-indicator name-evaluation method-calculation rule> and support dynamic updates.

[0055] Furthermore, the single-mode specific index includes at least: parameter deviation rate, used to match numerical mode-process parameter data; its calculation formula is: In the formula For the i-th group of actual values, This is the standard value.

[0056] Furthermore, the multimodal fusion metric includes at least: image-text information consistency rate, used to match image-text bimodal defect detection data; its calculation formula is: .

[0057] Furthermore, when selecting matching evaluation metrics, based on the output hierarchical classification results, the second-level data purpose and the third-level data attributes are read, and the corresponding evaluation metrics, evaluation methods, and calculation rules are automatically retrieved and matched from the evaluation metric library. When adapting to optional preset evaluation processes, based on the output hierarchical classification results, the first-level data modality classification results are read. If the classification result is a single text modality, image modality, numerical modality, or time-series modality, it is determined to be single-modality data, and the single-modality evaluation process is adapted. If the classification result involves a combination of two or more modalities, it is determined to be multimodal fusion data, and the multimodal fusion evaluation process is adapted. The single-modal evaluation process directly executes the evaluation based on the matched exclusive evaluation indicators and corresponding evaluation methods and calculation rules; The multimodal fusion evaluation process includes: A spatiotemporal alignment algorithm is used to synchronize and align different modal data in multimodal fusion data in both time and space dimensions. Key features are extracted from each aligned modal data; The extracted modal features are evaluated according to the multimodal fusion index and the corresponding evaluation method and calculation rules.

[0058] Furthermore, the evaluation results are graded and processed, including: For a single classification dataset, weights are assigned according to the importance of each evaluation indicator, and the scores of each indicator are weighted and summed to obtain the individual score of the classification dataset. The individual scores of all classification datasets are weighted according to the application priority of each classification dataset in production, and a second weighted average is performed to obtain the total score of the entire dataset. The grades are determined by the total score.

[0059] The present application provides a specific implementation example of the device as follows.

[0060] The data acquisition module, corresponding to the data input layer, is responsible for collecting various raw datasets from the entire high-end manufacturing process. It employs a multi-interface design, supporting interfaces with multiple data sources such as production equipment, testing instruments, and databases. It can acquire text, images, numerical data, time-series data, and multimodal fusion data. It features data caching to prevent data loss and supports data format conversion, transforming raw data in different formats (such as txt, jpg, csv, json) into a system-recognized format to ensure data can be correctly input into subsequent modules. Key parameters include: interface types such as RS485, Ethernet, and USB; data acquisition frequency ≥ 1 time / second; cache capacity ≥ 100GB; and support for ≥ 10 different format conversion types.

[0061] The metadata processing module, corresponding to the preprocessing layer, connects to the data acquisition module. It receives the collected raw data and performs data cleaning, standardization, and annotation operations. It incorporates a 3σ outlier detection algorithm, a linear interpolation algorithm, a hash deduplication algorithm, and an automated annotation algorithm, enabling fully automated preprocessing. It also supports manual verification to correct incorrectly labeled or incompletely cleaned data, ensuring the quality of the preprocessed data. Core functions: outlier detection accuracy ≥99%, missing data completion accuracy ≥98%, annotation accuracy ≥99.5%, preprocessing speed ≥1000 records / second.

[0062] The data classification module, corresponding to the classification layer, connects to the data preprocessing module. It receives preprocessed labeled data and employs a three-layer classification algorithm based on "modality-purpose-attribute" to achieve automatic hierarchical classification of the data. It has a built-in modality recognition model (capable of recognizing four core modalities: text, image, numerical, and time series), supports dynamic classification adjustments, and allows manual addition, deletion, or modification of classification categories through the system interface without altering the core algorithm, adapting to different scenario requirements. Core functions: classification accuracy ≥99%, classification speed ≥800 records / second, supports dynamic expansion of classification categories, and can store ≥100 classification categories.

[0063] The data evaluation module, corresponding to the evaluation layer, connects to the data classification module. It receives various types of categorized data and has a built-in unified evaluation indicator library and differentiated evaluation algorithms. Based on the data classification results, it can automatically select matching evaluation indicators from the indicator library, execute corresponding evaluation operations, and calculate evaluation results. It supports specialized evaluation of multimodal fusion data, with built-in spatiotemporal alignment and image-text verification algorithms to achieve alignment and verification of image-text dual-modal data. It also has an evaluation data recording function, automatically generating an evaluation data ledger. Core functions: evaluation accuracy ≥ 98.5%, evaluation speed ≥ 500 records / second, supports single-modal and multimodal data evaluation, and can automatically generate an evaluation data ledger.

[0064] The application module, corresponding to the application layer, connects with the data evaluation module, receives evaluation results, and performs operations such as result integration, classification, and application guidance generation. It has a built-in visualization chart generation algorithm that can automatically generate visual charts such as bar charts and line charts to present the evaluation results. It supports automatic generation of evaluation reports, and the report format can be customized according to user needs. At the same time, it connects to the enterprise production management system to push the evaluation results and application guidance to the corresponding production links, realizing the application of evaluation results.

[0065] The feedback optimization module, corresponding to the feedback optimization layer, connects to the application module. It receives application feedback data from the production process and analyzes the matching degree between evaluation indicators, classification standards, and production needs. It has a built-in feedback analysis algorithm that can automatically identify optimization directions and provide adjustment suggestions. It supports users to manually adjust classification categories, evaluation indicators, and thresholds, and simultaneously synchronizes the adjusted parameters to the corresponding modules to achieve closed-loop optimization of the system. It also has a parameter backup function that can save parameter configurations for different scenarios for easy switching.

[0066] Auxiliary modules: system control unit, display unit, and storage unit. The system control unit connects all core modules and is responsible for coordinating the working sequence; the display unit connects the application modules and the system control unit to present various types of information; and the storage unit connects all core modules to store various types of data.

[0067] The overall connection of the device is as follows: Data Acquisition Module → Data Preprocessing Module → Data Classification Module → Data Evaluation Module → Application Module → Feedback Optimization Module. Data flow is: Data Source → Data Acquisition Module → Data Preprocessing Module → Data Classification Module → Data Evaluation Module → Application Module → Production Stage; Feedback data from the production stage → Feedback Optimization Module → Data Classification Module / Data Evaluation Module, achieving closed-loop optimization. Each module is connected to the system control unit, which coordinates the working sequence of each module, receives user operation commands, and achieves overall system control. It is also equipped with a display unit and a storage unit. The display unit presents classification and evaluation results, system status, and other information, while the storage unit stores raw data, preprocessed data, classification results, and evaluation data, ensuring data traceability.

[0068] See Figure 5 This paper provides a system timing diagram to present the working timing of each module and the data interaction time nodes. Taking single batch data processing as an example, the specific timing description is as follows (unit: ms): 1.0ms: System initialization, all modules enter the ready state, and the system control unit sends a start command; 2.10-50ms: The data acquisition module acquires raw data, completes data format conversion and caching, and sends the data to the data preprocessing module; 3.50-150ms: The data preprocessing module performs data cleaning, standardization, and labeling operations, outputs the preprocessed data, and sends it to the data classification module; 4.150-250ms: The data classification module performs a three-level classification operation, outputs the classification results, and sends them to the data evaluation module; 5.250-450ms: The data evaluation module selects matching metrics, performs evaluation operations, records evaluation data, outputs evaluation results, and sends them to the application module. 6.450-550ms: The application module integrates the evaluation results, generates an evaluation report and application guidance, pushes them to the production stage, and simultaneously opens the application feedback interface; 7.550ms later: The production process provides feedback on the application effect. The feedback optimization module receives the feedback data, analyzes it, generates adjustment parameters, and synchronizes them to the data classification module and data evaluation module to prepare for the next batch of data processing. 8. The total processing time for a single batch of data is ≤550ms, meeting the needs of real-time processing of massive amounts of data in high-end manufacturing.

[0069] The above-described dataset classification and evaluation device implements the steps and processes of the above-described dataset classification and evaluation method embodiments, and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0070] Corresponding to the above-described dataset classification and evaluation method embodiments, this application provides an electronic device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps and processes of the above-described dataset classification and evaluation method embodiments and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0071] Memory can be used to store software programs and various data. Memory can primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area can store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, memory can include volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (Synchlink DRAM, SLDRAM), and direct memory bus RAM (DRRAM). The memory in the embodiments of this application includes, but is not limited to, these and any other suitable types of memory.

[0072] The processor may include one or more processing units; optionally, the processor integrates an application processor and a modem processor, wherein the application processor mainly handles operations related to the operating system, user interface, and applications, while the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into the processor.

[0073] Corresponding to the above-described dataset classification and evaluation method embodiments, this application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the steps and processes of the above-described dataset classification and evaluation method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0074] The processor is the processor in the electronic device described in the above embodiments of this application. The readable storage medium includes a computer-readable storage medium, such as a computer read-only memory (ROM), random access memory (RAM), a magnetic disk, or an optical disk.

[0075] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0076] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0077] It is understood that the embodiments of this application have been described above in conjunction with the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. As those skilled in the art will know, various changes or equivalent substitutions can be made to these features and embodiments without departing from the spirit and scope of the invention. Furthermore, those skilled in the art, under the guidance or instruction of this application, can modify these features and embodiments to adapt to specific situations and materials without departing from the spirit and scope of the invention. Therefore, this invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the protection scope of this invention.

Claims

1. A dataset classification evaluation method, characterized in that, include: Collect raw data from the entire process of the target product, and perform metadata sorting on the raw dataset to obtain raw data with metadata information; The metadata sorting includes metadata completion, data annotation, and metadata verification, with annotation information serving as a supplement to the metadata. The raw data is classified into three levels according to a preset <modality-purpose-attribute> classification system based on metadata information; each data item is labeled with three levels of classification tags: modality, purpose, and attribute. Based on the hierarchical classification results, matching evaluation indicators are selected from the preset evaluation indicator library for different modal data, and the evaluation results are output by adapting to the optional preset evaluation process. The evaluation indicators include single-modality specific indicators and multi-modality fusion indicators. The preset evaluation process includes single-modality evaluation process and multi-modality fusion evaluation process. The evaluation results are graded and processed, and corresponding application guidance is generated for datasets of different grades. A classification evaluation report is generated and pushed to the production stage. Collect feedback on the actual effects of the evaluation results, and dynamically adjust the classification categories and evaluation indicators based on the feedback.

2. The dataset classification and evaluation method according to claim 1, characterized in that, The aforementioned hierarchical classification of the raw data according to the <modality-use-attribute> three-level classification system based on metadata information includes: The first layer is classified by data modality, which includes at least text modality data, image modality data, numerical modality data, and time series modality data. The second layer: further classifies each type of modal data according to its purpose; The third layer: Further classify the data according to data attributes for each type of use.

3. The dataset classification and evaluation method according to claim 1, characterized in that, The evaluation index library stores the configured evaluation indicators in a structure of <modal classification-evaluation purpose-indicator name-evaluation method-calculation rule> and supports dynamic updates.

4. The dataset classification and evaluation method according to claim 1, characterized in that, The single-mode specific index includes at least: parameter deviation rate, used to match numerical mode-process parameter data; its calculation formula is: In the formula For the i-th group of actual values, This is the standard value.

5. The dataset classification evaluation method according to claim 1, characterized in that, The multimodal fusion index includes at least: image-text information consistency rate, used to match image-text bimodal defect detection data; its calculation formula is: 。 6. The dataset classification evaluation method according to claim 1, characterized in that, When selecting matching evaluation metrics, based on the output hierarchical classification results, the second-level data purpose and the third-level data attributes are read, and the corresponding evaluation metrics, evaluation methods, and calculation rules are automatically retrieved and matched from the evaluation metric library. When adapting to optional preset evaluation processes, based on the output hierarchical classification results, the first-level data modality classification results are read. If the classification result is a single text modality, image modality, numerical modality, or time-series modality, it is determined to be single-modality data, and the single-modality evaluation process is adapted. If the classification result involves a combination of two or more modalities, it is determined to be multimodal fusion data, and the multimodal fusion evaluation process is adapted. The single-modal evaluation process directly executes the evaluation based on the matched exclusive evaluation indicators and corresponding evaluation methods and calculation rules; The multimodal fusion evaluation process includes: A spatiotemporal alignment algorithm is used to synchronize and align different modal data in multimodal fusion data in both time and space dimensions. Key features are extracted from each aligned modal data; The extracted modal features are evaluated according to the multimodal fusion index and the corresponding evaluation method and calculation rules.

7. The dataset classification evaluation method according to claim 1, characterized in that, The evaluation results are graded and processed, including: For a single classification dataset, weights are assigned according to the importance of each evaluation indicator, and the scores of each indicator are weighted and summed to obtain the individual score of the classification dataset. The individual scores of all classification datasets are weighted according to the application priority of each classification dataset in production, and a second weighted average is performed to obtain the total score of the entire dataset. The grades are determined by the total score.

8. A dataset classification evaluation device, characterized in that, include: The data acquisition and metadata sorting module is used to collect raw data from the entire process of the target product, sort the raw dataset for metadata, and obtain raw data with metadata information. The metadata sorting includes metadata completion, data annotation, and metadata verification, with annotation information serving as a supplement to the metadata. The data classification module is used to classify the raw data according to a preset three-level classification system of <modality-purpose-attribute> based on metadata information; each data item is labeled with three-level classification tags: modality, purpose, and attribute. The data evaluation module is used to select matching evaluation indicators for different modal data from a preset evaluation indicator library based on the hierarchical classification results, adapt to optional preset evaluation processes, and output evaluation results; the evaluation indicators include single-modality specific indicators and multi-modality fusion indicators; the preset evaluation processes include single-modality evaluation processes and multi-modality fusion evaluation processes. The application module is used to classify the evaluation results, generate corresponding application guidance for datasets of different levels, and generate classification evaluation reports to push to the production process. The feedback optimization module is used to collect actual feedback on the evaluation results and dynamically adjust the configuration of classification categories and evaluation indicators based on the feedback results.

9. An electronic device, characterized in that, The electronic device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the dataset classification evaluation method as described in any one of claims 1 to 7.

10. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the dataset classification evaluation method as described in any one of claims 1 to 7.

Images