Construction method and system of high-quality data set, electronic equipment and storage medium

Abstract

The invention provides a construction method and system of a high-quality data set, electronic equipment and a storage medium. The method is applied to a high-quality data set construction system. The method comprises the following steps: acquiring an original data set in a target application scene, and processing the original data set in sequence based on intelligent agents corresponding to preprocessing nodes and data processing nodes respectively to obtain a processed data set; annotating the processing data set through the intelligent agent of the data annotation node to obtain an annotated data set, and learning the to-be-learned processing data in the processing data set by the intelligent agent of the data annotation node; and performing quality evaluation processing on the labeled data set through the intelligent agent of the quality evaluation node, and when the quality evaluation result is that the quality condition is not met, re-executing the construction process on the processed data set until the labeled data set meeting the quality condition is obtained. According to the scheme, full-process automatic processing from preprocessing to quality evaluation is realized, and the data processing efficiency and the accuracy of the labeled data set are improved.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a method, system, electronic device, and storage medium for constructing a high-quality data set. Background Art

[0002] Against the backdrop of a new round of scientific and technological revolution and industrial transformation led by artificial intelligence, the amount of data and computing power are growing explosively. Obtaining high-quality data from large amounts of data is an urgent problem to be solved.

[0003] Currently, common data governance platforms and data processing tools are basically oriented towards the data governance needs of enterprises. They only have certain functions including data integration and processing, metadata management, master data management, data quality management, data security and compliance, and data asset catalogs, and lack the ability to automate the entire process. There is a lot of manual intervention in each data processing link, which not only leads to a large amount of repetitive work and increases labor costs, but also greatly reduces the efficiency and accuracy of data processing, making it difficult to meet users' demand for "obtaining high-quality data quickly and well" for high-quality data sets. Summary of the Invention

[0004] The present disclosure provides a method, system, electronic device, and storage medium for constructing a high-quality dataset to address the problems of lack of fully automated data processing and inaccurate data processing results.

[0005] According to one aspect of the present disclosure, a method for constructing a high-quality dataset is provided, which is applied to a high-quality dataset construction system. The high-quality dataset construction system includes an intelligent agent corresponding to each processing node in the high-quality dataset construction process, wherein the processing result of the intelligent agent of the previous processing node is transmitted to the intelligent agent of the next processing node and serves as trigger information for the intelligent agent of the next processing node to start the next processing node until the high-quality dataset construction process is completed. The processing process includes the following processing nodes in sequence: a preprocessing node, a data processing node, a data annotation node, and a quality assessment node;

[0006] Methods include:

[0007] Obtain the original data set in the target application scenario, and process the original data set in sequence based on the intelligent agents corresponding to the preprocessing nodes and data processing nodes to obtain the processed data set;

[0008] The intelligent agent of the data annotation node performs an annotation process on each processed data in the processed data set to obtain an annotated data set. In the data annotation process, the processed data to be learned is identified, and the intelligent agent of the data annotation node re-learns the reviewed processed data to be learned. After learning, the intelligent agent continues to annotate other processed data.

[0009] The quality assessment node's intelligent agent performs quality assessment on the labeled dataset to obtain a quality assessment result. When the quality assessment result shows that the quality condition is not met, the construction process is re-executed on the processed dataset until a labeled dataset that meets the quality condition is obtained.

[0010] Optionally, the data labeling node includes an intelligent agent and a review end; the method also includes: generating a labeling task based on the processed data to be labeled in the processed data set, distributing the labeling task in sequence to the intelligent agent of the data labeling node for labeling processing, and obtaining first labeled data and corresponding confidence; receiving the first labeled data and the confidence of the first labeled data, and when the confidence is less than the set confidence threshold, using the first labeled data as the processed data to be learned; generating a review task for the processed data to be learned, and distributing the review task to the review end; receiving second labeled data fed back by the review end, generating a learning task based on the second labeled data, and distributing it to the intelligent agent of the data labeling node for learning; the second labeled data is the reviewed processed data to be learned, and the second labeled data includes standard labeling information of the processed data to be learned.

[0011] Optionally, the method also includes: sampling the processed data to be labeled in the processed data set to obtain sampled processed data, distributing the sampled processed data to the review end so that the review end performs labeling on the sampled processed data to obtain third labeled data fed back by the review end; obtaining first labeled data obtained by the intelligent agent of the data labeling node on the sampled processed data, and determining the accuracy of the spot check data based on the first labeled data and third labeled data rows corresponding to the sampled processed data; when the accuracy of the spot check data does not meet the sampling quality inspection conditions, the intelligent agent of the data labeling node learns based on the third labeled data, and the intelligent agent after learning re-labels the processed data set.

[0012] Optionally, the review end includes at least one level of processing ends, and the processing ends at different levels have different permissions; the received processing data is transmitted to the first processing end to obtain the labeled data output by the first processing end; when the labeled data output by the first processing end includes a pending identifier, the labeled data output by the first processing end is transmitted to the second processing end to obtain the labeled data output by the second processing end, wherein the authority of the second processing end is higher than the authority of the first processing end; the confidence of the labeled data output by the second processing end is higher than the labeled data output by the first processing end.

[0013] Optionally, there are multiple processing terminals at the same level; the method also includes: transmitting the received processing data to multiple first processing terminals respectively, obtaining and displaying the labeling information of the processing data of at least one first processing terminal; when the labeling information of the processing data of different first processing terminals is inconsistent, transmitting the processing data and multiple labeling information to the second processing terminal to obtain the labeling data output by the second processing terminal.

[0014] Optionally, the quality assessment process of the labeled data set is performed through the intelligent agent of the quality assessment node, including: constructing a quality assessment task based on the labeled data set, and sampling in the labeled data set to obtain sampled labeled data corresponding to the quality assessment task, and the sampled labeled data meets the coverage of the labeled data set in the data layer dimension and the time dimension; the quality assessment process of the sampled labeled data is performed based on the loaded evaluation rules by the intelligent agent of the quality assessment node, and the evaluation rules are loaded by adopting a dynamic hot loading mechanism.

[0015] Optionally, it is characterized in that the intelligent agent of the preprocessing node performs at least one of data cleaning, document parsing, data format conversion, anomaly detection and data deduplication on the original data; the intelligent agent of the data processing node performs at least one of data enhancement, multilingual processing, feature screening and data segmentation on the data set output by the preprocessing node.

[0016] According to another aspect of the present disclosure, a system for constructing a high-quality dataset is provided, comprising:

[0017] The dataset preprocessing module is used to obtain the original dataset in the target application scenario and process the original dataset based on the agent corresponding to the preprocessing node to obtain the preprocessed dataset;

[0018] A processing data set determination module is used to process the pre-processing data set based on the intelligent agent corresponding to the data processing node to obtain a processing data set;

[0019] The data set annotation module is used to annotate each processed data in the processed data set through the intelligent agent of the data annotation node to obtain an annotated data set. In the data annotation process, the processed data to be learned are identified, and the intelligent agent of the data annotation node re-learns the reviewed processed data to be learned. After learning, the intelligent agent continues to annotate other processed data.

[0020] The quality assessment module is used to perform quality assessment on the labeled data set through the intelligent agent of the quality assessment node to obtain the quality assessment result. When the quality assessment result does not meet the quality conditions, the construction process is re-executed on the processed data set until a labeled data set that meets the quality conditions is obtained.

[0021] According to another aspect of the present disclosure, an electronic device is provided, the electronic device including:

[0022] at least one processor; and

[0023] a memory communicatively connected to at least one processor; wherein,

[0024] The memory stores a computer program that can be executed by at least one processor. The computer program is executed by the at least one processor so that the at least one processor can perform the method for constructing a high-quality dataset according to any embodiment of the present disclosure.

[0025] According to another aspect of the present disclosure, a computer-readable storage medium is provided, which stores computer instructions. The computer instructions are used to enable a processor to implement the method for constructing a high-quality dataset according to any embodiment of the present disclosure when executed.

[0026] The technical solution of the embodiment of the present disclosure is applied to a high-quality dataset construction system, which includes intelligent agents corresponding to each processing node in the construction process of the high-quality dataset, wherein the processing result of the intelligent agent of the previous processing node is transmitted to the intelligent agent of the next processing node, and serves as trigger information for the intelligent agent of the next processing node to start the next processing node until the construction process of the high-quality dataset is completed; the processing process includes the following processing nodes in sequence: a preprocessing node, a data processing node, a data annotation node and a quality assessment node; by obtaining the original dataset in the target application scenario, the original dataset is processed in sequence based on the intelligent agents corresponding to the preprocessing node and the data processing node to obtain a processed dataset; the original dataset is preprocessed in sequence by the intelligent agents corresponding to the two nodes to obtain a processed dataset for subsequent annotation processing, thereby avoiding the problem that useless data in the original dataset affects the annotation processing and causes inaccurate annotation data results; each processed data in the processing dataset is annotated by the intelligent agent of the data annotation node to obtain an annotation data The method comprises the following steps: identifying the data to be learned and processed during the data labeling process, relearning the reviewed data to be learned and processed by the intelligent agent of the data labeling node, and continuing to label other processed data after learning, realizing the method of labeling processing by the intelligent agent of the data labeling node. At the same time, the intelligent agent of the data labeling node can also learn according to the data to be learned in the processed data set, thereby obtaining an intelligent agent with more accurate labeling processing, which is used to label other processed data in the processed data set to improve the accuracy of the labeling processing results of the data labeling node; performing quality assessment processing on the labeled data set by the intelligent agent of the quality assessment node to obtain a quality assessment result. When the quality assessment result is that the quality condition is not met, the construction process is re-executed on the processed data set until a labeled data set that meets the quality condition is obtained, realizing the quality assessment processing of the labeled data set by the intelligent agent of the quality assessment node to obtain a labeled data set that meets the quality condition, and further ensuring that the obtained labeled data set can meet the quality requirements of the labeled data set in the target application scenario. This solution realizes the full-process automated processing of the original data set from preprocessing to quality assessment through the intelligent agents corresponding to the preprocessing nodes, data processing nodes, data labeling nodes and quality assessment nodes respectively, solving the problems of lack of fully automated data processing and inaccurate data processing results, improving the efficiency of automated data processing and the accuracy of labeled data sets.

[0027] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0029] Figure 1 is a flowchart of a method for constructing a high-quality data set provided by an embodiment of the present disclosure;

[0030] Figure 2 is a flowchart of a method for constructing a high-quality data set provided by an embodiment of the present disclosure;

[0031] Figure 3 is a schematic diagram of the structure of a high-quality data set construction system provided by an embodiment of the present disclosure;

[0032] Figure 4 3 is a schematic diagram of the structure of an electronic device for implementing the method for constructing a high-quality data set according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0033] In order to enable those skilled in the art to better understand the solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the embodiments described are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present disclosure.

[0034] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0035] Figure 1This is a flowchart of a method for constructing a high-quality dataset provided by an embodiment of the present disclosure. This embodiment is applicable to the situation of constructing a high-quality dataset. The method can be executed by a high-quality dataset construction system. The high-quality dataset construction system can be implemented in the form of hardware and / or software. The high-quality dataset construction system can be configured in electronic devices such as servers and computers. The method is applied to a high-quality dataset construction system. The high-quality dataset construction system includes intelligent agents corresponding to each processing node in the construction process of the high-quality dataset, wherein the processing result of the intelligent agent of the previous processing node is transmitted to the intelligent agent of the next processing node and serves as trigger information for the intelligent agent of the next processing node to start the next processing node until the construction process of the high-quality dataset is completed; the processing flow includes the following processing nodes in sequence: preprocessing node, data processing node, data annotation node and quality assessment node.

[0036] In this embodiment, the process constructed to obtain a high-quality data set can be customized according to actual data characteristics or business needs. The processing process is constructed through a process construction interactive platform to obtain a data processing process for a high-quality data set that meets data processing needs. Exemplarily, the Apache Airflow work management platform can be called to display the interactive page of the work management platform so that business personnel can independently configure the key nodes and process sequence of data processing, support drag-and-drop adjustment, and follow the default data processing workflow if no settings are made. Abnormal data and reasons are configured to facilitate manpower to find the source of the abnormality. In some embodiments of the present disclosure, each processing node in the processing flow may include multiple sub-processing nodes, and the sub-processing flow corresponding to each processing node in the processing flow can be set through the interactive page of the above-mentioned work management platform. The sub-processing flow of any processing node includes the temporal relationship between the multiple sub-processing nodes of the processing node. In different target application scenarios, the sub-processing nodes included in the same processing node can be different. Accordingly, the sub-processing flows corresponding to the same processing node can be different. This can be set according to the process configuration operation settings on the interactive page of the work management platform. Specifically, the drag operation on the sub-processing node selects the sub-processing node executed by the processing node, and the data flow relationship between the sub-processing nodes is set to determine the timing relationship between the selected sub-processing nodes to form a sub-processing flow. It supports multiple parameter sources. For example, it supports multi-format data input such as Parquet / JSON / TFRecord / Avro. Once the workflow is confirmed, there is no need to readjust the initial settings for different data types, achieving "one-time setting, multiple reuse". Event-driven or timed scheduling can be used to automatically start intelligent workflows.

[0037] like Figure 1 As shown, the method includes:

[0038] S110 , obtaining an original data set in a target application scenario, and processing the original data set in sequence based on the intelligent agents corresponding to the preprocessing nodes and the data processing nodes to obtain a processed data set.

[0039] The target application scenario can be specifically understood as an application scenario that requires the construction of a high-quality data set. For example, the target application scenario can be a government processing scenario, a government management scenario, or a financial business processing scenario. It is understandable that different target application scenarios generate different raw data. The raw data set can be specifically understood as a basic data set that is directly collected or preliminarily collected in the target application scenario and has not been deeply processed to meet the specific business needs, research purposes, or application directions in the target application scenario. For example, the raw data set can be raw data including multiple data formats and / or multiple language types.

[0040] The intelligent agent can be specifically understood as a software entity, system or data processing model that can automatically perform data processing tasks and has the ability to perceive, make decisions and take actions. It can perceive input data, analyze contextual information in a specific environment, and take proactive actions to complete data processing tasks, which helps to improve the efficiency, accuracy and intelligence level of data processing. In this embodiment, corresponding intelligent agents can be set at different data processing nodes to perform corresponding functional processing on the data set received by the processing node. Optionally, the intelligent agent of the preprocessing node performs at least one of data cleaning, document parsing, data format conversion, anomaly detection and data deduplication on the raw data. The intelligent agent of the data processing node performs at least one of data enhancement, multilingual processing, feature screening and data segmentation on the data set output by the preprocessing node.

[0041] Specifically, in a target application scenario, data is collected from various data sources within the target application scenario. Based on the collected data, an unprocessed raw dataset is formed. The raw dataset is then processed by agents corresponding to preprocessing nodes to obtain a preprocessed dataset. These data sources may originate from different devices, systems, or platforms, and the data formats and quality may vary. For example, the agents corresponding to preprocessing nodes can perceive the format, missing values, and outliers of the raw data and perform corresponding operations, such as data cleaning. Corresponding data processing operations include, but are not limited to, deduplication, filling missing values, correcting erroneous data, and data conversion. Data conversion specifically refers to converting the raw data into a unified format and encoding. The preprocessed dataset is then transmitted to a data processing node, where the agents corresponding to the data processing nodes further process the preprocessed dataset. Based on the specific application requirements, the agents corresponding to the data processing nodes select corresponding algorithms or data processing models, analyze and process the data, and extract valuable information from the data to meet the needs of the target application scenario. After both preprocessing and data processing, a processed dataset is ultimately obtained.

[0042] In this embodiment, the original data set is processed in sequence by the intelligent agents corresponding to the preprocessing nodes and the data processing nodes respectively. Each intelligent agent can flexibly select appropriate processing methods and algorithms according to the characteristics of the data and application requirements, thereby improving flexibility. The intelligent agent can automatically perform data processing tasks, reduce manual intervention, and improve the efficiency of data processing. The data processing process is divided into two stages: preprocessing and data processing. Each stage is responsible for the corresponding intelligent agent, making the data processing process clearer and more standardized, which is conducive to data management and maintenance, and facilitates monitoring and adjustment of the data processing process to ensure the reliability and stability of data processing.

[0043] S120 , the intelligent agent of the data labeling node labels each processed data in the processed data set to obtain a labeled data set.

[0044] In the data labeling process, the data to be learned and processed are identified, and the intelligent agent of the data labeling node re-learns the verified data to be learned and processed, and the intelligent agent after learning continues to label other processed data.

[0045] In some embodiments of the present disclosure, a labeling workbench is used to set the processing process of the data labeling node. The labeling workbench is an interactive workbench. The labeled data range, labeling content, labeling rules and the labeling model to be called can be set through the interactive page of the labeling workbench. Specifically, the labeled data range, labeling content, labeling rules and the labeling model to be called can be set and modified through interactive operations on the interactive page. Among them, the labeled data range can be understood as the range of the processed data to be labeled in the processing data set, which is used to distinguish the processed data to be labeled from the labeled processed data; the labeling content can be understood as including multiple labels of the processing data set, which can be determined according to the target application scenario to which the processing data set belongs. The labeling model to be called can be understood as an intelligent entity that automatically labels the processing data set, wherein different target application scenarios can correspond to different labeling models, or different data types of the processing data in the processing data set (such as image, text, audio and other data types) can correspond to different labeling models.

[0046] In the annotation workbench, you can also view the annotation process and check the problems in the data annotation process. Staff with modification authority can intervene in time to modify the problems in the annotation process to ensure the accuracy of the labeled data.

[0047] Specifically, the data annotation node can be understood as a node for annotating data. The data annotation node is provided with a corresponding intelligent agent for performing automated annotation processing on the data received by the data annotation node. The processed data to be learned can be understood as the processed data with learning value in the processing data set. The intelligent agent of the data annotation node can be used to identify each processed data in the processing data set to obtain the processed data with learning value, thereby obtaining the processed data to be learned in the processing data set. Other processed data can be understood as the unannotated processed data other than the processed data to be learned in the processing data set.

[0048] Specifically, the agent at the data annotation node annotates each piece of processed data in the processing dataset, adding corresponding labels based on pre-set calibration rules and algorithms, thereby forming an annotated dataset. During the data annotation process, the agent identifies in real time potential learning data within the processing dataset, screens these data, and submits them for review. Once verified as correct, the data is again submitted to the agent at the data annotation node for learning. The agent then absorbs new knowledge and optimizes the annotation model and strategy. After learning is complete, the agent uses its updated capabilities and experience to continue annotating the remaining processed data.

[0049] In this embodiment, the intelligent agent of the data labeling node is used to label the processed data in the processing data set. Through the automatic labeling of the intelligent agent, the labeling efficiency is greatly improved, and the labor cost and time consumption are reduced. In addition, during the labeling process, the data to be learned is mined and the intelligent agent is allowed to learn again, so as to achieve self-optimization and capacity improvement of the intelligent agent, making subsequent labeling more accurate, effectively reducing labeling errors, and ensuring the quality of the labeled data set. In addition, this self-learning mechanism can also enable the intelligent agent to adapt to the diversity and changes of data, enhance the flexibility and adaptability of labeling work, and lay a solid foundation for subsequent machine learning, data analysis and other tasks based on the labeled data set.

[0050] Optionally, the data labeling node includes an intelligent agent and a review end; the method also includes: generating a labeling task based on the processed data to be labeled in the processed data set, distributing the labeling task in sequence to the intelligent agent of the data labeling node for labeling processing, and obtaining first labeled data and corresponding confidence; receiving the first labeled data and the confidence of the first labeled data, and when the confidence is less than the set confidence threshold, using the first labeled data as the processed data to be learned; generating a review task for the processed data to be learned, and distributing the review task to the review end; receiving second labeled data fed back by the review end, generating a learning task based on the second labeled data, and distributing it to the intelligent agent of the data labeling node for learning; the second labeled data is the reviewed processed data to be learned, and the second labeled data includes standard labeling information of the processed data to be learned.

[0051] In the data annotation node, the review end refers to the part that audits, verifies, and confirms the data annotated by the agent or the data identified as being to be learned. It can be set up for manual review or a specific review algorithm for review. Its main function is to ensure the accuracy and consistency of the annotation, check whether the agent's annotation results meet standards and requirements, correct any errors, and provide an accurate reference for the agent's learning and subsequent annotation work, thereby improving the quality of the annotated dataset. The processed data to be annotated in the processing dataset can be understood as all data in the processing dataset or data with an identification to be annotated. The specific determination is based on actual needs and is not limited here. The first annotated data refers to the data obtained after the agent at the data annotation node performs the initial annotating of the data to be annotated in the processing dataset based on the received annotation task. Upon completing the annotation, the agent also generates a corresponding confidence level for the annotated data, which represents the agent's estimate of the reliability of its own annotation results. The second annotated data is the data obtained after the review end verifies the data identified as being to be learned. First annotated data with a confidence level less than a set confidence threshold can be determined as data requiring review. The reviewer re-labels or reviews the reviewed data based on pre-set review standards and methods. This second labeling data contains the standard labeling information for the data to be learned. For example, the reviewer re-labels an image that the agent labeled as "cat" with a low confidence level and determines that the object is actually a "dog." This "dog" serves as the second labeling data, providing an accurate reference for the agent's learning, helping it correct errors and improve its labeling capabilities.

[0052] Standard labeling information can be specifically understood as pre-set label data. Specifically, it can be achieved by pre-building a label management module and managing the standard labeling information through the label management module to ensure the scientificity, consistency and scalability of the labeling system. The label management module includes a label setting unit, a label management unit and a label application unit. The label setting unit is used to set up the label system according to business needs and clarify the definition and classification of labels; the label management unit is used to store, maintain and update labels to ensure continuous optimization and updating of the label library; the label application unit is used to apply labels to data analysis and applications, supporting business scenarios such as data mining and intelligent decision-making. In the case of re-labeling the processed data on the review end, appropriate standard labeling information can be selected to label the processed data to ensure the consistency and rationality of the second labeling information.

[0053] Specifically, labeling tasks are generated based on the data to be labeled in the processed data set, and the tasks are sent one by one to the intelligent agent of the data labeling node. After the intelligent agent completes the labeling, the first labeled data and its corresponding confidence level are obtained. For example, the data to be labeled can be classified and organized according to the labeling goals and requirements, and the labeling content and labeling method corresponding to each data to be labeled can be clarified. Then, the data to be labeled can be converted into specific labeling tasks, such as the category of labeled entities in text data, the location and attributes of labeled targets in image data, etc. When generating labeling tasks, factors such as task priority and the processing capacity of the intelligent agent can also be considered to reasonably plan and arrange the labeling tasks. For example, the data to be processed can be grouped according to the type or difficulty level of each processed data, and labeling tasks can be generated based on the grouped data. The first labeled data and the corresponding confidence level are received. When the confidence level is lower than a set confidence threshold, the first labeled data with a confidence level lower than the set confidence threshold can be determined as the data to be learned and processed. Then, the data to be learned and processed are processed according to the task generation method to generate a review task for the data to be learned and processed, and sent to the review end. The review end receives the review task of the data to be learned and processed, reviews and re-labels the data according to preset review standards and specifications, and feeds back the second labeled data containing standard labeling information to the next data processing end of the data labeling node. After receiving the second labeled data, it generates a learning task based on the standard labeling information in the second labeled data, clarifies the learning objectives and content, and distributes the learning task to the intelligent agent of the data labeling node for the intelligent agent of the data labeling node to carry out learning improvement, so as to realize supervised learning of the intelligent agent of the data labeling node.

[0054] In this embodiment, a closed-loop mechanism of "identification-review-learning" is constructed at the data labeling node to effectively improve the data labeling quality and the labeling ability of the intelligent agent. Specifically, low-confidence data is handed over to the review end for processing, and is reviewed by manual or professional algorithms to ensure the accuracy and consistency of the labeling results, providing a reliable basis for subsequent learning; learning tasks are generated based on the reviewed standard labeling information, which allows the intelligent agent to correct labeling deviations in a targeted manner, optimize labeling strategies, and improve the accuracy and adaptability of labeling for complex data in continuous learning. While reducing the cost of manual comprehensive review, it ensures the quality of the labeled data set and lays a solid foundation for subsequent data analysis and model training.

[0055] Optionally, the method also includes: sampling the processed data to be labeled in the processed data set to obtain sampled processed data, distributing the sampled processed data to the review end so that the review end performs labeling on the sampled processed data to obtain third labeled data fed back by the review end; obtaining first labeled data obtained by the intelligent agent of the data labeling node on the sampled processed data, and determining the accuracy of the spot check data based on the first labeled data and third labeled data rows corresponding to the sampled processed data; when the accuracy of the spot check data does not meet the sampling quality inspection conditions, the intelligent agent of the data labeling node learns based on the third labeled data, and the intelligent agent after learning re-labels the processed data set.

[0056] The third labeled data specifically represents the data obtained by the verification end after sampling the processed data to be labeled in the processed data set. During the data labeling process, the third labeled data is used to compare with the first labeled data generated by the intelligent agent of the data labeling node. This is an important basis for determining the accuracy of the sampled data. Furthermore, if the accuracy of the sampled data does not meet the sampling quality inspection requirements, the third labeled data serves as standard data for learning and optimization of the intelligent agent of the data labeling node, helping the intelligent agent improve its labeling capabilities and thus improving the quality of the entire data labeling process.

[0057] Specifically, the intelligent agent of the data annotation node extracts a certain proportion of the data to be labeled from the processing data set as the sampled processing data to ensure that the sampled processing data can represent the overall characteristics of the processing data set; the sampled processing data is sent to the review end, which labels the sampled processing data and returns it. The labeling result returned by the review end is the third labeled data. When the third labeled data is obtained, the first labeled data generated by the intelligent agent for the same sampled processing data is obtained; then the first labeled data and the third labeled data are compared to calculate the accuracy of the sampling data. For example, the two sets of labeled data are compared and the proportion of samples with consistent labels is calculated as the sampling accuracy; if the accuracy does not meet the pre-set sampling quality inspection conditions, the intelligent agent will use the third labeled data as the learning content to learn, optimize its own labeling capabilities and strategies, and re-label the entire processing data set after learning is completed.

[0058] In this embodiment, the intelligent agent labeling effect is evaluated with a smaller amount of data through a sampling method, which greatly saves the time and manpower cost of comprehensive review; the third-party labeling data generated by the review end is used as a standard to accurately measure the accuracy of the intelligent agent labeling and discover labeling deviations; when the accuracy rate does not meet the standard, the intelligent agent can learn in a targeted manner, quickly correct labeling errors, and effectively improve the labeling quality; the closed-loop optimization process ensures the improvement of the overall labeling level of the processed data set, which not only guarantees the labeling quality but also improves the labeling efficiency, making data labeling work more efficient and reliable.

[0059] Based on the above embodiment, the review end includes at least one level of processing ends, and the processing ends at different levels have different permissions; the received processing data is transmitted to the first processing end to obtain the labeled data output by the first processing end; when the labeled data output by the first processing end includes a pending identifier, the labeled data output by the first processing end is transmitted to the second processing end to obtain the labeled data output by the second processing end, wherein the authority of the second processing end is higher than the authority of the first processing end; the confidence of the labeled data output by the second processing end is higher than the labeled data output by the first processing end.

[0060] Among them, the pending mark can be specifically understood as a mark set to characterize the labeled data with uncertainty or complexity in the labeled data output by the first processing end. In short, it can be understood as a pending review mark, which can be represented by characters, numbers or labeling boxes. The representation method is not limited here. Authority specifically refers to the different capabilities and permission scopes that different levels of processing ends have when annotating the processed data. In this embodiment, the first processing end has lower authority and can perform preliminary annotation on conventional processed data, but for some difficult to determine situations, it can only be marked as pending, while the second processing end has higher authority, which means that it can access more resources, such as a richer knowledge base, more advanced annotation tools or a more professional manual review team, so that it can perform more in-depth and accurate processing of the processed data with the pending mark and output labeled data with higher confidence, such as access to additional knowledge bases, use more complex algorithms or manual expert review authority.

[0061] Specifically, the review end uses a multi-level processing architecture. After receiving the processed data, it is first transmitted to the first processing end with the lowest authority for annotation. The first processing end outputs the annotated data. If the annotated data contains pending identifiers, the system automatically transfers the data to the second processing end with higher authority for secondary annotation, obtaining the annotated data output by the second processing end. The second processing end uses its higher authority to deeply process the data and ultimately outputs an annotation result with higher confidence. Throughout the entire process, data flows between different levels as needed, forming a step-by-step processing chain.

[0062] In this embodiment, the dual goals of resource optimization and quality assurance are achieved through a hierarchical processing mechanism. Through authority grading, simple tasks are assigned to low-level processing terminals, and complex tasks are handed over to high-level processing terminals, which greatly improves the overall processing efficiency; the multi-level review mechanism significantly reduces the labeling error rate, especially for the secondary processing of difficult samples, which effectively improves the reliability of the labeled data; the hierarchical improvement of confidence ensures the authority of the final labeling results and provides more accurate training data for subsequent intelligent learning; the dynamic flow mechanism avoids the waste of resources that require high-level processing of all data, and controls processing costs while ensuring quality, especially for scenarios with large differences in labeling difficulty and high requirements for data quality.

[0063] Optionally, there are multiple processing terminals at the same level; the method also includes: transmitting the received processing data to multiple first processing terminals respectively, obtaining and displaying the labeling information of the processing data of at least one first processing terminal; when the labeling information of the processing data of different first processing terminals is inconsistent, transmitting the processing data and multiple labeling information to the second processing terminal to obtain the labeling data output by the second processing terminal.

[0064] Specifically, within the review end, there are multiple processing ends at the same level. After receiving the processed data, the system will distribute the processed data to multiple first processing ends at the same time, so that multiple first processing ends can carry out labeling work in parallel. After the labeling of multiple first processing ends is completed, the system collects the labeling information output by at least one first processing end and displays it for viewing and comparison. If there are differences in the labeling information of the same processed data by different first processing ends, the system will transmit the processed data together with multiple labeling information to a second processing end with higher authority. The second processing end integrates all the labeling information, and with higher-level authority and processing capabilities, conducts in-depth analysis and judgment, and finally outputs unified and accurate labeling data.

[0065] In this embodiment, multiple processing terminals are set at the same level to process the received processing data. This method fully utilizes the advantages of parallel work of multiple processing terminals. By distributing the processed data to multiple first processing terminals at the same time, the labeling efficiency is greatly improved and the overall processing time is shortened. The labeling results of multiple first processing terminals are compared with each other, which can timely discover the differences and errors in the labeling process and avoid deviations caused by subjective judgment or ability limitations of a single processing terminal. When inconsistent labeling information occurs, the problem data is handed over to the second processing terminal with higher authority and stronger capabilities for processing, which ensures the accuracy and authority of the final labeled data and effectively improves the quality of the labeled data output by the review terminal. At the same time, this architecture is flexible and scalable, and the number of processing terminals at the same level can be increased or decreased according to actual needs to adapt to data labeling tasks of different scales.

[0066] S130. Perform quality assessment on the labeled dataset through the intelligent agent of the quality assessment node to obtain a quality assessment result. When the quality assessment result does not meet the quality conditions, re-execute the construction process on the processed dataset until a labeled dataset that meets the quality conditions is obtained.

[0067] The quality assessment node is a key link in the data processing process, used to assess the quality of the labeled dataset to ensure that the data quality meets specific standards and requirements. The corresponding agent is the intelligent program or algorithm model that plays a role in the quality assessment node. The agent in the quality assessment node has learning and analysis capabilities. It can conduct comprehensive and detailed inspection and analysis of the data in the labeled dataset based on preset quality indicators and rules, and evaluate the quality of the data from multiple dimensions. These multiple dimensions include but are not limited to checking the accuracy, consistency, and completeness of the data, and provide a corresponding quality assessment report based on the evaluation results. By continuously learning and optimizing its own evaluation strategy, the agent can more accurately identify problems in the data, providing strong support for ensuring data quality.

[0068] Specifically, the intelligent agent in the quality assessment node conducts a comprehensive evaluation of the labeled dataset. Based on pre-set quality assessment standards and algorithms, it can analyze multiple dimensions such as data accuracy, annotation consistency, and completeness to produce a quality assessment result. If the evaluation result shows that the pre-set quality conditions are not met, the system restarts the dataset construction process, starting with the original dataset and passing through the preprocessing, data processing, and data annotation nodes in sequence until the labeled dataset is generated again and undergoes quality assessment. This cycle repeats until a labeled dataset that meets the quality conditions is obtained.

[0069] In this embodiment, through automated testing of the quality assessment node agent, strict control of the quality of the labeled dataset is achieved, preventing unqualified data from flowing into subsequent links and affecting the accuracy of tasks such as data analysis and model training. When quality issues arise, timely backtracking and re-execution of the build process can systematically correct errors in the data processing and annotation process, ensuring that a high-quality labeled dataset is ultimately obtained. Continuous quality assessment and iterative optimization not only improve the availability and reliability of data resources, but also help optimize the data processing and annotation process in the long term, improve overall data processing efficiency, and reduce project risks and costs caused by data quality issues.

[0070] The technical solution of this embodiment is to obtain the original data set in the target application scenario, and process the original data set in sequence based on the intelligent agents corresponding to the preprocessing nodes and the data processing nodes to obtain a processed data set; the intelligent agent of the data annotation node annotates each processed data in the processed data set to obtain an annotated data set, wherein the data to be learned is identified during the data annotation process, and the intelligent agent of the data annotation node re-learns the reviewed processed data to be learned, and the intelligent agent after learning continues to annotate other processed data; the intelligent agent of the quality assessment node performs quality assessment on the annotated data set to obtain a quality assessment result, and when the quality assessment result does not meet the quality conditions, the construction process is re-executed on the processed data set until an annotated data set that meets the quality conditions is obtained. This solution realizes the full process automation of the original data set from preprocessing to quality assessment by the intelligent agents corresponding to the preprocessing nodes, data processing nodes, data annotation nodes and quality assessment nodes, solves the problem of lack of fully automated data processing and inaccurate data processing results, and improves the efficiency of automated data processing and the accuracy of annotated data sets.

[0071] Figure 2 This is a flowchart of a method for constructing a high-quality data set provided by an embodiment of the present disclosure. The method of this embodiment is a further optimization of the method of the above embodiment. Optionally, a quality assessment task is constructed based on the labeled data set, and sampling is performed in the labeled data set to obtain sampled labeled data corresponding to the quality assessment task. The sampled labeled data meets the coverage of the labeled data set in the data layer dimension and the time dimension; the intelligent body of the quality assessment node performs quality assessment processing on the sampled labeled data based on the loaded assessment rules to obtain the quality assessment results. The assessment rules are loaded using a dynamic hot loading mechanism. Figure 2 As shown, the method includes:

[0072] S210 , obtaining an original data set in a target application scenario, and processing the original data set in sequence based on the intelligent agents corresponding to the preprocessing nodes and the data processing nodes to obtain a processed data set.

[0073] S220. The intelligent agent of the data labeling node labels each processed data in the processed data set to obtain a labeled data set, wherein the processed data to be learned is identified during the data labeling process, and the intelligent agent of the data labeling node re-learns the reviewed processed data to be learned, and the intelligent agent after learning continues to label other processed data.

[0074] S230. Construct a quality assessment task based on the annotated dataset, and perform sampling in the annotated dataset to obtain sampled annotated data corresponding to the quality assessment task. The sampled annotated data satisfies the coverage of the annotated dataset in the data layer dimension and the time dimension.

[0075] The data stratification dimension refers to dividing the entire dataset into different levels or categories based on different data characteristics or attributes. For example, data stratification dimensions include, but are not limited to, data source, data type, data quality level, and business domain. This allows for more detailed data analysis and management, and understanding of the characteristics and patterns of data at different levels. The time dimension refers to the time when data is generated or collected. It can be used to assist in observing data trends over time, periodic patterns, and correlations between data at different time points. It is important for analyzing data timeliness and predicting future trends. Coverage is a measure of the representativeness of sampled data or an analysis result to the overall data. Coverage in the data stratification and time dimensions indicates the scope and proportion of the sampled data in each data stratum and at different time stages. The higher the coverage, the more comprehensively and accurately the sampled data reflects the characteristics and distribution of the original annotated dataset in different dimensions, thus providing a more reliable basis for subsequent analysis and evaluation.

[0076] Specifically, according to the overall situation of the annotated data set, the evaluation objectives and quality requirements, a detailed quality assessment task can be formulated, and the evaluation standards, indicators and methods can be clarified. Then, a sampling operation can be performed in the annotated data set to obtain the sampled annotated data corresponding to the quality assessment task. For example, to ensure that the sampling results are representative, the data stratification dimension and the time dimension are comprehensively considered during the sampling process. Among them, the considerations of the data stratification dimension include but are not limited to the data type, data source and data characteristics, and the considerations of the time dimension include but are not limited to the time coverage. Through a scientific and reasonable sampling algorithm, the corresponding processed data are selected from the data of each stratification and different time stages, and finally the sampled annotated data corresponding to the quality assessment task are obtained. These data can comprehensively reflect the characteristics and distribution of the annotated data set in the data stratification and time dimension.

[0077] In this embodiment, by constructing quality assessment tasks, the direction of quality assessment is clarified, laying the foundation for the efficient completion of the assessment work. The sampling strategy based on data stratification and time dimension can ensure that the sampled annotation data comprehensively and evenly covers the various characteristics and changes of the annotation dataset, avoiding distortion of the evaluation results due to sampling bias, and making the evaluation results more objective and credible. At the same time, by replacing the evaluation of all annotated data with sampling evaluation, the amount of data processing is greatly reduced while ensuring the quality of the evaluation, effectively improving the evaluation efficiency and reducing manpower and time costs. It is especially suitable for quality evaluation scenarios of large-scale annotated datasets.

[0078] S240. The intelligent agent of the quality assessment node performs quality assessment on the sampled labeled data based on the loaded assessment rules to obtain quality assessment results. The assessment rules are loaded using a dynamic hot loading mechanism.

[0079] Among them, the loaded evaluation rules refer to a series of detailed criteria and specifications formulated in advance based on the goals, requirements and relevant standards of data quality assessment. The evaluation rules are loaded into the intelligent agent of the quality assessment node and are used to perform quality assessment on the sampled and labeled data. They clarify the evaluation standards that the data should meet in all aspects, including but not limited to data accuracy requirements, consistency judgment conditions and integrity measurement indicators, and provide specific evaluation basis and methods for the intelligent agent to ensure the scientific and standardized nature of the quality assessment process. The dynamic hot loading mechanism refers to a technical means that can load or update evaluation rules in real time during system operation without stopping or restarting the system. It allows new evaluation rules to be loaded into the system in a timely manner according to changes in actual conditions, such as the evolution of data characteristics, new business needs or adjustments to quality standards, without affecting the business continuity of data quality assessment, so that the intelligent agent can evaluate the data according to the latest evaluation rules.

[0080] Specifically, the intelligent agent at the quality assessment node obtains sampled annotated data and, through a dynamic hot-loading mechanism, loads pre-set evaluation rules without interrupting system operation. These evaluation rules cover multiple criteria, including data accuracy, consistency, and completeness. Based on the loaded evaluation rules, the intelligent agent conducts a detailed analysis of various dimensions of the sampled annotated data. For example, it checks whether the annotated data conforms to the specified format, whether the annotated content accurately reflects the actual data, and whether there are any contradictions between different annotations. After comprehensive analysis and processing, the intelligent agent produces a quality assessment result that comprehensively reflects the quality of the sampled annotated data.

[0081] In this embodiment, a dynamic hot loading mechanism is used to load the evaluation rules, making the evaluation process highly flexible and adaptable. Without stopping the system operation, the evaluation rules can be updated and adjusted in a timely manner to cope with changes in data characteristics or new evaluation requirements, which greatly improves the flexibility and adaptability of the system, can quickly respond to various changes, ensure that the quality evaluation work always matches the actual needs, and effectively improve the effect and efficiency of data quality evaluation; the intelligent body of the quality evaluation node evaluates the sampled labeled data based on the loaded rules, and can efficiently quantify the data quality and quickly discover problems in the data. By evaluating the sampled labeled data, the quality of the entire labeled data set is reflected from point to surface. While ensuring the evaluation effect, the amount of data processing is reduced, the evaluation efficiency is improved, and a reliable quality reference is provided for the subsequent processing and application of the data, reducing the risks caused by data quality problems.

[0082] S250: When the quality evaluation result shows that the quality condition is not met, the construction process is re-executed on the processed data set until a labeled data set that meets the quality condition is obtained.

[0083] The technical solution of this embodiment is to obtain the original data set in the target application scenario, and process the original data set in sequence based on the intelligent agents corresponding to the preprocessing nodes and the data processing nodes to obtain a processed data set; the intelligent agent of the data labeling node labels each processed data in the processed data set to obtain a labeled data set, wherein, in the data labeling process, the processed data to be learned is identified, and the intelligent agent of the data labeling node re-learns the reviewed processed data to be learned, and the intelligent agent after learning continues to label other processed data; a quality assessment task is constructed based on the labeled data set, and sampling is performed in the labeled data set to obtain sampled labeled data corresponding to the quality assessment task, and the sampled labeled data meets the coverage of the labeled data set in the data layer dimension and the time dimension; the intelligent agent of the quality assessment node performs quality assessment on the sampled labeled data based on the loaded evaluation rules to obtain a quality assessment result, and the evaluation rules are loaded by a dynamic hot loading mechanism; when the quality assessment result does not meet the quality conditions, the construction process is re-executed on the processed data set until a labeled data set that meets the quality conditions is obtained. This solution realizes the full-process automated processing of the original data set from preprocessing to quality assessment through the intelligent agents corresponding to the preprocessing nodes, data processing nodes, data labeling nodes and quality assessment nodes respectively, solving the problems of lack of fully automated data processing and inaccurate data processing results, improving the efficiency of automated data processing and the accuracy of labeled data sets.

[0084] Figure 3 Schematic diagram of a high-quality dataset construction system provided by an embodiment of the present disclosure. Figure 3 As shown, the system includes:

[0085] The data set preprocessing module 310 is used to obtain the original data set in the target application scenario, and process the original data set based on the intelligent agent corresponding to the preprocessing node to obtain a preprocessed data set;

[0086] a processed data set determination module 320 for processing the pre-processed data set based on the agent corresponding to the data processing node to obtain a processed data set;

[0087] The data set annotation module 330 is configured to annotate each processed data in the processed data set through an agent at a data annotation node to obtain an annotated data set. During the data annotation process, the processed data to be learned is identified, and the verified processed data to be learned is relearned by the agent at the data annotation node. The learned agent then continues to annotate other processed data.

[0088] The quality assessment module 340 is used to perform quality assessment on the labeled data set through the intelligent agent of the quality assessment node to obtain a quality assessment result. When the quality assessment result does not meet the quality conditions, the construction process is re-executed on the processed data set until a labeled data set that meets the quality conditions is obtained.

[0089] The technical solution of this embodiment is to obtain the original data set in the target application scenario through the data set preprocessing module, and process the original data set based on the intelligent agent corresponding to the preprocessing node to obtain the preprocessed data set; the processing data set determination module processes the preprocessed data set based on the intelligent agent corresponding to the data processing node to obtain the processed data set; the data set annotation module annotates each processed data in the processing data set through the intelligent agent of the data annotation node to obtain the annotated data set, wherein, in the data annotation process, the processing data to be learned is identified, and the reviewed processing data to be learned is re-learned by the intelligent agent of the data annotation node, and the intelligent agent after learning continues to annotate other processing data; the quality assessment module performs quality assessment on the annotated data set through the intelligent agent of the quality assessment node to obtain a quality assessment result. When the quality assessment result does not meet the quality conditions, the construction process is re-executed on the processing data set until a annotated data set that meets the quality conditions is obtained. This solution realizes the full-process automated processing of the original data set from preprocessing to quality assessment through the intelligent agents corresponding to the preprocessing nodes, data processing nodes, data labeling nodes and quality assessment nodes respectively, solving the problems of lack of fully automated data processing and inaccurate data processing results, improving the efficiency of automated data processing and the accuracy of labeled data sets.

[0090] On the basis of the above embodiment, optionally, the data labeling node includes an intelligent agent and a review end; the data set labeling module 330 is also used to generate a labeling task based on the processed data to be labeled in the processed data set, and distribute the labeling task to the intelligent agent of the data labeling node in sequence for labeling processing to obtain first labeled data and corresponding confidence; receive the first labeled data and the confidence of the first labeled data, and when the confidence is less than the set confidence threshold, use the first labeled data as the processed data to be learned; generate a review task for the processed data to be learned, and distribute the review task to the review end; receive the second labeled data fed back by the review end, generate a learning task based on the second labeled data, and distribute it to the intelligent agent of the data labeling node for learning; the second labeled data is the reviewed processed data to be learned, and the second labeled data includes standard labeling information of the processed data to be learned.

[0091] Optionally, the data set labeling module 330 is also used to perform sampling processing on the processed data to be labeled in the processed data set to obtain sampled processed data, and distribute the sampled processed data to the review end so that the review end performs labeling processing on the sampled processed data to obtain third labeled data fed back by the review end; obtain the first labeled data obtained by the intelligent agent of the data labeling node on the sampled processed data, and determine the accuracy of the spot check data based on the first labeled data and third labeled data rows corresponding to the sampled processed data; when the accuracy of the spot check data does not meet the sampling quality inspection conditions, the intelligent agent of the data labeling node learns based on the third labeled data, and the intelligent agent after learning re-labels the processed data set.

[0092] Optionally, the review end includes at least one level of processing ends, and the permissions of processing ends at different levels are different; the data set annotation module 330 is also used to transmit the received processed data to the first processing end to obtain the annotation data output by the first processing end; when the annotation data output by the first processing end includes a pending identifier, the annotation data output by the first processing end is transmitted to the second processing end to obtain the annotation data output by the second processing end, wherein the authority of the second processing end is higher than the authority of the first processing end; the confidence of the annotation data output by the second processing end is higher than the annotation data output by the first processing end.

[0093] Optionally, there are multiple processing terminals at the same level; the data set annotation module 330 is also used to transmit the received processed data to multiple first processing terminals respectively, obtain and display the annotation information of the processed data of at least one first processing terminal; when the annotation information of the processed data at different first processing terminals is inconsistent, the processed data and multiple annotation information are transmitted to the second processing terminal to obtain the annotation data output by the second processing terminal.

[0094] Optionally, the quality assessment module 340 is specifically used to construct a quality assessment task based on the labeled data set, and to perform sampling in the labeled data set to obtain sampled labeled data corresponding to the quality assessment task, and the sampled labeled data meets the coverage of the labeled data set in the data layer dimension and the time dimension; the intelligent agent of the quality assessment node performs quality assessment processing on the sampled labeled data based on the loaded evaluation rules, and the evaluation rules are loaded using a dynamic hot loading mechanism.

[0095] Optionally, the intelligent agent of the preprocessing node performs at least one of data cleaning, document parsing, data format conversion, anomaly detection and data deduplication on the original data; the intelligent agent of the data processing node performs at least one of data enhancement, multilingual processing, feature screening and data segmentation on the data set output by the preprocessing node.

[0096] The high-quality dataset construction system provided by the embodiments of the present disclosure can execute the high-quality dataset construction method provided by any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0097] Figure 4 1 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. The electronic device 10 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0098] like Figure 4 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., which is communicatively connected to the at least one processor 11. The memory stores a computer program that can be executed by the at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. Various programs and data required for the operation of the electronic device 10 can also be stored in the random access memory (RAM) 13. The processor 11, the read-only memory (ROM) 12, and the random access memory (RAM) 13 are connected to each other via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0099] Multiple components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0100] The processor 11 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the method for constructing a high-quality dataset.

[0101] In some embodiments, the method for constructing a high-quality dataset may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via a read-only memory (ROM) 12 and / or a communication unit 19. When the computer program is loaded into a random access memory (RAM) 13 and executed by the processor 11, one or more steps of the method for constructing a high-quality dataset described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the method for constructing a high-quality dataset in any other appropriate manner (e.g., by means of firmware).

[0102] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0103] The computer programs for implementing the methods for constructing high-quality datasets disclosed herein can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer programs are executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer programs can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0104] The present disclosure also provides a computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a processor to execute a method for constructing a high-quality dataset, the method comprising:

[0105] Obtain the original data set in the target application scenario, and process the original data set in sequence based on the intelligent agents corresponding to the preprocessing nodes and data processing nodes to obtain the processed data set;

[0106] The intelligent agent of the data annotation node performs an annotation process on each processed data in the processed data set to obtain an annotated data set. In the data annotation process, the processed data to be learned is identified, and the intelligent agent of the data annotation node re-learns the reviewed processed data to be learned. After learning, the intelligent agent continues to annotate other processed data.

[0107] The quality assessment node's intelligent agent performs quality assessment on the labeled dataset to obtain a quality assessment result. When the quality assessment result shows that the quality condition is not met, the construction process is re-executed on the processed dataset until a labeled dataset that meets the quality condition is obtained.

[0108] In the context of the present disclosure, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. A computer-readable storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0109] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0110] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0111] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0112] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions of this disclosure can be achieved, and this document is not limited here.

[0113] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A method for constructing a high-quality dataset, characterized in that: Applied to a high-quality dataset construction system, the high-quality dataset construction system includes an intelligent agent corresponding to each processing node in the high-quality dataset construction process, wherein the processing result of the intelligent agent of the previous processing node is transmitted to the intelligent agent of the next processing node and serves as trigger information for the intelligent agent of the next processing node to start the next processing node until the high-quality dataset construction process is completed; the processing flow sequentially includes the following processing nodes: a preprocessing node, a data processing node, a data annotation node, and a quality assessment node; The method comprises: Obtaining an original data set in a target application scenario, and processing the original data set in sequence based on the intelligent agents corresponding to the preprocessing nodes and the data processing nodes to obtain a processed data set; The intelligent agent of the data annotation node performs an annotation process on each processed data in the processed data set to obtain an annotated data set, wherein the processed data to be learned is identified in the data annotation process, the intelligent agent of the data annotation node re-learns the reviewed processed data to be learned, and the intelligent agent after learning continues to perform annotating process on other processed data; The quality assessment process of the labeled data set is performed by the intelligent agent of the quality assessment node to obtain a quality assessment result. When the quality assessment result does not meet the quality conditions, the construction process is re-executed on the processed data set until a labeled data set that meets the quality conditions is obtained.

2. The method according to claim 1, characterized in that The data annotation node includes an intelligent agent and a review terminal; The method further comprises: Generate a labeling task based on the processed data to be labeled in the processed data set, and distribute the labeling task to the intelligent agent of the data labeling node in sequence for labeling processing to obtain first labeled data and corresponding confidence levels; receiving the first labeled data and a confidence level of the first labeled data, and using the first labeled data as data to be learned and processed when the confidence level is less than a set confidence level threshold; generating a review task for the data to be learned and processed, and distributing the review task to the review terminal; Receive the second labeled data fed back by the review end, generate a learning task based on the second labeled data, and distribute it to the intelligent agent of the data labeling node for learning; the second labeled data is the reviewed data to be learned and processed, and the second labeled data includes standard labeling information of the data to be learned and processed.

3. The method according to claim 2, characterized in that The method further comprises: Sampling the processed data to be labeled in the processed data set to obtain sampled processed data, and distributing the sampled processed data to a review end so that the review end performs labeling on the sampled processed data to obtain third labeled data fed back by the review end; Obtaining first labeled data obtained by processing the sampled processed data by the agent of the data labeling node, and determining the accuracy of the spot-checked data based on the first labeled data and the third labeled data rows corresponding to the sampled processed data; When the accuracy of the sampled data does not meet the sampling quality inspection conditions, the intelligent agent of the data labeling node learns based on the third labeled data, and the intelligent agent re-labels the processed data set after learning.

4. The method according to claim 2 or 3, characterized in that The review end includes at least one level of processing ends, and the processing ends at different levels have different permissions; Transmitting the received processed data to the first processing end to obtain the labeled data output by the first processing end; In the case where the labeled data output by the first processing end includes a pending identifier, the labeled data output by the first processing end is transmitted to the second processing end to obtain the labeled data output by the second processing end, wherein the authority of the second processing end is higher than the authority of the first processing end; and the confidence of the labeled data output by the second processing end is higher than that of the labeled data output by the first processing end.

5. The method according to claim 4, characterized in that There are multiple processing terminals at the same level; The method further comprises: transmitting the received processed data to the plurality of first processing terminals respectively, obtaining and displaying the labeling information of the processed data of at least one of the first processing terminals; When the labeling information of the processed data at different first processing ends is inconsistent, the processed data and the plurality of labeling information are transmitted to the second processing end to obtain the labeling data output by the second processing end.

6. The method according to claim 1, characterized in that The quality assessment process of the labeled data set is performed by the intelligent agent of the quality assessment node, including: Constructing a quality assessment task based on the annotated dataset, and sampling the annotated dataset to obtain sampled annotated data corresponding to the quality assessment task, wherein the sampled annotated data satisfies the coverage of the annotated dataset in the data layer dimension and the time dimension; The intelligent agent of the quality evaluation node performs quality evaluation processing on the sampled labeled data based on the loaded evaluation rules, and the evaluation rules are loaded using a dynamic hot loading mechanism.

7. The method according to claim 1, characterized in that The intelligent agent of the pre-processing node performs at least one of data cleaning, document parsing, data format conversion, anomaly detection and data deduplication on the raw data; The intelligent agent of the data processing node performs at least one of data enhancement, multilingual processing, feature screening and data segmentation on the data set output by the preprocessing node.

8. A system for constructing a high-quality dataset, characterized in that: include: A data set preprocessing module is used to obtain an original data set in a target application scenario, and process the original data set based on the agent corresponding to the preprocessing node to obtain a preprocessed data set; a processing data set determining module, configured to process the pre-processing data set based on the agent corresponding to the data processing node to obtain a processed data set; a data set annotation module, configured to annotate each processed data in the processed data set by the agent of the data annotation node to obtain an annotated data set, wherein the processed data to be learned is identified during the data annotation process, the verified processed data to be learned is re-learned by the agent of the data annotation node, and the learned agent continues to annotate other processed data; The quality assessment module is used to perform quality assessment processing on the labeled data set through the intelligent agent of the quality assessment node to obtain a quality assessment result. When the quality assessment result does not meet the quality conditions, the construction process is re-executed on the processed data set until a labeled data set that meets the quality conditions is obtained.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the method for constructing a high-quality dataset according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the method for constructing a high-quality dataset according to any one of claims 1 to 7 when executed.

Images