Data annotation method and system based on big data management

Abstract

According to the big data management-based data labeling method and system provided by the invention, efficient, accurate and intelligent data labeling is realized by integrating multi-virtual machine collaborative labeling, multi-model fusion labeling, adaptive data preprocessing and hierarchical cleaning and a multi-dimensional task scheduling and dynamic load balancing mechanism. The method comprises the steps that self-adaptive preprocessing and layered cleaning are conducted on original big data, noise is removed, missing values are filled up, and format conversion is completed; dividing the preprocessed data into a plurality of subtasks according to data characteristics; distributing the sub-tasks to different virtual machine instances for parallel processing by utilizing a multi-virtual machine technology; labeling the same data fragment on each virtual machine by adopting a multi-model labeling mechanism; adopting a weighted fusion or voting mechanism to generate a comprehensive annotation result for the output results of the multiple models in each virtual machine; and summarizing the labeling results of the virtual machines to form a global data labeling result.

Description

Technical Field

[0001] The present invention relates to the field of data management, and in particular to a data annotation method and system based on big data management. Background Art

[0002] Data labeling is an important prerequisite for machine learning, deep learning and artificial intelligence applications, and directly affects the accuracy and effect of subsequent model training and prediction. Traditional data labeling methods mainly rely on manual labeling or automatic labeling technology based on a single labeling model, which has the following shortcomings: Manual labeling is costly and inefficient: In large-scale data scenarios, relying on manual labeling of data is not only time-consuming and laborious, but also easily affected by subjective factors, resulting in inconsistency and low accuracy of labeling results.

[0003] Limitations of a single model: Automatic labeling methods based on a single machine learning or deep learning model can usually only adapt to limited data types and scenarios, and cannot fully cope with the complexity and diversity of data. A single model is easily affected by data noise, outliers, or uneven data distribution, which reduces the overall labeling quality.

[0004] Insufficient data preprocessing: Traditional methods often lack automation and adaptability in data cleaning and preprocessing. The labeling requirements of different data sources and types cannot be effectively distinguished and processed, and it is easy to pass noise and erroneous information to the subsequent labeling process, further affecting the accuracy of the labeling results.

[0005] Resource scheduling and load balancing issues: With the surge in data volume, single-machine or traditional parallel processing methods often suffer from insufficient computing resource utilization or unbalanced scheduling when facing large-scale data. The system response speed and processing efficiency are difficult to meet real-time or near real-time business needs.

[0006] Therefore, how to achieve efficient, accurate and automated data annotation in a big data environment has become a technical problem that needs to be solved urgently. This provides an innovative idea for a data annotation method and system based on big data management. By introducing multiple virtual machines, multiple models, adaptive data preprocessing and layered cleaning, as well as multi-dimensional task scheduling and dynamic load balancing mechanisms, it aims to solve the deficiencies in the existing technology and improve the overall efficiency and accuracy of data annotation. Summary of the invention

[0007] The present invention provides a data annotation method based on big data management, comprising the following steps: Adaptively preprocess and clean the raw big data in layers to remove noise, fill in missing values, and complete format conversion; Divide the preprocessed data into multiple subtasks according to data characteristics; Use multi-virtual machine technology to assign each subtask to different virtual machine instances for parallel processing; A multi-model annotation mechanism is used on each virtual machine to annotate the same data fragment separately; The output results of multiple models in each virtual machine are subjected to weighted fusion or voting mechanism to generate comprehensive annotation results; Summarize the annotation results of each virtual machine to form a global data annotation result.

[0008] As described above, a data labeling method based on big data management is performed, wherein adaptive preprocessing and hierarchical cleaning are performed, specifically, corresponding cleaning rules are automatically selected according to data type, data quality and data source to realize hierarchical processing of structured and unstructured data.

[0009] As described above, a data labeling method based on big data management is described, in which multi-virtual machine technology is used to allocate each subtask to different virtual machine instances for parallel processing, specifically including the dynamic creation of virtual machine instances, task scheduling and resource monitoring by the virtualization platform to achieve large-scale data parallel labeling.

[0010] As described above, a data labeling method based on big data management is used, in which a multi-model labeling mechanism is used to label the same data fragments separately, specifically including the use of rule models, traditional machine learning models and deep learning models, and the use of integrated learning methods to fuse the output results of each model.

[0011] As described above, a data annotation method based on big data management is used, in which a weighted fusion or voting mechanism is used to generate a comprehensive annotation result. Specifically, when the annotation results of each model are integrated, the final annotation output is determined according to preset weights or statistical voting rules.

[0012] As described above, a data annotation method based on big data management also includes dynamically adjusting task allocation and resource allocation according to task priority by real-time monitoring of CPU, memory and I / O resource usage of each virtual machine.

[0013] A data labeling method based on big data management as described above, which also includes automatic statistical analysis of labeling results and user feedback, and adaptive optimization of multi-model fusion weights and virtual machine scheduling strategies based on the feedback results.

[0014] The present invention also provides a data annotation system based on big data management, the system comprising: a central management platform and a virtual machine cluster; The central management platform is connected to the virtual machine cluster through a data communication interface, responsible for unified task scheduling, real-time monitoring of virtual machine resources and processing status, and achieving fault tolerance and task redistribution during the execution of the virtual machine, ensuring the efficient and stable operation of the entire system; The virtual machine cluster is used to divide the data sent by the central management platform into several subtasks and evenly distribute each subtask to each independent virtual machine instance. Each virtual machine is equipped with a multi-model annotation module to perform parallel annotation processing on the received data subtasks. The central management platform collects all local annotation results, and standardizes, normalizes and integrates the formats of each result through the result summary module, and finally generates a globally unified data annotation result. The processed annotation data is stored in the distributed data storage system for subsequent query and feedback optimization.

[0015] The beneficial effects achieved by the present invention are as follows: The present invention realizes efficient and accurate labeling of large-scale data by introducing innovative mechanisms such as multi-virtual machine parallel processing, multi-model integrated labeling, adaptive data preprocessing and layered cleaning, multi-dimensional task scheduling and dynamic load balancing. The system not only greatly improves processing efficiency and labeling accuracy, but also uses an automatic feedback mechanism to continuously optimize model parameters and resource allocation strategies, ensuring stable operation in a high-concurrency environment, and has good scalability and adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0017] Figure 1 This is a flow chart of a data annotation method based on big data management provided in Example 1 of the present application. DETAILED DESCRIPTION

[0018] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0019] Embodiment 1 like Figure 1 As shown, the first embodiment of the present application provides a data annotation method based on big data management, which is applicable to intelligent annotation tasks in a large-scale, multi-source data environment, including the following steps: Step S10: Adaptively preprocess and clean the original big data, remove noise, fill missing values ​​and complete format conversion; First, the original data to be annotated is obtained from multiple heterogeneous data sources, which may include but are not limited to text, images, and voice. The system performs preprocessing operations on data from different sources, including cleaning, missing filling, format conversion, and data quality assessment to ensure that the input data for subsequent annotation stages has high accuracy and consistency.

[0020] During the cleaning process, the system performs multi-level denoising and correction on the data by determining the noise ratio and missing conditions. Subsequently, in order to quantitatively evaluate the overall quality after cleaning, the system calculates the data quality index Q for the batch of data. Specifically, the completeness rate of each dimension can be combined with the detection results of abnormal data, and calculated using the following formula: , where N is the total amount of current data; It represents the number of data judged as valid under the i-th quality dimension; is the weight coefficient corresponding to the dimension; L is the total number of quality dimensions; Indicates the number of data judged as abnormal under the jth anomaly detection dimension; is the penalty coefficient of this dimension; M is the number of anomaly detection dimensions; It is an exponential factor used to adjust the overall sensitivity. The calculation result of this indicator Q can directly reflect the overall cleaning effect of the data. If the Q value is low, the system will automatically enhance the denoising and filling measurement rate to further improve the data quality.

[0021] After completing the above assessment, the system will perform appropriate filling operations on missing values ​​according to the data type, and convert data in different formats into a unified standardized format (such as a unified timestamp format, numerical unit or text encoding method). Through adaptive preprocessing and layered cleaning, potential noise and anomalies in the original data can be eliminated or corrected, and the formats between fields can be standardized, thus providing a reliable and high-quality input foundation for the subsequent data annotation process.

[0022] Step S20, dividing the preprocessed data into multiple subtasks according to data characteristics; The system divides the data set into fine-grained categories according to the preprocessed data characteristics (including multi-dimensional features such as data source, data type, timestamp, and geographic location) to achieve efficient parallel processing. Specifically, let the preprocessed data set be D, where each data point All have characteristic vectors In order to divide the data reasonably, the system constructs a segmentation metric, and its calculation formula is described as follows: ,in, is the mean of the i-th feature in the k-th subtask; is the standard deviation of the i-th feature in the k-th subtask; and is the weight coefficient of the i-th feature, reflecting the importance of different features in the division; To prevent the denominator from being zero, k represents the preset or dynamically determined subtask number, k=1,2,...K. For each data point x, the system will calculate its metric for all K subtasks ; Then determine which subtask x should belong to according to the following formula: ,in is the allowable tolerance coefficient, and the subtask k that satisfies the above inequality is selected as the task to which the data point x belongs. The meaning of this inequality is that if the comprehensive metric of the data point x in a subtask k is lower than the average comprehensive metric of other subtasks and is lower than a certain tolerance If the adjusted value of is , then the data point x is considered to have a high similarity with the feature center of the subtask, and thus x is assigned to subtask k.

[0023] Through the above segmentation metrics, the system can divide the preprocessed data set D into K subtasks based on full consideration of the normalized differences and weighted influences of each feature. This method ensures that the data features within the same subtask are highly similar, while different subtasks are clearly distinguished, thereby achieving task load balancing and efficient parallel processing, providing a data foundation for subsequent multi-virtual machine parallel labeling and multi-model collaborative work.

[0024] Step S30: using multi-virtual machine technology to assign each subtask to different virtual machine instances for parallel processing; In this step, the system uses multi-virtual machine technology to assign the subtasks obtained in step S20 to different virtual machine instances for parallel processing. First, the system dynamically creates or schedules a sufficient number of virtual machine instances based on the amount of preprocessed data and the number of subtasks to meet the needs of large-scale data processing. Each virtual machine instance acts as an independent processing unit. After startup, it will load the pre-configured task processing module and wait for the specific subtask to be assigned.

[0025] When assigning subtasks, the system uses an intelligent resource scheduling algorithm to comprehensively consider multiple indicators such as the real-time load, CPU utilization, memory usage, I / O status, etc. of each virtual machine to ensure that the amount of tasks assigned to each virtual machine is balanced in performance. This not only improves the overall processing efficiency, but also prevents system bottlenecks caused by overloading of a single virtual machine. After the assignment is completed, each virtual machine independently processes the received subtasks, including local data preprocessing, feature extraction, and subsequent labeling operations, to ensure that the data will not interfere with each other during parallel processing.

[0026] In addition, during the processing, virtual machines have good isolation and fault tolerance. If a virtual machine fails or has processing delays, the system will promptly reschedule its unfinished subtasks to other healthy instances to ensure the stability and efficiency of the overall process. Finally, after completing their respective tasks, each virtual machine uploads the processing results to the central management platform, which integrates the local results and subsequently integrates the multi-model annotations. Through this distributed and parallel processing method, the system can significantly improve the processing speed and stability of large-scale data annotation tasks.

[0027] Step S40: using a multi-model annotation mechanism on each virtual machine to respectively annotate the same data segment; In step S40, after each virtual machine receives the assigned subtask, it will process the same data segment using a multi-model annotation mechanism. This step specifically includes the following three stages: 1. Independent model annotation stage. In this stage, a variety of fully trained annotation models are deployed inside the virtual machine. These models use different algorithms and feature processing strategies, such as rule-based models, traditional machine learning models, and deep learning models. Each model independently receives the same data input and performs preprocessing, feature extraction, semantic analysis and other processing steps on the data according to the established process to generate its own annotation results. This design ensures that each model can analyze the data from different angles and different strategies to form independent and complete annotation outputs.

[0028] 2. Multi-model collaboration and data consistency assurance. The core of adopting a multi-model annotation mechanism is to make full use of the advantages of each model to make up for the limitations that may exist in a single model. By running multiple models in parallel, the system can simultaneously capture subtle and diverse feature information in the data, thereby improving the accuracy and robustness of the overall annotation. During the annotation process, the system strictly monitors the execution status of each model to ensure that they can make independent judgments according to the predetermined process under the same data input, and ensure that the output results have data consistency and high reliability. This design can not only improve the accuracy of data annotation, but also effectively reduce the risks caused by single model errors.

[0029] 3. Result integration and complementary advantages. Another important part of the multi-model annotation mechanism is the fusion of subsequent results. The annotation results generated by each model are centrally stored and transmitted to the central management module, where the respective judgments are integrated through a preset fusion strategy to form the final annotation output. This fusion process can make full use of the advantages of each model in different data scenarios, make up for the shortcomings of a single model in a specific task, and ensure that the final result has higher accuracy and stronger adaptability. The advantage of multi-model collaboration is that the misjudgments between different models are often inconsistent, and their complementarity can effectively offset their respective defects, thereby achieving the effect of improving the overall annotation performance.

[0030] Among them, each virtual machine labels the data, which specifically includes the following sub-steps: Step S41: Use the rule engine to preliminarily annotate the processed data using domain expert knowledge and historical annotation data to generate candidate labels, providing annotated data for deep learning model training; The rule engine uses the knowledge of domain experts and historical annotation data to perform preliminary annotation on the preprocessed data. The rule engine generates candidate labels based on preset annotation rules and data features, providing preliminary annotation data for subsequent deep learning model training. The specific implementation steps are as follows: According to the type and characteristics of the data, the system initializes the rule engine and loads the predefined annotation rules. The rules include but are not limited to threshold rules, classification rules, and association rules based on data characteristics. The design of the rules is defined by domain experts based on task requirements and data characteristics.

[0031] The rule engine analyzes each piece of data based on the preprocessed data features through rule matching and inference algorithms to generate candidate tags. The candidate label set of , where each label Both represent data A possible classification result.

[0032] The rule engine uses fuzzy logic to label data. Fuzzy logic compares each piece of data with the rule set. The fuzzy reasoning formula is as follows: ,in, For data Candidate labels for For rules The weight of represents the importance of the rule to the annotation result; For rules Data The fuzzy reasoning result indicates that the rule is in the data The purpose of fuzzy reasoning is to deal with uncertainty so that the rules can generate multiple possible labels based on the fuzzy features of the data, thereby increasing the flexibility of labeling.

[0033] For each candidate label , the system calculates its corresponding credibility value . Credibility value Reflects the label With data The matching degree, the calculation formula of the credibility value is as follows: ,in, The credibility of the label; For label The weight coefficient indicates the importance of the label in the annotation; It is a measure of similarity between data and labels. By calculating the credibility, the system can select the most suitable label for each data point and provide multiple candidate labels for subsequent steps. Finally, the system outputs a candidate label set for each data point based on the reasoning results of the rule engine. Each label set contains multiple candidate labels and their credibility, and the system provides these labels as annotated data sets to the deep learning model for training.

[0034] Step S42: Based on the preliminary annotation, the data is automatically annotated using a deep learning model, and the accuracy of the annotation results is calculated through precision evaluation to facilitate subsequent verification and optimization; The generated candidate label data is automatically annotated through the trained deep learning model. The deep learning model uses the characteristics of the input data to assign a label to each piece of data by calculating the predicted score of each label. The specific steps are as follows: First, the system encodes each label in the generated candidate label set to ensure that the data format is unified and suitable for input into the deep learning model. The preprocessed data features are processed by standardization and normalization, converted into feature vectors, and input into the deep learning model for label prediction.

[0035] The deep learning model processes the features of the input data at multiple levels, extracts useful high-dimensional features from the data, and calculates the prediction score for each label. It reflects the matching degree between the label and the feature. The calculation formula is as follows: ,in, is the label of the i-th data The predicted score of is the feature vector of the i-th data; For labels The corresponding k-th order regression coefficient; For label The regularization coefficient of ; is the weight coefficient of the regularization term; is the feature vector The gradient term represents the relationship between features; m is the order of the regression model; p is the order of the regularization term, which controls the weight of the relationship between high-order features. Through this formula, the system calculates the prediction score of each label, indicating the degree of matching of the model to each label.

[0036] After calculating the predicted score of each label, the system selects the label with the maximum predicted score as the final labeling result for each piece of data. In this way, the "credibility" of the label is judged by the maximum score, and the label is assigned to the data.

[0037] After calculating the predicted score for each label, the system performs an accuracy assessment on the labeling results. The accuracy assessment mainly evaluates the accuracy of the labeling results by calculating the error between the predicted label and the true label. The accuracy assessment formula for each sample is as follows: ,in, is the true label of the i-th data; is the predicted score of the predicted label of the i-th data; m is the number of feature dimensions of the data sample. This formula calculates the error between the predicted label value and the true label of each data point to evaluate the accuracy of each sample. The system summarizes the accuracy evaluation of all data points to obtain the labeling accuracy of the entire data set.

[0038] Through accuracy assessment, the system can identify accurate data and possible mislabeled data in the labeling results. The evaluation results provide a basis for subsequent automatic verification and optimization.

[0039] Step S40: automatically identifying and correcting errors in the automatic annotation results through an automated verification module to ensure the accuracy of the annotation results; In step S40, the system automatically identifies and corrects errors in the automatic annotation results generated in step S30 through the automatic verification module to ensure the accuracy and consistency of the annotation results. The specific implementation steps are as follows: The system first performs an accuracy assessment on the labeling results in step S30. The purpose of the accuracy assessment is to automatically detect mislabeled data and correct them through different standards. The accuracy assessment formula is as follows: ,in, is the true label of the i-th data; is the predicted score of the predicted label of the i-th data; n is the number of feature dimensions of the data sample; represents the absolute error between the predicted label and the true label. The accuracy evaluation formula calculates the error between the predicted label and the true label through this method and normalizes it to further evaluate the accuracy of the labeling result. The smaller the error of the sample, the higher the calculated accuracy. For mislabeled data, the error in the formula will be magnified to help the system automatically identify mislabeled data.

[0040] Through the above accuracy evaluation formula, the system automatically identifies the mislabeled data that may exist in the labeling results. Mislabeled data refers to data samples with large errors calculated through accuracy evaluation. The identified mislabeled data will be automatically marked as "pending correction" and enter the next step of the automatic correction process.

[0041] The system automatically selects data samples with low accuracy and large errors for processing based on the evaluation results, and corrects the labels of mislabeled data through the automatic correction module.

[0042] For mislabeled data, the system uses a deep learning model to predict labels again. The system uses the currently trained model to re-estimate the predicted score of each label to generate new labeling results. When re-estimating labels, the system will apply the preset labeling rule engine to ensure that the labeling of each piece of data conforms to the characteristics and contextual information of the data through the reasoning and logical judgment of the rule engine. The corrected data will be automatically stored and marked as "corrected". These data will not only be saved as new labeling results for subsequent use, but will also be returned to the system as correction feedback data for subsequent model optimization and labeling strategy adjustment; the corrected data will be automatically stored in the database and associated with the original data, marked as "corrected" for tracking and management; the corrected data will be input into the optimization module as feedback information to provide support for subsequent model training and optimization. The system further adjusts the model parameters by analyzing the corrected data to improve the accuracy of the labeling.

[0043] Each corrected data will be used in the system's adaptive optimization process. The system uses an adaptive optimization algorithm to update the model based on the corrected data, gradually improving the accuracy of the annotation. The optimization process includes: Model adaptive update: Corrected data is used to optimize the training set. The system will update the deep learning model based on the corrected data, automatically adjust the model parameters, and improve the performance of the model.

[0044] Rule and strategy adjustment: If a pattern of repeated errors occurs during the correction process, the system will automatically adjust the labeling rules or strategies to optimize the label prediction process and avoid similar errors in the future.

[0045] Through continuous verification, correction and feedback, the system can automatically optimize the labeling process, continuously improve the labeling accuracy, and ultimately generate a high-quality labeling data set.

[0046] Step S43: According to the corrected data, the model parameters are automatically adjusted and the data is re-labeled to ensure the accuracy and consistency of the labeling, and finally form a high-quality labeled data set.

[0047] The system automatically adjusts the parameters of the deep learning model based on the generated correction data, and optimizes the accuracy of the model through re-annotation. This step aims to continuously improve the annotation accuracy, reduce mislabeling and omissions, and improve the consistency and accuracy of the annotation results through a feedback mechanism. The specific implementation steps are as follows: The system has automatically identified and corrected the mislabeled data in the annotation results, and returned the corrected data to the system. The system first collects and analyzes the corrected data set to identify which labels have prediction errors. The corrected data includes the original data, the annotation results, and the corresponding correction information.

[0048] The system analyzes these correction data, identifies common error patterns in annotations, and provides a basis for subsequent model optimization. After receiving the correction data, the system automatically adjusts the parameters of the deep learning model through an optimization algorithm. The optimization process improves performance by calculating the gradient of the loss function and updating the model parameters. The optimization formula is as follows: ,in, and are the parameters of the model at the tth and t+1th iterations respectively; is the learning rate, which is dynamically adjusted in each iteration; and are the learning weights and regularization coefficients of the model parameters respectively; L is the loss function; and are the gradient and second-order derivative of the loss function with respect to the parameters; K is the number of dimensions of the model parameters; is a dynamic feedback factor used to adjust the learning speed; is the time decay factor, which indicates the adjustment speed of the model over time; is the start time of the feedback. Through this formula, the system iteratively updates the model parameters according to the correction data, making the model more accurate when processing the labeling task again.

[0049] Based on the optimized model, the system re-labels all unlabeled data. The optimized model can better handle complex data and reduce mislabeling and missing labels. The label of each data point will be re-predicted and updated by the system. The new label prediction value will be stored as the final labeling result for subsequent accuracy evaluation and verification.

[0050] After re-labeling is completed, the system evaluates the accuracy of the new labeling results. Based on the accuracy evaluation and correction feedback, the system will continue to adjust the model parameters and retrain with the optimized data set. This process ensures that the model can continuously improve the accuracy and consistency of labeling after each feedback, thereby providing higher quality labeling results in new labeling tasks.

[0051] Step S50: using a weighted fusion or voting mechanism to generate a comprehensive annotation result for the output results of multiple models in each virtual machine; In step S50, each virtual machine integrates the output results independently generated by the previous multiple annotation models for the same data segment to form the final annotation result. To this end, the system adopts two fusion strategies: weighted fusion and voting mechanism. These two methods can be used alone or in combination to give full play to the advantages of each model and reduce the risk of single model error.

[0052] In the weighted fusion process, each model generates a category probability distribution when annotating data. The system pre-sets a weight for each model to reflect the credibility and historical performance of the model in the overall annotation task. During processing, the system first weights the probability distribution of each model according to its weight, and then superimposes the weighted results of all models to form a comprehensive scoring vector. Specifically, let M be the number of annotation models deployed inside the virtual machine. Each model j outputs a probability vector after feature extraction for the input data segment x. The calculation formula is as follows: ,in, represents the preset weight of model j; f(x) is the feature vector of the input data; and are the weight matrix and bias vector of the lth sublayer in model j respectively; represents the activation function; is the number of sublayers in model j; It reflects the importance of the output of each sub-layer in the overall calculation. This formula performs a weighted product of the multi-level information output by each model and then takes the normalized power mean, ensuring that the comprehensive score can not only reflect the fine-grained probability information of individual models, but also achieve a balanced fusion between models. Finally, by comparing the scores of each category in the comprehensive score vector, the category with the highest score is selected as the final labeling result of the data segment. This method can make full use of the advantages of each model in fine-grained information and reflect the relative contribution of the output results of different models.

[0053] In the voting mechanism, each model selects the most likely category based on its internal calculations and uses this category as its "voting result". The system counts the votes of all models and weights and aggregates the votes according to the preset weights of each model. In the end, the category with the most votes (or the highest weighted votes) is determined as the final labeling result. This strategy effectively suppresses the accidental errors that may be caused by a single model through a clear "majority decision" method, and improves the robustness of labeling decisions to a certain extent.

[0054] The use of these two fusion strategies enables the system to give full play to the strengths of each model when facing diverse data and complex scenarios, obtaining detailed probabilistic information while ensuring the clarity of decision-making, thereby generating accurate and reliable comprehensive labeling results. The entire process is strictly carried out in accordance with the predetermined process, and the fusion results are summarized and verified through the central management platform to ensure that the final output labeling results are highly consistent and reliable.

[0055] Step S60: Aggregate the labeling results of each virtual machine to form a global data labeling result.

[0056] In step S60, after each virtual machine completes the labeling of its assigned subtask, the system centrally aggregates the labeling results of all virtual machines through the central management platform to generate global data labeling results. First, the central platform receives the local labeling outputs from each virtual machine, which include the final labeling category of each data segment, the relevant probability distribution, and the necessary metadata information. The system performs preliminary verification and formatting on the data submitted by each virtual machine to ensure that the data conforms to the unified data format and standard before being aggregated.

[0057] Next, the central management module uses a distributed data integration algorithm to normalize and fuse the annotation results from different virtual machines. For multiple local annotation results of the same data segment, the system uses a pre-set conflict resolution strategy and priority rules to compare, integrate and confirm the annotation opinions from different sources, thereby generating a unified and reliable annotation output. This process not only considers the accuracy and confidence of each local result, but also makes full use of the redundant information of the data to reduce the impact of a single virtual machine error on the global result.

[0058] Finally, the central platform will save the global annotation results after fusion processing to a unified data storage system and provide an interface for subsequent query, verification and feedback iteration.

[0059] Embodiment 2 Embodiment 2 of the present application provides a data annotation system based on big data management, including: a central management platform and a virtual machine cluster; The central management platform is connected to the virtual machine cluster through a data communication interface, responsible for unified task scheduling, real-time monitoring of virtual machine resources and processing status, and achieving fault tolerance and task redistribution during the execution of the virtual machine, ensuring the efficient and stable operation of the entire system; The virtual machine cluster is used to divide the data sent by the central management platform into several subtasks and evenly distribute each subtask to each independent virtual machine instance. Each virtual machine is equipped with a multi-model annotation module to perform parallel annotation processing on the received data subtasks. The central management platform collects all local annotation results, and standardizes, normalizes and integrates the formats of each result through the result summary module, and finally generates a globally unified data annotation result. The processed annotation data is stored in the distributed data storage system for subsequent query and feedback optimization.

[0060] The central management platform includes a data preprocessing and layered cleaning module, which is used to receive raw data from multiple data sources and perform denoising, missing fill, format conversion and layered processing on the data; The data preprocessing and layered cleaning module is responsible for receiving raw data from different data sources (including text, images, voice, etc.), and performing format conversion, denoising, missing fill-in and standardization on the data. Through the adaptive preprocessing algorithm, the data quality indicators are calculated according to the integrity and abnormality of the data, and the data is layered and processed. The high-quality data and the data to be cleaned are sent to the subsequent process respectively, thus providing unified and accurate data input for the entire annotation system.

[0061] Based on the data processed by the preprocessing module, the virtual machine cluster divides the data into several subtasks and evenly distributes each subtask to each independent virtual machine instance. Each virtual machine is configured with a multi-model annotation module to perform parallel annotation processing on the received data subtasks. The preprocessed data is divided into several subtasks according to its characteristics, and then processed in parallel by the virtual machine cluster. The system uses an intelligent task scheduling algorithm to evenly distribute subtasks to each virtual machine instance based on real-time monitoring of virtual machine resources (such as CPU, memory, I / O and other indicators). A multi-model annotation module is pre-deployed inside each virtual machine, which can parallelize and multi-angle annotate the received subtask data, thereby achieving efficient completion of local annotation tasks.

[0062] The system dynamically creates or schedules a sufficient number of virtual machine instances based on the amount of preprocessed data and the number of subtasks to meet the needs of large-scale data processing. Each virtual machine instance acts as an independent processing unit. After startup, it will load the pre-configured task processing module and wait for the specific subtask to be assigned.

[0063] When assigning subtasks, the system uses an intelligent resource scheduling algorithm to comprehensively consider multiple indicators such as the real-time load, CPU utilization, memory usage, I / O status, etc. of each virtual machine to ensure that the amount of tasks assigned to each virtual machine is balanced in performance. This not only improves the overall processing efficiency, but also prevents system bottlenecks caused by overloading of a single virtual machine. After the assignment is completed, each virtual machine independently processes the received subtasks, including local data preprocessing, feature extraction, and subsequent labeling operations, to ensure that the data will not interfere with each other during parallel processing.

[0064] In addition, during the processing, virtual machines have good isolation and fault tolerance. If a virtual machine fails or has processing delays, the system will promptly reschedule its unfinished subtasks to other healthy instances to ensure the stability and efficiency of the overall process. Finally, after completing their respective tasks, each virtual machine uploads the processing results to the central management platform, which integrates the local results and subsequently integrates the multi-model annotations. Through this distributed and parallel processing method, the system can significantly improve the processing speed and stability of large-scale data annotation tasks.

[0065] Each virtual machine integrates the output results independently generated by multiple previous annotation models for the same data segment to form the final annotation result. To this end, the system adopts two fusion strategies: weighted fusion and voting mechanism. These two methods can be used alone or in combination to give full play to the advantages of each model and reduce the risk of single model error.

[0066] The use of these two fusion strategies enables the system to give full play to the strengths of each model when facing diverse data and complex scenarios, obtaining detailed probabilistic information while ensuring the clarity of decision-making, thereby generating accurate and reliable comprehensive labeling results. The entire process is strictly carried out in accordance with the predetermined process, and the fusion results are summarized and verified through the central management platform to ensure that the final output labeling results are highly consistent and reliable.

[0067] The central management platform is connected to the virtual machine cluster through a data communication interface, responsible for unified task scheduling, real-time monitoring of virtual machine resources and processing status, and achieving fault tolerance and task redistribution during the execution of the virtual machine, ensuring the efficient and stable operation of the entire system; The central management platform serves as the command center of the system and communicates with the virtual machine cluster through a high-speed data interface. The platform is responsible for unified task scheduling, resource monitoring, status feedback, and fault tolerance. When an abnormality occurs in a virtual machine, the central management platform can promptly reallocate its unfinished tasks and monitor and adjust the working status of the virtual machine in real time. The platform is also responsible for collecting the local annotation results uploaded by each virtual machine, and providing the necessary metadata information and quality indicator support for the global result summary.

[0068] After each virtual machine has completed the labeling, the central management platform collects all local labeling results, and standardizes, normalizes and integrates the results through the result summary module, and finally generates a globally unified data labeling result. The processed labeling data is stored in the distributed data storage system for subsequent query and feedback optimization.

[0069] After all virtual machines have completed the labeling task, the central management platform performs preliminary verification, format standardization, and normalization on the local labeling results returned by each virtual machine. Using the distributed data integration algorithm, the platform integrates the results, resolves possible conflicts between local labeling results, and forms a globally unified labeling output. Finally, the integrated global labeling results are stored in a high-performance distributed data storage system for query, statistics, feedback, and continuous optimization.

[0070] Corresponding to the above embodiment, an embodiment of the present invention provides a computer storage medium, including: at least one memory and at least one processor; The memory is used to store one or more program instructions; A processor is used to run one or more program instructions to execute a data labeling method based on big data management.

[0071] Corresponding to the above-mentioned embodiments, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer storage medium contains one or more program instructions, and the one or more program instructions are used by a processor to execute a data labeling method based on big data management.

[0072] The embodiments disclosed in the present invention provide a computer-readable storage medium, in which computer program instructions are stored. When the computer program instructions are executed on a computer, the computer executes the above-mentioned data labeling method based on big data management.

[0073] In the embodiment of the present invention, the processor may be an integrated circuit chip having the ability to process signals. The processor may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0074] The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present invention can be directly embodied as a hardware decoding processor for execution, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The processor reads the information in the storage medium and completes the steps of the above method in combination with its hardware.

[0075] The storage medium may be a memory, which may be, for example, a volatile memory or a nonvolatile memory, or may include both volatile and nonvolatile memory.

[0076] Among them, the non-volatile memory can be a read-only memory (ROM), a programmable ROM (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory.

[0077] The volatile memory may be a random access memory (RAM) which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDRSDRAM), enhanced synchronous DRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus DRAM (DRRAM).

[0078] The storage media described in the embodiments of the present invention are intended to include, but are not limited to, these and any other suitable types of memory.

[0079] Those skilled in the art will appreciate that in one or more of the above examples, the functions described in the present invention can be implemented using a combination of hardware and software. When software is used, the corresponding functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any media that facilitates the transmission of computer programs from one place to another. Storage media can be any available media that can be accessed by general or special-purpose computers.

[0080] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solution of the present invention should be included in the scope of protection of the present invention.

Claims

1. A data annotation method based on big data management, characterized in that: The following steps are involved: Adaptively preprocess and clean the raw big data in layers to remove noise, fill in missing values, and complete format conversion; Divide the preprocessed data into multiple subtasks according to data characteristics; Use multi-virtual machine technology to assign each subtask to different virtual machine instances for parallel processing; A multi-model annotation mechanism is used on each virtual machine to annotate the same data fragment separately; The output results of multiple models in each virtual machine are subjected to weighted fusion or voting mechanism to generate comprehensive annotation results; Summarize the annotation results of each virtual machine to form a global data annotation result.

2. A data annotation method based on big data management as claimed in claim 1, characterized in that: Perform adaptive preprocessing and hierarchical cleaning, specifically automatically selecting corresponding cleaning rules according to data type, data quality and data source, to achieve hierarchical processing of structured and unstructured data.

3. A data annotation method based on big data management as claimed in claim 1, characterized in that: Multi-virtual machine technology is used to assign each subtask to different virtual machine instances for parallel processing, including the dynamic creation of virtual machine instances, task scheduling and resource monitoring by the virtualization platform, to achieve large-scale data parallel labeling.

4. The data annotation method based on big data management according to claim 1, characterized in that: A multi-model annotation mechanism is used to annotate the same data fragment separately, including the use of rule models, traditional machine learning models and deep learning models, and the output results of each model are integrated using an integrated learning method.

5. A data annotation method based on big data management as claimed in claim 4, characterized in that: A weighted fusion or voting mechanism is used to generate comprehensive annotation results. Specifically, when integrating the annotation results of each model, the final annotation output is determined according to preset weights or statistical voting rules.

6. A data annotation method based on big data management as claimed in claim 1, characterized in that: It also includes real-time monitoring of the CPU, memory and I / O resource usage of each virtual machine, and dynamically adjusting task allocation and resource allocation based on task priority.

7. The data annotation method based on big data management according to claim 1, characterized in that: It also includes automatic statistical analysis of annotation results and user feedback, and adaptive optimization of multi-model fusion weights and virtual machine scheduling strategies based on the feedback results.

8. A data annotation system based on big data management, characterized in that: The system includes: a central management platform and a virtual machine cluster; The central management platform is connected to the virtual machine cluster through a data communication interface, responsible for unified task scheduling, real-time monitoring of virtual machine resources and processing status, and achieving fault tolerance and task redistribution during the execution of the virtual machine, ensuring the efficient and stable operation of the entire system; The virtual machine cluster is used to divide the data sent by the central management platform into several subtasks and evenly distribute each subtask to each independent virtual machine instance. Each virtual machine is equipped with a multi-model annotation module to perform parallel annotation processing on the received data subtasks. The central management platform collects all local annotation results, and standardizes, normalizes and integrates the formats of each result through the result summary module, and finally generates a globally unified data annotation result. The processed annotation data is stored in the distributed data storage system for subsequent query and feedback optimization.

Images