A method and system for automatic data annotation based on deep learning

Abstract

The present invention discloses an automatic data annotation method and system based on deep learning, which relates to the field of data annotation technology, including collecting text data from multi-source data sources for preprocessing and extracting data word vectors, performing preliminary manual annotation on small-scale text data and constructing a dynamic prompt template; constructing a deep learning model based on the dynamic prompt template for automatic data annotation and evaluating the quality of the annotated data; storing the annotated data in a database and implementing access control. The present invention preprocesses multi-source data and constructs a vocabulary co-occurrence matrix and a hybrid word vector model to deeply extract word vectors of multi-source data for fusion, constructs a dynamic prompt template based on manually annotated samples, and constructs a deep learning model based on dynamic template data for automatic data annotation, provides high-quality word vectors for model training, helps the model to perform annotation tasks more accurately, and improves the automation level of data annotation and the quality of annotated data.

Description

Technical Field

[0001] The present invention relates to the field of data annotation technology, and in particular to an automated data annotation method and system based on deep learning. Background Art

[0002] With the rapid development of information technology, the amount of data has exploded, and text data has accumulated rapidly on the Internet. The amount is huge. These text data come from a wide range of sources, including social media, online news, academic papers, etc. They have become an indispensable resource for machine learning and data analysis. In this context, automated data labeling technology is particularly critical because it can provide training data for machine learning models, thereby supporting various natural language processing applications such as sentiment analysis and topic identification. Deep learning, as a powerful learning tool, has shown great potential in processing these complex text data. Although deep learning has been widely used, most systems still rely on a single data source or a simple word vector model, which limits the model's ability to process diverse text data. The feature fusion capabilities of traditional methods in a multi-source data environment need to be improved, and the accuracy and consistency of data labeling also need to be further improved. Summary of the Invention

[0003] In view of the problems existing in the above-mentioned existing deep learning-based automatic data annotation methods and systems, the present invention is proposed.

[0004] Therefore, the problem to be solved by the present invention is that most systems still rely on a single data source or a simple word vector model, which limits the model's ability to process diverse text data. The feature fusion capability of traditional methods in a multi-source data environment needs to be improved, and the accuracy and consistency of data annotation also need to be further improved.

[0005] To solve the above technical problems, the present invention provides the following technical solutions: a method for automatic data labeling based on deep learning, which includes collecting text data from multiple data sources, preprocessing and extracting data word vectors, performing preliminary manual labeling on small-scale text data and constructing dynamic prompt templates; constructing a deep learning model based on the dynamic prompt template to perform automatic data labeling and evaluate the quality of the labeled data; storing the labeled data in a database and implementing access control.

[0006] As a preferred solution of the deep learning-based automatic data annotation method of the present invention, wherein: the collecting text data from multiple data sources for preprocessing and extracting data word vectors refers to collecting text data from various data sources, preprocessing the text data, normalizing the text, converting all characters to lowercase, removing redundant spaces and punctuation, and using Spacy to decompose the text into separate vocabulary units and mark the part of speech of each word;

[0007] Generate a word co-occurrence matrix based on the preprocessed text data to capture the semantic associations between words. Select a window size of 5 words, slide the window through the text data, record the co-occurrence of word pairs within the window, count the number of times word pairs co-occur within the window, and generate a co-occurrence matrix.

[0008] Use graph data structure to create a vocabulary relationship graph, with each word in the co-occurrence matrix as a node, the co-occurrence relationship between words as an edge, and the co-occurrence frequency of word pairs as the weight of the edge;

[0009] Use TF-IDF to adjust the edge weights and calculate the adjusted edge weights:

[0010]

[0011] Where w bc is the word pair v b and v c The edge weight, C bc is the word pair v b and v c The number of co-occurrences of b ) and f(v c ) are word pairs v b and v c Frequency of occurrence in the text;

[0012] Build a hybrid word embedding model, including the GloVe model and the BERT model. Use the pre-trained GloVe model to map each word node in the semantic relationship graph to the corresponding GloVe word embedding. Use the pre-trained BERT model to process the text through the BERT tokenizer, converting the words in the semantic relationship graph into tokens that BERT can understand. Input the processed tokens into the BERT model to obtain the context-based embedding output vector for each token.

[0013] Configure two main input channels, one channel processes static word vectors extracted from GloVe, and the other channel processes dynamic word vectors obtained from the BERT model;

[0014] Configure a linear transformation layer for the GloVe channel, convert the 300-dimensional GloVe vector to a dimension compatible with the BERT vector, and initialize the weights and bias of the linear layer;

[0015] Configure a fully connected layer for the BERT channel to keep the dimensions of the BERT vector consistent with the GloVe vector dimension;

[0016] Construct a graph neural network and initialize it using the features of each node in the vocabulary relationship graph. The initialized node feature of each node a is the row vector of the vocabulary in the co-occurrence matrix. Use the average aggregation method to update the feature of each node to the average of its neighboring node features:

[0017]

[0018] Where, is the node p in the l+1th layer q is the eigenvector, N(q) is the node p q The set of neighbor nodes of q, |N(q)| is the number of neighbor nodes, σ is the ReLU activation function, and y is the neighbor node of q;

[0019] The same aggregation operation is performed on each layer of node features to allow information to be passed layer by layer to obtain the final node features. The obtained node features are input into the MSLFN network, and features of different scales are extracted through multi-scale convolutional layers;

[0020] The multi-scale features are fused by weighted averaging to generate the final multi-scale word vector representation, and the obtained comprehensive word vector is output through the output layer;

[0021] Use the preprocessed text data as training data to input the hybrid word vector model for iterative training. Define the loss function and Adam optimizer to iteratively optimize the model parameters. When the loss of the hybrid word vector model no longer decreases significantly during continuous iterations, stop the iteration, output the model parameters, and update the hybrid word vector model.

[0022] The data to be labeled is input into the hybrid word vector model to obtain the comprehensive word vector of the data, and a word vector set is generated based on the comprehensive word vector.

[0023] As a preferred solution of the deep learning-based automatic data annotation method described in the present invention, the preliminary manual annotation of small-scale text data refers to using LightTag as an annotation tool, selecting data of different categories and topics from the preprocessed data and randomly extracting 1% as annotation samples, and each text is independently annotated by two annotators on entity type, sentiment and classification, and the text with inconsistent annotations is analyzed and modified to unify the annotation results.

[0024] As a preferred solution of the deep learning-based automatic data annotation method of the present invention, the construction of the dynamic prompt template includes:

[0025] Use a clustering algorithm to cluster word vectors, test the number of clusters required by the clustering algorithm, determine the test range of the number of clusters, gradually increase the number of clusters g, use the K-means algorithm to cluster each g value, and calculate the total within-cluster sum of squares for each cluster;

[0026] Use the data visualization library to draw a line graph of the g value and the corresponding WSS, identify the elbow point where the rate of decrease of WSS with increasing g suddenly slows down, and obtain the g value;

[0027] Apply the K-means algorithm to cluster the word vector dataset and calculate the silhouette coefficient to evaluate the quality of each cluster;

[0028] The value range of the silhouette coefficient is [-1, 1]. Clusters with silhouette coefficients in the range of [0.5, 1] ​​are selected for further analysis. Frequently occurring entity and emotion combinations are extracted from each selected cluster, and dynamic prompt templates are created. Each template contains multiple entity type and emotion combinations. A periodic evaluation process is set up. If new entity-emotion combinations are detected in new data, the existing dynamic prompt templates are updated according to the new trends discovered.

[0029] As a preferred solution of the deep learning-based automatic data labeling method of the present invention, wherein: the deep learning model constructed based on the dynamic prompt template for automatic data labeling includes:

[0030] Choose to use the GPT-3 model provided by OpenAI, obtain access rights to GPT-3, and initialize the basic configuration of the model;

[0031] Combine small-scale manually annotated data with data from dynamic prompt templates to generate training samples containing annotation instructions. Use the training samples to fine-tune GPT-3, adjust fine-tuning hyperparameters, including the learning rate and number of training rounds, define a cross-entropy loss function, calculate the gradient of the loss function with respect to each weight through the backpropagation algorithm, and apply gradient descent to update the weights. During training, calculate the model loss in real time. If the model loss does not decrease in three consecutive cycles, stop training to obtain the GPT-3 model.

[0032] The trained GPT-3 model is integrated into the annotation system through the API, and the unannotated text and related dynamic prompt template data are sent to the GPT-3 model. The GPT-3 model generates corresponding text annotations based on the provided template content.

[0033] As a preferred embodiment of the deep learning-based automated data annotation method of the present invention, the method of evaluating the quality of the annotated data comprises randomly selecting 10% of the annotated data as a validation set, and calculating the accuracy, precision, recall, and F1 score of the validation set to evaluate the quality of the annotation.

[0034] Correct the detected labeling errors and analyze the error types. Adjust the model training parameters and labeling guidelines based on the errors found. Update the dynamic prompt template. Record the evaluation results and corrective measures and generate an evaluation report.

[0035] As a preferred solution of the deep learning-based automated data annotation method of the present invention, wherein: the storing of the annotated data in a database and implementing access control refers to selecting PostgreSQL as the database system for storing the annotated data, establishing a table structure to store various annotated data and an evaluation report of incorrectly annotated data, wherein the annotated data includes the annotated text, annotation type, standardizer information, and timestamp;

[0036] Implement access control policies to allow only authorized users to access sensitive data, and perform regular data backups, including full and incremental backups.

[0037] Another object of the present invention is to provide an automatic data annotation system based on deep learning, which comprises:

[0038] The data collection module is used to collect text data from various data sources, perform preliminary processing on the text data, and build a hybrid word vector model to generate comprehensive word vectors for the text data;

[0039] Dynamic prompt template module, which is used to manually annotate small-scale text data and construct dynamic prompt templates based on the generated comprehensive word vectors;

[0040] The automatic annotation module is used to build the GPT-3 model in the deep learning model and combine manually annotated data with dynamic prompt templates for deep learning training, and use the trained model for automatic data annotation;

[0041] The quality assessment module is used to extract a portion of data from the results of automated annotation to verify data quality, correct errors based on the results, and generate an assessment report;

[0042] The data storage module is used to store the labeled data and the incorrectly labeled assessment reports and perform access control and regular backup.

[0043] A computer device comprises: a memory and a processor; the memory stores a computer program, and the processor implements the steps of an automatic data labeling method based on deep learning when executing the computer program.

[0044] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of an automated data labeling method based on deep learning.

[0045] The beneficial effects of the present invention are as follows: the present invention preprocesses multi-source data and constructs a vocabulary co-occurrence matrix and a hybrid word vector model to deeply extract word vectors from multi-source data for fusion, constructs a dynamic prompt template based on manually annotated samples, and constructs a deep learning model based on the dynamic template data for automatic data annotation, providing high-quality word vectors for model training, which helps the model to perform annotation tasks more accurately, thereby improving the automation level of data annotation and the quality of annotated data. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0047] Figure 1 Schematic diagram of the process of automatic data labeling method based on deep learning.

[0048] Figure 2 Schematic diagram of the structure of the automatic data labeling system based on deep learning. DETAILED DESCRIPTION

[0049] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0050] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0051] Secondly, the term "one embodiment" or "embodiment" herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in various places throughout this specification does not necessarily refer to the same embodiment, nor does it designate a separate or selective embodiment that is mutually exclusive with other embodiments.

[0052] Example 1

[0053] Reference Figure 1, is the first embodiment of the present invention, which provides an automatic data labeling method based on deep learning. The automatic data labeling method based on deep learning includes:

[0054] S1. Collect text data from multiple data sources, preprocess it, and extract word vectors. Perform preliminary manual annotation on small-scale text data and build dynamic prompt templates.

[0055] Specifically, collecting text data from multiple data sources for preprocessing and extracting data word vectors refers to collecting text data from various data sources, preprocessing the text data, normalizing the text, converting all characters to lowercase, removing redundant spaces and punctuation, and using Spacy to decompose the text into separate vocabulary units and mark the part of speech of each word;

[0056] Generate a word co-occurrence matrix based on the preprocessed text data to capture the semantic associations between words. Select a window size of 5 words, slide the window through the text data, record the co-occurrence of word pairs within the window, count the number of times word pairs co-occur within the window, and generate a co-occurrence matrix:

[0057]

[0058] Where C bc is the word pair v b and v c The number of co-occurrences, the total number of windows N, w k is the kth window, f(v b ,v c |w k ) is the word pair v in the k-th window b and v c The co-occurrence frequency of

[0059] Use graph data structure to create a vocabulary relationship graph, with each word in the co-occurrence matrix as a node, the co-occurrence relationship between words as an edge, and the co-occurrence frequency of word pairs as the weight of the edge;

[0060] Use TF-IDF to adjust the edge weights and calculate the adjusted edge weights:

[0061]

[0062] Where w bc is the word pair v b and v c The edge weight, C bc is the word pair v b and v c The number of co-occurrences of b ) and f(v c ) are word pairs v band v c Frequency of occurrence in the text;

[0063] Build a hybrid word embedding model, including the GloVe model and the BERT model. Use the pre-trained GloVe model to map each word node in the semantic relationship graph to the corresponding GloVe word embedding. Use the pre-trained BERT model to process the text through the BERT tokenizer, converting the words in the semantic relationship graph into tokens that BERT can understand. Input the processed tokens into the BERT model to obtain the context-based embedding output vector for each token.

[0064] Configure two main input channels, one channel processes static word vectors extracted from GloVe, and the other channel processes dynamic word vectors obtained from the BERT model;

[0065] Configure a linear transformation layer for the GloVe channel, convert the 300-dimensional GloVe vector to a dimension compatible with the BERT vector, and initialize the weights and bias of the linear layer;

[0066] Configure a fully connected layer for the BERT channel to keep the dimensions of the BERT vector consistent with the GloVe vector dimension;

[0067] Construct a graph neural network and initialize it using the features of each node in the vocabulary relationship graph. The initialized node feature of each node a is the row vector of the vocabulary in the co-occurrence matrix. Use the average aggregation method to update the feature of each node to the average of its neighboring node features:

[0068]

[0069] Where, is the node p in the l+1th layer q is the eigenvector, N(q) is the node p q The set of neighbor nodes of q, |N(q)| is the number of neighbor nodes, σ is the ReLU activation function, and y is the neighbor node of q;

[0070] The same aggregation operation is performed on each layer of node features to pass information layer by layer to obtain the final node features. The obtained node features are input into the MSLFN network, and features of different scales are extracted through the multi-scale convolution layer:

[0071]

[0072] Where, represents the features extracted by scale s at position (i, j), is the weight of the convolution kernel at position (m,n), b (s) is the bias term of scale s, k s is the convolution kernel size, Xi+m-1,j+n-1 is the value of the input vector at position (i+m-1,j+n-1);

[0073] The multi-scale features are fused by weighted averaging to generate the final multi-scale word vector representation:

[0074]

[0075] Where L is the value of the fused feature at position (i, j), α s is the weight at scale s, S is the total number of scales;

[0076] The obtained comprehensive word vector is output through the output layer;

[0077] Use the preprocessed text data as training data to input the hybrid word vector model for iterative training. Define the loss function and Adam optimizer to iteratively optimize the model parameters. When the loss of the hybrid word vector model no longer decreases significantly during continuous iterations, stop the iteration, output the model parameters, and update the hybrid word vector model.

[0078] The data to be labeled is input into the hybrid word vector model to obtain the comprehensive word vector of the data, and a word vector set is generated based on the comprehensive word vector.

[0079] By integrating multi-source data sources, the system can process text data from different fields and styles, enhance the versatility and adaptability of the model, use graph data structure to create a vocabulary relationship graph, adjust edge weights through TF-IDF, effectively highlight important vocabulary relationships, reduce the influence of high-frequency, low-information vocabulary, and improve the expressiveness of the graph structure. It combines the static word vectors of the GloVe model and the dynamic word vectors of the BERT model to form a hybrid word vector model, which has both global semantic information and context dependency, and improves the richness and accuracy of word vector representation. It aggregates neighbor node information through graph neural networks, transfers and updates node features layer by layer, and combines multi-scale convolutional layers to extract features at different levels, and finally generates multi-scale word vector representation, which improves the comprehensiveness and depth of feature extraction. It uses the Adam optimizer and loss function to iteratively train the model to ensure the effective update and optimization of model parameters to achieve the optimal state, improve the accuracy and robustness of the model, and generate comprehensive word vectors based on the hybrid word vector model to capture complex semantic relationships in the text, support efficient and accurate data labeling, and improve labeling quality.

[0080] Furthermore, preliminary manual annotation of small-scale text data refers to using LightTag as an annotation tool, selecting data of different categories and topics from the preprocessed data and randomly sampling 1% as annotation samples. Each text is independently annotated by two annotators on entity type, sentiment, and classification. Texts with inconsistent annotations are analyzed and modified to unify the annotation results.

[0081] LightTag provides an efficient annotation platform that supports multiple annotation tasks, with an intuitive user interface and team collaboration capabilities. By using LightTag, annotators can complete annotation tasks quickly and accurately, improving annotation efficiency and quality. By randomly sampling 1% of the data for annotation, the annotation workload can be reduced and costs can be reduced while ensuring data representativeness. The random sampling method ensures the diversity and comprehensiveness of the selected samples and avoids bias. Each text is independently annotated by two annotators to ensure the accuracy and consistency of the annotation results. When the annotation results of the two annotators are inconsistent, the annotation results are unified through further analysis and discussion, which improves the annotation quality. The inconsistently annotated texts are analyzed and modified, and the annotation results are unified to ensure the consistency and accuracy of the data. This process not only improves the annotation quality, but also helps annotators improve their skills and consistency through continuous feedback.

[0082] Furthermore, building a dynamic prompt template includes,

[0083] Use a clustering algorithm to cluster word vectors, test the number of clusters required by the clustering algorithm, determine the test range of the number of clusters, gradually increase the number of clusters g, use the K-means algorithm to cluster each g value, and calculate the total within-cluster sum of squares WSS for each cluster:

[0084]

[0085] Where g is the number of clusters, Q d is the dth cluster, x is the cluster Q d Data points, μ d is the center of the d-th cluster;

[0086] Use the data visualization library to draw a line graph of the g value and the corresponding WSS, identify the elbow point where the rate of decrease of WSS with increasing g suddenly slows down, and obtain the g value;

[0087] Apply the K-means algorithm to cluster the word vector dataset and calculate the silhouette coefficient to evaluate the quality of each cluster:

[0088]

[0089] Where C is the silhouette coefficient, r is the average distance between samples in the same cluster, and t is the average distance between the samples in the nearest cluster and the current sample;

[0090] The value range of the silhouette coefficient is [-1, 1]. Clusters with silhouette coefficients in the range of [0.5, 1] ​​are selected for further analysis. Frequently occurring entity and emotion combinations are extracted from each selected cluster, and dynamic prompt templates are created. Each template contains multiple entity type and emotion combinations. A periodic evaluation process is set up. If new entity-emotion combinations are detected in new data, the existing dynamic prompt templates are updated according to the new trends discovered.

[0091] By clustering word vectors using a clustering algorithm, semantic patterns in text data can be effectively identified and classified. Clustering algorithms group similar words together, which helps to extract the main themes and emotional features of the text. By testing different numbers of clusters, the optimal number of clusters g can be determined, thereby improving the quality and accuracy of the clustering results. Using the elbow rule and silhouette coefficient method for evaluation, the number of clusters can be selected scientifically and reasonably. By calculating the WSS corresponding to each cluster number g and drawing a line graph, the trend of clustering effect changing with the number of clusters can be intuitively observed. By identifying the elbow point, the optimal value of the number of clusters can be determined. The silhouette coefficient comprehensively considers the cohesion and separation of the clusters and is an effective indicator for evaluating clustering quality. By calculating the silhouette coefficient, high-quality clustering results can be screened out to ensure the accuracy and reliability of the dynamic prompt template. Frequently occurring entity and emotion combinations are extracted from high-quality clustering results, and dynamic prompt templates are created. This can provide structured guidance information for subsequent text processing and analysis, improving the efficiency and accuracy of annotation and analysis.

[0092] S2. Build a deep learning model based on dynamic prompt templates to automatically label data and evaluate the quality of the labeled data;

[0093] Specifically, building a deep learning model based on dynamic prompt templates for automatic data annotation includes:

[0094] Choose to use the GPT-3 model provided by OpenAI, obtain access rights to GPT-3, and initialize the basic configuration of the model;

[0095] Combine small-scale manually annotated data with data from dynamic prompt templates to generate training samples containing annotation instructions. Use the training samples to fine-tune GPT-3, adjust fine-tuning hyperparameters, including the learning rate and number of training rounds, define a cross-entropy loss function, calculate the gradient of the loss function with respect to each weight through the backpropagation algorithm, and apply gradient descent to update the weights. During training, calculate the model loss in real time. If the model loss does not decrease in three consecutive cycles, stop training to obtain the GPT-3 model.

[0096] The trained GPT-3 model is integrated into the annotation system through the API, and the unannotated text and related dynamic prompt template data are sent to the GPT-3 model. The GPT-3 model generates corresponding text annotations based on the provided template content.

[0097] Choosing the GPT-3 model provided by OpenAI for automated data annotation can fully utilize its powerful language generation and comprehension capabilities to improve the accuracy and efficiency of annotation. The GPT-3 model has rich semantic understanding capabilities and can handle complex text annotation tasks. By obtaining access to GPT-3 and initializing the basic configuration of the model, you can ensure that the model runs in an optimal state and fully utilizes its performance advantages. The initialization configuration includes steps such as setting model parameters and loading pre-trained weights. Generating training samples by combining small-scale manually annotated data with data from dynamic prompt templates can improve the quality and diversity of training data and enhance the generalization and adaptability of the model. The dynamic prompt template provides structured guidance information to help the model better understand and process text data. Fine-tuning GPT-3 using training samples containing annotation instructions can make the model better adapt to specific task requirements and further improve the accuracy and reliability of annotation. During the fine-tuning process, model performance is optimized by adjusting hyperparameters such as the learning rate and number of training rounds.

[0098] Furthermore, the quality of the labeled data is evaluated by randomly selecting 10% of the labeled data as a validation set.

[0099] Calculate the accuracy, precision, recall and F1 score of the validation set to evaluate the quality of the annotation;

[0100] Correct the detected labeling errors and analyze the error types. Adjust the model training parameters and labeling guidelines based on the errors found. Update the dynamic prompt template. Record the evaluation results and corrective measures and generate an evaluation report.

[0101] Randomly selecting 10% of the labeled data as a validation set ensures sample diversity and representativeness, making the evaluation results more objective and accurate. Random sampling reduces sample selection bias and improves the reliability of model performance evaluation. By calculating the accuracy, precision, recall, and F1 score of the validation set, the quality of the labeled data and the performance of the model can be comprehensively evaluated. These metrics can reveal the strengths and weaknesses of the model in different aspects and guide subsequent model optimization and improvement. Correcting detected labeling errors and analyzing the error types can help identify common problems and weak links in the labeling process. By analyzing the error types, the labeling guidelines and model training can be improved in a targeted manner to reduce the occurrence of similar errors. Adjusting the model training parameters and labeling guidelines based on the discovered labeling errors can improve the model's adaptability and labeling accuracy. Dynamically adjusting the labeling guidelines ensures that the labeling process always meets the latest quality standards. Based on the labeling errors and model optimization results, the dynamic prompt template is updated to reflect the latest labeling standards and data characteristics. Updating the dynamic prompt template helps improve labeling efficiency and quality and reduce the workload of annotators.

[0102] S3. Store the labeled data in the database and implement access control;

[0103] Specifically, implementing access control on the annotated data stored in the database involves selecting PostgreSQL as the database system for storing the annotated data, establishing a table structure to store various annotated data and an evaluation report on incorrectly annotated data, and the annotated data includes the annotated text, annotation type, standardizer information, and timestamp;

[0104] Implement access control policies to allow only authorized users to access sensitive data, and perform regular data backups, including full and incremental backups.

[0105] PostgreSQL was chosen as the database system. Its powerful data processing capabilities and rich functions allow for efficient storage and management of labeled data. PostgreSQL supports complex queries and transaction processing to ensure data consistency and integrity. By establishing a table structure, various labeled data and assessment reports on incorrectly labeled data can be systematically stored and managed. The table structure facilitates data classification and query, improving data organization and retrieval efficiency. When storing labeled data, detailed information such as the labeled text, annotation type, standardizer information, and timestamps is included, which helps to comprehensively record and track the labeling process. The timestamp can accurately record the generation and modification time of the data, facilitating the management and query of historical data.

[0106] Example 2

[0107] Reference Figure 2, which is the second embodiment of the present invention, is different from the previous embodiment and provides an automatic data labeling system based on deep learning, which includes:

[0108] The data collection module is used to collect text data from various data sources, perform preliminary processing on the text data, and build a hybrid word vector model to generate comprehensive word vectors for the text data;

[0109] Dynamic prompt template module, which is used to manually annotate small-scale text data and construct dynamic prompt templates based on the generated comprehensive word vectors;

[0110] The automatic annotation module is used to build the GPT-3 model in the deep learning model and combine manually annotated data with dynamic prompt templates for deep learning training, and use the trained model for automatic data annotation;

[0111] The quality assessment module is used to extract a portion of data from the results of automated annotation to verify data quality, correct errors based on the results, and generate an assessment report;

[0112] The data storage module is used to store the labeled data and the incorrectly labeled assessment reports and perform access control and regular backup.

[0113] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0114] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0115] More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering, or processing in another suitable manner as necessary, and then stored in a computer memory.

[0116] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used to implement: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

Claims

1. An automated data annotation method based on deep learning, characterized by: include, Collect text data from multiple data sources, pre-process it, and extract word vectors. Perform preliminary manual annotation on small-scale text data and build dynamic prompt templates. Build a deep learning model based on dynamic prompt templates to automatically label data and evaluate the quality of labeled data; Store the annotated data in the database and implement access control; The collecting of text data from multiple data sources for preprocessing and extracting data word vectors refers to collecting text data from various data sources, preprocessing the text data, normalizing the text, converting all characters to lowercase, removing redundant spaces and punctuation, and using Spacy to decompose the text into separate vocabulary units and mark the part of speech of each word; Generate a word co-occurrence matrix based on the preprocessed text data to capture the semantic associations between words. Select a window size of 5 words, slide the window through the text data, record the co-occurrence of word pairs within the window, count the number of times word pairs co-occur within the window, and generate a co-occurrence matrix. Use graph data structure to create a vocabulary relationship graph, with each word in the co-occurrence matrix as a node, the co-occurrence relationship between words as an edge, and the co-occurrence frequency of word pairs as the weight of the edge; Use TF-IDF to adjust the edge weights and calculate the adjusted edge weights: , Where w bc is the word pair v b and v c The edge weight, C bc is the word pair v b and v c The number of co-occurrences of b ) and f(v c ) are word pairs v b and v c Frequency of occurrence in the text; Build a hybrid word embedding model, including the GloVe model and the BERT model. Use the pre-trained GloVe model to map each word node in the semantic relationship graph to the corresponding GloVe word embedding. Use the pre-trained BERT model to process the text through the BERT tokenizer, converting the words in the semantic relationship graph into tokens that BERT can understand. Input the processed tokens into the BERT model to obtain the context-based embedding output vector for each token. Configure two main input channels, one channel processes static word vectors extracted from GloVe, and the other channel processes dynamic word vectors obtained from the BERT model; Configure a linear transformation layer for the GloVe channel, convert the 300-dimensional GloVe vector to a dimension compatible with the BERT vector, and initialize the weights and bias of the linear layer; Configure a fully connected layer for the BERT channel to keep the dimensions of the BERT vector consistent with the GloVe vector dimension; Construct a graph neural network and initialize it using the features of each node in the vocabulary relationship graph. The initialized node feature of each node a is the row vector of the vocabulary in the co-occurrence matrix. Use the average aggregation method to update the feature of each node to the average of its neighboring node features: , Where, is the node p in the l+1th layer q is the eigenvector, N(q) is the node p q The set of neighbor nodes, |N(q)| is the number of neighbor nodes, is the ReLU activation function, y is the neighbor node of q; The same aggregation operation is performed on each layer of node features to allow information to be passed layer by layer to obtain the final node features. The obtained node features are input into the MSLFN network, and features of different scales are extracted through multi-scale convolutional layers; The multi-scale features are fused by weighted averaging to generate the final multi-scale word vector representation, and the obtained comprehensive word vector is output through the output layer; Use the preprocessed text data as training data to input the hybrid word vector model for iterative training. Define the loss function and Adam optimizer to iteratively optimize the model parameters. When the loss of the hybrid word vector model no longer decreases significantly during continuous iterations, stop the iteration, output the model parameters, and update the hybrid word vector model. Input the data to be labeled into the hybrid word vector model to obtain the comprehensive word vector of the data, and generate a word vector set based on the comprehensive word vector; The preliminary manual annotation of small-scale text data refers to using LightTag as an annotation tool, selecting data of different categories and topics from the preprocessed data and randomly sampling 1% as annotation samples. Each text is independently annotated by two annotators on entity type, sentiment, and classification. Texts with inconsistent annotations are analyzed and modified to unify the annotation results.

2. The deep learning-based automated data annotation method according to claim 1, wherein: The construction of the dynamic prompt template includes: Use a clustering algorithm to cluster word vectors, test the number of clusters required by the clustering algorithm, determine the test range of the number of clusters, gradually increase the number of clusters g, use the K-means algorithm to cluster each g value, and calculate the total within-cluster sum of squares for each cluster; Use the data visualization library to draw a line graph of the g value and the corresponding WSS, identify the elbow point where the rate of decrease of WSS with increasing g suddenly slows down, and obtain the g value; Apply the K-means algorithm to cluster the word vector dataset and calculate the silhouette coefficient to evaluate the quality of each cluster; The value range of the silhouette coefficient is [-1, 1]. Clusters with silhouette coefficients in the range of [0.5, 1] ​​are selected for further analysis. Frequently occurring entity and emotion combinations are extracted from each selected cluster, and dynamic prompt templates are created. Each template contains multiple entity type and emotion combinations. A periodic evaluation process is set up. If new entity-emotion combinations are detected in new data, the existing dynamic prompt templates are updated according to the new trends discovered.

3. The deep learning-based automated data annotation method according to claim 2, wherein: The method of constructing a deep learning model based on a dynamic prompt template to automatically label data includes: Choose to use the GPT-3 model provided by OpenAI, obtain access rights to GPT-3, and initialize the basic configuration of the model; Combine small-scale manually annotated data with data from dynamic prompt templates to generate training samples containing annotation instructions. Use the training samples to fine-tune GPT-3, adjust fine-tuning hyperparameters, including the learning rate and number of training rounds, define a cross-entropy loss function, calculate the gradient of the loss function with respect to each weight through the backpropagation algorithm, and apply gradient descent to update the weights. During training, calculate the model loss in real time. If the model loss does not decrease in three consecutive cycles, stop training to obtain the GPT-3 model. The trained GPT-3 model is integrated into the annotation system through the API, and the unannotated text and related dynamic prompt template data are sent to the GPT-3 model. The GPT-3 model generates corresponding text annotations based on the provided template content.

4. The deep learning-based automated data annotation method according to claim 3, wherein: The evaluation of the quality of the labeled data refers to randomly selecting 10% of the labeled data as a validation set, and calculating the accuracy, precision, recall and F1 score of the validation set to evaluate the quality of the annotation; Correct the detected labeling errors and analyze the error types. Adjust the model training parameters and labeling guidelines based on the errors found. Update the dynamic prompt template. Record the evaluation results and corrective measures and generate an evaluation report.

5. The deep learning-based automated data annotation method according to claim 4, wherein: The storing of the annotated data in the database and implementing access control refers to selecting PostgreSQL as the database system for storing the annotated data, establishing a table structure to store various annotated data and an evaluation report of incorrectly annotated data, wherein the annotated data includes the annotated text, the annotation type, the standardizer information, and the timestamp; Implement access control policies to allow only authorized users to access sensitive data, and perform regular data backups, including full and incremental backups.

6. A deep learning-based automated data labeling system based on the deep learning-based automated data labeling method according to any one of claims 1 to 5, characterized in that: include, The data collection module is used to collect text data from various data sources, perform preliminary processing on the text data, and build a hybrid word vector model to generate comprehensive word vectors for the text data; Dynamic prompt template module, which is used to manually annotate small-scale text data and construct dynamic prompt templates based on the generated comprehensive word vectors; The automatic annotation module is used to build the GPT-3 model in the deep learning model and combine manually annotated data with dynamic prompt templates for deep learning training, and use the trained model for automatic data annotation; The quality assessment module is used to extract a portion of data from the results of automated annotation to verify data quality, correct errors based on the results, and generate an assessment report; The data storage module is used to store the labeled data and the incorrectly labeled assessment reports and perform access control and regular backup.

7. A computer device comprising: memory and processor; The memory stores a computer program, characterized in that when the processor executes the computer program, the steps of the deep learning-based automatic data labeling method described in any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the deep learning-based automatic data labeling method according to any one of claims 1 to 5 are implemented.

Images