Intelligent data labeling method and system

Abstract

The invention relates to a big data processing technology, in particular to an intelligent data labeling method and system. The intelligent data labeling method comprises the following steps: preprocessing public opinion data; semantic deduplication of the historical sample library is carried out, and the sample library is updated; an intelligent auxiliary labeling method based on a historical work order library and a large model; model updating and sample increment updating; compared with the prior art, the method has the advantages that a multi-dimensional data cleaning method is adopted, the quality of the to-be-labeled data is improved, and interference of repeated semantic information is reduced through semantic duplicate removal; potential values of similar work orders in historical work orders are mined, a large model cue word project used for auxiliary labeling is constructed, an available auxiliary labeling cue word template is provided, and the cue word construction and auxiliary labeling process is opened; a PDCA thought is adopted, and a data-annotation-model-work order closed-loop management and PDCA mechanism spiral data quality method is constructed.

Description

Technical Field

[0001] The present invention relates to big data processing technology, and in particular to a data intelligent labeling method and system. Background Art

[0002] At present, small models in intelligent scenarios are often delayed in model updates, data accumulated from real business cannot be used for model training in a timely manner, data quality is poor, and deployment annotation is not timely, resulting in a mismatch between model capabilities and business needs. There is an urgent need to improve the matching degree between model capabilities and business through regular model training. Therefore, a set of efficient and intelligent annotation methods is needed to achieve closed-loop management of data and models, solve the problems of data annotation and model training, and achieve continuous improvement of model capabilities. Disadvantages of existing technologies:

[0003] In the current small model data labeling process, there are generally problems such as a large amount of dirty data, unclean data cleaning, ineffective removal of semantically duplicated data, low manual labeling volume and intelligent assisted labeling performance, which lead to problems such as low data labeling quality, large manual labeling volume, and low labeling accuracy.

[0004] Explanation of relevant terms:

[0005] Big model: refers to the large prediction model, which completes the auxiliary data labeling task through prompt word engineering and model fine-tuning.

[0006] Self-learning: By learning the semantic features of text through historical work order data, sample library data and model capabilities, it can provide auxiliary annotation capabilities during auxiliary annotation. Summary of the invention

[0007] This invention aims to solve the problems that affect the quality, timeliness and large amount of manual annotation of data annotation. Based on the PDCA quality engineering method, this invention realizes data annotation and model training to improve data quality through continuous and regular self-learning. The specific technical solution is as follows:

[0008] In a first aspect, a data intelligent labeling method includes the following process:

[0009] S100: Preprocessing of public opinion data; Specific process:

[0010] S110: Synchronize the public opinion history work order database data to the database to be annotated through a scheduled task;

[0011] S120: Use multi-dimensional data cleaning methods to obtain high-quality preliminary data;

[0012] S200: After the data to be labeled is cleaned, it is compared with the historical sample library data one by one, and a high-quality sample set is obtained after semantic deduplication. The specific process is as follows:

[0013] Use a word embedding model to process the text into text vectors and save them to a vector database. When saving, it is necessary to check the semantic similarity between the current data and the data in the sample library item by item, and only retain the data with a semantic similarity less than the threshold; that is, only retain the new data and not the similar data;

[0014] S300: Extract data from the sample set item by item and compare the semantic similarity with the work order library to find the data annotation content and results of historical similar work orders, and then construct a completion prompt. The prompt is sent to the large model for analysis to obtain the final annotation result, realizing intelligent assisted annotation; when manually annotating, refer to the intelligent assisted annotation result and the details of historical similar work orders for annotation to form a sample;

[0015] S400: Adopt a timed task method to regularly process the latest annotated samples into the format required for model training, and automatically train the model regularly. After the model accuracy, recall rate, F1 value and other indicators meet the requirements, save them locally to realize regular automatic training of the model;

[0016] After the final annotation result is confirmed, it will be added to the sample library as new sample data to further enrich the system's historical work order library and annotation samples, providing a reference for subsequent tasks.

[0017] Preferably, the data cleaning method includes the following processes:

[0018] S121: Deduplication processing; specifically: delete duplicate sentences, paragraphs or documents in the dataset to reduce data redundancy; use a hash algorithm or string comparison to detect duplicates; use a hash algorithm or string comparison to detect duplicates;

[0019] S122: Remove irrelevant or noisy content; specifically: clean up irrelevant content such as advertisements, navigation bars, scripts, and HTML tags; use regular expressions or an HTML parser to remove irrelevant HTML tags and advertisement content;

[0020] S123: Use a language detection tool to identify and filter text in non-target languages;

[0021] S124: Delete low-quality text; the specific process is: identify and remove low-quality content such as incomplete grammar, broken sentences, overly short content, or generative text;

[0022] Set a minimum number of characters or minimum number of words TT, and delete text with a length below this threshold;

[0023] Use a syntactic analysis tool to detect sentences with incomplete grammar;

[0024] Set a text length threshold T. If the text length L(D i <T), then delete it;

[0025] S125: Stop word and invalid character filtering; the specific process is: remove stop words according to specific task requirements, delete excessive symbols, emoticons and other invalid characters; use a stop word list to filter meaningless common words; use regular expressions to remove invalid characters and special symbols;

[0026] S126: Remove unsafe or harmful content; the specific process is: use classifiers or regular expressions to remove non-compliant content containing violence, pornography, and hate speech; use sensitive word libraries to detect and delete unsafe content;

[0027] S127: Fix text encoding and formatting issues, ensure that all texts are encoded consistently, remove encoding errors or non-standard symbols, and unify all texts to UTF-8;

[0028] S128: Spelling correction and vocabulary normalization, use spelling check tools to correct spelling errors and standardize unusual vocabulary and expressions;

[0029] S129: Process low-frequency and long-tail words; the specific process is: count the frequency of words in the data set, and words below the set threshold fff can be used <unk>(Unknown word) is used instead to prevent overfitting; for the word w i , if the word frequency f(w i ) < f, then replace w i with <unk>;

[0030] S130: Context consistency check, ensuring that the context of each data is complete and deleting isolated or truncated paragraphs;

[0031] S131: Content standardization; standardize non-standard units and symbols, convert non-standard expressions into standard forms, and convert non-standard units into standard units.

[0032] Preferably, in S121, the hash algorithm uses MD5 or SHA-1 to generate a hash value for each record, and compares whether the hash values ​​are the same; for each data D i , calculate the hash value H(D i ), if H(D i )=H(D j ) and i≠j, then delete D j .

[0033] Preferably, in S123, the probability P (language|D i ), retain the target language text with the highest probability; the core calculation formula is:

[0034] P(Chinese|D i )=max(P(Chinese 1|D i ),P(English 1|D i ), …).

[0035] Preferably, in S128, the spelling check tool PyCorrector is used to automatically correct spelling errors; and the morphological restoration technology is used to restore the vocabulary to a standard form.

[0036] Preferably, in S200, the specific operations of word embedding operation, data similarity calculation and sample storage include the following process:

[0037] S210: load bge-large-zh model;

[0038] Input: pre-trained model bge-large-zh, loaded through HuggingFace;

[0039] Operation: Use the SentenceTransformer class to load the bge-large-zh model;

[0040] Intermediate representation: After the model is loaded, the system can generate the corresponding embedding vector based on the input text data;

[0041] Output: The model is ready to receive input data and generate vectors;

[0042] S220: prepare data;

[0043] Input: a set of text data for which word vectors are to be generated;

[0044] Operation: Take these text data as input and prepare to generate their vector representation;

[0045] Intermediate representation: Input data is passed to the model and further processed;

[0046] Output: The text data is prepared and input into the model to generate word vectors;

[0047] S230: Generate word vectors; the specific process is:

[0048] Input: text data in S300;

[0049] Operation: Use the bge-large-zh model to encode the text and generate its embedding vector;

[0050] Intermediate representation: Each input text will generate a 1024-dimensional vector;

[0051] Formula: Assume the input text is x i , the generated word vector is v(x i ), then the vector generation formula is:

[0052] V(x i )=Model.encode(x i ),

[0053] Output: The generated vector shape is [n_samples, embedding_dim], where n_samples is the number of input texts and embedding_dim is the vector dimension. The result is:

[0054] Embedding=[V(x1),V(x2),…,V(x n )];

[0055] S240: Connect to Milvus database; the specific process is:

[0056] Input: The address and port of Milvus vector database;

[0057] Operation: Use the pymilvus client to connect to the Milvus database instance and enter the connection parameters;

[0058] Make sure the Milvus instance is started and can accept connections;

[0059] Intermediate representation: database connection status;

[0060] Output: Successfully connected to Milvus database, ready to perform subsequent storage operations;

[0061] S250: Create a Milvus collection; the specific process is:

[0062] Input: the name of the vector set and the dimension of the vector;

[0063] Operation: Create a new collection to store word vectors; set a unique identifier (such as an ID field) and a vector field for the collection;

[0064] Formula: Define a vector set C, with a vector dimension of D;

[0065] C=CreateCollection(name=embeddingcollection,dimension=1024),

[0066] Intermediate representation: The collection is successfully created and can store vector data;

[0067] Output: A vector collection named embedding_collection is created in Milvus, and vectors are ready to be inserted into it;

[0068] S260: define vector fields; the specific process is:

[0069] Input: structure definition of vector set;

[0070] Operation: Define the vector field and ensure that the vector dimension matches the generated word vector dimension (1024); the formula is as follows:

[0071] F=Field(name="embedding", type="FLOAT_ECTOR", dimension="1024),

[0072] Intermediate representation: The fields in the collection are defined, and the vector fields have storage characteristics;

[0073] Output: The vector field is defined and the data structure meets the requirements for word vector storage;

[0074] S270: Execute vector search; the specific process is:

[0075] Input: vector to be retrieved and parameters for similarity search;

[0076] Operation: Input a word vector, use Milvus to perform similarity search, retrieve similar data in the recent period, and return the most similar top k Results

[0077] Formula: Search(C,query_vector=V(x q ),top k =5),

[0078] Where V(x q ) is the query vector, top k Return the 5 most similar results;

[0079] Intermediate representation: The system returns a similarity score and the corresponding vector ID;

[0080] Output: Returns a list of vectors most similar to the query vector and their corresponding similarity scores and IDs;

[0081] If there is data in the returned result, skip the data, otherwise continue to execute;

[0082] S280: insert data into Milvus; the specific process is as follows:

[0083] Input: generated word vector and unique identifier;

[0084] Operation: Insert the generated word vector into the Milvus collection together with the corresponding ID; each vector has a unique ID;

[0085] Intermediate representation: the process of inserting data into a collection, Milvus associates each vector with its unique identifier;

[0086] Output: The word vector is successfully inserted into the Milvus database and can be queried and retrieved by its ID;

[0087] S290: Get the latest sample; after performing S270 and S280 operations on the new data, the new data will be saved in the sample library.

[0088] Preferably, in S300, the detailed operation method of intelligent auxiliary labeling is as follows:

[0089] S310: Data extraction; the specific process is as follows:

[0090] In the data extraction stage, data needs to be extracted from the data to be annotated and historical work order data sources:

[0091] S311: extracting sample data to be labeled one by one from the sample library, where the sample content includes public opinion information and feedback submitted by users;

[0092] S312: extracting work orders with semantic similarity to the data to be annotated from the historical work order database, where each historical work order usually includes a text description and annotated results for subsequent comparison;

[0093] These data will be used for subsequent prompt word construction to help the system automatically generate annotation results;

[0094] S320: prompt word construction; when constructing prompt words, the data to be annotated, the historical work order data and the pre-defined annotation rules are included; S330: large model analysis; the constructed prompt words are input into the large model to generate the automatic annotation result R;

[0095] S340: The manual annotator makes final confirmation or adjustment by referring to the annotation results generated by the large model, the annotation details and prompt words of the historical work orders.

[0096] Preferably, in S400, the detailed process of sample incremental update and model automatic training is as follows:

[0097] S410: Data extraction and preprocessing; the specific process is as follows:

[0098] S411: extract newly annotated sample data from the sample library at regular intervals to generate an incremental sample set; set a sample time threshold T, and use the new sample data within the past T time period to form an incremental sample set D new ;

[0099] D new ={x∈D|t(x)>t0},

[0100] Where t(x) represents the annotation time of sample x, and t0 is the time threshold of the last update;

[0101] S412: The newly extracted samples undergo necessary data cleaning, standardization, and format conversion to ensure that they meet the input requirements for model training; specific steps include deduplication, invalid character filtering, and label standardization;

[0102] S420: Sample format conversion,

[0103] Assuming that the training format adopted by the model is feature vector X and label y, the conversion process of the new sample set can be expressed as:

[0104] X new ,y new =Transform(D new ),

[0105] The Transform function is used to convert the original text data into a feature vector representation X, and convert the annotation result into the corresponding label y;

[0106] S430: The system sets a scheduled task and regularly uses the latest incremental sample data for model training. This process can be completed by the following steps:

[0107] Set up regular scheduling tasks to start the model training process at regular intervals. The task scheduler will check whether the incremental sample set in the sample library is large enough. If the incremental sample data volume meets the training requirements, the training process will be started.

[0108] S440: Each time the model is trained, the full sample set D is used all , including all historical samples and incremental samples, to optimize the model; assuming that the full sample set is:

[0109] D all =D old ∪D new ,

[0110] Among them, D old is a historical sample, D new is the latest incremental sample; model training updates the model parameter θ by optimizing the objective function L(θ):

[0111] θ * =argminL(θ; D all ),

[0112] Common objective functions L(θ) may include cross entropy loss function and mean square error;

[0113] S450: After training is completed, the system will use the validation set to evaluate the model and calculate the key performance indicators of the model, such as accuracy, recall, and F1 value;

[0114] S460: If the model evaluation index reaches the preset threshold, the system will automatically save the model version and apply it to the actual labeling task; the saved model version M new Will overwrite the old model;

[0115] Preferably, in S400, the update cycle of the model is set to be monthly or quarterly.

[0116] In a second aspect, a data intelligent labeling system includes:

[0117] The data preprocessing module is used to synchronize the public opinion historical work order database data to the database to be annotated, and use multi-dimensional data cleaning methods to obtain high-quality preliminary data;

[0118] The data comparison module is used to compare the cleaned data with the historical sample library data one by one, and obtain a high-quality sample set after semantic deduplication;

[0119] The intelligent auxiliary labeling module is used to extract data from the sample set one by one and compare the semantic similarity with the work order database, find the data labeling content and results of historically similar work orders, and then build the prompt words. The prompt words are sent to the big model for analysis to obtain the final labeling results, realizing intelligent auxiliary labeling;

[0120] The model training module is used to regularly process the latest labeled samples into the format required for model training, regularly and automatically train the model, and save the model to the local computer when the model accuracy, recall rate, F1 value and other indicators meet the requirements, so as to realize regular and automatic model training.

[0121] The advantages of the present invention over the prior art are:

[0122] A multi-dimensional data cleaning method is used to improve the quality of the data to be annotated, and semantic deduplication is used to reduce the interference of repeated semantic information.

[0123] The value of historical work orders and historical samples is explored, and strict sample cleaning methods are used to improve the quality of the sample set. Large models and historical work orders are used to assist in labeling clean data to be labeled, thereby ensuring the quality of the sample set data while achieving intelligent auxiliary labeling, thereby greatly reducing the interference of dirty data and providing auxiliary labeling results with higher accuracy for manual reference.

[0124] Adopting the PDCA concept, we build a closed-loop management of data-annotation-model-work order and a spiral data quality method based on the PDCA mechanism. BRIEF DESCRIPTION OF THE DRAWINGS

[0125] Figure 1 It is a schematic diagram of the principle of a data intelligent labeling method. DETAILED DESCRIPTION

[0126] A data intelligent labeling method includes the following process:

[0127] S100: Preprocessing of public opinion data; Specific process:

[0128] S110: Synchronize the public opinion history work order database data to the database to be annotated through a scheduled task;

[0129] S120: Use multi-dimensional data cleaning methods to obtain high-quality preliminary data; specifically, the following processes are included:

[0130] S121: Deduplication processing; specifically: deleting repeated sentences, paragraphs or documents in the data set to reduce data redundancy; using hash algorithms or string comparisons to detect duplicates. Using hash algorithms or string comparisons to detect duplicates;

[0131] The hash algorithm uses MD5 or SHA-1 to generate a hash value for each record and compares whether the hash values ​​are the same; for each data D i , calculate the hash value H(D i ), if H(D i )=H(D j ) and i≠j, then delete D j ;

[0132] S122: Remove irrelevant or noisy content; specifically: remove irrelevant content such as advertisements, navigation bars, scripts, HTML tags, etc.; use regular expressions or HTML parsers such as BeautifulSoup; remove irrelevant HTML tags, advertisement content, etc., such as 、 <script>、等标签；

[0133] S123：使用语言检测工具识别并过滤非目标语言的文本；使用langdetect或CLD2语言检测工具，基于字符分布和词汇特征，计算各个语言的概率P(语言|Di))，保留概率最高的目标语言文本；核心计算公式为：

[0134] P(中文|Di)＝max(P(中文1|Di),P(英文1|Di)，…)；

[0135] S124：删除低质量文本；具体过程为：识别语法不完整、句子断裂、内容过短或生成式文本等低质量内容并清除；

[0136] 设置最小字符数或最少词数TT，删除长度低于该阈值的文本；

[0137] 使用句法分析工具spaCy检测语法不完整的句子；

[0138] 设置文本长度阈值T,若文本长度L(Di<T)，则删除；

[0139] S125：停用词与无效字符过滤；具体过程为：根据具体任务需求去除停用词，删除过多的符号、表情等无效字符；使用中文停用词库过滤无意义的常用词；使用正则表达式去除无效字符、特殊符号，比如，"的”、"地”、"了”、"啊”和标点符号；

[0140] S126：去除不安全或有害内容；具体过程为：使用分类器或正则表达式来去除含有暴力、色情、仇恨言论等不合规内容；使用敏感词库检测和删除不安全内容；

[0141] S127：使用iconv或chardet等工具检测文本编码，修复文本编码和格式问题，确保所有文本的编码一致，清理掉编码错误或非标准符号，并将所有文本统一为UTF-8；

[0142] S128：使用拼写检查工具PyCorrector自动修正拼写错误；使用词形还原技术将词汇还原为标准形式；

[0143] S129：处理低频和长尾词汇；具体过程为：对数据集中词汇进行词频统计，低于设定阈值fff的词汇可以用<unk>(未知词)代替，防止过拟合；对于词汇wi，若词频f(wi)<f，则替换wi为<unk>；

[0144] S130：上下文一致性检查，确保每条数据上下文完整，删除孤立或截断的段落；使用句法分析或上下文窗口方法，检测并删除上下文不连贯或被截断的段落；

[0145] S131：内容规范化；将非标准的单位、符号等进行规范化，如将缩写扩展为完整单词；通过规则匹配或查找表，将非标准表达转换为标准形式，比如将缩写词扩展，非标准单位转换为标准单位；

[0146] S200：使用词嵌入模型将文本处理为文本向量并保存指向量数据库，在保存时需要逐条检查当前数据与样本库数据的语义相似度，只保留语义相似度小于阈值的数据；也即只保留新数据，不保留相似数据；包括如下过程：

[0147] S210：加载bge-large-zh模型；

[0148] 输入：预训练模型bge-large-zh，通过HuggingFace进行加载；

[0149] 操作：使用SentenceTransformer类加载bge-large-zh模型；

[0150] 中间表示：模型加载后，系统能够根据输入的文本数据生成对应的嵌入向量；

[0151] 输出：模型准备就绪，可以接收输入数据生成向量。

[0152] S220：准备数据；

[0153] 输入：一组待生成词向量的文本数据；如单词、短语或句子，假设输入数据为：["天气","晴朗","出门"]；

[0154] 操作：将这些文本数据作为输入，准备生成它们的向量表示；

[0155] 中间表示：输入数据被传递到模型，并会被进一步处理；

[0156] 输出：文本数据准备完毕，输入到模型进行词向量生成；

[0157] S230：生成词向量；具体过程为：

[0158] 输入：S300中的文本数据；

[0159] 操作：使用bge-large-zh模型对文本进行编码，生成其嵌入向量；

[0160] 以bge-large-zh模型为例，嵌入向量的维度通常为1024；

[0161] 中间表示：每个输入文本都会生成一个1024维的向量；

[0162] 公式：假设输入文本为xi，生成的词向量为v(xi)，则向量生成公式为：

[0163] V(xi)＝Model.encode(xi)，

[0164] 输出：生成的向量形状为[n_samples,embedding_dim]，其中n_samples是输入文本的数量，embedding_dim是向量维度，结果为：

[0165] Embedding＝[V(x1),V(x2),…,V(xn)]；

[0166] 例如：[1024维向量]；

[0167] S240：连接Milvus数据库；具体过程为：

[0168] 输入：Milvus向量数据库的地址和端口；

[0169] 操作：使用pymilvus客户端连接到Milvus数据库实例，输入连接参数；如host＝'127.0.0.1'，port＝'19530'；

[0170] 确保Milvus实例已启动，并且可以接受连接；

[0171] 中间表示：数据库连接状态；

[0172] 输出：成功连接到Milvus数据库，准备执行后续的存储操作；

[0173] S250：创建Milvus集合；具体过程为：

[0174] 输入：向量集合的名称和向量的维度；

[0175] 操作：创建一个新的集合，存储词向量；假设集合名称为embedding_collection，向量维度为1024；

[0176] 为该集合设置唯一标识符，如ID字段、embeddingfield向量字段；

[0177] 公式：定义向量集合C，向量维度为D；

[0178] C＝CreateCollection(name＝embeddingcollection,dimension＝1024)，

[0179] 中间表示：集合被成功创建，可以存储向量数据；

[0180] 输出：Milvus中创建了一个名为embedding_collection的向量集合，准备向其中插入向量；

[0181] S260：定义向量字段；具体过程为：

[0182] 输入：向量集合的结构定义；

[0183] 操作：定义向量字段，确保向量维度与生成的词向量维度(1024)匹配；公式如下：

[0184] F＝Field(name＝"embedding",type＝"FLOAT_ECTOR",dimension＝1024)，

[0185] 中间表示：集合中已定义字段，向量字段具备存储特性；

[0186] 输出：向量字段定义完毕，数据结构符合词向量存储要求；

[0187] S270：执行向量检索；具体过程为：

[0188] 输入：待检索的向量和相似性搜索的参数，如topk；

[0189] 操作：输入一个词向量，使用Milvus执行相似性检索，检索最近一段时间的相似数据，返回最相似的topk个结果；

[0190] 公式：Search(C,query_vector＝V(xq),topk＝5)，

[0191] 其中V(xq)是查询向量，topk是返回最相似的5个结果；

[0192] 中间表示：系统返回相似性分数和相应的向量ID；

[0193] 输出：返回与查询向量最相似的向量列表及其对应的相似性得分和ID；

[0194] 如果返回结果中有数据，则跳过该条数据，反之则继续执行；

[0195] S280：向Milvus插入数据；具体过程为：

[0196] 输入：生成的词向量和唯一标识符；

[0197] 操作：将生成的词向量与相应的ID一起插入到Milvus集合中；每个向量都有唯一的ID，如自增ID或手动指定ID；

[0198] 中间表示：数据插入集合的过程，Milvus将每个向量与其唯一标识符进行关联；

[0199] 输出：词向量成功插入Milvus数据库，并可以通过其ID进行查询和检索；

[0200] S290：得到最新样本；对新数据执行S270和S280操作之后，新数据将保存至样本库；

[0201] S300：从样本集中逐条抽取数据与工单库进行语义相似度对比，找到历史相似工单的数据标注内容和结果，然后构建完成提示词，提示词送入大模型分析得到最后标注结果，实现智能辅助标注；人工标注时参考智能辅助标注结果和历史相似工单详情进行标注，形成样本；智能辅助标注详细操作方法如下：

[0202] S310：数据抽取；具体过程如下：

[0203] 在数据抽取阶段，需从待标注数据、历史工单数据来源中抽取数据：

[0204] S311：从样本库中逐条抽取当前待标注的样本数据，样本内容包括用户提交的舆情信息、反馈等；

[0205] S312：从历史工单库中提取与待标注数据语义相似的工单，每条历史工单通常包含文本描述及已标注的结果，供后续对比使用；数据抽取采用S127方法执行向量检索，检索得到待标注工单相似的历史工单和历史相似样本数据；

[0206] 这些数据将被用于后续的提示词构建，帮助系统自动生成标注结果；

[0207] S320：提示词构建；构建提示词时，将待标注数据、历史工单数据及事先定义的标注规则一并纳入；提示词的模板如下：

[0208] 提示词模板：

[0209]

任务

[0210]

当前待标注数据

[0211]

相似样本数据

[0212]

相似工单标注内容

[0213]

标注规则

[0214] 请基于以上信息生成最终标注结果。

[0215] 利用上一步得到的当前待标注收据、相似样本数据、相似工单数据；

[0216] 示例：

[0217]

任务

[0218]

当前待标注数据

[0219]

相似样本数据

[0220]

相似工单标注内容

[0221]

标注规则

[0222] 请基于以上信息生成最终标注结果。

[0223] S330：大模型分析；将构建的提示词输入大模型，生成自动标注结果R；

[0224] S340：人工标注员参考大模型生成的标注结果、历史工单的标注详情及提示词，进行最终确认或调整。此步骤通过结合自动标注结果和人工经验，确保标注的准确性和一致性。

[0225] S400：采用定时任务方式，定期将最新标注的样本处理成模型训练所需格式，定期自动训练模型，当模型准确率、召回率、F1值等指标符合要求之后保存至本地，实现模型定期自动训练；样本增量更新、模型自动训练详细过程如下：

[0226] S410：数据抽取与预处理；具体过程如下：

[0227] S411：每隔一段时间从样本库中提取新近标注的样本数据，生成增量样本集；设定样本时间阈值T，以过去T时间段内的新样本数据构成增量样本集Dnew；

[0228] Dnew＝{x∈D∣t(x)>t0}，

[0229] 其中，t(x)表示样本x的标注时间，t0为上次更新的时间阈值；

[0230] S412：新提取的样本经过必要的数据清洗、标准化和格式转换，确保符合模型训练的输入要求；该预处理过程与之前的数据清洗流程一致，具体步骤包括去重、无效字符过滤、标签标准化等；

[0231] S420：样本格式转换

[0232] 模型训练通常需要特定的输入格式，如特征向量表示、标签等。因此，在样本增量更新过程中，需将新标注的样本数据转换为适合训练的数据格式。假设模型采用的训练格式为特征向量X和标签y，则新样本集的转换过程可表示为：

[0233] Xnew,ynew＝Transform(Dnew)，

[0234] 其中，Transform函数用于将原始文本数据转换为特征向量表示X，并将标注结果转为相应的标签y；

[0235] S430：系统通过设定定时任务，定期使用最新的增量样本数据进行模型训练；该过程可以通过以下步骤完成：

[0236] 设定定期调度任务，每隔一定时间段(如每周或每月)启动模型训练流程；任务调度器会检查样本库中的增量样本集是否足够大，如果增量样本数据量满足训练要求，则启动训练过程；

[0237] S440：每次模型训练时，使用全量样本集Dall，包括所有历史样本和增量样本，进行模型优化；假设全量样本集为：

[0238] Dall＝Dold∪Dnew，

[0239] 其中，Dold为历史样本，Dnew为最新的增量样本；模型训练通过最优化目标函数L(θ)，对模型参数θ进行更新：

[0240] θ*＝argminL(θ；Dall)，

[0241] 常见的目标函数L(θ)可能包括交叉熵损失函数、均方误差等，具体取决于模型类型和任务；

[0242] S450：在训练完成后，系统会使用验证集对模型进行评估，计算模型的关键性能指标，如准确率、召回率、F1值等；

[0243] S460：如果模型评估指标达到预设阈值，系统将自动保存该模型版本并应用到实际标注任务中；保存的模型版本Mnew会覆盖旧模型；

[0244] 整个模型更新与样本增量更新过程是定期进行的，确保模型能够持续学习最新的标注数据，并且随着标注样本的不断积累，模型的性能将不断提升。模型的更新周期Ttrain取决于样本增量数据的增长速度和系统的性能需求，常见的更新周期可以设定为每月或每季度。

[0245] 最终的标注结果经过确认后，将作为新的样本数据加入样本库，进一步丰富系统的历史工单库和标注样本，为后续任务提供参考。

[0246] 面所述仅为本发明的较佳实施例,并不用以限制本发明,凡在本发明的精神和原则之内,所作的任何修改,等同替换、改进等，均应包含在本发明的保护范围之内。< / script> < / unk> < / unk>

Claims

1. A data intelligent labeling method, a data intelligent labeling method and system are characterized by: The process includes the following: S100: Preprocessing of public opinion data; Specific process: S110: Synchronize the public opinion history work order database data to the database to be annotated through a scheduled task; S120: Use multi-dimensional data cleaning methods to obtain high-quality preliminary data; S200: After the data to be labeled is cleaned, it is compared with the historical sample library data one by one, and a high-quality sample set is obtained after semantic deduplication. The specific process is as follows: Use the word embedding model to process the text into a text vector and save it in the pointer database. When saving, it is necessary to check the semantic similarity between the current data and the sample library data one by one, and only retain the data with a semantic similarity less than the threshold; that is, only retain the new data, not the similar data; S300: Extract data from the sample set one by one and compare the semantic similarity with the work order database to find the data annotation content and results of historical similar work orders, then build prompt words, send the prompt words to the big model for analysis to obtain the final annotation results, and realize intelligent auxiliary annotation; When manually labeling, refer to the intelligent auxiliary labeling results and the details of similar historical work orders to form samples; S400: Using a scheduled task method, the latest labeled samples are regularly processed into the format required for model training, and the model is automatically trained regularly. When the model accuracy, recall rate, F1 value and other indicators meet the requirements, they are saved locally to achieve regular automatic model training; after the final labeling results are confirmed, they will be added to the sample library as new sample data.

2. According to claim 1, a data intelligent labeling method is characterized in that: The data cleaning method includes the following process: S121: deduplication processing; specifically: deleting repeated sentences, paragraphs or documents in the data set to reduce data redundancy; using a hash algorithm or string comparison to detect duplicate items; using a hash algorithm or string comparison to detect duplicate items; S122: Remove irrelevant or noisy content; specifically: remove advertisements, navigation bars, scripts, and irrelevant content of HTML tags; use regular expressions or HTML parsers to remove irrelevant HTML tags and advertisement content; S123: Use language detection tools to identify and filter texts in non-target languages; S124: Delete low-quality texts; the specific process is: identify and delete incomplete grammar, broken sentences, too short content or low-quality generated texts; Set a minimum number of characters or words TT and delete texts whose length is below the threshold; Use syntactic analysis tools to detect grammatically incomplete sentences; Set the text length threshold T. If the text length L(D i <T), then delete it; S125: Stop word and invalid character filtering; the specific process is: remove stop words according to specific task requirements, delete excessive symbols, emoticons and other invalid characters; use a stop word list to filter meaningless common words; use regular expressions to remove invalid characters and special symbols; S126: Remove unsafe or harmful content; the specific process is: use classifiers or regular expressions to remove non-compliant content containing violence, pornography, and hate speech; Use sensitive word libraries to detect and remove unsafe content; S127: Fix text encoding and formatting issues, ensure that all texts are encoded consistently, remove encoding errors or non-standard symbols, and unify all texts to UTF-8; S128: Spelling correction and vocabulary normalization, use spelling check tools to correct spelling errors and standardize unusual vocabulary and expressions; S129: Process low-frequency and long-tail words; the specific process is: count the frequency of words in the data set, and words below the set threshold fff can be used <unk>Replace (unknown word) to prevent overfitting; for the word w i , if the word frequency f(w i ) < f, then replace w i with <unk> ；< / unk> < / unk> S130: Context consistency check, ensuring that the context of each data is complete and deleting isolated or truncated paragraphs; S131: Content standardization; standardize non-standard units and symbols, convert non-standard expressions into standard forms, and convert non-standard units into standard units.

3. According to claim 2, a data intelligent labeling method is characterized in that: In S121, the hash algorithm uses MD5 or SHA-1 to generate a hash value for each record, and compares whether the hash values ​​are the same; for each data D i , calculate the hash value H(D i ), if H(D i )=H(D j ) and i≠j, then delete D j .

4. According to claim 2, a data intelligent labeling method is characterized in that: In S123, the probability P (language|D i ), retain the target language text with the highest probability; the core calculation formula is: P(Chinese|D i )=max(P(Chinese 1|D i ),P(English 1|D i ), …).

5. According to claim 2, a data intelligent labeling method is characterized in that: In S128, the spelling check tool PyCorrector is used to automatically correct spelling errors; and the morphological restoration technology is used to restore the vocabulary to a standard form.

6. According to claim 1, a data intelligent labeling method is characterized in that: In S200, the word embedding operation, data similarity calculation and sample storage operations include the following processes: S210: load bge-large-zh model; Input: pre-trained model bge-large-zh, loaded through HuggingFace; Operation: Use the SentenceTransformer class to load the bge-large-zh model; Intermediate representation: After the model is loaded, the system can generate the corresponding embedding vector based on the input text data; Output: The model is ready to receive input data and generate vectors; S220: prepare data; Input: a set of text data for which word vectors are to be generated; Operation: Take these text data as input and prepare to generate their vector representation; Intermediate representation: Input data is passed to the model and further processed; Output: The text data is prepared and input into the model to generate word vectors; S230: Generate word vectors; the specific process is: Input: text data in S300; Operation: Use the bge-large-zh model to encode the text and generate its embedding vector; Intermediate representation: Each input text will generate a 1024-dimensional vector; Formula: Assume the input text is x i , the generated word vector is v(x i ), then the vector generation formula is: V(x i )=Model.encode(x i ), Output: The generated vector shape is [n_samples, embedding_dim], where n_samples is the number of input texts and embedding_dim is the vector dimension. The result is: Embedding=[V(x1),V(x2),…,V(x n )]; S240: Connect to Milvus database; the specific process is: Input: The address and port of Milvus vector database; Operation: Use the pymilvus client to connect to the Milvus database instance and enter the connection parameters; Make sure the Milvus instance is started and can accept connections; Intermediate representation: database connection status; Output: Successfully connected to Milvus database, ready to perform subsequent storage operations; S250: Create a Milvus collection; the specific process is: Input: the name of the vector set and the dimension of the vector; Operation: Create a new collection to store word vectors; set a unique identifier (such as an ID field) and a vector field for the collection; Formula: Define a vector set C, with a vector dimension of D; C=createCollection(name=embeddingcollection,dimension=1024), Intermediate representation: The collection is successfully created and can store vector data; Output: A vector collection named embedding_collection is created in Milvus, and vectors are ready to be inserted into it; S260: define vector fields; the specific process is: Input: structure definition of vector set; Operation: Define the vector field and ensure that the vector dimension matches the generated word vector dimension (1024); the formula is as follows: F=Field(name="embedding", type="FLOAT_ECTOR", dimension="1024), Intermediate representation: The fields in the collection are defined, and the vector fields have storage characteristics; Output: The vector field is defined and the data structure meets the requirements for word vector storage; S270: Execute vector search; the specific process is: Input: vector to be retrieved and parameters for similarity search; Operation: Input a word vector, use Milvus to perform similarity search, retrieve similar data in the recent period, and return the most similar top k Results Formula: Search(C, query_vector = V(x q ), top k = 5), Where V(x q ) is the query vector, top k Return the 5 most similar results; Intermediate representation: The system returns a similarity score and the corresponding vector ID; Output: Returns a list of vectors most similar to the query vector and their corresponding similarity scores and IDs; If there is data in the returned result, skip the data, otherwise continue to execute; S280: insert data into Milvus; The specific process is: Input: generated word vector and unique identifier; Operation: Insert the generated word vector into the Milvus collection together with the corresponding ID; each vector has a unique ID; Intermediate representation: the process of inserting data into a collection, Milvus associates each vector with its unique identifier; Output: The word vector is successfully inserted into the Milvus database and can be queried and retrieved by its ID; S290: Get the latest sample; after performing S270 and S280 operations on the new data, the new data will be saved in the sample library.

7. According to claim 1, a data intelligent labeling method is characterized in that: In S300, the detailed operation method of intelligent auxiliary labeling is as follows: S310: Data extraction; the specific process is as follows: In the data extraction stage, data needs to be extracted from the data to be annotated and historical work order data sources: S311: extracting sample data to be labeled one by one from the sample library, where the sample content includes public opinion information and feedback submitted by users; S312: extracting work orders with semantic similarity to the data to be annotated from the historical work order database, where each historical work order usually includes a text description and annotated results for subsequent comparison; These data will be used for subsequent prompt word construction to help the system automatically generate annotation results; S320: prompt word construction; when constructing prompt words, the data to be annotated, the historical work order data and the pre-defined annotation rules are included; S330: large model analysis; the constructed prompt words are input into the large model to generate the automatic annotation result R; S340: The manual annotator makes final confirmation or adjustment by referring to the annotation results generated by the large model, the annotation details and prompt words of the historical work orders.

8. The method for intelligent data labeling according to claim 1, characterized in that: In S400, the detailed process of sample incremental update and model automatic training is as follows: S410: Data extraction and preprocessing; the specific process is as follows: S411: extract newly annotated sample data from the sample library at regular intervals to generate an incremental sample set; set a sample time threshold T, and use the new sample data within the past T time period to form an incremental sample set D new ; D new ={x∈D∣t(x)>t0}, Where t(x) represents the annotation time of sample x, and t0 is the time threshold of the last update; S412: The newly extracted samples undergo necessary data cleaning, standardization, and format conversion to ensure that they meet the input requirements for model training; specific steps include deduplication, invalid character filtering, and label standardization; S420: Sample format conversion Assuming that the training format adopted by the model is feature vector X and label y, the conversion process of the new sample set can be expressed as: X new ,y new =Transform(D new ), The Transform function is used to convert the original text data into a feature vector representation X, and convert the annotation result into the corresponding label y; S430: The system sets a scheduled task and regularly uses the latest incremental sample data for model training. This process can be completed by the following steps: Set up regular scheduling tasks to start the model training process at regular intervals. The task scheduler will check whether the incremental sample set in the sample library is large enough. If the incremental sample data volume meets the training requirements, the training process will be started. S440: Each time the model is trained, the full sample set D is used all , including all historical samples and incremental samples, to optimize the model; assuming that the full sample set is: D all =D old ∪D new , Among them, D old is a historical sample, D new is the latest incremental sample; model training updates the model parameter θ by optimizing the objective function L(θ): θ * =argminL(θ;D all ), Common objective functions L(θ) may include cross entropy loss function and mean square error; S450: After training is completed, the system will use the validation set to evaluate the model and calculate the key performance indicators of the model, such as accuracy, recall, and F1 value; S460: If the model evaluation index reaches the preset threshold, the system will automatically save the model version and apply it to the actual labeling task; the saved model version M new The old model will be overwritten.

9. The method for intelligent data labeling according to claim 1, characterized in that: In S400, the update cycle of the model is set to be monthly or quarterly.

10. A data intelligent labeling system, characterized in that: include: The data preprocessing module is used to synchronize the public opinion historical work order database data to the database to be annotated, and use multi-dimensional data cleaning methods to obtain high-quality preliminary data; The data comparison module is used to compare the cleaned data with the historical sample library data one by one, and obtain a high-quality sample set after semantic deduplication; The intelligent auxiliary labeling module is used to extract data from the sample set one by one and compare the semantic similarity with the work order database, find the data labeling content and results of historically similar work orders, and then build the prompt words. The prompt words are sent to the big model for analysis to obtain the final labeling results, realizing intelligent auxiliary labeling; The model training module is used to regularly process the latest labeled samples into the format required for model training, regularly and automatically train the model, and save the model to the local computer when the model accuracy, recall rate, F1 value and other indicators meet the requirements, so as to realize regular and automatic model training.

Images