A method and system for automatic data labeling based on model training

Abstract

This invention belongs to the technical field of specific computing models and specifically designs a method and system for automatic data labeling based on model training. The method includes: collecting target content information; screening based on information timeliness and relevance to obtain first filtered data; proofreading the first filtered data, generating historical information and providing feedback; extracting differential features as data to be labeled and automatically labeling them; verifying and updating the labeled results based on historical error rates; calculating deviations and generating monitoring evaluation intervals, and classifying the output results as frozen or unfrozen data. The system includes collection, screening, proofreading, labeling, and deviation analysis modules. This solution improves labeling efficiency and accuracy and is suitable for multimodal data processing scenarios.

Description

Technical Field

[0001] The present invention belongs to the technical field of specific computing models, and specifically designs a method and system for automatic data labeling based on model training. Background Art

[0002] With the rapid development of technologies such as artificial intelligence, machine learning, and natural language processing, training high-precision intelligent models has placed a higher demand on large-scale, high-quality annotated data. In data-driven tasks such as image recognition, semantic understanding, and speech recognition, the quality of annotated data directly determines model performance. However, traditional data annotation methods, which primarily rely on manual labor, are not only costly and inefficient, but also prone to inconsistent annotations due to subjective judgment, which can affect model training effectiveness.

[0003] Therefore, there is an urgent need to propose a data automatic labeling method and system that can combine the model training process, automatically generate labeling information, and continuously optimize the labeling quality and model performance through a feedback mechanism, thereby reducing manual dependence, improving labeling efficiency and consistency, and providing a high-quality and scalable data foundation for intelligent models. Summary of the Invention

[0004] In view of the above problems, the present invention aims to propose a method for automatic data labeling based on model training, comprising:

[0005] S1. Target content information collection: collecting target content information using a collection model; wherein the target content information includes at least one of text, image, and video;

[0006] The acquisition model is a RAG large language model, which performs semantic expansion based on the annotation tags input by the user to obtain an extended vocabulary with similar semantics to the annotation tags. Based on the extended vocabulary, keyword search is performed on the text, images and videos of the acquisition website. The hit text, images and videos constitute the target content information;

[0007] S2. Information screening and preliminary evaluation: Screening the target content information according to a first evaluation condition to obtain first screening data; wherein the first evaluation condition includes information timeliness and information relevance;

[0008] S3, Screening Data Proofreading and Feature Feedback: Proofreading the first screening data to obtain historical information, and feeding it back to the subsequent target content information collection stage; wherein the historical information includes proofread information and unproofread information;

[0009] S4, difference feature extraction and labeling preparation: overflow screening is performed on the historical information to obtain difference information, and the difference information is marked as data to be labeled;

[0010] S5, automatic labeling and error rate verification and update: segmenting the data to be labeled;

[0011] The segmentation process specifically refers to: based on the annotation label input by the user, extending the hit point of the data to be annotated forward and backward by a necessary length to obtain a semantically complete text, image or video segment, while cutting off the text, image or video segment containing non-annotated information;

[0012] After segmentation processing, multiple segments to be labeled are obtained, and then the segments to be labeled are input one by one into the automatic labeling model to obtain labeled data. The labeled data are then verified based on the error rate of historical information, and the error rate is updated based on the verification result;

[0013] S6. Deviation calculation and monitoring interval generation: perform deviation calculation on the updated error rate, and then generate a monitoring evaluation interval based on the deviation. The error rate is updated during the next data annotation, and the output results of the automatic annotation model are classified into frozen data and unfrozen data based on the instantaneous value of the updated error rate and the monitoring evaluation interval.

[0014] As a preferred solution, S1 specifically includes: building a collection website list and issuing collection instructions to each collection website in the collection website list; calling the interface of the collection website, accessing each collection website, and collecting output information from the collection website to obtain target content information.

[0015] As a preferred solution, S2 specifically includes: first, evaluating and processing the timeliness of each target content information based on the information timeliness in the first evaluation condition, and eliminating data that does not meet the conditions; the information timeliness is the timeliness interval of the target information set by the user; then calculating the information relevance of the remaining data, and filtering out content with a relevance not lower than a threshold as the first filtering data; the information relevance is the semantic cosine value between the target information and the annotation label input by the user.

[0016] As a preferred solution, S3 includes: constructing a list of information to be matched, and adding each of the first screening data to the list; using a matching model to match the information one by one; if the match is successful, pre-marking is performed synchronously, if it fails, proofreading is continued, and historical information is optimized and updated according to a preset strategy.

[0017] As a preferred solution, the historical information includes first historical information, which represents data in the first filtered data that is still consistent with the first filtered data after proofreading; second historical information, which represents data in the first filtered data that is found to be inconsistent with the first filtered data after proofreading; and third historical information, which represents newly added data that does not match the historical annotation label.

[0018] As a preferred solution, the error rate update process in S5 includes: constructing a list of label information to be verified, extracting the update time and model input of each label information; determining unlabeled or erroneous labels based on the matching degree with historically labeled labels, setting the total number of labeling errors, calculating the deviation rate, and dynamically updating the error rate based on the unlabeled frequency, and re-incorporating the updated results into the historical information.

[0019] As a preferred solution, the deviation measurement in S6 includes: dividing the updated error rate into multiple value segments according to the time dimension, performing discretization preprocessing and sorting, inputting the deviation measurement model in combination with the set smoothing coefficient, and outputting the deviation result.

[0020] As a preferred solution, the method for generating the monitoring evaluation interval includes: inputting the error rate into the verification model, obtaining verification information and performing wavelet transform and feature sorting, then performing first-order difference and inverse normalization processing, calculating the risk value based on the updated bias and the set risk threshold, and generating high interval values and low interval values based on this to construct the final monitoring evaluation interval.

[0021] The present invention also provides a system for automatic data annotation based on model training, which is used to implement the method described above, including:

[0022] Information collection module: used to build a collection website list, issue collection instructions, and call the interface to capture target content information such as text, images, and videos.

[0023] Information screening module: used to evaluate and process target content information based on the timeliness and relevance of the information, remove data that does not meet the conditions, and output the first screening data.

[0024] Data proofreading module: used to call the matching model to compare the first screening data with historical information, generate proofread and unproofread information, and feed back the comparison results to the acquisition model.

[0025] Data overflow screening module: used to extract difference information from historical information, identify change trends, and mark it as data to be labeled for use by the automatic labeling model.

[0026] Automatic labeling module: used to input the data to be labeled into the model in segments for labeling, verify and correct the labeling results based on the historical error rate, and dynamically update the error rate parameters.

[0027] Deviation analysis module: used to perform deviation measurement on the updated error rate, generate monitoring evaluation intervals, and classify the status of model output results according to the degree of deviation to achieve dynamic control of annotation quality.

[0028] The present invention also provides an electronic device, comprising at least one processor and a memory communicatively connected to the processor, wherein the memory stores a computer program executable by the processor, and the program is used to execute the method described.

[0029] Beneficial effects:

[0030] The present invention provides a method and system for automatic data labeling based on model training, and their application in electronic devices, which have the following beneficial effects:

[0031] From a methodological perspective, this paper establishes a five-stage closed-loop annotation process: "collection - screening - proofreading - annotation - evaluation." This process leverages historical information for dynamic feedback and error rate tracking. By combining differential feature screening with segmented processing within an automatic annotation model, the accuracy and automation of data processing are enhanced. The introduction of error rate verification and deviation monitoring mechanisms after annotation dynamically adjusts the model output quality and effectively categorizes the annotation results, improving the overall stability and robustness of the method and significantly reducing the cost of manual intervention.

[0032] From a system perspective, the proposed system boasts a clear structure, with distinct functional divisions across modules, encompassing the entire process of information collection, screening, proofreading, annotation, and monitoring and evaluation. The system's automatic annotation module and deviation analysis module work collaboratively to implement closed-loop control and risk management for annotated data, ensuring the continuous and stable operation of the data processing pipeline. Furthermore, the system's highly modular nature facilitates deployment and scalability, adapting to data processing needs of varying scale and complexity.

[0033] From an application perspective: This invention can be widely applied to multimodal data scenarios such as text comprehension, image recognition, and video processing, and possesses excellent generalization capabilities and practical feasibility in big data environments. By embedding this method within the processor and memory modules of electronic devices, it can achieve real-time processing capabilities for simultaneous collection and annotation, meeting the integration requirements of various devices such as smart terminals, server clusters, and industrial collection platforms. This technology effectively solves the problems of low efficiency, high error rates, and untimely model updates that exist in traditional annotation, and has significant promotional value in fields such as artificial intelligence, knowledge graph construction, and search engine training. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 The figure is a flow chart of a method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0035] In order to deepen the understanding of the present invention, the present invention will be further described in detail below with reference to the examples. The examples are only used to explain the present invention and do not constitute a limitation on the scope of protection of the present invention.

[0036] Example 1

[0037] like Figure 1 As shown, this embodiment provides a method for automatic data labeling based on model training, specifically including:

[0038] S1. Target content information collection: Target content information is collected using a collection model; wherein the target content information includes at least one of text, images, and videos; specifically, the steps include: constructing a collection website list and issuing a collection instruction to each collection website in the collection website list; calling an interface of the collection website, accessing each collection website, and collecting output information from the collection website to obtain the target content information.

[0039] S2. Information screening and preliminary evaluation: The target content information is screened according to the first evaluation condition to obtain the first screening data; wherein, the first evaluation condition includes information timeliness and information relevance; specifically, the process includes: firstly, evaluating and processing the timeliness of each target content information according to the information timeliness in the first evaluation condition, and eliminating data that does not meet the condition; the information timeliness is the timeliness interval of the target information set by the user; then calculating the information relevance of the remaining data, and screening out content with a relevance not lower than a threshold as the first screening data; the information relevance is the semantic cosine value between the target information and the annotation label input by the user.

[0040] S3. Screening data proofreading and feature feedback: proofread the first screening data to obtain historical information, and feed it back to the subsequent target content information collection stage; wherein, the historical information includes proofread information and unproofread information; including: constructing a list of information to be matched, and adding each of the first screening data to the list; using the matching model to match the information one by one; if the match is successful, pre-marking is performed synchronously, if it fails, proofreading is continued, and the historical information is optimized and updated according to the preset strategy; the historical information includes: first historical information, indicating data in the first screening data that is still consistent with the first screening data after proofreading; second historical information, indicating data in the first screening data that is found to be inconsistent with the first screening data after proofreading; third historical information, indicating newly added data that does not match the historical annotation label.

[0041] S4, difference feature extraction and labeling preparation: overflow screening is performed on the historical information to obtain difference information, and the difference information is marked as data to be labeled;

[0042] S5, automatic labeling and error rate verification and update: segmenting the data to be labeled;

[0043] The segmentation process specifically refers to: based on the annotation label input by the user, extending the hit point of the data to be annotated forward and backward by a necessary length to obtain a semantically complete text, image or video segment, while cutting off the text, image or video segment containing non-annotated information;

[0044] After segmentation processing, multiple segments to be labeled are obtained, and then the segments to be labeled are input into the automatic labeling model one by one to obtain labeled data. The labeled data are verified according to the error rate of historical information, and the error rate is updated according to the verification result; the error rate update process includes: constructing a list of label information to be verified, extracting the update time and model input of each label information; determining unlabeled or erroneous labels based on the matching degree with the historical labeled labels, setting the total number of labeling errors, calculating the deviation rate, and dynamically updating the error rate according to the unlabeled frequency, and re-incorporating the updated results into the historical information.

[0045] S6. Deviation measurement and monitoring interval generation: perform deviation measurement on the updated error rate, and then generate a monitoring evaluation interval based on the deviation, and update the error rate at the next data annotation, and classify the output results of the automatic annotation model into frozen data and unfrozen data based on the instantaneous value of the updated error rate and the monitoring evaluation interval; the deviation measurement includes: dividing the updated error rate into multiple value segments according to the time dimension, performing discretization preprocessing and sorting, combining the set smoothing coefficient to input the deviation measurement model, and outputting the deviation result; the generation method of the monitoring evaluation interval includes: inputting the error rate into the verification model, obtaining verification information and performing wavelet transform and feature sorting, and then performing first-order difference and inverse normalization processing, calculating the risk value based on the updated deviation and the set risk threshold, and generating high interval value and low interval value accordingly to construct the final monitoring evaluation interval.

[0046] Example 2

[0047] This embodiment provides a system for automatic data labeling based on model training, including:

[0048] An information collection module, configured to collect target content information using a collection model, wherein the target content information includes at least one of text, images, and videos;

[0049] An information screening module, configured to screen target content information according to a first evaluation condition to obtain first screening data, wherein the first evaluation condition includes timeliness and relevance of the information;

[0050] a data proofreading module, configured to proofread the first screened data to obtain historical information, and provide feedback to a subsequent target content information collection stage, wherein the historical information includes proofread information and unproofread information;

[0051] A data overflow screening module, the data overflow screening module is used to perform overflow screening on the historical information to obtain difference information, and mark the difference information as data to be marked;

[0052] An automatic labeling module, wherein the automatic labeling module is used to segment the data to be labeled to obtain multiple segments to be labeled, and input the segments to be labeled one by one into the automatic labeling model to obtain labeled data, and verify the labeled data based on the error rate of historical information, and update the error rate according to the verification result;

[0053] A deviation analysis module is used to measure the deviation of the updated error rate and generate a monitoring evaluation interval based on the deviation, update the error rate during the next data annotation, and classify the output results of the automatic annotation model into frozen data and unfrozen data based on the instantaneous value of the updated error rate and the monitoring evaluation interval.

[0054] The basic principles, main features, and advantages of the present invention are shown and described above. Those skilled in the art should understand that the present invention is not limited to the foregoing embodiments. The foregoing embodiments and descriptions are merely illustrative of the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and modifications are intended to fall within the scope of the present invention. The scope of protection claimed in the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for automatic data annotation based on model training, characterized in that: include: S1. Target content information collection: Based on the annotation tags input by the user, the target content information is collected using the RAG large language model as the collection model; wherein the target content information includes at least one of text, image, and video; S2. Information screening and preliminary evaluation: Screening the target content information according to a first evaluation condition to obtain first screening data; wherein the first evaluation condition includes information timeliness and information relevance; S3. Screening Data Verification and Feature Feedback: Verify the first screening data to obtain historical information, and provide feedback to the subsequent target content information collection stage; S4, difference feature extraction and labeling preparation: overflow screening is performed on the historical information to obtain difference information, and the difference information is marked as data to be labeled; S5. Automatic labeling and error rate verification and update: segmenting the data to be labeled with the data to be labeled to obtain labeled data, verifying the labeled data based on historical information error rates and updating the error rates; S6. Deviation measurement and monitoring interval generation: The deviation of the updated error rate is measured, and the monitoring evaluation interval is generated based on the deviation. The error rate is updated during the next data annotation, and the output results of the automatic annotation model are classified into frozen data and unfrozen data based on the instantaneous value of the updated error rate and the monitoring evaluation interval.

2. The method for automatic data labeling based on model training according to claim 1, characterized in that: Said S1 specifically includes: A collection website list is constructed, and a collection instruction is issued to each collection website in the collection website list; an interface of the collection website is called, each collection website is accessed, and output information of the collection website is collected to obtain target content information.

3. The method for automatic data labeling based on model training according to claim 1, characterized in that: The S2 specifically includes: First, the timeliness of each target content information is evaluated based on the information timeliness in the first evaluation condition, and data that does not meet the condition is eliminated; the information timeliness is the timeliness interval of the target information set by the user; Then, information relevance is calculated for the remaining data, and content with information relevance not less than a threshold is screened out as the first screening data; the information relevance is the semantic cosine value between the target information and the annotation label input by the user.

4. The method for automatic data labeling based on model training according to claim 1, characterized in that: The S3 includes: Construct a list of information to be matched, and add each of the first screening data to the list; use the matching model to match the information one by one; if the match is successful, perform pre-marking synchronously; if it fails, continue proofreading, and optimize and update historical information according to the preset strategy.

5. The method for automatic data labeling based on model training according to claim 4, characterized in that: The historical information includes: The first historical information represents data in the first filtered data that remains consistent with the first filtered data after verification; The second historical information indicates data in the first screening data that is found to be inconsistent with the first screening data after verification; The third historical information represents newly added data that does not match the historical annotation labels.

6. The method for automatic data labeling based on model training according to claim 1, characterized in that: The error rate update process in S5 includes: Build a list of label information to be verified, extract the update time and model input of each label information; determine unlabeled or incorrect labels based on the matching degree with historically labeled labels, set the total number of labeling errors, calculate the deviation rate, and dynamically update the error rate based on the unlabeling frequency, and re-incorporate the updated results into the historical information.

7. The method for automatic data labeling based on model training according to claim 1, characterized in that: The deviation calculation in S6 includes: The updated error rate is divided into multiple value segments according to the time dimension, and discretization preprocessing and sorting are performed. The deviation measurement model is input with the set smoothing coefficient to output the deviation result.

8. The method for automatic data labeling based on model training according to claim 1, characterized in that: The method for generating the monitoring evaluation interval includes: The error rate is input into the verification model to obtain verification information and perform wavelet transform and feature sorting. Then, first-order difference and inverse normalization are performed. The risk value is calculated based on the updated bias and the set risk threshold, and the high interval value and low interval value are generated accordingly to construct the final monitoring evaluation interval.

9. A system for automatic data annotation based on model training, used to implement the method according to any one of claims 1 to 8, characterized in that: include: Information collection module: used to build a collection website list, issue collection instructions, and call the interface to capture text, image, and video target content information; Information screening module: used to evaluate and process target content information based on the timeliness and relevance of the information, remove data that does not meet the conditions, and output first screened data; Data proofreading module: used to call the matching model to compare the first screening data with the historical information, generate proofread and unproofread information, and feed the comparison results back to the acquisition model; Data overflow screening module: used to extract difference information from historical information, identify change trends, and mark it as data to be labeled for use by the automatic labeling model; Automatic labeling module: used to input the data to be labeled into the model in segments for labeling, verify and correct the labeling results based on the historical error rate, and dynamically update the error rate parameters; Deviation analysis module: used to perform deviation measurement on the updated error rate, generate monitoring evaluation intervals, and classify the status of model output results according to the degree of deviation to achieve dynamic control of annotation quality.

10. An electronic device, characterized in that: The method comprises at least one processor and a memory in communication with the processor, wherein the memory stores a computer program executed by the processor, and the program is used to execute the method according to any one of claims 1 to 8.

Images