A deep learning model training sample intelligent labeling method

Abstract

The application discloses a kind of deep learning model training sample intelligent labeling method, comprising: if labeling task is not in business system, then outside business system labeling is carried out;If labeling task is in business system, then inside business system labeling is carried out;According to the number of labeling sample obtained by outside business system labeling and inside business system labeling, it is judged whether it needs incremental labeling;If it does not need incremental labeling, then labeling task ends;If it needs incremental labeling, then outside business system labeling is optimized by incremental labeling.The application improves the data labeling efficiency of labeling personnel, and also integrates the business experience and knowledge of labeling personnel into the business data processing process, realizes the "business data processing-feedback-model updating" cyclic iteration, and improves the performance and efficiency of intelligent processing system.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data intelligent labeling, in particular to a deep learning model training sample intelligent labeling method. BACKGROUND

[0002] Data is the basis for improving the effect of a deep learning model, and rich labels are the labels for successful modeling of a deep learning model. Sample data labeling is a key link for most artificial intelligence algorithms to effectively run. It mainly labels voice, images, text and the like, and mainly labels data sets through marking, highlighting, tagging, framing objects, annotating and the like, and then trains and learns the data sets by a machine.

[0003] At present, data labeling of domestic teams often adopts a crowdsourcing and manual labeling mode, such as a Baidu data crowdsourcing service and an Ant crowdsourcing. The crowdsourcing mode can quickly collect a large amount of labeled data, but the quality of the labeled sample data is often uneven. Meanwhile, in the field of security, due to the reason of data confidentiality, public labeling often cannot be performed. In order to improve the quantity and quality of the training data of a field deep learning model, a business personnel often needs to spend a large amount of time and energy to complete it, which will bring a huge workload to the business personnel. SUMMARY

[0004] In view of this, the present application provides a deep learning model training sample intelligent labeling method, which is used in a machine learning and deep learning model training data preparation stage, uses a business system external labeling, business data internal labeling and incremental labeling process method, improves the training sample acquisition efficiency, and further improves the effect of a deep learning model.

[0005] The present application discloses a deep learning model training sample intelligent labeling method, which comprises the following steps:

[0006] If the labeling task is not in the business system, business system external labeling is performed;

[0007] If the labeling task is in the business system, business system internal labeling is performed;

[0008] According to the number of labeled samples obtained by the business system external labeling and the business system internal labeling, it is determined whether incremental labeling is needed;

[0009] If the incremental labeling is not needed, the labeling task is ended;

[0010] If the incremental labeling is needed, the business system external labeling is optimized through the incremental labeling.

[0011] Further, the business system external labeling comprises the following steps:

[0012] A to-be-labeled sample is obtained from a sample material;

[0013] manually annotating or machine pre-annotating the to-be-annotated sample, to obtain annotated sample data;

[0014] After manual modification and confirmation of the annotated sample data, the annotated sample data is stored in a corresponding sample library.

[0015] Further, the to-be-annotated sample obtained from the sample material comprises:

[0016] According to the specific task mission, application scene requirement, and annotation theme, the to-be-annotated sample is obtained from the sample material.

[0017] Further, the manually annotating or machine pre-annotating the to-be-annotated sample comprises:

[0018] If there is a machine extraction model corresponding to the to-be-annotated sample, machine pre-annotation is performed.

[0019] If there is no machine extraction model corresponding to the to-be-annotated sample, manual annotation is performed.

[0020] Further, the process of manual annotation comprises:

[0021] According to the annotation task requirement, the to-be-annotated sample is manually annotated one by one until the annotation of all samples in the current task is completed.

[0022] Further, the process of machine pre-annotation comprises:

[0023] For the obtained to-be-annotated sample, a machine extraction model corresponding to the annotation task is used to generate a machine annotation result.

[0024] Further, the machine extraction model is a deep learning algorithm model.

[0025] Further, the number of annotated samples obtained by the business system outside annotation and the business system inside annotation is used to determine whether incremental annotation is needed, comprising:

[0026] The annotated samples obtained by the business system outside annotation and the business system inside annotation are used as training samples of the machine extraction model, and when the number of training samples of the machine extraction model reaches a preset threshold, incremental annotation is needed, otherwise, the annotation task is ended.

[0027] Further, the optimization of the business system outside annotation through incremental annotation comprises:

[0028] The machine extraction model is incrementally trained, and the original machine extraction model is updated and optimized; after the model training is completed, the updated and optimized machine extraction model is used to continue machine pre-annotation in the business system outside annotation.

[0029] Further, the process of marking in the business system is:

[0030] The data marking process is integrated into the use process of the business system by using the "man-in-loop" marking method. The data processed by the algorithm service is fed back and corrected. The corrected result and the corresponding business data text are stored in the sample library as marked samples.

[0031] Due to the above technical solutions, the present application has the following advantages:

[0032] (1) The business system marking proposed in the present application integrates data marking work into the daily use work of business personnel in the business system. Business personnel correct, feed back and mark algorithm output data while reading reports. The marked data flows back to the training sample set. In this way, the data marking efficiency of the marking personnel is improved, and the business experience and knowledge of the marking personnel are integrated into the business data processing process. The "business data processing-feedback-model updating" cycle iteration is realized, and the performance and efficiency of the intelligent processing system are improved.

[0033] (2) The incremental learning marking proposed in the present application is used for marking historical data. A large number of newly added marking data in the data marking process is used to optimize the pre-marking model. Then, in the subsequent data marking process, the optimized pre-marking model is used to re-mark the subsequent data to be marked or verified. Incremental learning marking not only deepens the model's memory of the marked data type, but also optimizes the newly marked data type, further improving the data marking efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments described in the present application. Other drawings can be obtained by those skilled in the art based on these drawings.

[0035] Figure 1 A flowchart of a deep learning model training sample intelligent marking method according to an embodiment of the present application;

[0036] Figure 2 An entity recognition model structure diagram according to an embodiment of the present application. DETAILED DESCRIPTION

[0037] The present application is further illustrated in combination with the drawings and embodiments. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. All other embodiments obtained by those skilled in the art should belong to the scope of protection of the embodiments of the present application.

[0038] In order to more specifically express the method for implementing the present application, the present application takes the annotation of the training corpus of the entity recognition model in the company financial statement analysis scenario in the economic field as an example, and the following is described, the corpus material text content theme is "asset liability increase and decrease analysis situation", and the entity types that need to be annotated include time, company name, fixed asset amount, growth rate, change trend, etc.

[0039] As shown in Figure 1 , the deep learning model training sample intelligent annotation method provided by the embodiment of the present application comprises the following steps:

[0040] 1. External annotation of the business system: retrieve and obtain "asset liability increase analysis situation" related content text materials from the corpus, not less than 50, and the number of sentences is not less than 500. Set the color displayed on the front-end page after annotating the time, company name, fixed asset amount, growth rate, change trend, etc. According to whether the above type entity extraction model is used, the method of machine pre-annotation is used, otherwise manual annotation is used to complete the annotation task.

[0041] 1.1 Manual annotation: select a text corpus to be annotated in the current text material entity type annotation task; for each sentence in the text corpus, manually judge whether the sentence contains time, company name, fixed asset amount, growth rate, change trend, etc. Type entity, and annotate the entity in the sentence in the word division manner, and each type of entity is represented by a different color; after annotation, generate an annotation sample as shown in Table 1, and store it in the sample library. Wherein sen_text is the sentence to be annotated, entities is the annotation result, text is the annotated entity text, type is the entity type, start is the start position of the entity in the sentence, and end is the end position of the entity in the sentence.

[0042] After annotating this text corpus, select the next document for annotation until all the annotations of the corpus in the current task are completed.

[0043] 1.2 Machine pre-annotation: use the entity recognition model as shown in Figure 2 to complete the "asset liability increase and decrease analysis situation" text entity type annotation. Select any one text corpus to be annotated in the current text material; for each sentence in the selected text corpus, call the entity recognition model service as shown in Figure 2 , generate an annotation result sample as shown in Table 1, and manually modify and confirm the result sample in the word division manner, and then store it in the sample library.

[0044] Table 1 Entity recognition corpus sample format

[0045]

[0046] 2. Intra-business system annotation: sample annotation is a link in the business system process, and the annotated data is directly used as business system information flow data. The entity recognition task annotation is embedded into the business data keyword information extraction link, the keywords extracted by the business system are manually modified, and the keyword types are supplemented, and the system rewrites the supplemented and modified business data format into an entity recognition model training format, that is, generates an entity recognition model training corpus, and stores it in a corpus library.

[0047] 3. Incremental learning annotation: based on the number of newly added model training corpora in the entity recognition corpus, when the newly added sample data reaches a certain threshold, use most of the newly added corpora to start Figure 2 The entity recognition model training incrementally trains the existing model to update and optimize the training of the deep learning model; after the model training is completed, the remaining small part of the newly added corpora is used to evaluate the accuracy and recall rate of the trained model, and the F1 value is calculated, wherein the F1 value is the harmonic mean of the accuracy and recall rate. If the F1 value is greater than the F1 value of the original model, update the original entity recognition model file, and update Figure 2 The model.

[0048] The above describes the preferred embodiments of the present application, and it should be noted that the above embodiments are used to illustrate the present application, but the present application is not limited thereto, and those skilled in the art can design alternative embodiments without departing from the scope of the appended claims. For those skilled in the art, various modifications and improvements can be made without departing from the spirit and essence of the present application, and these modifications and improvements are also considered within the protection scope of the present application.

Claims

1. A method for intelligent annotation of training samples for deep learning models, characterized in that, include: If the annotation task is not within the business system, then perform annotation outside the business system; If the annotation task is located in a business system, then the annotation will be performed within the business system. Determine whether incremental annotation is needed based on the number of annotation samples obtained from external annotation and internal annotation within the business system; If incremental annotation is not required, the annotation task ends. If incremental annotation is required, optimize the annotation outside the business system through incremental annotation; The external annotations of the business system include: Obtain the samples to be labeled from the sample materials; The samples to be labeled are manually labeled or pre-labeled by machine to obtain labeled sample data; After the labeled sample data is manually modified and confirmed, it is stored in the corresponding sample library; The step of determining whether incremental annotation is needed based on the number of annotation samples obtained from external annotations and internal annotations within the business system includes: The labeled samples obtained from external and internal annotations of the business system are used as training samples for the machine extraction model. When the number of training samples for the machine extraction model reaches a preset threshold, incremental annotation is required; otherwise, the annotation task ends. The optimization of external annotations in the business system through incremental annotation includes: Incrementally train the machine extraction model and update and optimize the original machine extraction model; after the model training is completed, use the updated and optimized machine extraction model to continue machine pre-labeling in the external annotation of the business system. The annotation process within the business system is as follows: The "human-in-the-loop" annotation method is adopted to integrate the data annotation process into the use of the business system. Feedback and correction are provided on the data processed by the algorithm service, and the corrected results and the corresponding business data text are stored as annotated samples in the sample library.

2. The method according to claim 1, characterized in that, The step of obtaining the sample to be labeled from the sample material includes: Based on the specific mission, application scenario requirements, and annotation theme, retrieve the samples to be annotated from the sample materials.

3. The method according to claim 2, characterized in that, The manual or machine pre-labeling of the samples to be labeled includes: If a machine extraction model exists that corresponds to the sample to be labeled, then machine pre-labeling is performed; If no machine extraction model exists for the sample to be labeled, then manual labeling is performed.

4. The method according to claim 3, characterized in that, The manual annotation process is as follows: According to the annotation task requirements, each sample to be annotated is manually annotated until all samples in the current task have been annotated.

5. The method according to claim 3, characterized in that, The machine pre-labeling process is as follows: For the obtained samples to be labeled, the machine extraction model corresponding to the labeling task is used to generate machine labeling results.

6. The method according to any one of claims 3-5, characterized in that, The machine extraction model is a deep learning algorithm model.

Images