An intelligent labeling method and system based on natural language processing

Abstract

The present invention discloses an intelligent labeling method and system based on natural language processing, including: constructing a training set and a test set; constructing a labeling model based on the training set and the test set; wherein the training set and the test set are multiple field texts with known labels; encapsulating the constructed labeling model and generating an interface; configuring the interface; obtaining text data to be processed; calling the configuration to label the natural language to be processed. The implementation of the present invention enables the text labeling task to realize the automation of the entire process of model construction, testing, online, execution and text labeling, greatly improving work efficiency and saving the company's labor costs. At the same time, it has been verified that the invention also achieves a high accuracy rate.

Description

Technical Field

[0001] The present invention relates to the field of smart labeling technology, and in particular to a smart labeling method and system based on natural language processing. Background Art

[0002] The statements in this section merely mention background art related to the present invention and do not necessarily constitute prior art.

[0003] In the era of big data, a large amount of text information data has emerged, and more and more text data is stored in the database for unified preservation. Therefore, how to uniformly manage and label the massive text data in the database is a key issue that needs to be solved at present. However, most of the existing text labeling methods focus on processing only the text content of a single field, and have not realized the construction of a complete process for labeling text information in all fields in the database, and also lack closed-loop management of text labeling tasks. Therefore, it is necessary to combine the actual application scenarios of text labeling tasks, build an automated text labeling method to label massive texts, and at the same time, realizing the closed loop of text labeling tasks is the core issue of this task. Summary of the invention

[0004] In order to solve the deficiencies of the prior art, the present invention provides an intelligent labeling method and system based on natural language processing; first, a database text labeling task flow is constructed, then a text labeling processing model based on natural language processing is constructed, the corresponding model is selected for matching the text data to be processed in the database to complete the labeling task creation, and finally, the text is automatically labelled intelligently according to the task execution rules. Thus, the work of intelligently labelling text information data in the database based on natural language processing is completed.

[0005] In a first aspect, the present invention provides an intelligent labeling method based on natural language processing;

[0006] An intelligent labeling method based on natural language processing, comprising:

[0007] Constructing a training set and a test set; constructing a label model based on the training set and the test set; wherein the training set and the test set are multiple field texts with known labels;

[0008] Encapsulate the constructed label model and generate an interface;

[0009] Configure the interface;

[0010] Get the text data to be processed; call the configuration to label the natural language to be processed.

[0011] In a second aspect, the present invention provides an intelligent labeling system based on natural language processing;

[0012] An intelligent labeling system based on natural language processing, comprising:

[0013] A construction module is configured to: construct a training set and a test set; and construct a label model based on the training set and the test set; wherein the training set and the test set are multiple field texts with known labels;

[0014] The encapsulation module is configured to: encapsulate the constructed label model and generate an interface;

[0015] A configuration module, which is configured to: configure the interface;

[0016] The tagging module is configured to: obtain text data to be processed; call configuration to perform tagging processing on the natural language to be processed.

[0017] In a third aspect, the present invention further provides an electronic device, comprising:

[0018] a memory for non-transitory storage of computer-readable instructions; and

[0019] a processor for executing the computer readable instructions,

[0020] When the computer-readable instructions are executed by the processor, the method described in the first aspect is executed.

[0021] In a fourth aspect, the present invention further provides a storage medium that non-temporarily stores computer-readable instructions, wherein when the non-temporary computer-readable instructions are executed by a computer, the instructions of the method described in the first aspect are executed.

[0022] Compared with the prior art, the present invention has the following beneficial effects:

[0023] In actual work, manual data labeling is time-consuming and laborious, but after the introduction of the labeling model, only part of the manual operation can be reduced. Especially after the model is updated or the labeling data is updated, the existing labeling model and labeling method cannot be reused, which results in a waste of the company's manpower and development costs. The implementation of the present invention enables the text labeling task to label multiple fields at the same time, realizes the automation of the entire process of model construction, testing, online, execution and text labeling, greatly improves work efficiency, saves the company's manpower costs, and it has been verified that the invention also achieves a higher accuracy rate.

[0024] Advantages of additional aspects of the present invention will be partially given in the following description or learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0026] Figure 1 This is a flow chart of the method of the first embodiment. DETAILED DESCRIPTION

[0027] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0028] It should be noted that the terms used herein are only for describing specific embodiments, and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that the terms "include" and "have" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0029] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0030] In this embodiment, all data is obtained in compliance with laws and regulations and based on the user's consent, and is used legally.

[0031] Embodiment 1

[0032] This embodiment provides an intelligent labeling method based on natural language processing;

[0033] like Figure 1 As shown, an intelligent labeling method based on natural language processing includes:

[0034] S101: Construct a training set and a test set; construct a label model based on the training set and the test set; wherein the training set and the test set are multiple field texts with known labels;

[0035] S102: Encapsulate the constructed label model and generate an interface;

[0036] S103: configure the interface;

[0037] S104: Acquire text data to be processed; call configuration to perform labeling on the natural language to be processed.

[0038] For example, in the basic information data A of a company's employees, there are multiple fields: department, place of origin, school of graduation, education and title, etc.; current technical research mainly focuses on obtaining a label based on only one field, for example: obtaining a "graduate student" label based on "education level"; this patent combines specific business scenarios to achieve obtaining a label based on multiple fields, for example: obtaining a company's dimensional labels such as "graduate student with intermediate professional title" or "undergraduate with junior professional title" based on "department", "education level" and "title", thereby increasing the dimension of data analysis. Further, S101: constructing training sets and test sets; specifically including:

[0039] Get text data with known labels;

[0040] Perform data cleaning on the text data with known labels. During the data cleaning process, delete spaces, line breaks, exclamation marks, and special characters in the text data; special characters refer to "*" or "#";

[0041] Divide the cleaned data into several text entries;

[0042] The obtained text entries are divided into training sets and test sets in proportion.

[0043] Exemplarily, the data cleaning is implemented by using a replace function.

[0044] Exemplarily, the data after data cleaning is divided into a plurality of text entries, which is implemented by using the spilt function.

[0045] Further, S101: constructing a label model based on the training set and the test set; specifically including:

[0046] The language representation model (Bidirectional Encoder Representations from Transformers, BERT) is used to extract features from the text data in the training set to obtain text features;

[0047] Based on the text features and corresponding labels in the training set, the convolutional neural network is trained to obtain a trained convolutional neural network, and the trained convolutional neural network is used as a label model;

[0048] Based on the text features and corresponding labels in the test set, the trained convolutional neural network is tested to obtain the accuracy of the model's labeling;

[0049] If the accuracy rate exceeds the set threshold, the training ends and the trained convolutional neural network is used as the label model; if the accuracy rate is lower than the set threshold, retraining is performed.

[0050] Further, S102: encapsulate the constructed label model and generate an interface; specifically including:

[0051] S1021: define a label model request method, implement the definition of request parameters, output of the return parameter text label label and the score corresponding to the return parameter text label label, and complete the encapsulation of the label model request method;

[0052] S1022: The constructed label model and the encapsulated label model request method are started to generate a label model calling interface through a python command.

[0053] Further, S103: configuring the interface; specifically including:

[0054] S1031: Calling the API interface according to the tag model, and using the ElementUI method through the VUE framework to build the model visualization configuration function;

[0055] S1032: According to the address of the model calling interface generated in S1022 and the label model request method defined in S1021, obtain the label model return parameters, and save them according to the parameters of the defined text label label and the score corresponding to the text label label to complete the visualization interface configuration of the label model.

[0056] Further, S104: obtaining text data to be processed; calling configuration to perform labeling processing on the natural language to be processed; specifically including:

[0057] S1041: Introduce the VUE framework, use the ElementUI method, set the label task name, model call interface, request parameters, return parameters, and complete the label task function configuration;

[0058] S1042: According to the label task configuration function constructed in S1041, the label model interface constructed in S1022 and the text content to be labeled, configure the label task name, model call interface, request parameters, return parameters and label field information;

[0059] S1043: Obtain input data information to be labeled, and configure the output table name and output table fields of the labeled data;

[0060] S1044: DolphinScheduler is used to build a distributed workflow scheduling diagram to automatically start and stop labeling tasks and complete labeling.

[0061] The present invention designs an intelligent labeling system and method based on natural language processing. First, a labeling model based on Bert+ rules is constructed using natural language processing technology, and then a text labeling task is constructed. The task construction is completed by selecting the data to be labeled and the labeling model. Finally, the intelligent labeling task is automatically executed to complete the labeling operation.

[0062] Embodiment 2

[0063] This embodiment provides an intelligent labeling system based on natural language processing;

[0064] An intelligent labeling system based on natural language processing, comprising:

[0065] A construction module is configured to: construct a training set and a test set; and construct a label model based on the training set and the test set; wherein the training set and the test set are multiple field texts with known labels;

[0066] The encapsulation module is configured to: encapsulate the constructed label model and generate an interface;

[0067] A configuration module, which is configured to: configure the interface;

[0068] The tagging module is configured to: obtain text data to be processed; call configuration to perform tagging processing on the natural language to be processed.

[0069] It should be noted that the above-mentioned building module, encapsulation module, configuration module and labeling module correspond to steps S101 to S104 in Embodiment 1, and the examples and application scenarios implemented by the above-mentioned modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above-mentioned Embodiment 1. It should be noted that the above-mentioned modules as part of the system can be executed in a computer system such as a set of computer executable instructions.

[0070] The description of each embodiment in the above embodiments has different emphases. For parts not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0071] The proposed system can be implemented in other ways. For example, the system embodiment described above is only illustrative, and the division of the modules is only a logical function division. In actual implementation, there may be other division methods, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed.

[0072] Embodiment 3

[0073] This embodiment also provides an electronic device, including: one or more processors, one or more memories, and one or more computer programs; wherein the processor is connected to the memory, and the one or more computer programs are stored in the memory. When the electronic device is running, the processor executes the one or more computer programs stored in the memory so that the electronic device executes the method described in the above embodiment one.

[0074] It should be understood that in this embodiment, the processor may be a central processing unit CPU, and the processor may also be other general-purpose processors, digital signal processors DSP, application-specific integrated circuits ASIC, off-the-shelf programmable gate arrays FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0075] The memory may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type.

[0076] In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in a processor or an instruction in the form of software.

[0077] The method in the first embodiment can be directly embodied as a hardware processor, or a combination of hardware and software modules in the processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware. To avoid repetition, it will not be described in detail here.

[0078] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with this embodiment can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0079] Embodiment 4

[0080] This embodiment further provides a computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, the method described in the first embodiment is completed.

[0081] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An intelligent labeling method based on natural language processing, characterized in that: include: Construct training and test sets; Based on the training set and test set, a label model is constructed, including: The language representation model is used to extract features from the text data in the training set to obtain text features; Based on the text features and corresponding labels in the training set, the convolutional neural network is trained to obtain a trained convolutional neural network, and the trained convolutional neural network is used as a label model; Based on the text features and corresponding labels in the test set, the trained convolutional neural network is tested to obtain the accuracy of the model's labeling; If the accuracy rate exceeds the set threshold, the training ends and the trained convolutional neural network is used as the label model; if the accuracy rate is lower than the set threshold, retraining is performed; The training set and the test set are both multiple field texts with known labels; the construction of the training set and the test set specifically includes: obtaining text data with known labels; performing data cleaning on the text data with known labels, during which spaces, line feed characters, exclamation marks and special characters in the text data are deleted; dividing the cleaned data into a number of text entries; and dividing the obtained text entries into the training set and the test set in proportion; Encapsulate the constructed label model and generate an interface, including: Define the label model request method, implement the definition of request parameters, return parameter text label label, and output the score corresponding to the return parameter text label label, completing the encapsulation of the label model request method; Use the python command to start generating the label model calling interface by using the constructed label model and the encapsulated label model request method; Configure the interface, including: Call the API interface according to the tag model, and use the ElementUI method to build the model visualization configuration function through the VUE framework; According to the address of the generated model call interface and the defined label model request method, the label model return parameters are obtained, and the parameters of the defined text label label and the corresponding score of the text label label are saved to complete the visualization interface configuration of the label model; Get the text data to be processed and get a label based on multiple fields; call the configuration to label the natural language to be processed, including: Introduce the VUE framework, use the ElementUI method to set the label task name, model call interface, request parameters, and return parameters to complete the label task function configuration; According to the constructed labeling task configuration function, the constructed labeling model calling interface and the text content to be labeled, configure the labeling task name, model calling interface, request parameters, return parameters and label field information; Get the input data information to be labeled, and configure the output table name and output table fields of the labeled data; By using DolphinScheduler to build a distributed workflow scheduling diagram, we can automatically start and stop labeling tasks and complete labeling.

2. The intelligent labeling method based on natural language processing as claimed in claim 1, characterized in that: The data cleaning is implemented by using the replace function; the data after the data cleaning is divided into a number of text items, which is implemented by using the spilt function.

3. An intelligent labeling system based on natural language processing, using the intelligent labeling method based on natural language processing as described in any one of claims 1-2, characterized in that: include: A construction module, which is configured to: construct a training set and a test set; Based on the training set and the test set, a label model is constructed; wherein the training set and the test set are multiple field texts with known labels; an encapsulation module is configured to: encapsulate the constructed label model and generate an interface; A configuration module, which is configured to: configure the interface; The tagging module is configured to: obtain text data to be processed; call configuration to perform tagging processing on the natural language to be processed.

4. An electronic device, comprising: a memory for non-transitory storage of computer readable instructions; as well as a processor for executing the computer readable instructions, When the computer-readable instructions are executed by the processor, an intelligent labeling method based on natural language processing as described in any one of claims 1 to 2 is executed.

5. A storage medium characterized by being non-temporary The computer-readable instructions are stored in a non-temporary manner, wherein when the non-temporary computer-readable instructions are executed by a computer, the instructions of the intelligent labeling method based on natural language processing as described in any one of claims 1-2 are executed.

Images