Text extraction model training method, device, system and storage medium

Abstract

This application provides a training method, device, system, and storage medium for a text extraction model, relating to the field of artificial intelligence. In the training method, a server receives a model training request from a client. Based on the request, the server retrieves an existing first text extraction model from a model library. Then, based on this first model, a second text extraction model is created. The second model is trained using a new training dataset from the training request, enabling it to extract new tags in addition to the text extraction capabilities of the first model. After completing the training of the second text extraction model, the server returns it to the client, allowing users to utilize it in practical applications. This approach shortens model training time, reduces server resource consumption during training, and ensures data security.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a training method, device, system, and storage medium for a text extraction model. Background Technology

[0002] The Natural Language Processing (NLP) self-learning platform supports customized NLP algorithm capabilities such as text entity extraction, text classification, key phrase extraction, sentiment analysis, relation extraction, product evaluation parsing, resume extraction, and contract element extraction. Users do not need to have extensive algorithm background; they only need to annotate or upload a suitable amount of document data to quickly create and use algorithm models through the platform.

[0003] Currently, NLP self-learning platforms offer some pre-trained models, but these cannot fully meet user needs. For example, the platform provides a model for extracting contract elements, which can effectively extract the 10 entity types it supports, such as time, location, and internship content. If users want the model to extract new entity types, such as whether someone has a certain R&D experience, the current approach is for users to rebuild a new model and train it using newly built training data. The newly trained model can then extract more entity types, thereby improving its extraction capabilities.

[0004] However, if the above methods involve a large amount of data, the training time and resources will be consumed, resulting in a poor user experience. Summary of the Invention

[0005] This application provides a training method, device, system, and storage medium for a text extraction model, which can shorten the training time and reduce resource consumption during training.

[0006] The first aspect of this application provides a method for training a text extraction model.

[0007] Receive a model training request from the client. The model training request is used to request the update of the first text extraction model. The first text extraction model is a pre-trained base model. The first text extraction model can extract text corresponding to preset labels.

[0008] Based on the model training request, a second text extraction model is created on the basis of the first text extraction model, and the second text extraction model is trained. The second text extraction model can extract the text corresponding to the newly added label.

[0009] A model training response is sent to the client, the model training response including a trained second text extraction model.

[0010] In an optional embodiment of the first aspect of this application, receiving a model training request from a client includes: in response to a first operation by a user on a model creation interface, receiving the model training request from the client;

[0011] The first operation includes the user selecting the first text extraction model on the model creation interface, and the user adding the new training dataset on the model creation interface.

[0012] The model training request includes the identifier of the first text extraction model and a new training dataset for updating the first text extraction model.

[0013] In an optional embodiment of the first aspect of this application, the step of creating a second text extraction model based on the model training request and the first text extraction model includes:

[0014] Based on the identifier of the first text extraction model in the model training request, the first text extraction model is obtained from the model library;

[0015] Based on the first text extraction model, a new decoding unit is added to obtain the second text extraction model; the new decoding unit is used to extract the text corresponding to the newly added tag.

[0016] In an optional embodiment of the first aspect of this application, the training method further includes:

[0017] Obtain the model parameters of the first text extraction model from the model library;

[0018] Assign the model parameters of the first text extraction model to the model structure corresponding to the second text extraction model;

[0019] The model parameters of the new decoding unit are randomly initialized.

[0020] In an optional embodiment of the first aspect of this application, training the second text extraction model includes:

[0021] Based on the new training dataset in the model training request and the created second text extraction model, train the second text extraction model;

[0022] When the loss function of the second text extraction model converges, training of the second text extraction model is stopped;

[0023] The new training dataset includes multiple training texts and corresponding annotation information for the training texts, wherein the annotation information includes the label data of the training texts.

[0024] In an optional embodiment of the first aspect of this application, the second text extraction model includes an encoder and a decoder, the encoder being used to obtain contextual features of the input text of the model, and the decoder being used to predict the labels of each field in the input text, the labels including preset labels and newly added labels.

[0025] In an optional embodiment of the first aspect of this application, the decoder includes a first decoding unit and a second decoding unit, the first decoding unit corresponding to the preset tag, and the second decoding unit being a newly added decoding unit corresponding to the newly added tag.

[0026] In an optional embodiment of the first aspect of this application, the encoder in the second text extraction model includes any of the following model structures: BERT, LSTM, BiLSTM, RoBERTa, XLNet; and the decoder in the second text extraction model includes any of the following model structures: BiLSTM and / or CRF, softmax.

[0027] A second aspect of this application provides a training apparatus for a text extraction model, comprising:

[0028] The receiving unit is used to receive a model training request from the client. The model training request is used to request the update of the first text extraction model. The first text extraction model is a pre-trained base model. The first text extraction model can extract text corresponding to preset labels.

[0029] The processing unit is configured to create a second text extraction model based on the first text extraction model, and train the second text extraction model, based on the model training request. The second text extraction model is capable of extracting the text corresponding to the newly added tag.

[0030] The sending unit is used to return the trained second text extraction model to the client.

[0031] A third aspect of this application provides an electronic device, including: a memory, a processor, and a computer program; the computer program is stored in the memory and configured to be executed by the processor to implement the method as described in any of the first aspects of this application.

[0032] A fourth aspect of this application provides a data processing system, including: an end-side device and a cloud-side device;

[0033] The edge device is used to send a model training request to the cloud device. The model training request is used to request the update of the first text extraction model. The first text extraction model is a pre-trained base model. The first text extraction model can extract text corresponding to preset labels.

[0034] The cloud-side device is used to create a second text extraction model based on the first text extraction model, and train the second text extraction model, which is capable of extracting the text corresponding to the newly added tag, based on the model training request.

[0035] The cloud-side device is also used to send a model training response to the end-side device, the model training response including a trained second text extraction model.

[0036] A fifth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the method as described in any of the first aspects of this application.

[0037] A sixth aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the method described in any of the first aspects of this application.

[0038] A seventh aspect of this application provides a chip, the chip including a processor, the processor being configured to invoke a computer program in memory to perform the method described in any of the first aspects of this application.

[0039] This application provides a training method, device, system, and storage medium for a text extraction model, relating to the field of artificial intelligence. In this training method, a server receives a model training request from a client. Based on the request, the server retrieves an existing first text extraction model from a model library. Then, based on this first model, a second text extraction model is created. The second model is trained using a new training dataset from the training request. This allows the second model to extract text corresponding to newly added tags, in addition to the text extraction capabilities of the first model. After completing the training of the second text extraction model, the server returns it to the client so that the user can use it for practical applications. This approach significantly shortens model training time and improves training efficiency. By utilizing existing model capabilities and directly transferring parameters from existing models, not only can server resource consumption during training be reduced, but data security can also be ensured. Attached Figure Description

[0040] Figure 1 A schematic diagram illustrating an application scenario of the training method for the text extraction model provided in this application embodiment;

[0041] Figure 2 A flowchart illustrating a training method for a text extraction model provided in an embodiment of this application;

[0042] Figure 3 These are schematic diagrams illustrating the structures of two text extraction models provided in embodiments of this application.

[0043] Figure 4 A schematic diagram illustrating the initialization process of the model parameters for the second text extraction model provided in this application embodiment;

[0044] Figure 5 A user interface diagram provided for an embodiment of this application;

[0045] Figure 6 A user interface diagram provided for an embodiment of this application;

[0046] Figure 7 A user interface diagram provided for an embodiment of this application;

[0047] Figure 8 A user interface diagram provided for an embodiment of this application;

[0048] Figure 9 A schematic diagram illustrating data processing for a text extraction model provided in this application embodiment;

[0049] Figure 10 This is a schematic diagram of the structure of a decoder provided in an embodiment of this application;

[0050] Figure 11 A schematic diagram of the structure of a training device for a text extraction model provided in an embodiment of this application;

[0051] Figure 12 This is a hardware structure diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0053] To facilitate a clear description of the technical solutions in the embodiments of this application, the terms "first" and "second" are used to distinguish identical or similar items with essentially the same function and purpose. For example, "first text extraction model" and "second text extraction model" are merely used to distinguish different text extraction models and do not limit their order. Similar descriptions include "first decoding unit" and "second decoding unit," etc. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and that "first" and "second" do not necessarily imply that they are different.

[0054] In this application embodiment, "at least one" refers to one or more, and "multiple" refers to two or more. "And / or" describes the relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following associated objects are in an "or" relationship. "At least one of the following (kind / items)" or similar expressions refer to any combination of these items, including any combination of single (kind / items) or multiple (kind / items). For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0055] It should be noted that, in the embodiments of this application, the words "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design scheme described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0056] The following is a brief introduction to the technical terms used in the embodiments of this application.

[0057] First, BERT is a pre-trained language model (PLM), short for Bidirectional Encoder Representations from Transformers. The BERT model is based on Transformers, which can be understood as a neural network module with a complex internal network structure. The module achieves rapid parallelism through self-attention mechanisms, improving training speed and allowing for increased model depth and accuracy.

[0058] Second, LSTM: Long Short-Term Memory, is a special type of Recurrent Neural Network (RNN) model that can easily process time series data and has wide applications in fields such as NLP.

[0059] Third, BiLSTM: This stands for Bidirectional LSTM. A unidirectional LSTM model can only capture information passed from front to back, while a bidirectional LSTM model can capture both forward and backward information simultaneously, making the utilization of text information more comprehensive and effective.

[0060] Fourth, RoBERTa: An improved version of BERT (A Robustly Optimized BERT Pretraining Approach), which makes the following adjustments based on BERT: longer training time, larger batch size, more training data, removal of next-sentence prediction (NSP), longer training sequences, and dynamic masking.

[0061] Fifth, XLNet: a generalized autoregressive pre-trained model capable of learning bidirectional contextual information. Furthermore, XLNet incorporates Transformer-XL (an improvement on the Transformer model), utilizing broader contextual information.

[0062] Sixth, CRF: Conditional Random Fields, is a model for solving the conditional probability distribution of the output sequence given an input sequence.

[0063] Seventh, softmax: is a fully connected layer whose function is to map the outputs of multiple neurons after computation by the convolutional neural network to the (0, 1) interval, giving the probability of multiple classifications.

[0064] Eighth, entity extraction: a subtask of information extraction, which refers to extracting predefined entity information, such as time, place, and people, from text data.

[0065] Ninth, incremental learning: When new training data is added, there is no need to train the model from scratch; the parameters of the old model can be continued for training.

[0066] Currently, users encounter the following problems when training text extraction models (also known as entity extraction models) using products such as Natural Language Processing (NLP) self-learning platforms:

[0067] First, NLP self-learning platforms offer pre-trained models, such as generic named entity extraction, resume extraction, and contract element extraction, but these may not fully meet user needs. For example, users might want models capable of extracting new entity types. For instance, taking a contract element extraction model, the NLP self-learning platform supports 10 entity types. If a user needs to add two more entity types, based on the existing pre-trained model, the user must re-label data containing all 12 entity types and retrain the model. In some embodiments, entity types are also called label types or field types.

[0068] Secondly, if a user trains model V1 using a batch of data D1, and then adds a new batch of data D2, updating the model can only be done by retraining the model based on D1+D2. If the amount of data is large, the time and resource costs will be high.

[0069] Third, in some scenarios, due to data security requirements, data cannot be copied to other environments. For example, in a judicial setting, data D1 in environment E1 (court 1) and data D2 in environment E2 (court 2) cannot be transferred between different environments, so it is impossible to use D1 and D2 simultaneously to train the model.

[0070] To address the aforementioned issues, this application proposes a training method for a text extraction model. The main idea is as follows: Based on a pre-trained text extraction model, a new text extraction model is constructed, along with a new training dataset. Based on the new training dataset and the newly constructed text extraction model, model training continues. The new text extraction model constructed in this embodiment supports adding new entity types (i.e., adding new tags that the model can recognize). This training process involves adding data training to a pre-trained text extraction model, which shortens training time, improves model training efficiency, reduces server resource consumption during training, and supports migrating only model parameters without transferring data, ensuring data security.

[0071] To facilitate understanding, the application scenarios of the technical solutions provided in the embodiments of this application will be briefly introduced below. Figure 1 This is a schematic diagram illustrating an application scenario of the training method for the text extraction model provided in this application embodiment. For example... Figure 1 As shown, the scenario includes a client 110 and a server 120, with the client 110 connected to the server 120 via a communication network.

[0072] Server 120 can be any suitable server for storing information, data, programs, and / or any other suitable type of content. In some embodiments, server 120 can perform any suitable function. For example, in some embodiments, server 120 deploys executable code for implementing the technical solutions of this application, for training a text extraction model, and for returning the trained text extraction model to client 110.

[0073] The communication network can be any suitable combination of one or more wired and / or wireless networks. For example, the communication network can include any one or more of the following: the Internet, intranet, wide area network (WAN), local area network (LAN), wireless network, digital subscriber line (DSL) network, frame relay network, asynchronous transfer mode (ATM) network, virtual private network (VPN), and / or any other suitable communication network. Client 110 can connect to the communication network via one or more communication links, which can be linked to server 120 via one or more communication links. The communication link can be any communication link suitable for transmitting data between client 110 and server 120, such as a network link, dial-up link, wireless link, hardwired link, any other suitable communication link, or any suitable combination of such links.

[0074] Client 110 may include any one or more user devices suitable for receiving training datasets. In some embodiments, client 110 may include any suitable type of device. For example, in some embodiments, client 110 may include mobile devices, tablet computers, laptop computers, desktop computers, wearable computers, or any other suitable type of user device, and this application embodiment does not impose any limitation on this.

[0075] Although server 120 is illustrated as a single device, in some embodiments, any suitable number of devices may be used to perform the functions performed by server 120. For example, in some embodiments, multiple devices may be used to implement the functions performed by server 120. Alternatively, cloud services may be used to implement the functions of server 120.

[0076] based on Figure 1 The scenario shown is an example. Server 120 receives a model training request from client 110. Based on the model training request and the existing text extraction models in the model library, server 120 creates a new text extraction model. Based on the training dataset in the model training request, server 120 trains the new text extraction model. When the model loss function converges, server 120 outputs the trained text extraction model. Server 120 returns the trained text extraction model (including model parameters) to client 110 via a model training response.

[0077] The training scheme for the text extraction model provided in this application will be described in detail below through specific embodiments. It should be noted that the technical solutions provided in the embodiments of this application may include some or all of the following contents. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0078] Figure 2 This is a flowchart illustrating a training method for a text extraction model provided in an embodiment of this application. This text extraction model training method can be applied to... Figure 1 The server, or other execution device shown, such as Figure 2 As shown, the method includes the following steps:

[0079] Step 201: The server receives a model training request from the client.

[0080] The model training request is used to request an update of the first text extraction model, which is a pre-trained base model capable of extracting text corresponding to preset labels.

[0081] Step 202: Based on the model training request, the server creates a second text extraction model on top of the first text extraction model. This second text extraction model is capable of extracting the text corresponding to the newly added tags.

[0082] As an example, the model training request includes an identifier for a first text extraction model. Based on this identifier, the server retrieves the first text extraction model from its model library. Then, it adds a new decoding unit to the first text extraction model, resulting in a second text extraction model. This new decoding unit is used to extract the text corresponding to the newly added tags. It should be noted that the server's model library stores text extraction models of different types (or different domains), such as models for extracting contract elements.

[0083] To facilitate understanding, the following will be combined with... Figure 3 The model structures of the first and second text extraction models are described in detail. For example, Figure 3 The diagram shows the structure of two text extraction models provided in the embodiments of this application.

[0084] like Figure 3 As shown in (a), the first text extraction model includes an encoder and a decoder. The encoder is used to obtain the contextual features of the text input to the first text extraction model, and the decoder is used to predict the labels of each field in the text based on the contextual features output by the encoder. The labels of the first text extraction model include preset labels, which can be regarded as the pre-set labels in the basic model provided by the NLP self-learning platform, for example... Figure 3 Label 1, label 2, ..., label n are shown in (a). As an example, the decoder of the first text extraction model includes multiple decoding units, such as... Figure 3 The decoding units 1, 2, ..., n shown in (a) are used to predict their corresponding tags, such as decoding unit 1 corresponding to tag 1, decoding unit 2 corresponding to tag 2, etc.

[0085] like Figure 3 As shown in (b), the second text extraction model includes an encoder and a decoder. The encoder is used to obtain the contextual features of the text input to the second text extraction model, and the decoder is used to predict the labels of each field in the text based on the contextual features output by the encoder. The labels of the second text extraction model include preset labels and newly added labels, where the newly added labels can be user-defined labels, for example... Figure 3 Labels n+1 and n+2 are shown in (b). As an example, unlike the first text extraction model, the decoder of the second text extraction model also includes two additional decoding units, namely decoding unit n+1 and decoding unit n+2. Decoding unit n+1 corresponds to the newly added label n+1, and decoding unit n+2 corresponds to the newly added label n+2.

[0086] based on Figure 3 As shown in the text extraction model, the second text extraction model includes an encoder and a decoder. The encoder is used to obtain the contextual features of the input text, and the decoder is used to predict the labels of each field in the input text. The labels include preset labels and newly added labels. The decoder includes a first decoding unit and a second decoding unit. The first decoding unit corresponds to the preset labels and can be... Figure 3 The decoding units 1, 2, ..., n shown in (b) can have preset tags that are: Figure 3 (b) shows labels 1 to n. The second decoding unit is a newly added decoding unit, corresponding to the newly added label. The second decoding unit can be... Figure 3 The newly added tag can be one of the decoding units n+1 and n+2 shown in (b). Figure 3 Labels n+1 and n+2 are shown in (b).

[0087] As an example, the encoder in the second text extraction model includes any of the following model architectures: BERT, LSTM, BiLSTM, RoBERTa, XLNet. The decoder in the second text extraction model includes any of the following model architectures: BiLSTM and / or CRF, softmax. It should be understood that the model architectures of the encoder and decoder in the first text extraction model are the same as those in the second text extraction model.

[0088] Step 203: The server trains the second text extraction model.

[0089] The server trains the second text extraction model based on the new training dataset and the created second text extraction model in the model training request. The training of the second text extraction model stops when the loss function of the second text extraction model converges.

[0090] As an example, stopping the training of the second text extraction model when the loss function of the second text extraction model converges includes: stopping the training of the second text extraction model when the loss function of a new decoding unit in the second text extraction model converges.

[0091] As an example, training of the second text extraction model is stopped when the loss function of the second text extraction model converges, including: stopping training of the second text extraction model when the loss functions of the decoding units in the second text extraction model all converge.

[0092] In this embodiment, the new training dataset includes: multiple training texts and corresponding annotation information for the training texts. The annotation information includes the label data of the training texts, and the label data indicates the label value of each field in the training texts.

[0093] As an example, the training text in the new training dataset is used as the input of the second text extraction model, and the annotation information corresponding to the training text is used as the output of the second text extraction model. The various modules or units in the second text extraction model are trained, and training is stopped when the loss function of the second text extraction model converges.

[0094] It should be noted that before training the second text extraction model on the server, the model parameters of the second text extraction model need to be initialized. To fully utilize the text extraction capabilities of the already trained first text extraction model, the model parameters of the first text extraction model can be transferred to the second text extraction model. The following section will explain this in conjunction with... Figure 4 The initialization process of the model parameters is explained in detail.

[0095] For example, Figure 4 This is a schematic diagram illustrating the initialization process of the model parameters for the second text extraction model provided in this application embodiment. The initialization process for the model parameters of the second text extraction model includes the following steps: the server retrieves the model parameters of the first text extraction model from the model library, assigns the model parameters of the first text extraction model to the model structure corresponding to the second text extraction model, and randomly initializes the model parameters of new decoding units in the second text extraction model. For example... Figure 4As shown, the model parameters of the encoder in the first text extraction model are assigned to the encoder in the second text extraction model, thus synchronizing the encoder's model parameters. Similarly, the model parameters of decoding unit 1 in the first text extraction model are assigned to decoding unit 1 in the second text extraction model, and the same applies to decoding unit 2 in the first text extraction model. The model parameters of decoding unit n in the first text extraction model are also assigned to decoding unit n in the second text extraction model, thus synchronizing the model parameters of the same decoding units in the decoder. Simultaneously, the model parameters of the newly added decoding units n+1 and n+2 in the second text extraction model are randomly initialized.

[0096] In the example above, by using the existing model structure of the first text extraction model on the server, when it is necessary to extract text corresponding to a new entity type (or tag type), the model structure changes, and can be used in the first text extraction model.

[0097] A new decoding unit is configured in the multi-decoder, and the parameters of the new decoding unit are randomly initialized to 5. Simultaneously, the parameters of the original first text extraction model (including the encoder and decoder parameters) are copied to...

[0098] The new second text extraction model achieves the transfer of capabilities from the original model. In this way, when training the second text extraction model using a new training dataset, it directly transfers the capabilities of the original model on the one hand, and also supports the extraction of new entity types (or label types) on the other.

[0099] Step 204: The server sends the model training response to the client.

[0100] 0. Here, the model training response includes the trained second text extraction model. As an example, the model training response...

[0101] It should include the model structure and model parameters of the trained second text extraction model.

[0102] The text extraction model training method illustrated in this application is an incremental learning method for entity extraction that supports the addition of new entities. It addresses the issue of new training datasets containing new entities during incremental model training.

[0103] In this process, the server receives a model training request from the client, retrieves the first text extraction model from the model library 5 based on the request, and then creates a second text extraction model based on this first text extraction model.

[0104] The server trains a second text extraction model using the new training dataset from the model training request. This second model, in addition to possessing the text extraction capabilities of the first model, also has the ability to extract text corresponding to newly added tags. After completing the training task of the second text extraction model, the server returns the second text extraction model to the client.

[0105] This allows users to easily apply the pre-trained second text extraction model in practical applications. The above solution significantly reduces model training time and improves training efficiency. Because it leverages existing model capabilities, parameters from the existing model can be directly transferred.

[0106] This not only reduces resource consumption during server training but also ensures data security.

[0107] Based on the above embodiments, the model training products provided by the server are described below in conjunction with the user interface.

[0108] The trigger condition for the server to receive a model training request sent by the client is:

[0109] The first action taken by the user on the model creation interface provided by the server, i.e., the fifth action taken by the user on the model creation interface, is the server receiving a model training request from the client.

[0110] As an example, the first operation includes the user selecting a first text extraction model on the model creation interface, and the user adding a new training dataset on the model creation interface. Correspondingly, the model training request includes an identifier for the first text extraction model and a new training dataset for updating the first text extraction model.

[0111] For example, Figure 5 This is a user interface diagram provided for an embodiment of this application. Figure 5 The interface 500 shown is the model creation interface provided by the server. Interface 500 includes a text box 501, a selection box 502, a data upload control 503, a submit control, and a reset control. Users can customize the name of the new text extraction model in the text box 501, select various existing models provided by the server in the selection box 502, and add a new training dataset by clicking the data upload control 503. After completing the above operations, users can click the submit control to trigger training of the new model. To modify the settings, users can click the reset control and re-execute the above operations.

[0112] 5. In addition, before users upload the training dataset, the training text in the training dataset needs to be labeled, or an existing labeled training dataset needs to be obtained. If users want to extract more entity types, they can set more entity types (field names or labels) and label the text corresponding to the new entity types in the training dataset to obtain a new training dataset. The new training dataset contains text labeling information for the new entity types.

[0113] For example, Figure 6 This is a user interface diagram provided for an embodiment of this application. Figure 6 Interface 600 shown is the label setting interface for the training dataset. Interface 600 includes general labels that users can select, which correspond to the preset labels of the model mentioned above. Taking the legal document element extraction model as an example, general labels include labels such as time, location, crime, person, card number, means of transportation, vehicle license plate number, frequency, etc. Users can select some or all of the general labels before training the model. In addition, interface 600 also includes a control 601 for custom labels. Users can click on control 601 to input custom labels, which correspond to the newly added labels mentioned above. For example, if a user wants to extract relevant legal citations from legal documents, they can add a new label "Law"; or, if a user needs to extract victim information, they can add a new label "Victim's Name," etc.

[0114] In this example, users can submit their model training label requirements on interface 600, enabling them to independently select and customize model labels, thus improving the user experience during model training.

[0115] In addition to providing users with an interface for creating models and setting labels, the server also provides an evaluation interface for model training, which displays the accuracy of model extraction at the current training progress.

[0116] For example, Figure 7 This is a user interface diagram provided for an embodiment of this application. Figure 7Interface 700 shown is the model evaluation interface. Users can view evaluation parameters for training with all labels, built-in model labels (general or preset labels), or custom labels (newly added labels) on interface 700. Evaluation parameters include accuracy, recall, and F1 score. List 701 in interface 700 displays various evaluation parameters for training with built-in model labels. Taking the "person" label as an example, the accuracy of this label refers to the ratio of the number of samples correctly identified by the model to the total number of samples identified by the model. The recall of this label refers to the ratio of the number of samples correctly identified by the model to the total number of samples identified by the model. The F1 score of this label can be the harmonic mean of the accuracy and recall of this label. It should be noted that the total accuracy, total recall, and total F1 score above list 701 refer to the average accuracy, average recall, and average F1 score of all labels in list 701, respectively.

[0117] After receiving the pre-trained text extraction model from the server, the client can obtain the text extraction results through the text extraction interface provided by the server. For example, Figure 8 This is a user interface diagram provided for an embodiment of this application. Figure 8 Interface 800 shown is a text extraction interface. Users enter the text to be extracted into input box 801. They can directly copy the text into input box 801 or add text by adding a file. Users simply click the submit control to view the text extraction results in output box 802. Output box 802 can display field names (tag names) and corresponding text in a list format. For example, if the text extraction model extracts the input text from input box 801, and the tags of the text extraction model include "person," "crime," and "law," then the corresponding text extraction results displayed in output box 802 will include: "Person: Zhang San," "Criminal: Traffic Accident," and "Law: xx Law." In some embodiments, output box 802 can also display the text extraction results in the form of text annotations, which are not shown in the figure.

[0118] Based on the above embodiments, the following example illustrates the model structure and processing of an existing text extraction model (first text extraction model) and a new text extraction model (second text extraction model).

[0119] For example, Figure 9 This is a schematic diagram illustrating data processing for a text extraction model provided in an embodiment of this application.

[0120] Figure 9The text extraction model shown in (a) can be considered as an existing text extraction model on the server. This text extraction model includes a decoder and an encoder. The encoder adopts the BERT model structure, and the decoder includes decoding unit 1 and decoding unit 2. Decoding unit 1 corresponds to the preset label "person" and is used to extract the text corresponding to the label "person". Decoding unit 2 corresponds to the preset label "event" and is used to extract the text corresponding to the label "event". For example, the input text "Zhang San had two car accidents" is input into the encoder of the text extraction model. After processing by the encoder, the context feature vector corresponding to the input text is obtained. Then, the context feature vector is input into decoding unit 1 and decoding unit 2 of the decoder respectively. The output sequence of decoding unit 1 is "BIOOOOOO", which indicates that the first two fields of the input text are person names. In this sequence, B represents the beginning of the label "person" and I represents the middle part of the label "person". Similarly, the output sequence of decoding unit 2 is "OOOOOOBI", which indicates that the seventh and eighth fields of the input text are event names. In this sequence, B represents the beginning of the label "event" and I represents the middle part of the label "event". This example combines the output sequences of decoding unit 1 and decoding unit 2 to output the overall sequence, indicating the label information of the input text.

[0121] It should be noted that, regardless of the type of decoding unit mentioned above, the output sequence typically includes three representations: B, I, and O. B represents the beginning of an entity, I represents the middle part of an entity, and O represents non-entity parts. Additionally, E can be used to represent the end of an entity, and S can represent a single-word entity. Here, the entity can be understood as the tag corresponding to the decoding unit. That is, the output sequence of the decoding unit can represent tag information in the form of BIO, BIOES, or BIOES.

[0122] Figure 9 The text extraction model shown in (b) can be viewed as a new text extraction model, which is based on... Figure 9 Based on the text extraction model shown in (a), a new decoding unit, namely decoding unit 3, is added to the decoder section. Decoding unit 3 corresponds to the newly added label "frequency" and is used to extract the text corresponding to the label "frequency". For example, since the input text is "Zhang San had 2 car accidents", the output sequence of decoding unit 3 is "OOOOBIOO". This sequence indicates that the fifth and sixth fields of the input text are frequency information. In this sequence, B represents the beginning of the label "frequency" and I represents the middle part of the label "frequency". This example summarizes the output sequences of encoding unit 1, encoding unit 2, and encoding unit 3 to output the total sequence, indicating the label information of the input text.

[0123] based on Figure 9 As shown in the example, users can build new text extraction models based on existing models provided by the server by setting new labels, thus giving them broader text extraction capabilities. Training a new text extraction model requires constructing a training dataset containing text with the new labels. (Comparison) Figure 9 In (a) and (b), the new text extraction model adds a new encoding unit, namely encoding unit 3, to the model structure. This unit is used to extract new labeled text from the text, enabling incremental training based on the existing model training and improving the model's text extraction capabilities.

[0124] For example, Figure 10 This is a schematic diagram of the structure of a decoder provided in an embodiment of this application. Figure 9 The decoder of the text extraction model shown can be used Figure 10 The model structure of BiLSTM+CRF is shown below. The input text "Zhang San had two car accidents" is processed by the encoder to obtain the context feature vector of the input text. This context feature vector is then input into the first LSTM layer of the BiLSTM and simultaneously passed through to the second LSTM layer. The data transmission direction of the first LSTM layer is from left to right, while the data transmission direction of the second LSTM layer is from right to left. The data transmission directions of the two LSTM layers are opposite, and their data processing is relatively independent. The outputs of both the first and second LSTM layers are passed to the CRF, which ultimately solves for the labels corresponding to each field in the input text.

[0125] It should be understood that the text extraction model provided in this application embodiment may adopt other model structures, which will not be elaborated here.

[0126] It should be noted that some processes described in the above embodiments and accompanying drawings include multiple operations that appear in a specific order. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or may be executed in parallel. The sequence numbers are merely used to distinguish different operations, and the sequence numbers themselves do not represent any execution order. Furthermore, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel.

[0127] The training method of the text extraction model provided in the embodiments of this application has been described above. The training device of the text extraction model provided in the embodiments of this application will be described below.

[0128] This application embodiment can divide the training device of the text extraction model into functional modules according to the above method embodiment. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or software functional modules. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods. The following description uses the division of each functional module according to its own function as an example.

[0129] Figure 11 This is a schematic diagram of the structure of a training device for a text extraction model provided in an embodiment of this application. Figure 11 As shown, the training device 1100 for the text extraction model in this embodiment includes: a receiving unit 1101, a processing unit 1102, and a sending unit 1103.

[0130] The receiving unit 1101 is used to receive a model training request from a client. The model training request is used to request the update of a first text extraction model. The first text extraction model is a pre-trained base model. The first text extraction model can extract text corresponding to preset labels.

[0131] Processing unit 1102 is used to create a second text extraction model based on the first text extraction model, and train the second text extraction model, which is capable of extracting the text corresponding to the newly added label, based on the model training request.

[0132] The sending unit 1103 is used to send a model training response to the client, the model training response including a trained second text extraction model.

[0133] In an optional embodiment, the receiving unit 1101 is configured to: receive the model training request from the client in response to a first operation by the user on the model creation interface;

[0134] The first operation includes the user selecting the first text extraction model on the model creation interface, and the user adding the new training dataset on the model creation interface.

[0135] The model training request includes the identifier of the first text extraction model and a new training dataset for updating the first text extraction model.

[0136] In an optional embodiment, the processing unit 1102 is configured to:

[0137] Based on the identifier of the first text extraction model in the model training request, the first text extraction model is obtained from the model library;

[0138] Based on the first text extraction model, a new decoding unit is added to obtain the second text extraction model; the new decoding unit is used to extract the text corresponding to the newly added tag.

[0139] In an optional embodiment, the processing unit 1102 is further configured to:

[0140] Obtain the model parameters of the first text extraction model from the model library;

[0141] Assign the model parameters of the first text extraction model to the model structure corresponding to the second text extraction model;

[0142] The model parameters of the new decoding unit are randomly initialized.

[0143] In an optional embodiment, the processing unit 1102 is further configured to:

[0144] Based on the new training dataset in the model training request and the created second text extraction model, train the second text extraction model;

[0145] When the loss function of the second text extraction model converges, training of the second text extraction model is stopped;

[0146] The new training dataset includes multiple training texts and corresponding annotation information for the training texts, wherein the annotation information includes the label data of the training texts.

[0147] In an optional embodiment, the second text extraction model includes an encoder and a decoder. The encoder is used to obtain contextual features of the input text of the model, and the decoder is used to predict the labels of each field in the input text. The labels include preset labels and newly added labels.

[0148] In one optional embodiment, the decoder includes a first decoding unit and a second decoding unit, the first decoding unit corresponding to the preset tag, and the second decoding unit being a newly added decoding unit corresponding to the newly added tag.

[0149] In an optional embodiment, the encoder in the second text extraction model includes any of the following model structures: BERT, LSTM, BiLSTM, RoBERTa, XLNet; the decoder in the second text extraction model includes any of the following model structures: BiLSTM and / or CRF, softmax.

[0150] The text extraction model training device provided in this embodiment can be used to execute the model training method of the above method embodiment. Its implementation principle and technical effect are similar, and will not be described again in this embodiment.

[0151] This application also provides a data processing system that can be used to train and update a text extraction model. (See attached document.) Figure 1 The data processing system in this embodiment may include end-side devices (e.g., Figure 1 Client 110 in the middle) and cloud-side devices (e.g. Figure 1 The server (120) in the cloud stores various pre-trained text extraction models for users to choose from or optimize. Optimization mainly refers to improving the model's capabilities, such as adding new labels to enable the model to extract the text corresponding to those new labels.

[0152] The edge device is used to send a model training request to the cloud device. The model training request is used to request the update of the first text extraction model. The first text extraction model is a pre-trained base model. The first text extraction model can extract text corresponding to preset labels.

[0153] The cloud-based device is used to create a second text extraction model based on the first text extraction model, and train the second text extraction model. The second text extraction model can extract the text corresponding to the newly added label.

[0154] The cloud-side device is also used to send model training responses to the edge-side device, which include the trained second text extraction model.

[0155] In this embodiment, the cloud-side device is used to perform... Figure 2 The method steps of the server in the illustrated embodiment can be correspondingly implemented on the cloud-side device. Figure 11 The training device 1100 for the text extraction model in the embodiment shown can be referred to the above embodiment for its implementation principle and technical effects, which will not be repeated here.

[0156] Figure 12 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 12 As shown, the electronic device includes a memory 1201 and a processor 1202. The memory 1201 stores computer programs and can be configured to store various other data to support operation on a training device for the text extraction model. The processor 1202, coupled to the memory 1201, executes the computer programs in the memory 1201 to implement the technical solutions of the aforementioned method embodiments. Its implementation principle and technical effects are similar and will not be repeated here.

[0157] In some embodiments, such as Figure 12As shown, the electronic device also includes other components such as a firewall 1203, a load balancer 1204, a communication component 1205, and a power supply component 1206. Figure 12 The diagram only shows some components and does not mean that the electronic device includes only these components. Figure 12 The components shown.

[0158] This application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to implement the technical solution described in the foregoing method embodiments.

[0159] This application provides a computer program product, including a computer program / instructions, which, when executed by a processor, cause the processor to implement the technical solution as described in the foregoing method embodiments.

[0160] This application provides a chip, including a processing module and a communication interface. The processing module is capable of executing the technical solutions described in the foregoing method embodiments. In some embodiments, the chip further includes a storage module (e.g., a memory), which stores instructions. The processing module executes the instructions stored in the storage module, and the execution of the instructions stored in the storage module causes the processing module to execute the technical solutions described in the foregoing method embodiments.

[0161] The aforementioned storage can be object storage (OSS).

[0162] The aforementioned memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0163] The aforementioned communication components are configured to facilitate wired or wireless communication between the device containing the communication components and other devices. The device containing the communication components can access wireless networks based on communication standards, such as WiFi, 2G, 3G, 4G / LTE, 5G, and other mobile communication networks, or combinations thereof. In one exemplary embodiment, the communication components receive broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the communication components also include a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra-Wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0164] The aforementioned power supply components provide power to various components within the device in which they reside. These power supply components may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to the device in which they reside.

[0165] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0166] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0167] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0168] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0169] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0170] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0171] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0172] It should be noted that the training data in this application embodiment may involve user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.), all of which are information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0173] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0174] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A training method for a text extraction model, characterized in that, include: Receive a model training request from the client. The model training request is used to request the update of the first text extraction model. The first text extraction model is a pre-trained base model. The first text extraction model can extract text corresponding to preset labels. Based on the identifier of the first text extraction model in the model training request, the first text extraction model is obtained from the model library; Based on the first text extraction model, a new decoding unit is added to obtain the second text extraction model; the new decoding unit is used to extract the text corresponding to the newly added tags; Obtain the model parameters of the first text extraction model from the model library; Assign the model parameters of the first text extraction model to the model structure corresponding to the second text extraction model; Randomly initialize the model parameters of the new decoding unit; Based on the new training dataset in the model training request and the created second text extraction model, the second text extraction model is trained; the second text extraction model is able to extract the text corresponding to the newly added label; A model training response is sent to the client, the model training response including a trained second text extraction model.

2. The method according to claim 1, characterized in that, Receiving a model training request from a client includes: In response to the user's first operation on the model creation interface, the system receives the model training request from the client. The first operation includes the user selecting the first text extraction model on the model creation interface, and the user adding a new training dataset on the model creation interface. The model training request includes the identifier of the first text extraction model and a new training dataset for updating the first text extraction model.

3. The method according to claim 1 or 2, characterized in that, The method further includes: stopping the training of the second text extraction model when the loss function of the second text extraction model converges; The new training dataset includes multiple training texts and corresponding annotation information for the training texts, wherein the annotation information includes the label data of the training texts.

4. The method according to claim 1 or 2, characterized in that, The second text extraction model includes an encoder and a decoder. The encoder is used to obtain the contextual features of the input text of the model, and the decoder is used to predict the labels of each field in the input text. The labels include preset labels and newly added labels.

5. The method according to claim 4, characterized in that, The decoder includes a first decoding unit and a second decoding unit. The first decoding unit corresponds to the preset tag, and the second decoding unit is a newly added decoding unit that corresponds to the newly added tag.

6. The method according to any one of claims 1, 2, and 5, characterized in that, The encoder in the second text extraction model includes any of the following model structures: BERT, LSTM, BiLSTM, RoBERTa, XLNet; The decoder in the second text extraction model includes any of the following model structures: BiLSTM and / or CRF, softmax.

7. A training device for a text extraction model, characterized in that, include: The receiving unit is used to receive a model training request from the client. The model training request is used to request the update of the first text extraction model. The first text extraction model is a pre-trained base model. The first text extraction model can extract text corresponding to preset labels. The processing unit is configured to: obtain the first text extraction model from the model library based on the identifier of the first text extraction model in the model training request; add a new decoding unit to the first text extraction model to obtain a second text extraction model; the new decoding unit is used to extract the text corresponding to the newly added tag; obtain the model parameters of the first text extraction model from the model library; assign the model parameters of the first text extraction model to the model structure corresponding to the second text extraction model; and randomly initialize the model parameters of the new decoding unit. Based on the new training dataset in the model training request and the created second text extraction model, the second text extraction model is trained; the second text extraction model is able to extract the text corresponding to the newly added label; The sending unit is used to return the trained second text extraction model to the client.

8. An electronic device, characterized in that, include: A memory, a processor, and a computer program; the computer program is stored in the memory and configured to be executed by the processor to implement the method as claimed in any one of claims 1 to 6.

9. A data processing system, characterized in that, include: End-side equipment and cloud-side equipment; The edge device is used to send a model training request to the cloud device. The model training request is used to request the update of the first text extraction model. The first text extraction model is a pre-trained base model. The first text extraction model can extract text corresponding to preset labels. The cloud-side device is configured to: obtain the first text extraction model from the model library based on the identifier of the first text extraction model in the model training request; add a new decoding unit to the first text extraction model to obtain a second text extraction model; the new decoding unit is used to extract the text corresponding to the newly added tag; obtain the model parameters of the first text extraction model from the model library; assign the model parameters of the first text extraction model to the model structure corresponding to the second text extraction model; and randomly initialize the model parameters of the new decoding unit. Based on the new training dataset in the model training request and the created second text extraction model, the second text extraction model is trained; the second text extraction model is able to extract the text corresponding to the newly added label; The cloud-side device is also used to send a model training response to the end-side device, the model training response including a trained second text extraction model.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that is executed by a processor to implement the method as described in any one of claims 1 to 6.

11. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 6.

Images