Text analysis methods, devices, equipment, media, and program products

Abstract

This disclosure provides a text analysis method applicable to the fields of artificial intelligence and finance. The method includes: acquiring a text sequence to be analyzed; inputting the text sequence to be analyzed into a pre-trained text analysis model, the pre-trained text analysis model being trained on a dataset containing remote labels using a named entity recognition model; analyzing the text sequence based on the pre-trained text analysis model to obtain text analysis results, wherein the text analysis results include named entities and their corresponding entity labels, wherein the pre-trained text analysis model is trained based on an optimized loss function, the optimized loss function including a label weight factor, the label weight factor being used to determine the loss function corresponding to the current model. This disclosure also provides a text analysis device, apparatus, storage medium, and program product.

Description

Technical Field

[0001] This disclosure relates to the fields of artificial intelligence or finance, specifically to a text analysis method, apparatus, device, medium, and program product. Background Technology

[0002] In text analysis tasks, Named Entity Recognition (NER) models have gained widespread application due to their ability to automatically label named entities in text. Supervised training of NER models requires large labeled datasets. Therefore, some methods use datasets containing long-range labels to train NER models. However, datasets containing long-range labels are often perturbative. This is because the entity names in the named entity dictionary (or knowledge base) used to label long-range labels may not fully cover the entity names in the dataset, or the entity names in the dictionary (or knowledge base) may correspond to multiple types of labels, leading to potentially incorrect long-range labels in the dataset. Existing methods using such datasets for training models degrade the model's performance on unperturbed datasets, resulting in poor analytical accuracy for existing NER models in text analysis. Summary of the Invention

[0003] In view of the above problems, embodiments of this disclosure provide text analysis methods, apparatus, devices, media, and program products to improve the accuracy of text analysis.

[0004] According to a first aspect of this disclosure, a text analysis method is provided, comprising: acquiring a text sequence to be analyzed; inputting the text sequence to be analyzed into a pre-trained text analysis model, the pre-trained text analysis model being trained on a dataset containing remote labels using a named entity recognition model; analyzing the text sequence to be analyzed based on the pre-trained text analysis model to obtain text analysis results, wherein the text analysis results include named entities and entity labels corresponding to the named entities, wherein the pre-trained text analysis model is trained based on an optimized loss function, the optimized loss function including a label weight factor, the label weight factor being used to determine the loss function corresponding to the current model, the loss function corresponding to the current model including one of a first loss function to an nth loss function, wherein n is an integer greater than or equal to 2.

[0005] According to embodiments of this disclosure, pre-training a text analysis model includes: acquiring a labeled dataset and a remotely labeled dataset, wherein the remotely labeled dataset is obtained by generating remote labels from an unlabeled dataset using a knowledge base; pre-training the named entity recognition model based on the labeled dataset to obtain a model to be optimized; training the model to be optimized based on the remotely labeled dataset and the optimized loss function to obtain a text analysis model, wherein training the model to be optimized based on the remotely labeled dataset includes: training the model to be optimized based on a first loss function and updating model parameters, wherein the first loss function includes a full set of label weight factors; after updating the model parameters in the (i-1)th round, deleting perturbation labels from the remotely labeled data based on a threshold control strategy to obtain the i-th loss function, wherein the threshold control strategy includes a preset threshold, the threshold being used to determine whether a remote label is a perturbation label; stopping training when obtaining the n-th loss function, wherein the n-th loss function includes effective label weight factors after deleting perturbation labels, wherein n is the number of rounds of iterative training when the preset training cutoff condition is met, and i is an integer value in the range [2, n] from smallest to largest.

[0006] According to embodiments of this disclosure, deleting perturbation labels from remotely labeled data based on a threshold control strategy includes: setting the label weight factor in the i-th loss function that matches the perturbation label to zero.

[0007] According to an embodiment of this disclosure, the remote annotation dataset contains a text sequence containing n remotely annotated words, wherein the j-th annotated word corresponds to a pre-annotated p-th class remote label. The step of setting the label weight factor in the i-th loss function that matches the perturbation label to zero includes: determining whether the probability that the j-th annotated word in the text sequence corresponds to the p-th class remote label is greater than a preset threshold, where j satisfies 1≤j≤n, and p is an integer greater than or equal to 1; when the probability that the j-th annotated word in the text sequence corresponds to the p-th class remote label is less than the preset threshold, determining that the p-th class remote label is a perturbation label; and setting the label weight factor corresponding to the p-th class remote label to 0.

[0008] According to an embodiment of this disclosure, when the probability that the j-th labeled word in the text sequence corresponds to the p-th type of remote label is greater than or equal to a preset threshold, the p-th type of remote label is determined to be a valid label; and the label weight factor corresponding to the p-th type of remote label is kept unchanged.

[0009] According to embodiments of this disclosure, the preset training cutoff condition includes a preset number of iterations or a preset model recognition accuracy.

[0010] According to embodiments of this disclosure, the named entity recognition model is the BERT model.

[0011] A second aspect of this disclosure provides a text analysis apparatus, comprising: an acquisition module configured to acquire a text sequence to be analyzed; an input module configured to input the text sequence to be analyzed into a pre-trained text analysis model, the pre-trained text analysis model being trained on a dataset containing remote labels using a named entity recognition model; and an analysis module configured to analyze the text sequence to be analyzed based on the pre-trained text analysis model and obtain text analysis results, wherein the pre-trained text analysis model is trained based on an optimized loss function, the optimized loss function including a label weight factor, the label weight factor being used to determine the loss function corresponding to the current model, the loss function corresponding to the current model including one of a first loss function to an nth loss function, wherein n is an integer greater than or equal to 2.

[0012] A third aspect of this disclosure provides a training apparatus for a text analysis model, including a preparation module, a first training module, and a second training module. The preparation module is configured to acquire a labeled dataset and a remotely labeled dataset, wherein the remotely labeled dataset is obtained by generating remote labels from an unlabeled dataset using a knowledge base. The first training module is configured to pre-train the named entity recognition model based on the labeled dataset to obtain a model to be optimized. The second training module is configured to train the model to be optimized based on the remotely labeled dataset to obtain a text analysis model. The process involves training the model to be optimized based on a first loss function and updating the model parameters. The first loss function includes a weight factor for all labels. After the (i-1)th round of model parameter updates, perturbation labels in the remote labeled data are deleted based on a threshold control strategy to obtain the i-th loss function. The threshold control strategy includes a preset threshold, which is used to determine whether a remote label is a perturbation label. When obtaining the n-th loss function, training is stopped. The n-th loss function includes a weight factor for the effective labels after deleting perturbation labels. Here, n is the number of rounds of iterative training when the preset training cutoff condition is met, and i is an integer value in the range [2, n] from smallest to largest.

[0013] According to embodiments of this disclosure, the second training module includes a judgment submodule, a first confirmation submodule, and an update submodule. The judgment submodule is configured to determine whether the probability that the j-th labeled word in the text sequence corresponds to a p-th type of remote label is greater than a preset threshold, where j satisfies 1 ≤ j ≤ n, and p is an integer greater than or equal to 1. The first confirmation submodule is configured to determine that the p-th type of remote label is a perturbation label when the probability that the j-th labeled word in the text sequence corresponds to a p-th type of remote label is less than the preset threshold; and to determine that the p-th type of remote label is a valid label when the probability that the j-th labeled word in the text sequence corresponds to a p-th type of remote label is greater than or equal to the preset threshold. The update submodule is configured to set the label weight factor corresponding to the p-th type of remote label to 0 when the p-th type of remote label is determined to be a perturbation label.

[0014] A third aspect of this disclosure provides an electronic device comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors perform the text analysis method described above.

[0015] A fourth aspect of this disclosure also provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the text analysis method described above.

[0016] The fifth aspect of this disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described text analysis method.

[0017] The method provided in the embodiments of this disclosure can eliminate the impact of perturbation labels in the dataset on the recognition accuracy of the text analysis model by dynamically deleting perturbed remote labels through optimizing the loss function, thereby improving the model performance. Attached Figure Description

[0018] The foregoing contents, as well as other objects, features, and advantages of this disclosure, will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0019] Figure 1 The illustrations depict application scenarios of text analysis methods, apparatuses, devices, media, and program products according to embodiments of the present disclosure.

[0020] Figure 2 A flowchart illustrating a text analysis method according to an embodiment of the present disclosure is shown schematically.

[0021] Figure 3 A flowchart illustrating a method for pre-training a text analysis model according to an embodiment of the present disclosure is shown schematically.

[0022] Figure 4 A flowchart illustrating a method for training a model to be optimized based on a remotely labeled dataset according to an embodiment of the present disclosure is shown.

[0023] Figure 5 A flowchart illustrating a method for removing perturbation labels from remote annotation data based on a threshold control strategy according to an embodiment of the present disclosure is shown.

[0024] Figure 6 A flowchart illustrating a method for updating the label weight factor in the i-th loss function according to an embodiment of the present disclosure is shown.

[0025] Figure 7 A schematic diagram of the BERT model according to an embodiment of the present disclosure is shown.

[0026] Figure 8 A schematic block diagram of a text analysis apparatus according to an embodiment of the present disclosure is shown.

[0027] Figure 9 A schematic block diagram of a training apparatus for a text analysis model according to an embodiment of the present disclosure is shown.

[0028] Figure 10 A schematic block diagram of the structure of the second training module according to an embodiment of the present disclosure is shown.

[0029] Figure 11 A block diagram schematically illustrates an electronic device suitable for implementing a text analysis method according to an embodiment of the present disclosure. Detailed Implementation

[0030] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0031] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0032] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0033] When using expressions such as "at least one of A, B, and C", they should generally be interpreted in accordance with the meaning that is commonly understood by a person skilled in the art (e.g., "a system having at least one of A, B, and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B, and C, etc.).

[0034] Named Entity Recognition (NER) models can automatically label named entities in text and have wide applications in text analysis tasks. Supervised training of NER models requires a large labeled dataset. A common approach is to use a named entity dictionary or knowledge base to synthesize a dataset with remote labels by combining it with an unlabeled dataset, thus increasing the dataset size. However, remote labels often contain perturbation labels. This is because entity names in the knowledge base may not fully cover entity names in the dataset, or entity names in the knowledge base may correspond to multiple label types, leading to potentially incorrect remote labels in the dataset. Using such a dataset to train the model degrades its performance on unperturbed datasets, reducing model accuracy.

[0035] In view of the above, embodiments of this disclosure provide a text analysis method, comprising: acquiring a text sequence to be analyzed; inputting the text sequence to be analyzed into a pre-trained text analysis model, the pre-trained text analysis model being trained on a dataset containing remote labels using a named entity recognition model; analyzing the text sequence to be analyzed based on the pre-trained text analysis model to obtain text analysis results, wherein the text analysis results include named entities and entity labels corresponding to the named entities, wherein the pre-trained text analysis model includes an optimized loss function, the optimized loss function including label weight factors, and the label weight factors including effective label weight factors after deleting perturbation labels.

[0036] The text analysis method provided in the embodiments of this disclosure can eliminate the impact of perturbation labels in the dataset on the recognition accuracy of the text analysis model by dynamically deleting perturbed remote labels through optimizing the loss function, thereby improving the model performance.

[0037] It should be noted that the text analysis methods, apparatus, devices, media, and program products provided in this disclosure can be used in text analysis related to artificial intelligence technology, and can also be used in various fields other than artificial intelligence technology, such as the financial field. The application fields of the text analysis methods, apparatus, devices, media, and program products provided in this disclosure are not limited.

[0038] The above-described operations for achieving at least one objective of this disclosure will be described below in conjunction with the accompanying drawings and their descriptions.

[0039] Figure 1 The illustrations depict application scenarios of text analysis methods, apparatuses, devices, media, and program products according to embodiments of the present disclosure.

[0040] like Figure 1 As shown, application scenario 100 according to this embodiment may include terminal devices 101, 102, and 103. Network 104 is used as a medium to provide a communication link between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0041] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0042] Terminal devices 101, 102, and 103 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0043] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using terminal devices 101, 102, and 103 (for example only). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0044] It should be noted that the text analysis method provided in this embodiment can generally be executed by server 105. Correspondingly, the text analysis device provided in this embodiment can generally be located in server 105. The text analysis method provided in this embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105. Correspondingly, the text analysis device provided in this embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105.

[0045] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0046] The following will be based on Figure 1 The described scene, through Figures 2-7 The text analysis method of the disclosed embodiments will be described in detail.

[0047] Figure 2 A flowchart illustrating a text analysis method according to an embodiment of the present disclosure is shown schematically.

[0048] like Figure 2 As shown, the text analysis method of this embodiment includes operations S210 to S230. The text analysis method can be executed by a processor or by any electronic device including a processor.

[0049] In operation S210, the text sequence to be analyzed is obtained.

[0050] In operation S220, the text sequence to be analyzed is input into a pre-trained text analysis model, which is based on a dataset containing remote labels trained using a named entity recognition model.

[0051] In operation S230, the text sequence to be analyzed is analyzed based on the pre-trained text analysis model to obtain text analysis results, wherein the text analysis results include named entities and entity annotations corresponding to the named entities.

[0052] According to embodiments of this disclosure, the text analysis model is built upon Named Entity Recognition (NER). Named entity recognition, also known as proper noun recognition, is a fundamental task in natural language processing with a wide range of applications. Named entities generally refer to entities in text that have specific meaning or strong referentiality, typically including names of people, places, organizations, dates and times, proper nouns, etc. It usually includes two parts: entity boundary identification and determining the entity type (name of person, place, organization, or others). The NER system extracts these entities from unstructured input text and can identify more categories of entities according to business needs, such as product names, models, and prices. Therefore, the concept of an entity can be very broad; any specific text fragment required by the business can be called an entity. The named entities involved in NER generally include three main categories (entity, time, and number) and seven subcategories (name of person, place, organization, time, date, currency, and percentage). In practical applications, NER models typically only need to identify names of people, places, organizations, and dates / times. Some systems also provide results for proper nouns (such as abbreviations, conference names, product names, etc.). Numerical entities such as currency and percentages can be handled using regular expressions. Additionally, in some application scenarios, entities specific to a particular domain may be identified, such as book titles, song titles, and journal names. The text analysis model used in the embodiments of this disclosure is applicable to various text sequences containing named entities. The embodiments of this disclosure are particularly suitable for analyzing text in the financial field, and can identify keywords in the text, including named entities such as time, date, currency, percentage, and financial terminology.

[0053] In the embodiments of this disclosure, the pre-trained text analysis model is trained based on an optimized loss function, which includes a label weight factor. This label weight factor is used to determine the loss function corresponding to the current model. The loss function corresponding to the current model includes one of a first loss function to an nth loss function, where n is an integer greater than or equal to 2. In the process of training existing named entity recognition models using supervised methods, a large amount of labeled dataset is required. When the dataset size is insufficient, one solution is to increase the training set by adding remotely labeled data. This remotely labeled data contains remote labels, which are obtained by remotely labeling unlabeled data using a named entity dictionary or knowledge base. However, since the entity names in the named entity dictionary or knowledge base may not completely cover the entity names in the dataset, or the entity names in the named entity dictionary or knowledge base may correspond to multiple types of labels, the remote labels in the dataset may be incorrect. In this case, the remote label becomes a perturbation label. Training the model using the aforementioned dataset will reduce the model's performance on unperturbed datasets and decrease the model's accuracy in recognizing real text. The embodiments of this disclosure dynamically remove perturbation labels by optimizing the loss function, enabling the performance of the named entity recognition model to improve on both the remotely labeled dataset and the unperturbed labeled dataset. The optimized loss function includes label weight factors. It should be understood that at the initial stage of training, each class of remote labels contains initial label weight factors. During iterative training, the label weights in the optimized loss function change to remove perturbation labels whose predicted values ​​do not match the remotely labeled values. After each round of model training, the model can be updated for the next round based on the loss function corresponding to the current model. Therefore, the nth loss function obtained after the last round of training includes effective label weight factors after removing perturbation labels, but excludes perturbation label weight factors. This ensures that at the end of training, the parameters of the text analysis model are minimally affected by perturbation labels, thereby improving model accuracy and optimizing model performance.

[0054] Figure 3 A flowchart illustrating a method for pre-training a text analysis model according to an embodiment of the present disclosure is shown schematically.

[0055] like Figure 3 As shown, the method for obtaining a text analysis model through pre-training in this embodiment includes operations S310 to S330.

[0056] In operation S310, a labeled dataset and a remotely labeled dataset are obtained, wherein the remotely labeled dataset is obtained by using a knowledge base to annotate the unlabeled dataset to generate remote labels.

[0057] In a typical example, CoNLL03 is chosen as the labeled dataset. Wikipedia is used as the knowledge base, and SPARAL is used to query Wikipedia to obtain the category labels for entity names. Then, the unlabeled dataset is labeled to obtain remote labels. During training, the dataset is split into training and testing sets. The ratio of data in the training and testing sets can be determined empirically.

[0058] In operation S320, the named entity recognition model is pre-trained based on the labeled dataset to obtain the model to be optimized.

[0059] In operation S330, the model to be optimized is trained based on the remote labeled dataset and the optimized loss function to obtain a text analysis model.

[0060] The text analysis model training method in the embodiments of this disclosure includes two stages: the first stage is to train an initial named entity recognition model using a labeled dataset. After training, a more accurate model can be obtained based on the existing labeled dataset.

[0061] The aforementioned model has room for further optimization due to limitations in data volume. In the second stage, further training of the model using a remotely labeled dataset can improve the model's accuracy while increasing the amount of training data.

[0062] Figure 4 A flowchart illustrating a method for training a model to be optimized based on a remotely labeled dataset according to an embodiment of the present disclosure is shown.

[0063] like Figure 4 As shown, the method for training the model to be optimized based on the remote labeled dataset in this embodiment includes operations S410 to S430.

[0064] In operation S410, the model to be optimized is trained based on the first loss function, and the model parameters are updated, wherein the first loss function includes the full label weight factor.

[0065] In operation S420, after the model parameters are updated in the (i-1)th round, perturbation labels in the remote labeled data are removed based on the threshold control strategy to obtain the i-th loss function.

[0066] In operation S430, when obtaining the nth loss function, training is stopped and the nth loss function is used as the optimized loss function, where n is the number of iterations of training when the preset training cutoff condition is met, and i is taken from integer values ​​in [2, n] in ascending order.

[0067] In the embodiments of this disclosure, at the initial moment of the second stage of training, an initial loss function, namely the first loss function, can be set.

[0068] A typical first loss function is shown in equation (1):

[0069]

[0070] Among them, L GCE Here, f is the loss function, θ is the model parameter, and f is the model parameter. j，yj (x; θ) represents the model's predicted probability that the j-th labeled word xj in sequence x belongs to category p. q is a hyperparameter with a value range of 0 < q < 1; for example, q = 0.7. j is the tag weight factor, which can be an integer. From equation (1), we can see that the tag weight factor for the j-th labeled word is w. j Initially, no adjustment of label weight factors is involved, therefore the first loss function includes all label weight factors. It can be preset that the label weight factor corresponding to each labeled word is the same; in a preferred embodiment, w can be taken as... j The value is 1.

[0071] In embodiments of this disclosure, when further training the model to be optimized using a remotely labeled dataset, a threshold control strategy can be used to optimize the loss function. It should be understood that for datasets containing remote labels, it is difficult to automatically detect these labels if it is impossible to know which entity names are incorrectly labeled. Embodiments of this disclosure optimize the loss function using a threshold control strategy to achieve automatic removal of perturbation labels. The model can undergo n-1 rounds of iterative training before a preset training cutoff condition is met. After each round of iterative training, the model updates its parameters. After the parameter update, perturbation labels in the remotely labeled data are removed based on the threshold control strategy, and the i-th loss function is obtained. In embodiments of this disclosure, the threshold control strategy includes a step of setting a preset threshold, which is used to determine whether a remote label is a perturbation label. Perturbation labels include remote labels whose predicted labels are inconsistent with the remote labels. When a remote label is determined to be a perturbation label, the perturbation label is deleted. Repeat the above training process until the preset training cutoff condition is reached. At this point, after n-1 rounds of training, the nth loss function is obtained. The nth loss function does not contain perturbation label weight factors, and the obtained text analysis model is not affected by perturbation labels, further improving the accuracy of model analysis.

[0072] Figure 5 A flowchart illustrating a method for removing perturbation labels from remote annotation data based on a threshold control strategy according to an embodiment of the present disclosure is shown.

[0073] like Figure 5As shown, the method for deleting perturbation labels in remote annotation data based on a threshold control strategy in this embodiment includes operation S510.

[0074] In operation S510, the label weight factors matching the perturbation label in the i-th loss function are set to zero. As mentioned earlier, in the loss function, each labeled term contains a label and has a corresponding label weight factor matching the label. At the beginning of step S330, the label weight factors corresponding to each labeled term can be the same and non-zero integers. When it is determined that a certain remote label in the i-th loss function is a perturbation label, the label weight factors matching the perturbation label can be set to zero, thereby realizing the automatic deletion of the perturbation label.

[0075] As mentioned earlier, the remote annotation dataset contains text sequences, each containing n remotely annotated words, where the j-th annotated word corresponds to a pre-annotated remote label of class p. This can be based on... Figure 6 The method updates the label weight factor in the i-th loss function.

[0076] Figure 6 A flowchart illustrating a method for updating the label weight factor in the i-th loss function according to an embodiment of the present disclosure is shown.

[0077] like Figure 6 As shown, the method for updating the label weight factor in the i-th loss function in this embodiment includes operations S610 to S630.

[0078] In operation S610, it is determined whether the probability that the j-th labeled word in the text sequence corresponds to the p-th type of remote label is greater than or equal to a preset threshold, where j satisfies 1≤j≤n, and p is an integer greater than or equal to 1.

[0079] In operation S620, when the probability that the j-th labeled word in the text sequence corresponds to the p-th type of remote label is less than a preset threshold, the p-th type of remote label is determined to be a perturbation label.

[0080] In operation S630, the tag weight factor corresponding to the p-th type remote tag is set to 0.

[0081] According to embodiments of this disclosure, a preset threshold can be set to τ, and the probability that the j-th labeled word corresponds to the p-th type of remote tag can be f. j，yj (x; θ), when f j，yj When (x; θ) < τ, the p-th class remote label corresponding to the j-th labeled word is determined to be a perturbation label. At this time, the label weight factor corresponding to the p-th class remote label is updated to 0, so that the perturbation label will not be calculated in the loss function L. GCEIn this case, the effect is equivalent to the remote label being deleted. The value of τ can be set empirically, for example, setting τ = 0.7. The resulting text analysis model is minimally affected by the perturbed label and has high model accuracy.

[0082] Accordingly, when the probability that the j-th labeled word in the text sequence corresponds to the p-th type of remote label is greater than or equal to a preset threshold, operations S640 to S650 are executed.

[0083] In operation S640, it is determined that the p-th type of remote tag is a valid tag.

[0084] In operation S650, the label weight factor corresponding to the p-th class of remote labels remains unchanged. For example, the label weight factor corresponding to the p-th class of remote labels is still 1. This yields an effective label weight factor included in the optimized loss function.

[0085] According to embodiments of this disclosure, the preset training cutoff condition includes a preset number of training iterations or a preset model recognition accuracy. For example, training can be preset to stop after 200 training iterations, or it can be preset to stop when the model recognition accuracy reaches 0.8.

[0086] In the embodiments of this disclosure, the named entity recognition model is the BERT model. The BERT model is a pre-trained language representation model widely used in natural language processing tasks. Its goal is to train on large-scale unlabeled corpora to obtain expressions of text containing rich semantic information, i.e., the semantic representation of the text, and then fine-tune this semantic representation for a specific natural language processing task before finally applying it to that task. The BERT model is pre-trained using a masked language model (MLM) and employs deep bidirectional Transformer components (where each token attends to all tokens) to construct the entire model, thus ultimately generating a deep bidirectional language representation that can integrate left and right contextual information.

[0087] Figure 7 A schematic diagram illustrating the structure of a BERT model according to an embodiment of this disclosure is shown. Figure 7As shown, the input to BERT is the representation corresponding to each word segment, where Tok1, Tok2...TokN are tokens, and E1, E2...EN are the representations corresponding to the tokens. The word dictionary is constructed using the WordPiece algorithm. To complete the specific classification task, in addition to the word tokens, a specific classification token ([CLS]) is inserted at the beginning of each input sequence. The output of the last Transformer layer corresponding to this classification token is used to aggregate the representation information of the entire sequence. C is the output of the last Transformer corresponding to the classification token ([CLS]), T1, T2...T... N This represents the output of the last Transformer corresponding to other tokens. In the named entity recognition model of this disclosure, given a sequence x = [x1, ..., xn] containing n words... n The sequence of category labels corresponding to it is y = [y1, ..., y2]. n ].

[0088] In one example, the text analysis model of this disclosure embodiment is tested using text from the financial field. The text analysis model can be used to label the categories of named entities in the financial field text and output the labeled text. Examples of some named entity recognition are shown in Tables 1 and 2:

[0089] Table 1. Text annotated using text analysis models

[0090]

[0091] Table 2 contains text with accurate labels.

[0092]

[0093] TIME, ORG, NUM, and FINANCE are the category labels for named entities.

[0094] As can be seen from Tables 1 and 2, the text analysis model using the embodiments of this disclosure can identify most named entities, and the labeled entity tags have a high accuracy rate.

[0095] Based on the above text analysis method, this disclosure also provides a text analysis device. The following will be combined with... Figure 8 The device is described in detail.

[0096] Figure 8 A schematic block diagram of a text analysis apparatus according to an embodiment of the present disclosure is shown.

[0097] like Figure 8 As shown, the text analysis device 800 of this embodiment includes an acquisition module 810, an input module 820, and an analysis module 830.

[0098] The acquisition module 810 is configured to acquire the text sequence to be analyzed.

[0099] The input module 820 is configured to input the text sequence to be analyzed into a pre-trained text analysis model. The pre-trained text analysis model is trained on a dataset containing long-range labels using a named entity recognition model. The pre-trained text analysis model is trained based on an optimized loss function, which includes label weight factors. These label weight factors determine the loss function corresponding to the current model. The loss function corresponding to the current model includes one of a first loss function to an nth loss function, where n is an integer greater than or equal to 2.

[0100] The analysis module 830 is configured to analyze the text sequence to be analyzed based on the pre-trained text analysis model and obtain text analysis results, wherein the text analysis results include named entities and entity annotations corresponding to the named entities.

[0101] Embodiments of this disclosure also provide a training apparatus for a text analysis model.

[0102] Figure 9 A schematic block diagram of a training apparatus for a text analysis model according to an embodiment of the present disclosure is shown.

[0103] like Figure 9 As shown, the training device 900 for the text analysis model in this embodiment includes a preparation module 910, a first training module 920, and a second training module 930.

[0104] The preparation module 910 is configured to acquire a labeled dataset and a remotely labeled dataset, wherein the remotely labeled dataset is obtained by using a knowledge base to annotate the unlabeled dataset to generate remote labels.

[0105] The first training module 920 is configured to pre-train the named entity recognition model based on the labeled dataset to obtain a model to be optimized. The model to be optimized includes a first loss function, which includes full label weight factors.

[0106] The second training module 930 is configured to train the model to be optimized based on the remote labeled dataset to obtain a text analysis model. Specifically, the model to be optimized is trained based on a first loss function, updating model parameters. The first loss function includes a weight factor for all labels. After the (i-1)th round of model parameter updates, perturbation labels in the remote labeled data are deleted based on a threshold control strategy to obtain the i-th loss function. The threshold control strategy includes a preset threshold used to determine whether a remote label is a perturbation label. When obtaining the n-th loss function, training stops. The n-th loss function includes the effective label weight factor after deleting perturbation labels. Here, n is the number of iterations required to meet the preset training cutoff condition, and i is an integer value within the range [2, n] from smallest to largest.

[0107] Figure 10 A schematic block diagram of the structure of the second training module according to an embodiment of the present disclosure is shown.

[0108] like Figure 10 As shown, the second training module 930 of this embodiment includes a judgment submodule 9301, a first confirmation submodule 9302, and an update submodule 9303.

[0109] The judgment submodule 9301 is configured to determine whether the probability that the j-th labeled word in the text sequence corresponds to the p-th type of remote label is greater than a preset threshold, where j satisfies 1≤j≤n and p is an integer greater than or equal to 1.

[0110] The first confirmation submodule 9302 is configured to determine that the p-th type of remote label is a perturbation label when the probability that the j-th labeled word in the text sequence corresponds to the p-th type of remote label is less than a preset threshold; and to determine that the p-th type of remote label is a valid label when the probability that the j-th labeled word in the text sequence corresponds to the p-th type of remote label is greater than or equal to the preset threshold.

[0111] The update submodule 9303 is configured to set the label weight factor corresponding to the p-th type remote label to 0 when it is determined that the p-th type remote label is a perturbation label.

[0112] According to embodiments of this disclosure, any and multiple modules among the acquisition module 810, input module 820, analysis module 830, preparation module 910, first training module 920, second training module 930, judgment submodule 9301, first confirmation submodule 9302, and update submodule 9303 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least some of the functions of one or more of these modules can be combined with at least some of the functions of other modules and implemented in one module. According to embodiments of this disclosure, at least one of the acquisition module 810, input module 820, analysis module 830, preparation module 910, first training module 920, second training module 930, judgment submodule 9301, first confirmation submodule 9302, and update submodule 9303 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), programmable logic array (PLA), system-on-a-chip, system-on-a-substrate, system-on-package, application-specific integrated circuit (ASIC), or any other reasonable method of integrating or packaging the circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three methods. Alternatively, at least one of the acquisition module 810, input module 820, analysis module 830, preparation module 910, first training module 920, second training module 930, judgment submodule 9301, first confirmation submodule 9302, and update submodule 9303 can be at least partially implemented as a computer program module, which, when run, can perform corresponding functions.

[0113] Figure 11 A block diagram schematically illustrates an electronic device suitable for implementing a text analysis method according to an embodiment of the present disclosure.

[0114] like Figure 11 As shown, an electronic device 1000 according to an embodiment of the present disclosure includes a processor 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage portion 1008 into a random access memory (RAM) 1003. The processor 1001 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 1001 may also include onboard memory for caching purposes. The processor 1001 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0115] RAM 1003 stores various programs and data required for the operation of electronic device 1000. Processor 1001, ROM 1002, and RAM 1003 are interconnected via bus 1004. Processor 1001 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 1002 and / or RAM 1003. It should be noted that the programs may also be stored in one or more memories other than ROM 1002 and RAM 1003. Processor 1001 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.

[0116] According to embodiments of this disclosure, the electronic device 1000 may further include an input / output (I / O) interface 1005, which is also connected to a bus 1004. The electronic device 1000 may also include one or more of the following components connected to the I / O interface 1005: an input section 1006 including a keyboard, mouse, etc.; an output section 1007 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN card, modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. A removable medium 1011, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 1010 as needed so that computer programs read from it can be installed into the storage section 1008 as needed.

[0117] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0118] According to embodiments of this disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this disclosure, the computer-readable storage medium may include ROM 1002 and / or RAM 1003 and / or one or more memories other than ROM 1002 and RAM 1003 described above.

[0119] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to cause the computer system to implement the methods provided in the embodiments of this disclosure.

[0120] When the computer program is executed by the processor 1001, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0121] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 1009, and / or installed from a removable medium 1011. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0122] In such an embodiment, the computer program can be downloaded and installed from a network via communication section 1009, and / or installed from removable medium 1011. When the computer program is executed by processor 1001, it performs the functions defined in the system of this disclosure embodiment. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0123] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0124] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0125] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0126] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A text analysis method, characterized in that, include: Obtain the text sequence to be analyzed; The text sequence to be analyzed is input into a pre-trained text analysis model; The text analysis model obtained through pre-training is used to analyze the text sequence to be analyzed, and the text analysis results are obtained. The text analysis results include named entities and the entity annotations corresponding to the named entities. The pre-trained text analysis model includes: Obtain labeled datasets and remotely labeled datasets, wherein the remotely labeled datasets are obtained by using a knowledge base to annotate unlabeled datasets to generate remote labels; The named entity recognition model is pre-trained based on the labeled dataset to obtain the model to be optimized. The model to be optimized is trained based on the remote labeled dataset and the optimized loss function to obtain a text analysis model. The optimized loss function includes a label weight factor. The training of the model to be optimized based on the remote labeled dataset and the optimized loss function includes: In each round of training, perturbation labels in remotely labeled data are identified based on a threshold control strategy, the label weight factors are adjusted, the perturbation labels are deleted, and an optimized loss function is obtained. The threshold control strategy includes a preset threshold.

2. The text analysis method according to claim 1, wherein, In each round of training, based on a threshold control strategy, perturbation labels in the remotely labeled data are identified, label weight factors are adjusted, and the perturbation labels are deleted to obtain an optimized loss function, including: The model to be optimized is trained based on the first loss function, and the model parameters are updated. The first loss function includes the full label weight factor. After the (i-1)th round of model parameter updates, perturbation labels in remotely labeled data are identified based on a threshold control strategy, the label weight factors are adjusted, the perturbation labels are deleted, and the i-th loss function is obtained. Training stops when the nth loss function is obtained. The nth loss function includes the effective label weight factor after removing the perturbation label. Here, n is the number of iterations of training when the preset training cutoff condition is met, and i is an integer value in [2, n] that is traversed from smallest to largest.

3. The text analysis method according to claim 2, wherein, The adjustment of the label weight factor and the deletion of the perturbation label include: setting the label weight factor in the i-th loss function that matches the perturbation label to zero.

4. The text analysis method according to claim 3, wherein, The remote annotation dataset contains text sequences, each containing n remotely annotated words, where the j-th annotated word corresponds to a pre-annotated p-th class remote label. The step of resetting the label weight factor in the i-th loss function that matches the perturbation label to zero includes: Determine whether the probability that the j-th labeled word in the text sequence corresponds to the p-th type of remote label is greater than or equal to a preset threshold, where j satisfies 1≤j≤n, and p is an integer greater than or equal to 1; When the probability that the j-th labeled word in the text sequence corresponds to the p-th type of remote label is less than a preset threshold, the p-th type of remote label is determined to be a perturbation label; and Set the label weight factor corresponding to the p-th class remote label to 0.

5. The text analysis method according to claim 4, wherein when the probability that the j-th labeled word in the text sequence corresponds to the p-th type of remote tag is greater than or equal to a preset threshold, the p-th type of remote tag is determined to be a valid tag; and Keep the label weight factor corresponding to the p-th type of remote label unchanged.

6. The text analysis method according to claim 2, wherein, The preset training cutoff conditions include the preset number of training iterations or the preset model recognition accuracy.

7. The text analysis method according to claim 1, wherein, The named entity recognition model is the BERT model.

8. A text analysis device, characterized in that, include: The acquisition module is configured to acquire the text sequence to be analyzed. The input module is configured to input the text sequence to be analyzed into a pre-trained text analysis model; The analysis module is configured to analyze the text sequence to be analyzed based on the pre-trained text analysis model, and obtain text analysis results, wherein the text analysis results include named entities and entity annotations corresponding to the named entities. The pre-trained text analysis model includes: Obtain labeled datasets and remotely labeled datasets, wherein the remotely labeled datasets are obtained by using a knowledge base to annotate unlabeled datasets to generate remote labels; The named entity recognition model is pre-trained based on the labeled dataset to obtain the model to be optimized. The model to be optimized is trained based on the remote labeled dataset and the optimized loss function to obtain a text analysis model. The optimized loss function includes a label weight factor. The training of the model to be optimized based on the remote labeled dataset and the optimized loss function includes: In each round of training, perturbation labels in remotely labeled data are identified based on a threshold control strategy, the label weight factors are adjusted, the perturbation labels are deleted, and an optimized loss function is obtained. The threshold control strategy includes a preset threshold.

9. An electronic device, comprising: One or more processors; Storage device for storing one or more programs. Wherein, when the one or more programs are executed by the one or more processors, the one or more processors perform the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 7.

11. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images