A text structuring method based on pre-training model and location information

Abstract

The present invention discloses a text structuring method based on a pre-trained language model and location information. The present invention comprises the following steps: Step 1: crawling text from a news website as raw material and submitting it to manual annotation, and pre-processing it; Step 2: using the pre-processed text to train BERT-CRF to identify common entities such as name, gender, physical features, height, etc.; Step 3: using the common entities in each text as input, training a multi-label classification model BERT-Dense, and outputting the type of relevant population to which each entity belongs; Step 4: using the name of the relevant population as the subject word, according to the structural characteristics of the text, obtaining the attributes of the subject word based on the location feature, and obtaining the final multi-subject text structuring. The present invention is applicable to domain-specific multi-subject text structuring tasks, and effectively improves the accuracy and efficiency of news text structuring through hierarchical structuring and forming multi-tuples based on location information.

Description

Technical Field

[0001] The present invention belongs to the technical field of natural language processing, and in particular relates to a text structuring method based on a pre-training model and position information. Background Art

[0002] With the rapid development of information technology, a vast amount of news and case information exists in text form. Faced with this ever-increasing amount of text data, business personnel currently face the challenge of spending a significant amount of time simply reading through it. Natural language processing technology has made considerable progress. Therefore, automatically extracting key information points from unstructured text and presenting them in a structured format not only saves business personnel from reading large amounts of text, saving time and effort, but also provides a solid foundation for subsequent data mining tasks, such as linking and merging similar cases.

[0003] In recent years, pre-trained language models have been widely used in natural language processing tasks. Pre-trained models learn the textual representation of words in specific contexts from large-scale unsupervised data in a self-supervised manner, and then apply them to specific tasks through fine-tuning. Pre-trained language models have many advantages: 1) They are an application of transfer learning, implicitly learning general grammatical and semantic knowledge; 2) They can transfer knowledge learned from open domains to downstream tasks to improve low-resource tasks; 3) They have achieved state-of-the-art results in almost all natural language processing tasks; and 4) They are highly scalable; when tackling a new task, they only need to fine-tune the model using the labeled data for that task.

[0004] In real-world applications, such as robbery cases, most news reports contain information about multiple relevant individuals. For example, a single paragraph may contain multiple witness names, genders, and physical characteristics, as well as the robber's name and gender. Current mainstream text structuring methods formalize this as a named entity recognition task, using a pre-trained language model (BERT) as a feature extractor and then directly identifying the various attributes of the relevant individuals through conditional random field (CRF) decoding. However, this approach suffers from two problems: 1) the excessive number of label categories required for named entity recognition leads to poor model performance; 2) it is unable to form multi-tuples such as <witness 1 name, witness 1 gender, witness 1 physical characteristics>, <witness 2 name, witness 2 gender, witness 2 physical characteristics>, <robber 1 name, robber 1 gender>, <robber 2 name, robber 2 gender>, and so on, making it impossible to structure multi-agent text.

[0005] In summary, this interdisciplinary field urgently needs to design a new text structuring method to solve the above problems. Summary of the Invention

[0006] In view of this, the present invention provides a text structuring method based on a pre-trained model and location information. First, to solve the problem of excessive number of named entity recognition label categories caused by directly identifying the attributes of relevant personnel, the present invention adopts a hierarchical structuring method, first using BERT-CRF to identify common entities such as name, gender, physical features, height, etc. in the text, and then constructing a multi-label classification model BERT-Dense for these entities, and associating them with different types of relevant people, such as witnesses, robbers, and robbed people. Second, to solve the problem of being unable to form a multi-tuple form with the names of relevant personnel as the main body, the present invention takes the names of relevant people as the main words, and obtains the attributes of the main words based on the position features according to the structural characteristics of the text.

[0007] The technical route of the present invention is implemented in the following ways: first, the robbery incident text is crawled from the news website as the original corpus and manually annotated, and then preprocessed; second, the preprocessed text is used to train BERT-CRF to identify common entities such as name, gender, physical features, height, etc.; third, these entities are used as input, and the relevant population types such as witnesses, robbers, and victims are used as category labels to train the multi-label classification model BERT-Dense, aiming to associate the common entities with different relevant population types respectively; finally, the names of the relevant population are used as the subject words, and according to the structural characteristics of the text, the attributes of the subject words are obtained based on the position features to obtain the final multi-subject text structuring.

[0008] This invention takes the multi-subject text structuring of a robbery news case as an example, and can also be used for multi-subject text structuring of other events. The specific steps are:

[0009] S1. Crawl event texts from news websites and manually annotate them to obtain a text-structured supervised dataset A, which is then preprocessed as input for subsequent models.

[0010] S2. Take the preprocessed supervised text as input, train the named entity recognition model BERT-CRF, and output the common entities in the text in the form of a BIO tag system.

[0011] S3. Take the common entities in each text as input, train the multi-label classification model BERT-Dense, and output the relevant population type to which each entity belongs.

[0012] S4. Take the names of the relevant people as the subject words, obtain the attributes of the subject words based on the location information according to the structural characteristics of the news text, and realize the final multi-subject text structuring of the robbery news case.

[0013] Furthermore, step S1 includes:

[0014] S11. Crawl robbery incident texts from news websites and manually annotate them as the original corpus for text structuring.

[0015] S12. Based on manually defined annotation specifications, manually annotate the extracted original corpus to form a text dataset A with supervised signals;

[0016] S13. Add special characters "[CLS]" to the beginning of each text in A and "[SEP]" to the end;

[0017] S14. Determine the maximum sequence length and fix each text to this length. That is, if the length of the original sequence is greater than this length, it is truncated; if it is less than this length, it is padded with "[PAD]" characters.

[0018] Furthermore, step S2 includes:

[0019] S21. Use the preprocessed supervised text as input to train the BERT-CRF model.

[0020] S22. Use the trained BERT-CRF model to predict the test set, obtaining a three-dimensional matrix S of dimension m*n*l, where m is the number of texts, n is the predefined maximum sequence length, and l is the number of types of manually annotated general entities under the BIO labeling system. A number S[i][j][t] in the three-dimensional matrix S, where 0≤i≤m-1, 0≤j≤, n-1≤t≤l-1, represents the probability that the j-th character in the i-th text belongs to the t-th label, where 0≤p[i][j][t]≤1;

[0021] S23. Convert the three-dimensional matrix S output by the BERT-CRF model to a two-dimensional matrix M, where M[i][j] = argmax(S[i][j]), where 0≤i≤m-1, 0≤j≤n-1. In other words, M[i][j] takes the index value corresponding to the maximum probability for the j-th character in the i-th text;

[0022] S24. Based on the index value corresponding to the maximum probability in the matrix M, convert it into the form of a BIO tag to obtain named entity recognition of the common entity in the text.

[0023] Furthermore, step S3 includes:

[0024] S31. Take the common entities in each text as input, use special characters "[unused1]", "[unused2]", "[unused3]"... to represent the common entities in the text, and train the multi-label classification model BERT-Dense;

[0025] S32. The loss function of the overall model uses cross entropy to represent the difference between the true probability distribution and the predicted probability distribution. The optimization goal is to minimize the cross entropy value. The objective function is:

[0026]

[0027] Among them, n represents the number of samples, p(x i ) represents the true probability distribution of the i-th sample, q(x i ) represents the predicted probability distribution;

[0028] S33. Use the trained multi-label classification model BERT-Dense to predict the test set, and obtain a two-dimensional matrix H of dimension x*y, where x represents the total number of general entities in the supervised text dataset A, and y represents the number of types of relevant populations in A. A number H[i][j] in the two-dimensional matrix H represents the probability that the i-th general entity belongs to the j-th type of relevant population;

[0029] S34. For the probability value in H[i], if the result is greater than 0.5, set the type label of the i-th general entity.

[0030] Furthermore, step S4 includes:

[0031] S41, using the names of the relevant people identified in steps S2 and S3 as subject words;

[0032] S42. For any subject word, filter out entities that belong to the same relevant group of people as the subject word;

[0033] S43. The entities that appear after the subject word and are closest to the subject word are used as attributes of the subject word to form a multi-tuple, thereby achieving the final multi-subject text structuring.

[0034] After adopting the above strategy, the positive effects of the present invention are:

[0035] (1) Aiming at the multi-subject text structuring task, the present invention proposes a text structuring method based on a pre-trained language model and position information. The names of relevant people are taken as the subject words. According to the structural characteristics of the news case text, the attributes of the subject words are obtained based on the position features. This effectively solves the problem that the current mainstream structuring method cannot form a multi-tuple form with the names of relevant people as the subject. The key information points in the unstructured text are displayed in a structured form, thereby improving the efficiency of business personnel in reading news case texts.

[0036] (2) The present invention adopts a hierarchical structured method to first use BERT-CRF to identify common entities such as name, gender, physical features, height, etc. in the text, and then constructs a multi-label classification model BERT-Dense for these entities, and associates them with different relevant groups, such as witnesses, robbers, and robbed persons. This solves the problem that directly identifying the attributes of relevant personnel leads to an excessive number of named entity recognition label categories and a decrease in model performance, thereby accurately identifying different types of relevant groups and improving the recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is a flowchart of a text structuring method based on a pre-trained model and location information provided by the present invention.

[0038] Figure 2 This is a schematic diagram of the structure of the hierarchical structured pre-trained language model in the method of the present invention.

[0039] Figure 3 This is a schematic diagram of an example of multi-subject text structuring in the method of the present invention. DETAILED DESCRIPTION

[0040] In order to enable those skilled in the art to better understand the present invention and to make the above-mentioned objectives, technical solutions and advantages of the present invention more obvious and understandable, the present invention is further described in detail below in conjunction with embodiments and accompanying drawings.

[0041] See Figure 1 , Figure 1 This is a flowchart of a text structuring method based on a pre-trained model and location information provided by an embodiment of the present invention. The method includes: first, crawling text from a news website as raw corpus and handing it over to manual annotation and pre-processing; second, training BERT-CRF to identify common entities such as name, gender, physical features, height, etc.; third, taking relevant groups such as witnesses, robbers, and robbed people as category labels, training a multi-label classification model BERT-Dense, and associating common entities with different types of relevant groups; finally, taking the names of relevant groups as subject words, according to the structural characteristics of the news text, the attributes of the subject words are obtained based on the location features, and the final multi-subject text structuring is obtained. The specific steps include:

[0042] S1. We crawled 3,000 robbery news articles from news websites and manually annotated them to obtain a text-structured supervised dataset A. We then preprocessed the dataset as input for subsequent models.

[0043] S11. Crawl 3,000 robbery news texts from news websites as the original corpus for text structuring;

[0044] S12. Based on manually defined annotation specifications, manually annotate the extracted original corpus to form a text dataset A with supervised signals;

[0045] S13. Add special characters "[CLS]" to the beginning of each text in A and "[SEP]" to the end;

[0046] S14. Determine the maximum sequence length to be 256, and fix each text to this length. That is, if the length of the original sequence is greater than 256, it is truncated; if it is less than 256, it is padded with "[PAD]" characters.

[0047] S2. Take the preprocessed supervised text as input, train the named entity recognition model BERT-CRF, and output the common entities in the text in the form of a BIO tag system.

[0048] The model structure diagram proposed by the present invention is as follows Figure 2 shown.

[0049] S21. Use the preprocessed supervised text as input to train the BERT-CRF model.

[0050] S22. Use the trained BERT-CRF model to predict the test set, obtaining a three-dimensional matrix S with a dimension of m*n*l, where m is the number of texts, i.e., 3000, n is the predefined maximum sequence length, i.e., 256, and l is the number of types of manually annotated general entities in the BIO label system. General entities include four types: name, gender, physical features, and height. When these are converted into BIO labels, l is 9. A number S[i][j][t] in the three-dimensional matrix S, where 0≤i≤2999, 0≤j≤255, and 0≤t≤8, represents the probability that the j-th character in the i-th text belongs to the t-th label, where 0≤p[i][j][t]≤1;

[0051] S23. Convert the three-dimensional matrix S output by the BERT-CRF model to a two-dimensional matrix M, where M[i][j] = argmax(S[i][j]), where 0≤i≤2999 and 0≤j≤255. M[i][j] takes the index value corresponding to the maximum probability for the j-th character in the i-th text.

[0052] S24. Based on the index value corresponding to the maximum probability in the matrix M, it is converted into the form of a BIO tag to obtain named entity recognition of the four common entities in the text.

[0053] S3. Take the common entities in each text as input, train the multi-label classification model BERT-Dense, and output the relevant population type to which each entity belongs.

[0054] S31. Take the common entities in each text as input, use the special characters "[unused1]", ..., "[unused4]" to represent the four common entities in the text, and train the multi-label classification model BERT-Dense.

[0055] S32. The loss function of the overall model uses cross entropy to represent the difference between the true probability distribution and the predicted probability distribution. The optimization goal is to minimize the cross entropy value. The objective function is:

[0056]

[0057] Among them, n represents the number of samples, p(x i ) represents the true probability distribution of the i-th sample, q(x i ) represents the predicted probability distribution;

[0058] S33. Use the trained multi-label classification model BERT-Dense to predict the test set, obtaining a two-dimensional matrix H of dimension x*y, where x represents the total number of general entities in the supervised text dataset A, and y represents the number of types of relevant groups in A. There are three types of relevant groups: witnesses, robbers, and people who report robbery, that is, y is 3. A number H[i][j] in this two-dimensional matrix represents the probability that the i-th general entity belongs to the j-th type of relevant group;

[0059] S34. For the probability value in H[i], if the result is greater than 0.5, set the type label of the i-th general entity.

[0060] S4. Take the names of relevant people as subject words, obtain the attributes of the subject words based on the location information according to the structural characteristics of the news text, and realize the final multi-subject text structuring.

[0061] For example, Figure 3 As shown. Figure 3 For example, "On a certain day in a certain month in 20XX, Zhang (male) said: On a certain day in a certain month in 20XX, Li (female) and Wang (female) were robbed of their mobile phones by Chen (male, wearing a blue top, black pants, about 170cm tall) and Liu (male, wearing black glasses) in Blue Sky Square."

[0062] S41. Use the names of the relevant people identified in steps S2 and S3 as subject words. In the above example, there are five subject words, namely, the name of the witness "Zhang", the name of the robbed person "Li", the name of the robbed person "Wang", the name of the robber "Chen", and the name of the robber "Liu".

[0063] S42. For any subject word, filter out entities that belong to the same relevant group as the subject word. For example, for the subject word "Chen Mou", filter out entities that belong to the same relevant group as the subject word, namely, the robber's gender is "male", the robber's physical characteristics are "wearing a blue shirt and black pants", the robber's height is "about 170cm", and the robber's physical characteristics are "wearing black glasses";

[0064] S43. The entities that appear after the subject word and are closest to the subject word are used as attributes of the subject word to form a multi-tuple, thereby achieving the final multi-agent text structuring. According to the example text, the robber's gender is "male", the robber's physical characteristics are "wearing a blue shirt and black pants", and the robber's height is "about 170cm". They appear after the subject word "Chen" and are closest to "Chen". Therefore, the multi-tuple <"Chen", "male", "wearing a blue shirt and black pants", "about 170cm"> can be formed. The same method can also form <"Zhang", "male">, <"Li", "female">, <"Wang", "female">, and <"Liu", "male", "wearing black glasses"> multi-tuples, thereby achieving the multi-agent structuring of the text.

[0065] Specific embodiments of the present invention have been described above with reference to the accompanying drawings. However, those skilled in the art will appreciate that various modifications and substitutions may be made to the specific embodiments of the present invention without departing from the spirit and scope of the present invention. Such modifications and substitutions are intended to fall within the scope of the claims.

Claims

1. A text structuring method based on a pre-trained model and location information, the specific steps are as follows: S1. Crawl event texts from news websites and manually annotate them to obtain a text-structured supervised dataset A. Preprocess the dataset as input for subsequent models. S2. Take the preprocessed supervised text as input and train the named entity recognition model BERT-CRF to output common entities in the text in the form of a BIO tag system. Specifically, it includes: S21. Use the preprocessed supervised text as input to train the BERT-CRF model. S22. Use the trained BERT-CRF model to predict the test set and obtain a three-dimensional matrix S with a dimension of m*n*l, where m is the number of texts, n is the predefined maximum sequence length, and l is the number of types of manually annotated general entities under the BIO label system. A number S[i][j][t] in the three-dimensional matrix S, where 0≤i≤m-1, 0≤j≤, n-1≤t≤l-1, represents the probability that the j-th character in the i-th text belongs to the t-th label, where 0≤p[i][j][t]≤1; S23. Convert the three-dimensional matrix S output by the BERT-CRF model into a two-dimensional matrix M, where M[i][j] = argmax(S[i][j]), where 0≤i≤m-1, 0≤j≤n-1, and M[i][j] takes the index value corresponding to the maximum probability for the j-th character in the i-th text; S24, based on the index value corresponding to the maximum probability in the matrix M, convert it into the form of a BIO tag to obtain named entity recognition of the common entity in the text; S3. Take the common entities in each text as input and train the multi-label classification model BERT-Dense to output the relevant population type of each entity, including: S31, take the common entities in each text as input, with the special character "[unused1]", "[unused2]", "[unused3]"... are used to represent common entities in the text and train the multi-label classification model BERT-Dense; S32. The loss function of the overall model uses cross entropy to represent the difference between the true probability distribution and the predicted probability distribution. The optimization goal is to minimize the cross entropy value. The objective function is: Among them, n represents the number of samples, p(x i ) represents the true probability distribution of the i-th sample, q(x i ) represents the predicted probability distribution; S33. Use the trained multi-label classification model BERT-Dense to predict the test set, and obtain a two-dimensional matrix H of x*y dimensions, where x represents the total number of general entities in the supervised text dataset A, y represents the number of types of relevant populations in A, and a number H[i][j] in the two-dimensional matrix H represents the probability that the i-th general entity belongs to the j-th type of relevant population; S34. For the probability value in H[i], if the result is greater than 0.5, set the type label of the i-th general entity; S4. Taking the names of the relevant people as subject words, according to the structural characteristics of the news text, the attributes of the subject words are obtained based on the location information to obtain the final multi-subject text structure, which specifically includes: S41, using the names of the relevant people identified in steps S2 and S3 as subject words; S42. For any subject word, filter out entities that belong to the same relevant group of people as the subject word; S43. The entities that appear after the subject word and are closest to the subject word are used as attributes of the subject word to form a multi-tuple, thereby obtaining the final multi-subject text structuring.

2. A text structuring method based on a pre-trained model and position information according to claim 1, characterized in that: The step S1 specifically includes: S11. Crawl event texts from news websites as the original corpus for text structuring; S12. Based on manually defined annotation specifications, manually annotate the extracted original corpus to form a text dataset A with supervised signals; S13. Add special characters "[CLS]" to the beginning of each text in A and "[SEP]" to the end; S14. Determine the maximum sequence length and fix each text to this length. That is, if the length of the original sequence is greater than this length, it is truncated; if it is less than this length, it is padded with "[PAD]" characters.

Images