A text encoding method

Abstract

The application discloses a text coding method, which comprises the following steps: data cleaning and preprocessing of a corpus, sentence division and word division of the corpus, extraction of relevant features, coding representation of the sentence, addition of three dimensions on the basis of coding, and construction of an upper model; compared with the prior art, the difference of the application lies in that the previous additional features are mostly intra-sentence features, such as part of speech, syntax dependency, relative position, absolute position and the like, while the application adds global statistical information, the overall operation is simple, the training cost is low, the prior knowledge injection based on global statistical information is carried out on the sentence level coding, and the accuracy of the upstream task is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of information technology, in particular to a text encoding method. BACKGROUND

[0002] In the existing natural language processing technology, when performing text representation, sequence labeling, named entity recognition and other tasks, sentence level input is often used, which is often limited by the exponential growth of related calculations accompanying the growth of sentence length, and sentence level input will cause the loss of background information, and even the most simple statistical learning based features. In the conventional field task, such as predicting the company information in "a group of business income still creates a new high", most models can easily solve it, and there is no ambiguity, "group" can clearly indicate the company in front, but in the professional field, such as "synthetic tear fluid contains 150 mM PBS (pH 7.2), 0.05 mM beta-D-glucose, 3 mM L-lactate, 0.18 mM L-ascorbate, 5.4 mM urea, 2.47 mg mL-1 lysozyme, 0.2 mg mL-1 BSA and 0.15 mg mL-1 mucin", the entity recognition model is difficult to establish the entity boundary, such as "L-lactate" is easily predicted as "lactate" or "mM L-lactate" or "3mM L-lactate", but combined with simple context statistical information, "L-lactate" has a higher condensation degree, which can be easily identified in statistical learning. This kind of problem is more common in the field of chemistry in the patent field, and the combination of numbers and letters before and after is difficult to confirm whether it is a unit, an appendage change or a proper noun. SUMMARY

[0003] The purpose of the present application is to solve the problems of the prior art and provide a text encoding method.

[0004] In order to achieve the above purpose, a text encoding method is designed, and the method is as follows:

[0005] S1. Data cleaning and preprocessing of corpus set;

[0006] S2. Sentence segmentation and word segmentation of corpus set;

[0007] S3. Extract relevant features;

[0008] S31. Extract N-gram features as candidate word groups;

[0009] S32. Traverse the candidate word groups and process each word group as follows:

[0010] According to the part of speech and the syntax template, judge the candidate word,

[0011] According to the formula Calculate the new word score

[0012] L cand is the length of the candidate word; C cand is the word frequency statistics of the candidate word; subcand is the sub-candidate word in the candidate word; is the word frequency cumulative statistics of the sub-word by traversing all sub-candidate words;

[0013] S33. According to the score, filtering according to a threshold value, obtaining a new candidate word group;

[0014] S34. Traversing the new candidate word group, processing each word group as follows:

[0015] Statistics of the total number of times of the previous word b appearing in the text, C b ,

[0016] Statistics of the total number of times of the previous word b appearing in the text, C a ,

[0017] Statistics of all candidate words of b as the previous word,

[0018] Statistics of all candidate words of a as the previous word,

[0019] According to the following formula, calculate the new word score

[0020]

[0021] L cands is the number of candidate words; L basb is the number of candidate words with b as the previous word; is the cumulative sum of the number of all candidate words with b as the previous word; L aasa is the number of candidate words with a as the previous word, is the cumulative sum of the number of all candidate words with a as the previous word;

[0022] S35. According to the formula is the candidate word with the previous joint probability feature,

[0023] is the candidate word with b as the previous word, and the frequency of the candidate word with the candidate word

[0024] S36. According to the formula is the candidate word with the previous joint probability feature,

[0025] is the candidate word with a as the previous word, and the frequency of the candidate word with the candidate word;

[0026] According to the score, filtering according to a threshold value, obtaining a new candidate word group;

[0027] S37. Mark the candidate word set obtained after filtering with an identifier;

[0028] S38. Append the features obtained in steps S35, S36 and S37 to the original data.

[0029] The present application also includes the following preferred technical solutions:

[0030] Further, the word list is obtained by adopting the minimum granularity word segmentation mode in step S2.

[0031] Further, the method further comprises:

[0032] S4. Encode the sentence;

[0033] S5. Append the three dimensions obtained in step S3 on the basis of the original encoding;

[0034] S6. Construct an upper model.

[0035] Compared with the prior art, the present application has the advantages that the features added in the past are mainly intra-sentence features such as word type, syntactic dependency, relative position, absolute position, etc., while the present application adds global statistical information, the overall operation is simple, the training cost is low, the sentence-level encoding is injected with prior knowledge based on global statistical information, and the accuracy of the upstream task is improved. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 The figure is an encoding mode taking the transformer structure as an example. DETAILED DESCRIPTION

[0037] The technical solution of the present application is summarized as follows: before the target corpus set is segmented, new word discovery is performed, and whether it is a new word identifier is added to the original text encoding, and the previous coagulation score and the subsequent coagulation score are also added. The specific steps are as follows:

[0038] ① Data cleaning, preprocessing, etc. are performed on the corpus set;

[0039] ② The corpus set is segmented and word segmented, and the minimum granularity word segmentation mode is adopted to obtain a word list; ③ relevant features are extracted according to the following steps, as follows:

[0040]

[0041] ④ The sentence is encoded and represented, such as one-hot encoding, one-hot, word2vec, embedding, etc.

[0042] ⑤ The three dimensions obtained in step ③ are appended on the basis of the original encoding;

[0043] ⑥According to the business requirements, build the upper layer model.

[0044] Referring to Figure 1 Taking the transformer structure as an example, the modified encoding mode further injects global statistical information into the original text encoding mode by only adding three dimensions, and the parameter amount of the three newly added dimensions is lower than the training cost of the table of the model structure, so that the parameter matrix of the general pre-training model can be used, and the training cost is lower.

[0045] By selecting the left and right joint probability as the injection of prior knowledge, the word boundary segmentation problem is solved, and better performance can be achieved under the professional field data set.

[0046] Compared with the injection of intra-sentence features such as part-of-speech and syntactic dependency, the present technical solution first proposes the injection of global statistical information.

[0047] In order for those skilled in the art to better understand the present application, the following examples are given for further description.

[0048] Embodiment: Taking the patent field named entity recognition task as an example.

[0049] (1) Collect domestic invention patents in 2019, and screen the related patents under C07 and C08 according to IPC.

[0050] (2) Clean the original patent text to remove random codes, HTML tags, etc.

[0051] (3) Segment, sentence, and word processing are performed on the text.

[0052] (4) The features are obtained according to the following steps:

[0053]

[0054] (5) Encode the sentence according to the tokenizer of the transformer structure to obtain token-embedding, position-embedding, and segment-embedding.

[0055] (6) Concatenate the obtained features, [token-embedding, position-embedding, segment-embedding, is_new, 1-solid, r_solid.

[0056] (7) Use concat as the joint representation to perform subsequent model operations, stack multiple layers of transformer and crf layer.

[0057] (8) training the model structure of (8) to perform a named entity recognition task.

[0058] (9) using the model trained in (9) to perform prediction for a downstream task.

[0059] The above merely provides a specific implementation of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art, according to the technical solution and novel concept of the present application, can make equivalent substitutions or changes within the technical range disclosed by the present application, which should be encompassed in the protection scope of the present application.

Claims

1. A text encoding method, characterized in that... The method is as follows: S1. Perform data cleaning and preprocessing on the corpus; S2. Segment and word segment the corpus; S3. Extract relevant features; S31. Extract N-gram features as candidate word groups; S32. Traverse the candidate word groups and process each word group as follows: Based on part of speech and syntactic template, it is judged to be a candidate word. According to the formula Calculate the score for new words; The length of the candidate words; This represents the word frequency statistics of candidate words; subcand represents the sub-candidate words among the candidate words. S33. Filter according to the score and threshold to obtain new candidate word groups; S34. Iterate through the new candidate word groups, and process each word group as follows: Count the total number of times the preceding word 'b' appears in the text, C b , Count the total number of times the word 'a' appears in the text, C a , Calculate the new word score using the following formula: ， The number of candidate words; The number of candidate words with 'b' as the preceding word; To calculate the cumulative sum of the number of all candidate words with 'b' as the preceding term; The number of candidate words with 'a' as the following term. To calculate the cumulative sum of the number of all candidate words that end with 'a'; S35. According to the formula Assign joint probability features to candidate words. S36. According to the formula The candidate words are assigned a joint probability feature of the subsequent term, and then filtered according to the score and a threshold to obtain new candidate word groups; S37. Mark the filtered candidate word set; S38. Add the features obtained in steps S35, S36 and S37 to the original data; The method further includes: S4. Encode and represent the sentences; S5. Add the three dimensions obtained in step S3 to the encoding in step S4; S6. Construct the upper-level model.

2. The text encoding method according to claim 1, characterized in that, In step S2, the word segmentation adopts the minimum granularity word segmentation mode to obtain a word list.

Images