Method for solving inconsistency of number of tokens and number of words in English entity recognition

Abstract

The invention belongs to the technical field of natural language processing, and discloses a method for solving the problem that the number of tokens and the number of words in English entity recognition are inconsistent, and the method comprises the following specific steps: S1, sentence preprocessing and word level labeling; s2, letter combination segmentation; s3, word embedding; s4, upstream model processing; s5, pooling operation; S6, downstream model processing; and S7, label sequence outputting. By optimizing the processing flow, the problem of resource waste caused by letter combination level labeling is successfully solved, unnecessary calculation overhead is avoided, meanwhile, the problem that in word level prediction, the word and letter combination corresponding relation is difficult to learn is solved, and the word level prediction efficiency is improved. Compared with a traditional method, the method ensures that the model more accurately captures semantic and morphological information in the text, and experimental results show that compared with the traditional method, the processing method adopting the letter combination level has the advantages that the performance is remarkably improved, and the accuracy is improved by 3%.

Description

Technical Field

[0001] The present invention belongs to the technical field of natural language processing, and specifically provides a method for solving the inconsistency between the number of English entity recognition tokens and the number of words. Background Art

[0002] Named entity recognition is crucial in natural language processing, especially when dealing with English text. Given its large vocabulary, entity recognition often focuses on letter combinations rather than complete words. This process first involves breaking down words into meaningful letter combinations, such as "What is your name?" is segmented into "wh", "##at", etc., where special markers are used to distinguish the beginning of words. These letter combinations are then converted into word embedding vectors and fed as input to deep learning models including IdCnn, RNN, Transformer, BERT, or models designed specifically for entity recognition such as CRF and Tplinker to achieve efficient and accurate named entity recognition. The current technical challenge is that although the goal is word-level named entity recognition, data annotation and model prediction remain at the letter combination level, resulting in serious resource consumption in downstream models such as CRF and Tplinker during processing. In addition, the irregularity of the positional relationship between letter combinations and words makes it an inefficient choice to directly transform sequences through a fully connected matrix, which increases the difficulty of model learning. The model is difficult to capture which letter combinations belong to the same word, which in turn affects the overall recognition accuracy and efficiency. Summary of the invention

[0003] The purpose of the present invention is to provide a method for solving the inconsistency between the number of English entity recognition tokens and the number of words, so as to solve the problems raised in the above background technology.

[0004] In order to achieve the above object, the present invention provides the following technical solution: a method for solving the inconsistency between the number of English entity recognition tokens and the number of words, the specific steps of the method are as follows:

[0005] S1: Sentence preprocessing and word-level labeling: "My nationality is the people's republic of China", after simplified segmentation, the nationality is marked as "ORG";

[0006] S2: Letter combination segmentation: segment words into letter combinations and record them in the matrix to prepare for subsequent word embedding;

[0007] S3: Word embedding: Customize the letter combination to convert the word embedding and build a [14,128] matrix for backup;

[0008] S4: Upstream model processing: input [14,128] matrix to the model, encode and output [14,n] new vector sequence;

[0009] S5: Pooling operation: group letter combinations, pool new vectors, and construct a [8,n] result matrix to represent words;

[0010] S6: Downstream model processing: input pooling matrix to downstream, full connection or CRF predict word label;

[0011] S7: Output tag sequence: decode the tag sequence into a readable format.

[0012] Preferably, the specific steps of sentence preprocessing and word-level labeling in S1 are as follows:

[0013] Step 1: Input sentences and word segmentation

[0014] Input sentence: "My nationality is the people's republic of China";

[0015] Word segmentation: Based on spaces and Chinese word segmentation algorithm, the sentence is decomposed into the following word sequence: "My", "nationality", "is", "the", "people's", "republic", "of", "China";

[0016] Step 2: Annotation sequence design

[0017] Understand tags: recognize the given tag sequence "O","O","O","B-ORG","M-ORG","M-ORG","M-ORG","E-ORG",

[0018] Labeling decision: "People's Republic of" should be regarded as the name of the entire organization in the real context and simplified. It is decided to treat "people's" separately and not include it in the continuous labeling of the organization name;

[0019] Step 3: Execute annotation

[0020] Execute annotation: Based on the above decision, annotate the word segmentation results:

[0021] My:O

[0022] nationality:B-ORG

[0023] is:

[0024] the:O

[0025] people's:

[0026] republic:M-ORG

[0027] of:O

[0028] China:E-ORG.

[0029] Preferably, the S2 letter combination segmentation refers to that each word is disassembled according to a predefined letter combination rule, the sentence is segmented according to the letter combination, and recorded in a matrix, and cut into a sequence of "my", "na", "##tion", "##ality", "is", "the", "peo", "##ple", "##'s", "re", "##pub", "##lic", "of", "china", letter combinations, wherein "##" represents a connector, which is used to identify that the segmented fragments are still part of the original word, and these segmented letter combinations will be recorded in sequence in a matrix or list, and each element corresponds to a combination.

[0030] Preferably, the word embedding in S3 refers to the use of a custom method to generate word embedding for the segmented letter combinations when processing text data, and the conversion is performed according to the custom letter combination characteristics. Each letter combination is mapped to a 128-dimensional vector space. The vector captures the semantic and morphological information between the letter combinations. The vectors of all letter combinations are organized in sequence to construct a matrix with a shape of [14,128]. The matrix provides rich feature representation for subsequent natural language processing tasks, such as named entity recognition or text classification.

[0031] Preferably, the specific steps of the upstream model processing in S4 are as follows:

[0032] Step 1: Input vector sequence

[0033] The shape of the letter combination word embedding matrix [14,128] is passed as input to the upstream model. The matrix contains 128-dimensional vectors of 14 letter combinations, each of which is rich in corresponding semantic and morphological information;

[0034] Step 2: Model processing and output

[0035] After the upstream model receives this vector sequence, it will encode it. The model uses complex calculations to capture the dependencies and context information in the sequence. After the processing is completed, the model will output a new vector sequence whose shape becomes [14,n].

[0036] Preferably, the specific steps of the pooling operation in S5 are as follows:

[0037] Step 1: Grouping

[0038] According to the letter combination segmentation results of each word in the original sentence, the corresponding letter combination vectors are grouped;

[0039] Step 2: Maximum Pooling

[0040] Perform a maximum pooling operation on all letter combination vectors corresponding to each word. For each component, select the maximum value of the component among all letter combination vectors of the word to obtain a new vector representing the word.

[0041] Step 3: Construct the result matrix

[0042] Arrange the vectors obtained after maximum pooling of all words in order to form the final result matrix. Assuming there are 8 words in the sentence, and each word is pooled to obtain an n-dimensional vector, the shape of the final result matrix is ​​[8,n], where each vector is the pooling result of a word in the original sentence.

[0043] Preferably, the specific steps of the downstream model processing in S6 are as follows:

[0044] Step 1: Input the pooled matrix

[0045] The [8,n] matrix processed by maximum pooling is passed as input to the downstream model. The matrix contains the pooled vectors of each word in the sentence, and each vector captures the key features of the word;

[0046] Step 2: Model prediction

[0047] After receiving the input vector sequence, the downstream model uses the learned parameters and functional relationships to predict the vector of each word and output the label corresponding to each word. The label is the entity type in the named entity recognition task, or the classification result in other NLP tasks.

[0048] Preferably, the output label sequence in S7 refers to that in the NLP task, after the downstream model processes the pooled matrix, it will output a series of label codes, and the codes represent the category or state of each word in the sentence. Next, these label codes are decoded and converted into a human-readable format. For example, "O" represents non-entity, "B-ORG" represents the beginning of entity organization, and "M-ORG" represents the middle part of entity organization. After decoding, the final label sequence is obtained, such as "OOOB-ORGM-ORGM-ORGE-ORGOOOO", which clearly shows the entity annotation results of each word in the sentence.

[0049] The beneficial effects of the present invention are as follows:

[0050] 1. The present invention successfully solves the problem of resource waste caused by letter combination-level annotation by optimizing the processing flow, avoiding unnecessary computational overhead. At the same time, the solution also overcomes the difficulty of learning the correspondence between words and letter combinations in word-level prediction, ensuring that the model can more accurately capture the semantic and morphological information within the text. The experimental results show that the letter combination-level processing method has achieved significant performance improvements compared to the traditional method, with an accuracy increase of 3 percentage points, providing a more efficient and accurate solution for natural language processing tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is a schematic diagram of the overall process of the present invention. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] like Figure 1 As shown, the embodiment of the present invention provides a method for solving the inconsistency between the number of English entity recognition tokens and the number of words. The specific steps of using the advanced method are as follows:

[0054] S1: Sentence preprocessing and word-level labeling: "My nationality is the people's republic of China", after simplified segmentation, the nationality is marked as "ORG";

[0055] S2: Letter combination segmentation: segment words into letter combinations and record them in the matrix to prepare for subsequent word embedding;

[0056] S3: Word embedding: Customize the letter combination to convert the word embedding and build a [14,128] matrix for backup;

[0057] S4: Upstream model processing: input [14,128] matrix to the model, encode and output [14,n] new vector sequence;

[0058] S5: Pooling operation: group letter combinations, pool new vectors, and construct a [8,n] result matrix to represent words;

[0059] S6: Downstream model processing: input pooling matrix to downstream, fully connected or CRF predicts word labels;

[0060] S7: Output tag sequence: decode the tag sequence into a readable format.

[0061] The specific steps of sentence preprocessing and word-level labeling in S1 are as follows:

[0062] Step 1: Input sentences and word segmentation

[0063] Input sentence: "My nationality is the people's republic of China";

[0064] Word segmentation: Based on spaces and Chinese word segmentation algorithms (processing continuous Chinese words such as "people'srepublicof"), the sentence is decomposed into the following word sequence: "My", "nationality", "is", "the", "people's", "republic", "of", "China";

[0065] Step 2: Annotation sequence design

[0066] Understand labels: Identify the given label sequence "O", "O", "O", "B-ORG", "M-ORG", "M-ORG", "M-ORG", "E-ORG", where "O" indicates a non-organization name, "B-ORG" indicates the beginning of an organization name, "M-ORG" indicates the middle of an organization name, and "E-ORG" indicates the end of an organization name. 1. The length of the label sequence is 8.

[0067] Labeling decision: "People's Republic of" should be considered as the name of the entire organization in the real context and simplified. It is decided to treat "people's" separately and not include it in the continuous labeling of the organization name;

[0068] Step 3: Execute annotation

[0069] Execute annotation: Based on the above decision, annotate the word segmentation results:

[0070] My:O

[0071] nationality:B-ORG (nationality as the beginning of the organization name)

[0072] is:O

[0073] the:O

[0074] people's:O (Although related to words such as "republic", it is treated separately here)

[0075] republic:M-ORG (the middle part of the organization name)

[0076] of:O

[0077] China: E-ORG (end of organization name).

[0078] Sentence preprocessing and word-level labeling ensure the accuracy of data input and the rationality of labels in NLP tasks through fine word segmentation and reasonable labeling strategies. This process not only enhances the model's ability to recognize text entities, but also promotes the correct understanding of complex text structures, laying a solid foundation for subsequent named entity recognition and other tasks, and effectively improving the performance and accuracy of the overall NLP processing flow.

[0079] Among them, the S2 letter combination segmentation refers to the decomposition of each word according to the predefined letter combination rules, the sentence is segmented according to the letter combination, and recorded in a matrix, cut into a sequence of "my", "na", "##tion", "##ality", "is", "the", "peo", "##ple", "##'s", "re", "##pub", "##lic", "of", "china", letter combinations, where "##" represents a connector, which is used to identify that the segmented fragments are still part of the original word. These segmented letter combinations will be recorded in order in a matrix or list, and each element corresponds to a combination for subsequent word embedding processing. This process not only helps to capture the morphological information inside the word, but also provides a more sophisticated input representation for the deep learning model.

[0080] The letter combination segmentation technology breaks down words in detail through predefined rules, uses connectors to retain the attribution of fragments, and records them in an orderly manner in the form of a matrix. This process deepens the mining of the internal morphological information of words, provides more delicate and richer input for word embedding processing, enhances the model's ability to understand and capture text features, and is an effective means to improve the performance and accuracy of NLP tasks.

[0081] The word embedding in S3 refers to a custom method used to generate word embedding for the segmented letter combinations when processing text data. The word embedding is converted according to the custom letter combination characteristics. Each letter combination is mapped to a 128-dimensional vector space. The vector captures the semantic and morphological information between the letter combinations. The vectors of all letter combinations are organized in sequence to construct a matrix with a shape of [14,128], where 14 represents the total number of letter combinations and 128 is the dimension of each vector. This matrix provides rich feature representation for subsequent natural language processing tasks, such as named entity recognition or text classification.

[0082] Word embedding technology accurately captures the semantic and morphological information of letter combinations by mapping them to high-dimensional vector space, and constructs an information-rich feature matrix. This matrix provides a solid foundation for subsequent NLP tasks such as named entity recognition and text classification. It not only simplifies the data processing process, but also significantly improves the accuracy and efficiency of task processing. It is one of the key technologies to promote the development of natural language processing.

[0083] The specific steps of the upstream model processing in S4 are as follows:

[0084] Step 1: Input vector sequence

[0085] The shape of the letter combination word embedding matrix [14,128] is passed as input to the upstream model (such as LSTM or Transformer). This matrix contains 128-dimensional vectors of 14 letter combinations, each of which is rich in corresponding semantic and morphological information;

[0086] Step 2: Model processing and output

[0087] After receiving this vector sequence, the upstream model (such as LSTM or Transformer) will encode it. The model uses complex calculations (such as LSTM's recurrent unit processing or Transformer's self-attention mechanism) to capture the dependencies and contextual information in the sequence. After processing, the model will output a new vector sequence, whose shape may become [14, n], where n may be different from the original 128 dimensions, depending on the specific structure and design of the model. This new vector sequence contains high-level features encoded by the model and can be used for subsequent text classification, sentiment analysis, or other NLP tasks.

[0088] The upstream model processes the input vector sequence and uses advanced models such as LSTM or Transformer to deeply mine the semantic and contextual information between letter combinations. The new vector sequence output after encoding not only retains the key features, but also incorporates high-level semantic information, providing richer and more accurate input for downstream tasks. This process significantly improves the performance and efficiency of NLP tasks and is the cornerstone of building an efficient and intelligent text processing system.

[0089] The specific steps of the pooling operation in S5 are as follows:

[0090] Step 1: Grouping

[0091] According to the segmentation results of the letter combination of each word in the original sentence, the corresponding letter combination vectors are grouped. For example, "nationality" is segmented into "na", "##tion", and "##ality", and the vectors of these three letter combinations will be grouped together.

[0092] Step 2: Maximum Pooling

[0093] Perform a maximum pooling operation on all letter combination vectors corresponding to each word. For each component (i.e., each dimension of the vector), select the maximum value of the component among all letter combination vectors of the word to obtain a new vector representing the word.

[0094] Step 3: Construct the result matrix

[0095] Arrange the vectors obtained after maximum pooling of all words in order to form the final result matrix. Assuming there are 8 words in the sentence, and each word is pooled to obtain an n-dimensional vector, the shape of the final result matrix is ​​[8,n], where each vector is the pooling result of a word in the original sentence.

[0096] The pooling operation effectively compresses word features and improves processing efficiency through three steps: grouping, maximum pooling, and result matrix construction. It retains the key information of the word while reducing data redundancy and providing more refined input for downstream models. This process not only simplifies data processing, but also enhances the generalization ability of the model. It is a key step in optimizing feature extraction in NLP tasks and helps improve overall system performance.

[0097] The specific steps of the downstream model processing in S6 are as follows:

[0098] Step 1: Input the pooled matrix

[0099] The [8,n] matrix processed by the maximum pooling is passed as input to the downstream model (such as the fully connected layer, CRF, etc.). This matrix contains the pooled vectors of each word in the sentence, and each vector captures the key features of the word;

[0100] Step 2: Model prediction

[0101] After receiving the input vector sequence, the downstream model uses the learned parameters and functional relationships to predict the vector of each word and output the label corresponding to each word. These labels may be entity types in named entity recognition tasks (such as names of people, places, and organizations), or classification results in other NLP tasks.

[0102] The processing flow of the mid- and downstream models is clear. First, the pooled [8,n] matrix is ​​input to capture the key features of the word; then the downstream model accurately predicts and outputs the corresponding label of the word. This process improves processing efficiency and prediction accuracy, while maintaining system flexibility and scalability, making it easier to adapt to different NLP task requirements and introduce new technologies. It is the key to improving the performance of NLP applications.

[0103] The output label sequence in S7 refers to the output of a series of label codes after the downstream model processes the pooled matrix in the NLP task. The code represents the category or state of each word in the sentence. Next, these label codes are decoded and converted into a human-readable format, such as "O" for non-entity, "B-ORG" for the beginning of entity organization, and "M-ORG" for the middle part of entity organization. After decoding, the final label sequence is obtained, such as "OOOB-ORGM-ORGM-ORGE-ORGOOOO". This sequence clearly shows the entity annotation results of each word in the sentence.

[0104] Based on the above content, the generation and decoding process of the label sequence is clearly explained, allowing readers to have a clear picture of the whole picture from the original text to the final annotation result. By explaining the meaning of the label in detail (such as "O" non-entity, "B-ORG" entity start), the understanding of NLP entity annotation is deepened and the effective application of knowledge is promoted. At the same time, the display of the intermediate steps of the model (pooling matrix, label encoding) enhances transparency and facilitates debugging and optimization. This process is not only a valuable case for education, but also a powerful guide in practice, helping developers and researchers to seamlessly connect theory with practical applications.

[0105] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0106] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for solving the inconsistency between the number of English entity recognition tokens and the number of words, characterized by: The specific steps of this method are as follows: S1: Sentence preprocessing and word-level labeling: "My nationality is the people's republic of China", after simplified segmentation, the nationality is marked as "ORG"; S2: Letter combination segmentation: segment words into letter combinations and record them in the matrix to prepare for subsequent word embedding; S3: Word embedding: Customize the letter combination to convert the word embedding and build a [14,128] matrix for backup; S4: Upstream model processing: input [14,128] matrix to the model, encode and output [14,n] new vector sequence; S5: Pooling operation: group letter combinations, pool new vectors, and construct a [8,n] result matrix to represent words; S6: Downstream model processing: input pooling matrix to downstream, full connection or CRF predict word label; S7: Output tag sequence: decode the tag sequence into a readable format.

2. The method for solving the inconsistency between the number of English entity recognition tokens and the number of words according to claim 1 is characterized in that: The specific steps of sentence preprocessing and word-level labeling in S1 are as follows: Step 1: Input sentences and word segmentation Input sentence: "My nationality is the people's republic of China"; Word segmentation: Based on spaces and Chinese word segmentation algorithm, the sentence is decomposed into the following word sequence: "My", "nationality", "is", "the", "people's", "republic", "of", "China"; Step 2: Annotation sequence design Understand tags: recognize the given tag sequence "O","O","O","B-ORG","M-ORG","M-ORG","M-ORG","E-ORG", Labeling decision: "People's Republic of" should be regarded as the name of the entire organization in the real context and simplified. It is decided to treat "people's" separately and not include it in the continuous labeling of the organization name; Step 3: Execute annotation Execute annotation: Based on the above decision, annotate the word segmentation results: My:O nationality:B-ORG is: the:O people's: republic:M-ORG of:O China:E-ORG.

3. The method for solving the inconsistency between the number of English entity recognition tokens and the number of words according to claim 1 is characterized in that: The S2 letter combination segmentation refers to that each word is disassembled according to the predefined letter combination rules, the sentence is segmented according to the letter combination, and recorded in a matrix, and cut into the sequence "my", "na", "##tion", "##ality", "is", "the", "peo", "##ple", "##'s", "re", "##pub", "##lic", "of", "china", letter combination; The "##" represents a connector, which is used to indicate that the segmented fragments are still part of the original word. These segmented letter combinations will be recorded in sequence in a matrix or list, with each element corresponding to a combination.

4. The method for solving the inconsistency between the number of English entity recognition tokens and the number of words according to claim 1, characterized in that: The word embedding in S3 refers to a custom method used to generate word embeddings for the segmented letter combinations when processing text data. The word embeddings are transformed according to the custom letter combination characteristics. Each letter combination is mapped to a 128-dimensional vector space. The vector captures the semantic and morphological information between the letter combinations. The vectors of all letter combinations are organized in sequence to construct a matrix with a shape of [14,128]. The matrix provides rich feature representation for subsequent natural language processing tasks, such as named entity recognition or text classification.

5. The method for solving the inconsistency between the number of English entity recognition tokens and the number of words according to claim 1, characterized in that: The specific steps of the upstream model processing in S4 are as follows: Step 1: Input vector sequence The shape of the letter combination word embedding matrix [14,128] is passed as input to the upstream model. The matrix contains 128-dimensional vectors of 14 letter combinations, each of which is rich in corresponding semantic and morphological information; Step 2: Model processing and output After the upstream model receives this vector sequence, it will encode it. The model uses complex calculations to capture the dependencies and context information in the sequence. After the processing is completed, the model will output a new vector sequence whose shape becomes [14,n].

6. The method for solving the inconsistency between the number of English entity recognition tokens and the number of words according to claim 1, characterized in that: The specific steps of the pooling operation in S5 are as follows: Step 1: Grouping According to the letter combination segmentation results of each word in the original sentence, the corresponding letter combination vectors are grouped; Step 2: Maximum Pooling Perform a maximum pooling operation on all letter combination vectors corresponding to each word. For each component, select the maximum value of the component among all letter combination vectors of the word to obtain a new vector representing the word. Step 3: Construct the result matrix Arrange the vectors of all words after maximum pooling in order to form the final result matrix; Assuming there are 8 words in the sentence, and each word is pooled to obtain an n-dimensional vector, the final result matrix shape is [8,n], where each vector is the pooling result of a word in the original sentence.

7. The method for solving the inconsistency between the number of English entity recognition tokens and the number of words according to claim 1, characterized in that: The specific steps of downstream model processing in S6 are as follows: Step 1: Input the pooled matrix The [8,n] matrix processed by maximum pooling is passed as input to the downstream model. The matrix contains the pooled vectors of each word in the sentence, and each vector captures the key features of the word; Step 2: Model prediction After receiving the input vector sequence, the downstream model uses the learned parameters and functional relationships to predict the vector of each word and output the label corresponding to each word. The label is the entity type in the named entity recognition task or the classification result in other NLP tasks.

8. The method for solving the inconsistency between the number of English entity recognition tokens and the number of words according to claim 1, characterized in that: The output label sequence in S7 refers to that in the NLP task, after the downstream model processes the pooled matrix, it will output a series of label codes, which represent the category or state of each word in the sentence. These label codes are then decoded and converted into a human-readable format. For example, "O" represents a non-entity, "B-ORG" represents the beginning of an entity organization, and "M-ORG" represents the middle part of an entity organization. After decoding, the final label sequence is obtained, such as "OOOB-ORGM-ORGM-ORGE-ORGOOOO". This sequence clearly shows the entity annotation results of each word in the sentence.

Images