A named entity extraction method based on term merging

Abstract

The application relates to the technical field of text word item recognition, and particularly discloses a named entity extraction method based on word item combination, which comprises the following steps in sequence: sequential combination judgment, whole text frequency statistics and word item combination, can automatically mine and combine continuous and high-frequency word item combinations from original text without relying on external dictionaries or training models, realizes extraction of named entities, is suitable for open domain, low resource or real-time text processing scenes, does not need model loading or training, and can efficiently run on resource-limited terminals or edge devices.

Description

Technical Field

[0001] This application relates to the field of text term recognition technology, and specifically discloses a named entity extraction method based on term merging. Background Technology

[0002] Named Entity Recognition (NER) is one of the core tasks in natural language processing, and its accuracy directly affects downstream applications such as text structuring, information extraction, and knowledge graph construction.

[0003] Most existing named entity recognition methods are based on machine learning or deep learning techniques. These methods typically rely on manually annotated corpora, large-scale pre-trained language models, or externally imported, manually constructed entity dictionaries for training and inference. In particular, they require large-scale pre-trained language models or manually constructed entity dictionaries as prior knowledge. Although they demonstrate high accuracy on standard datasets, they still face several technical challenges in practical applications, including:

[0004] In the process of segmenting Chinese proper nouns, due to the complex structure of Chinese proper nouns, standard word segmentation often incorrectly segments entities, resulting in incomplete entity recognition.

[0005] It has a weak ability to recognize long entities, especially in scenarios where there is a lack of context or prior knowledge.

[0006] The external models and dictionaries they support are costly to maintain and difficult to adapt to domain texts with frequent semantic changes or low-resource application scenarios.

[0007] This invention provides a named entity extraction method based on term merging to solve the above-mentioned problems. Summary of the Invention

[0008] The purpose of this invention is to provide a named entity extraction method based on term merging. Based on the term combination and frequency statistics after word segmentation, named entities are automatically identified through structural merging without any external labeled data or pre-trained models.

[0009] To achieve the above objectives, the basic solution of this invention provides a named entity extraction method based on term merging, comprising the following steps:

[0010] Step A1: Process the original text to obtain a global term set including a list of terms divided according to sentence order, wherein the term list includes a list of words divided according to string order;

[0011] Step A2: Establish a word segmentation position index based on the initial string position;

[0012] Step A3: Combine the word segment corresponding to the current word segmentation position index and the word segment at the next string position into a word group. Check whether the string length of the word group meets the minimum word length limit. If it does, jump to step A6. Otherwise, check the word merging condition of the word group. If it meets the condition, proceed to step A4. Otherwise, jump to step A6.

[0013] Step A4: Combine the word segments into a long entity and add it to the final entity set. Then, treat the word segment group as a new term to replace the word segment at the string position corresponding to the original word segment position index, and delete the word segment at the next string position.

[0014] Step A5: If the length of the current word segmentation position index is less than the length of the current word list minus one, repeat steps A3 to A5 based on the current word segmentation position index; otherwise, proceed to step A7.

[0015] Step A6: Move one string position backward to establish a new word segmentation position index. If the length of the new word segmentation position index is greater than the length of the current word list minus one, proceed to step A7; otherwise, return to step A3 and loop through steps A3 to A6.

[0016] Step A7: Repeat steps A3 to A7 until all term lists have been traversed, and output the final entity set.

[0017] Furthermore, the word segmentation process on the original text includes sentence-level segmentation based on punctuation marks, and processing each segmented sentence using a general Chinese word segmentation method, ultimately obtaining a global term set composed of all segmented words in string order.

[0018] Furthermore, the index of the word segmentation position corresponding to the initial string position is 0.

[0019] Furthermore, the default value for the minimum word length limit is 2.

[0020] Furthermore, the term merging condition is as follows:

[0021] The global frequency of word groups is counted in the global term set. A minimum frequency threshold is established as the merging condition. When the global frequency is greater than or equal to the minimum frequency threshold, the term merging condition is met; otherwise, the term merging condition is not met.

[0022] Furthermore, the specific statistical method for the global occurrence frequency is as follows:

[0023] Iterate through each term list in the global term set one by one. During this process, use a sliding window to extract all consecutive term pairs. Compare the extracted consecutive term pairs with the word segments in turn. If a match is found, increment the frequency counter by one. Continue until all term lists have been traversed. The count value of the frequency counter is the global frequency of the word segment in the current term set.

[0024] Furthermore, the sliding window has a window size of 2 and a step size of 1.

[0025] Furthermore, in step A3, the default value for the minimum frequency threshold is 5.

[0026] The principle and effect of this basic scheme are as follows:

[0027] Compared with existing technologies, this invention extracts named entities by performing a series of steps, namely, sequential combination judgment, full-text frequency statistics, and term merging, without relying on external dictionaries or training models, to automatically mine and merge continuous and high-frequency term combinations from the original text. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 A flowchart of a named entity extraction method based on term merging proposed in an embodiment of this application is shown. Detailed Implementation

[0030] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided below.

[0031] A named entity extraction method based on term merging, implementing, for example... Figure 1 As shown, it includes the following steps:

[0032] Step A1: Perform sentence-level segmentation and word segmentation on the original text to be extracted with named entities in sequence to obtain a global term set W_all composed of all word segmentation results. The global term set W_all includes several term lists, and the order of the term lists is the same as the order of the sentences. Each term list is composed of the word segments obtained after the corresponding sentence is processed and arranged in string order.

[0033] Specifically, the global term set W_all = [ ], This represents a list of terms composed of all the word segments in the k-th sentence. , =[ ], This represents the word segment in the word list corresponding to the k-th sentence, where m is the string in the m-th order. .

[0034] Specifically, the word segmentation process for the original text includes sentence-level segmentation based on punctuation marks, and then processing each segmented sentence using a general Chinese word segmentation method to ultimately obtain a global term set composed of all the segmented words. Punctuation marks include, but are not limited to, periods, question marks, and exclamation marks.

[0035] For example, when the original text obtained is:

[0036] "The article was published in 1987, and the author was quite old."

[0037] The word segmentation result obtained after word segmentation processing is as follows:

[0038] "The article was published in 1987, and the author was very old."

[0039] Its corresponding global term set W_all is represented as follows:

[0040] [Article, published in, 1987, [author, age, very old]].

[0041] Step A2: Establish the word segmentation position index j based on the initial string position;

[0042] Step A3: Starting from the initial word segmentation position index j=0, process the word list... Tokenization at the corresponding string position and the word segmentation of the following string position Merge and combine to form word groups [ ];

[0043] Establish a minimum word length limit min_len after merging, and combine it with the word segmentation group [ The lengths of the strings are compared, and when the word group [ If the string length of ] is less than the minimum word length limit min_len, proceed to step A6; otherwise, count the global occurrence frequency f of the word group in the global term set W_all, and establish a minimum frequency threshold freq as the term merging condition. When the global occurrence frequency f is greater than or equal to the minimum frequency threshold freq, it means that the term merging condition is met and proceed to step A4; otherwise, proceed to step A6.

[0044] In this embodiment, the established minimum word length limit is represented as min_len, and min_len = Term_min, where Term_min is of type int, and the default value of the minimum word length limit min_len is 2.

[0045] In this embodiment, for word segments exceeding the minimum word length limit min_len, [ The specific statistical method for its global occurrence frequency f in the global term set W_all is as follows:

[0046] Iterate through each term list in the global term set W_all. In this process, a sliding window method is used to extract all consecutive word pairs, specifically setting the window size to 2 and the step size to 1. The extracted consecutive word pairs are then sequentially compared with the word segmentation groups [ The comparison is performed, and if a match is found, the frequency counter is incremented by one until all the word lists are traversed. The count value of the frequency counter is the global occurrence frequency f of the word segment in the current word set.

[0047] In this embodiment, the established minimum frequency threshold is denoted as freq, of type int, with a default value of 5, and min_freq=freq.

[0048] Step A4: Combine the word segments that meet the merging criteria. [Merge them into a single long entity and add it to the final entity set entity_set, and add it to the list of related terms.] Lieutenant General word segmentation [ [Merge into a new term] And by this new term Replace the original word in the word list Including String merging The position of the string, and delete it at the same time. String.

[0049] In this example, consecutive terms are also merged synchronously in all sublists of terms in the global term set W_all: word groups [ At this time, when it appeared in other places For combinations, they are merged. When reaching this position during traversal, there is no need for recalculation.

[0050] Step A5: Count the length of the current term list, denoted as n. If j < n - 1, then while keeping the starting position j unchanged, repeat Step A3 to Step A5 to re-judge the new segmentation group at the current position ;

[0051] Otherwise, it means that all terms in the current term list have been judged, and proceed to Step A7;

[0052] Step A6: Move one string position backward to establish a new segmentation position index, let , and count the length of the current term list, denoted as n. If > n - 1, it means that the judgment of the current term list is completed, and proceed to Step A7. If < n - 1, return to Step A3 and loop through Step A3 to Step A6 until all continuous terms that can be merged in the current term list are merged;

[0053] Step A7: According to the order of the term lists in the global term set W_all, sequentially select the (k + 1)-th term list and repeat Step A3 to Step A7 until all term lists are traversed, complete the judgment of the sentences of all term lists, and output the final entity set entity_set. The type of the final entity set entity_set is list.

[0054] A named entity recognition method based on term merging proposed by the present invention, through a series of steps carried out in sequence: sequential combination judgment, full-text frequency statistics, and term merging. Without relying on external dictionaries or training models, it automatically mines and merges continuous and high-frequency term combinations from the original text to achieve the extraction of named entities.

[0055] Compared with existing named entity recognition methods, the present invention shows significant advantages in multiple aspects:

[0056] (1) No external dependencies and strong adaptability: It does not rely on any external dictionaries, pre-trained models or manually annotated corpora, and is suitable for open-domain, low-resource or real-time text processing scenarios;

[0057] (2) Low computational resource consumption and suitable for lightweight applications: It does not require model loading or training and can run efficiently on terminals or edge devices with limited resources.

[0058] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any indirect modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A named entity extraction method based on term merging, characterized in that, Includes the following steps: Step A1: Process the original text to obtain a global term set including a list of terms divided according to sentence order, wherein the term list includes a list of words divided according to string order; Step A2: Establish a word segmentation position index based on the initial string position; Step A3: Combine the word segment corresponding to the current word segmentation position index and the word segment at the next string position into a word group. Check whether the string length of the word group is less than the minimum word length limit. If it is less, jump to step A6. Otherwise, check the word merging condition of the word group. If it is satisfied, proceed to step A4. Otherwise, jump to step A6. The conditions for merging the terms are: The global frequency of word groups is counted in the global term set. A minimum frequency threshold is established as the merging condition. When the global frequency is greater than or equal to the minimum frequency threshold, it means that the term merging condition is met; otherwise, it means that the term merging condition is not met. Step A4: Combine the word segments into a long entity and add it to the final entity set. Then, treat the word segment group as a new term to replace the word segment at the string position corresponding to the original word segment position index, and delete the word segment at the next string position. Step A5: If the length of the current word segmentation position index is less than the length of the current word list minus one, repeat steps A3 to A5 based on the current word segmentation position index; otherwise, proceed to step A7. Step A6: Move one string position backward to establish a new word segmentation position index. If the length of the new word segmentation position index is greater than the length of the current word list minus one, proceed to step A7; otherwise, return to step A3 and loop through steps A3 to A6. Step A7: Repeat steps A3 to A7 until all term lists have been traversed, and output the final entity set.

2. The named entity extraction method based on term merging according to claim 1, characterized in that, The word segmentation process for the original text includes sentence-level segmentation based on punctuation marks, and processing each segmented sentence using a general Chinese word segmentation method, ultimately obtaining a global term set composed of all segmented words in string order.

3. The named entity extraction method based on term merging according to claim 1, characterized in that, The initial string position corresponds to the word segmentation position index of 0.

4. The named entity extraction method based on term merging according to claim 1, characterized in that, The default value for the minimum word length limit is 2.

5. The named entity extraction method based on term merging according to claim 1, characterized in that, The specific statistical method for global occurrence frequency is as follows: Iterate through each term list in the global term set one by one. During this process, use a sliding window to extract all consecutive term pairs. Compare the extracted consecutive term pairs with the word segments in turn. If a match is found, increment the frequency counter by one. Continue until all term lists have been traversed. The count value of the frequency counter is the global frequency of the word segment in the current term set.

6. The named entity extraction method based on term merging according to claim 5, characterized in that, The sliding window has a window size of 2 and a step size of 1.

7. The named entity extraction method based on term merging according to claim 5, characterized in that, In step A3, the minimum frequency threshold defaults to 5.

Images