Named entity recognition method and apparatus

Abstract

The present application provides a named entity recognition method and a related device. The method comprises: acquiring a first natural language text; on the basis of the first natural language text and a classification model, filtering one or more statements in the first natural language text that do not contain named entities, to obtain a second natural language text, wherein the second natural language text comprises one or more statements in the first natural language text that contain named entities, and the classification model is used for classifying the statements in the first natural language text as either containing named entities or not; and performing named entity recognition on the second natural language text to obtain a first named entity recognition result. In the technical solution of the present application, the amount of calculation of named entity recognition tasks is reduced, and the speed of named entity recognition is improved.

Description

A method and device for named entity recognition

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on March 21, 2024, with application number 202410333691.0 and invention name “A method and device for named entity recognition”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of artificial intelligence (AI), and more specifically, to a method and apparatus for named entity recognition. Background Art

[0003] In current information systems, there is a massive amount of multi-type unstructured data, such as text, images, audio, video, and other data that cannot be described in a specific structure (such as a table). Among them, text data, including personal information and user data, accounts for a considerable proportion. In this context, named entity recognition (NER) technology is needed to identify and classify named entities in text data in information systems. Named entities generally refer to entities with specific meaning or strong referentiality in text, usually including names of people, places, organizations, dates, times, proper nouns, etc. NER is a basic task in natural language processing and is widely used in tasks such as information extraction, information retrieval, and information recommendation.

[0004] However, in current data management systems, such as those involving large amounts of textual data, or in speech recognition processes, the speed of named entity recognition technology is often limited by hardware resources. Therefore, how to utilize limited hardware resources to quickly and accurately perform named entity recognition on massive amounts of unstructured text is a pressing challenge. Summary of the Invention

[0005] The present application provides a method, apparatus, computing device, chip system, computer-readable storage medium and computer program product for named entity recognition, which can reduce the computational complexity of named entity recognition tasks and improve the speed of named entity recognition.

[0006] In a first aspect, a method for named entity recognition is provided, the method comprising: obtaining a first natural language text; based on the first natural language text and a classification model, filtering one or more sentences in the first natural language text that do not contain named entities to obtain a second natural language text, the second natural language text including one or more sentences in the first natural language text that contain named entities, the classification model being used to classify sentences in the first natural language text as containing named entities or not containing named entities; performing named entity recognition on the second natural language text to obtain a first named entity recognition result.

[0007] In the embodiments of this application, through the natural language text screening process, one or more sentences that do not contain named entities are filtered based on the classification model, thereby reducing the amount of natural language text data used for named entity recognition. Compared with methods such as those based on neural networks that require hardware acceleration, the named entity recognition method in the embodiments of this application significantly reduces the amount of computation and increases the speed of named entity recognition.

[0008] In combination with the first aspect, in certain implementations of the first aspect, the method is applied to a computing platform that executes computing tasks through a central processing unit (CPU).

[0009] In other words, this method can be applied to computing platforms without hardware acceleration. This refers to platforms that do not contain any hardware other than the central processing unit (CPU) to perform computing tasks (such as a graphics processing unit (GPU) or a neural processing unit (NPU)).

[0010] In combination with the first aspect, in certain implementations of the first aspect, the classification model determines whether a sentence contains a named entity based on the frequency and order of words in the sentence.

[0011] Optionally, the classification model can also determine whether a sentence contains a named entity based on primary semantic features such as part of speech, root, length, etc. of words in the text paragraph or other high-level semantic features extracted by a pre-trained model.

[0012] In combination with the first aspect, in certain implementations of the first aspect, the named entity recognition is performed on the second natural language text to obtain a first named entity recognition result, including: obtaining multiple words to be recognized in the second natural language text; based on the multiple words to be recognized and a finite state machine, obtaining type labels of the multiple words to be recognized output by the finite state machine, the type labels being used to indicate whether the words to be recognized are named entities; and determining the first named entity recognition result based on the type labels of the multiple words to be recognized.

[0013] It should be understood that a large amount of text data may exist in a data management system or storage system. For this massive amount of unstructured text data, while reducing the processing load of the entity recognition algorithm and further improving overall processing performance, the named entity recognition process also needs to maintain a certain level of accuracy.

[0014] In the embodiments of the present application, the finite state machine model is more CPU-friendly than the neural network model because the reasoning of the neural network requires a large number of parallel floating-point operations, while the CPU is more suitable for performing serial operations. The degree of parallelization is relatively low compared to hardware such as GPU or NPU, and the ability to process such floating-point operations is insufficient. The finite state machine is a serial operation and the amount of computation required for state prediction and transfer is relatively small, so it is more suitable for CPU execution. Therefore, the feature encoding and prediction process of the NER algorithm based on the finite state machine is more CPU-friendly, and the CPU can efficiently complete complex computing tasks.

[0015] In combination with the first aspect, in certain implementations of the first aspect, based on the multiple words to be recognized and a finite state machine, obtaining the type labels of the multiple words to be recognized output by the finite state machine includes: determining the dynamic word vector and the static word vector mapped to each of the multiple words to be recognized, wherein the static word vector mapped to each word to be recognized in different context sentences has the same vector representation, and the dynamic word vector mapped to each word to be recognized in different context sentences has a different vector representation; based on the dynamic word vector and the static word vector mapped to the multiple words to be recognized and a finite state machine, obtaining the type labels of the multiple words to be recognized output by the finite state machine, the input of the finite state machine being a fusion word vector of the dynamic word vector and the static word vector mapped to each of the multiple words to be recognized.

[0016] In an embodiment of the present application, the type label of the word to be recognized can be predicted by performing vector fusion on the dynamic word vector and the static word vector mapped to the word to be recognized. Exemplarily, the vector fusion method in the embodiment of the present application can be weighted fusion, fusion based on feature selection, etc. For example, the result of the fusion vector can be obtained by adding the static word vector and the dynamic word vector. For another example, the static word vector and the dynamic word vector of each word to be recognized can be weighted averaged according to a certain weight, and the weight can be set according to the different importance of the static word vector and the dynamic word vector. For another example, a static word vector or a dynamic word vector can be selected according to the importance of the feature, and the most important feature vector can be selected through features such as mutual information and correlation coefficient to obtain a more accurate fusion result.

[0017] Optionally, the representation of dynamic word vectors and static word vectors can adopt algorithms such as word embedding in related existing technologies or improved algorithms.

[0018] In combination with the first aspect, in certain implementations of the first aspect, the static word vector mapped to each word to be recognized is determined based on a preset word vector library, which includes static word vectors mapped to multiple words, and the multiple words include the multiple words to be recognized.

[0019] In combination with the first aspect, in certain implementations of the first aspect, the static word vector mapped to each word to be recognized is determined based on a pre-trained static word vector model.

[0020] Optionally, the method of converting (mapping) the words to be recognized into static word vectors can be the word2vec algorithm based on local context prediction and / or the GloVe regression algorithm based on explicit global co-occurrence information. Alternatively, the corresponding static word vector of the word to be recognized can be obtained by querying the word vector library or dictionary.

[0021] In combination with the first aspect, in certain implementations of the first aspect, the dynamic word vector mapped to each word to be recognized is determined based on a pre-trained dynamic word vector model.

[0022] Optionally, the method of converting (mapping) the words to be recognized into dynamic word vectors can be based on a pre-trained dynamic word vector model such as ELMo, BERT or convolutional neural network (CNN).

[0023] In combination with the first aspect, in some implementations of the first aspect, the type tag is further used to indicate the meaning of the word to be recognized and / or the position of the word to be recognized in the named entity.

[0024] In combination with the first aspect, in certain implementations of the first aspect, after performing named entity recognition on the second natural language text to obtain a first named entity recognition result, the method further includes: based on the first named entity recognition result, determining one or more missing sentences from the one or more sentences that do not contain named entities, the first named entity recognition result includes one or more named entities in the second natural language text, and the one or more missing sentences include at least one of the one or more named entities; performing named entity recognition on the one or more missing sentences to obtain a second named entity recognition result.

[0025] It should be understood that in the text screening process, the classification model may make misjudgments, that is, sentences with named entities may be misclassified as sentences without named entities. Therefore, in view of this, in the embodiments of the present application, the processes of re-matching and re-recognition are added to reduce omissions in the named entity recognition results and improve the accuracy of named entity recognition.

[0026] In combination with the first aspect, in some implementations of the first aspect, the method further includes: determining a named entity matching result of the first natural language text based on the first natural language text and a preset matching rule.

[0027] In the embodiments of the present application, the additional rule matching process can improve the accuracy of the named entity recognition process without affecting the execution time and execution efficiency of the named entity recognition model.

[0028] In a second aspect, a device for named entity recognition is provided, which includes: an acquisition module for acquiring a first natural language text; a processing module for filtering, based on the first natural language text and a classification model, one or more sentences in the first natural language text that do not contain named entities to obtain a second natural language text, wherein the second natural language text includes one or more sentences in the first natural language text that contain named entities, and the classification model is used to classify sentences in the first natural language text as including named entities or not including named entities; the processing module is also used to perform named entity recognition on the second natural language text to obtain a first named entity recognition result.

[0029] In combination with the second aspect, in certain implementations of the second aspect, the device is applied to a computing platform that executes computing tasks through a central processing unit (CPU).

[0030] In combination with the second aspect, in some implementations of the second aspect, the classification model determines whether a sentence contains a named entity based on the word frequency and word order of the words in the sentence.

[0031] In combination with the second aspect, in certain implementations of the second aspect, the acquisition module is specifically used to: acquire multiple words to be recognized in the second natural language text; the processing module is specifically used to: obtain type labels of the multiple words to be recognized output by the finite state machine based on the multiple words to be recognized and the finite state machine, and the type labels are used to indicate whether the words to be recognized are named entities; and determine the first named entity recognition result based on the type labels of the multiple words to be recognized.

[0032] In combination with the second aspect, in certain implementations of the second aspect, the processing module is specifically used to: determine the dynamic word vector and the static word vector mapped to each of the multiple words to be recognized, wherein the static word vector mapped to each word to be recognized in different context sentences has the same vector representation, and the dynamic word vector mapped to each word to be recognized in different context sentences has a different vector representation; based on the dynamic word vectors and the static word vectors mapped to the multiple words to be recognized and a finite state machine, obtain the type labels of the multiple words to be recognized output by the finite state machine, and the input of the finite state machine is the fusion word vector of the dynamic word vector and the static word vector mapped to each of the multiple words to be recognized.

[0033] In combination with the second aspect, in certain implementations of the second aspect, the static word vector mapped to each word to be recognized is determined based on a preset word vector library, which includes static word vectors mapped to multiple words, including the multiple words to be recognized.

[0034] In combination with the second aspect, in certain implementations of the second aspect, the static word vector mapped to each word to be recognized is determined based on a pre-trained static word vector model.

[0035] In combination with the second aspect, in certain implementations of the second aspect, the dynamic word vector mapped to each word to be recognized is determined based on a pre-trained dynamic word vector model.

[0036] In combination with the second aspect, in some implementations of the second aspect, the type tag is further used to indicate the meaning of the word to be recognized and / or the position of the word to be recognized in the named entity.

[0037] In combination with the second aspect, in certain implementations of the second aspect, after the processing module performs named entity recognition on the second natural language text to obtain a first named entity recognition result, the processing module is also used to: based on the first named entity recognition result, determine one or more missing sentences from the one or more sentences that do not contain named entities, the first named entity recognition result includes one or more named entities in the second natural language text, and the one or more missing sentences include at least one of the one or more named entities; perform named entity recognition on the one or more missing sentences to obtain a second named entity recognition result.

[0038] In a third aspect, an embodiment of the present application provides a computing device, which includes a processor, which is used to couple with a memory, read and execute instructions and / or program codes in the memory to execute the first aspect or any possible implementation of the first aspect.

[0039] In a fourth aspect, an embodiment of the present application provides a chip system, which includes a logic circuit, which is used to couple with an input / output interface and transmit data through the input / output interface to execute the first aspect or any possible implementation method of the first aspect.

[0040] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a program code. When the program code runs on a computer, the computer executes the first aspect or any possible implementation method of the first aspect.

[0041] In a sixth aspect, an embodiment of the present application provides a computer program product, which includes: a computer program code, which, when running on a computer, enables the computer to execute the first aspect or any possible implementation of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] FIG1 is a schematic diagram of an application scenario provided in an embodiment of the present application.

[0043] FIG2 is a schematic diagram of a state transition of a finite state machine provided in an embodiment of the present application.

[0044] FIG3 is a schematic flowchart of a named entity recognition method 300 provided in an embodiment of the present application.

[0045] FIG4 is a schematic diagram of a training process and a recognition process of a named entity recognition method 300 provided in an embodiment of the present application.

[0046] FIG5 is a schematic diagram of a named entity recognition method 300 provided in an embodiment of the present application.

[0047] FIG6 is a schematic flowchart of another named entity recognition method 600 provided in an embodiment of the present application.

[0048] FIG7 is a flowchart of a prediction process of another named entity recognition method 600 provided in an embodiment of the present application.

[0049] FIG8 is a schematic diagram of a named entity recognition model training process provided in an embodiment of the present application.

[0050] FIG9 is a schematic diagram of a named entity recognition model recognition process provided in an embodiment of the present application.

[0051] FIG10 is a schematic diagram of another named entity recognition method 600 provided in an embodiment of the present application.

[0052] FIG11 is a flowchart of a prediction process of another named entity recognition method provided in an embodiment of the present application.

[0053] FIG12 is a schematic flowchart of another named entity recognition method 1200 provided in an embodiment of the present application.

[0054] FIG13 is a schematic diagram of another named entity recognition method 1200 provided in an embodiment of the present application.

[0055] FIG14 is a flowchart of another named entity recognition method 1400 provided in an embodiment of the present application.

[0056] FIG15 is a schematic structural diagram of a device for named entity recognition provided in an embodiment of the present application.

[0057] FIG16 is a schematic diagram of the hardware structure of a device for named entity recognition provided in an embodiment of the present application. DETAILED DESCRIPTION

[0058] The technical solution in this application will be described below with reference to the accompanying drawings.

[0059] Before introducing the embodiments, the terms involved in this application are described in detail. The terms used in the implementation methods of this application are only used to explain the specific embodiments of this application and are not intended to limit this application. Some embodiments provided in this application can be applied to speech recognition and natural language processing (NLP) technologies in the field of artificial intelligence (AI), and can be specifically applied to applications such as data storage management and voice assistants.

[0060] 1. Named entity recognition (NER):

[0061] Named entity recognition, also known as "proper name recognition" or "entity recognition", refers to the identification of named entities with specific meaning or strong referentiality in the text to be recognized. A named entity (also known as an entity) can be a word or a phrase. For example, a named entity can refer to a person's name, place name, organization name, date and time in general fields; it can also refer to specific fields, such as genes, proteins, drugs and diseases in the biomedical field; or it can be a broader entity, such as numbers, currencies, addresses, etc. NER is the process of locating and classifying named entities in text into predefined entity categories.

[0062] In the process of named entity recognition, the NER model (also referred to as the NER system) can identify and extract the above-mentioned named entities from the text to be recognized, and can identify more categories of named entities according to business needs. The NER model can adopt a model of an implementation method such as a feature template-based method or a neural network-based method in the existing related technology. In an optional example, the NER model can adopt a model of word embedding + long short-term memory network (long short-term memory, LSTM) / bidirectional long short-term memory network (BiLSTM) + conditional random field (conditional random field, CRF). Optionally, in an embodiment of the present application, the natural language text to be recognized can be obtained from a text file in a data management or data storage system or storage device, and can be a natural language text obtained after converting speech into text. The natural language text to be recognized can include Chinese, numbers, symbols, English and other characters.

[0063] Exemplarily, FIG1 shows a schematic diagram of a named entity recognition scenario provided by an embodiment of the present application. The named entity recognition scenario can be a speech recognition scenario. In an optional example, the named entity recognition method can be applied to a user device equipped with a sound receiver, and the user device can specifically be a mobile terminal (for example, a smart phone), a computer, a personal digital assistant, a wearable device, a vehicle-mounted device, an Internet of Things device, or other electronic device capable of receiving sound. In this example, the user voice collected by the sound receiver is converted into natural language text to be recognized by a data processing device (such as a server) as shown in FIG1. ​​In another optional example, the text to be recognized can be a text manually input by the user through the user device. In this example, the text input by the user is stored in the memory of the user device or the server through a data processing device (such as a server) as shown in FIG1.

[0064] For example, in the named entity recognition scenario shown in FIG1 , the named entity recognition method can also be applied to a data management system or a data storage system. For example, a user accesses a user device or a cloud (which can be a cluster of multiple electronic devices such as a public cloud or a private cloud) that stores user data and enters the data file path to be scanned. Then, a data processing device (such as a server) can scan the data file path entered by the user and query and read the text file content therein to obtain the natural language text to be recognized.

[0065] 2. Finite-state machine (FSM):

[0066] A finite state machine, also known as a finite automatic state machine or state machine, is a mathematical model used to represent a finite number of states and the transitions and actions between them. FSMs can decouple the model's multiple states and the transition conditions between them, reducing program coupling.

[0067] For ease of understanding, FIG2 shows a schematic diagram of the state transition of a finite state machine provided in an embodiment of the present application, wherein the finite state machine can be applied to the named entity recognition process. The form of the finite state machine is shown in FIG2 , where q0, q1, q2, and q3 all represent the states of a finite state machine (FSM), and are also entities and non-entities in the named entity recognition rules. For example, the state of the finite state machine or the type label of the word to be recognized included in the finite state machine model may include a non-entity label (q0) and entity labels such as place name-start (q1), place name-middle (q2), place name-end (q3), or person name-word. Among them, "place name", "person name", etc. in the entity label can be used to describe the meaning of the word to be recognized, and "start", "end", etc. in the entity label can be used to describe the position of the word to be recognized in the named entity. For example, the position of the word to be identified "China" in the named entity "Chaoyang District, Beijing, China" is the starting position, the position of the word to be identified "Beijing" in the named entity "Chaoyang District, Beijing, China" is the middle position or the position of the word to be identified in the named entity "Beijing, China" is the end position, and the position of the word to be identified "Chaoyang District" in the named entity "Chaoyang District, Beijing, China" is the end position.

[0068] As shown in Figure 2, the text to be recognized is "The meeting was held in XX Hotel, Chaoyang District, Beijing, China". When the finite state machine matches "meeting" and "held in" in the text to be recognized, the state or type label of the above two words to be recognized is determined to be "non-entity"; when "China" is matched in the text to be recognized, the state or type label of "China" is determined to be "place name-start". In other words, when the finite state machine matches the word to be recognized "China", it transfers from the "non-entity" state to the "place name-start" state; when "Beijing" and "Chaoyang District" are matched in the text to be recognized, the state or type label of the two words to be recognized is determined to be "place name-middle". In other words, the finite state machine transfers from the "place name-start" state to the "place name-middle" state; when "XX Hotel" is matched in the text to be recognized, the state or type label of the word to be recognized is determined to be "place name-end". In other words, the finite state machine transfers from the "place name-middle" state to the "place name-end" state.

[0069] It is worth noting that the finite state machine shown in Figure 2 is only an example. The finite state machine can also include more states, such as more types of entity tags, such as "company-word". In addition, during the state transition prediction process of the finite state machine, inappropriate transition actions will be automatically filtered out based on the current state of the finite state machine or the type tag of the current word to be recognized. For example, a named entity that appears after a named entity with the type tag "person name-start" cannot have the type tag "place name-middle".

[0070] 3. Word vector:

[0071] Natural language is a language that conforms to grammar, contains information, and is understandable and communicative to humans. For example, the text you are reading right now is natural language. However, when computers process natural language, Chinese characters, letters, punctuation marks, and other symbols must be digitized before they can be input into the computer for subsequent processing. Therefore, words need to be vectorized. Word vector representations mainly include word embedding, one-hot encoding, and distributed representation. For example, one-hot encoding can be a sparse representation [0 0 0 1 0 0…], while distributed representation can be a dense representation using low-dimensional real number vectors of a specific dimension [0.70 0.56 -0.13…].

[0072] Depending on whether the vector representation of a word changes in different contexts or linguistic contexts, word vectors can be divided into static word vectors and dynamic word vectors. For example, the co-occurrence information and contextual information of a word in the entire corpus are aggregated into the vector representation of the word. Then, for any word, its vector representation is constant and does not change with changes in its context. This constant vector representation is a static word vector. For another example, in natural language, the same word may have multiple different meanings, grammatical properties, or attributes in different contexts or linguistic contexts, that is, a word has multiple meanings. In this case, for any word, its vector representation is related to the context, and the vector representation that changes dynamically with the context is a dynamic word vector.

[0073] Currently, commonly used static word embedding pre-training models include Word2Vec and GloVe. These models are trained on large-scale corpora and map each word into a fixed-dimensional vector by learning co-occurrence relationships or contextual information between words. Commonly used dynamic word embedding pre-training models include ELMo. These models, through pre-training and fine-tuning, represent each word as a dynamic vector based on the specific context.

[0074] For example, when there is a sentence "OK, no problem" in the text to be recognized, if a static word vector is used to represent the word "OK", then in the entire text to be recognized, the vector representation of the word can be [0.70 0.56 -0.13 ...] (just an example), to indicate that the meaning of the word is agreement. However, assuming that there is a sentence "The product is good" in the text to be recognized, the word "OK" means good quality. If a static word vector is used, it indicates a different meaning. Therefore, a dynamic word vector [0.22 0.31 -0.66 ...] (just an example) can be used to represent the word "OK" in the sentence "The product is good".

[0075] In current information systems, there is a massive amount of multi-type unstructured data, such as text, images, audio, and video, which cannot be described in a specific structure (such as a table). Text data, including personal information and user data, accounts for a considerable proportion of this data. In this context, named entity recognition (NER) technology is needed to identify and classify named entities in text data in information systems.

[0076] Commonly used named entity recognition methods include rule-based and dictionary-based methods, statistical machine learning-based methods, and deep learning-based methods. Among them, the named entity recognition method based on deep learning algorithms has the best effect. For example, the deep learning-based method can perform end-to-end model training, avoiding the problem of manual selection and feature design. Deep learning algorithms can also be roughly divided into two types: non-language models and language models. Non-language model deep learning algorithms use fixed word vectors as input features and use forward neural networks (multi-layer perceptrons, convolutional neural networks, or recurrent neural networks) for prediction. The speed depends on the size of the neural network model, and the accuracy is significantly better than the machine learning method. On this basis, the language model deep learning algorithm uses a pre-trained language model such as the BERT pre-trained language model to further encode the features of the fixed word vectors, achieving better recognition accuracy. However, the feature encoding of the language model is extremely computationally intensive, and better performance can only be achieved with relevant hardware and hardware acceleration.

[0077] In general, to achieve high accuracy, currently used named entity recognition methods based on deep learning algorithms require heavy neural networks to encode the features of the sentences to be recognized. This is computationally intensive and relies on hardware acceleration for optimal performance. Therefore, how to utilize limited hardware resources to quickly and accurately perform named entity recognition in massive amounts of unstructured text is a pressing challenge.

[0078] In order to solve the above technical problems, an embodiment of the present application provides a method 300 for named entity recognition. Among them, method 300 includes steps 310 to 330. Compared with methods based on neural networks and other methods that require hardware acceleration, the named entity recognition method in method 300 greatly reduces the amount of calculation and improves the speed of named entity recognition. In addition, method 300 can be applied to a computing platform that performs computing tasks through a central processing unit (CPU), or to a computing platform without hardware acceleration. Among them, no hardware acceleration means that it does not contain other hardware for performing computing tasks (such as a graphics processing unit (GPU) or a neural network processor (NPU), etc.) in addition to the central processing unit CPU.

[0079] Step 310: Obtain a first natural language text.

[0080] As mentioned above, in the embodiments of the present application, the first natural language text may be obtained from a text file in a data management or data storage system or storage device, may be text obtained by converting speech to text, or may be text input by a user through a user device. The first natural language text may include Chinese characters, numbers, symbols, English characters, and the like.

[0081] The first natural language text may include multiple words to be recognized, and the words to be recognized may be Chinese words, English words, etc. obtained after word segmentation in the first natural language text. For example, if the first natural language text is "Company A is headquartered in Shenzhen, China," the words to be recognized are "Company A," "Headquarters," "Located in," "China," and "Shenzhen" after word segmentation in the first natural language text.

[0082] It is worth noting that the embodiment of the present application does not impose any limitation on the content of the first natural language text, and the first natural language text can be composed of any language.

[0083] Step 320: Based on the first natural language text and the classification model, filter one or more sentences in the first natural language text that do not contain named entities to obtain a second natural language text.

[0084] In other words, the second natural language text is determined by filtering or screening the first natural language text. When the first natural language text has a large amount of data, the named entity recognition process may take a long time and be inefficient. Therefore, method 300 filters or screens the text data of the first natural language text to obtain a second natural language text with a smaller amount of data, thereby reducing the workload and processing of the named entity recognition algorithm.

[0085] Figure 4 shows a flow chart of a text filtering process provided by an embodiment of the present application, that is, filtering a first natural language text based on a classification model. Wherein, the classification model is used to classify sentences in the first natural language text as containing named entities or not containing named entities. As shown in Figure 4, the embodiment of the present application can be based on the word frequency and word order in the text paragraphs in the training corpus as features to train the classification model, and then, the classification model can determine which sentences in the first natural language text are sentences containing named entities and which sentences are sentences not containing named entities based on the word frequency and word order of the words in the sentences of the first natural language text. In other embodiments of the present application, the embodiment of the present application can also train the classification model based on primary semantic features such as part of speech, root, length of words in the text paragraphs or other high-level semantic features extracted by a pre-training model. The present application does not impose any restrictions on the training process of the classification model.

[0086] As shown in Figure 4, the first natural language text is input into the above-mentioned classification model, and the classification model can classify the sentences in the first natural language text to obtain sentences classified as not containing named entities and sentences classified as containing named entities. Furthermore, the embodiment of the present application can filter out sentences classified as not containing named entities, retain or filter out sentences classified as containing named entities, to obtain the second natural language text, thereby reducing the amount of natural language text data required for named entity recognition.

[0087] For ease of understanding, FIG5 shows a schematic diagram of the text screening process provided by an embodiment of the present application. As shown in FIG5 , the first natural language text includes the four sentences shown on the left side of FIG5 . Then, after the four sentences are input into the classification model, the classification model determines whether there are named entities in the four sentences based on the word frequency, word order, part of speech and word root features of the words appearing in the four sentences. For example, the classification model determines that the sentences "A Company's headquarters is located in Shenzhen, China." and "In the area of ​​energy, Beijing's B Power Company, Ltd. obtained the approval." include named entities (such as the words "A Company", "China", "Shenzhen", "Beijing" and "B Power Company, Ltd." shown in bold and underlined in FIG5 ), and then filters out the two sentences and filters the remaining two sentences in FIG5 that do not contain named entities. Thus, only the unlined sentences shown on the right side of FIG5 , i.e., the unfiltered sentences, need to be subjected to named entity recognition, and the workload is less than that of the entire first natural language text.

[0088] Step 330: Perform named entity recognition on the second natural language text to obtain a first named entity recognition result.

[0089] The first named entity recognition result may be a display of type labels of all words to be recognized in the second natural language text, or may be a display of only words to be recognized whose type labels are named entities in the second natural language text.

[0090] In a data management system or storage system, there may be a large amount of text data. In view of the massive unstructured text data, while method 300 reduces the processing volume of the entity recognition algorithm and further improves the overall processing performance, it is also necessary to ensure a certain accuracy in the named entity recognition process. Therefore, in the named entity recognition process of step 330, an embodiment of the present application proposes a method 600 for named entity recognition. Among them, method 600 includes steps 610 to 630. In method 600, a finite state machine is used as a model for predicting the type label of each word to be recognized, and the computing platform that performs computing tasks through the CPU can efficiently complete the computing tasks of the finite state machine.

[0091] Step 610: Acquire multiple words to be recognized in the second natural language text.

[0092] Among them, after obtaining the second natural language text in step 320, step 610 can be to segment the second natural language text through a data preprocessing process, thereby obtaining a plurality of words to be recognized. Optionally, the segmentation process in the data preprocessing process can be a segmentation method such as a string matching method, an understanding method (such as performing syntactic and semantic analysis while segmenting words) or a statistical method (such as performing statistics on the frequency of word groups in the corpus to achieve segmentation). The goal of the segmentation process is to divide the sentences in the second natural language text into a plurality of segmentations, i.e., a plurality of words to be recognized. The division unit of the sentence division can be a single word or a character. Data preprocessing can also include other text standardization processes, such as text garbled characters removal, removal of redundant spaces and blank lines, unification of simplified and traditional Chinese characters, and unification of uppercase and lowercase characters, etc.

[0093] Step 620: According to the multiple words to be recognized and the finite state machine, obtain type labels of the multiple words to be recognized output by the finite state machine, where the type labels are used to indicate whether the words to be recognized are named entities.

[0094] The type tag may be a tag such as "non-entity" or "place name-start" as shown in FIG2 . In other words, the type tag is a state such as "non-entity" or "place name-start" of a finite state machine. In the embodiments of the present application, other type tags such as "word" may also be included, but this application does not limit this.

[0095] In an embodiment of the present application, the finite state machine can determine the type label of each word based on each word or the word vector mapped to the word to be recognized. For example, the finite state machine can determine the type label of the word to be recognized based on the fixed static word vector mapped to each word to be recognized. For another example, in order to make the accuracy of named entity recognition equal to the accuracy of the current neural network model, the present application can predict the type label of the word to be recognized by performing vector fusion on the dynamic word vector and the static word vector mapped to the word to be recognized. As mentioned above, the static word vectors mapped to different context sentences of the second natural language text for each word to be recognized have the same vector representation, and the dynamic word vectors mapped to different context sentences of the second natural language text for each word to be recognized have different vector representations. The embodiments below will introduce this process in detail.

[0096] It should be understood that the finite state machine model is more CPU-friendly than the neural network model because neural network reasoning requires a large number of parallel floating-point operations, while the CPU is more suitable for performing serial operations, with a lower degree of parallelism than hardware such as GPUs or NPUs, and is insufficiently capable of processing such floating-point operations. Finite state machines operate serially and require less computation for state prediction and transfer, making them more suitable for CPU execution. Therefore, in method 600, the feature encoding and prediction process of the finite state machine-based NER algorithm is more CPU-friendly, allowing the CPU to efficiently complete complex computational tasks.

[0097] Step 630: Determine the first named entity recognition result according to the type labels of the multiple words to be recognized.

[0098] Among them, the type labels of the multiple words to be recognized divide the multiple words to be recognized into named entities (as well as named entity labels indicating the meaning of the words and / or the position in the entity) and non-entities. The named entity recognition result can be to display the type labels of the multiple words to be recognized, or it can only display the words to be recognized whose type labels are named entities.

[0099] The following describes an embodiment of the named entity recognition process using vector fusion prediction described above, with reference to Figures 7 to 10. Method 600 utilizes feature encoding information by combining dynamic and static word vector fusion predictions, maintaining high named entity recognition accuracy with a low computational load.

[0100] FIG7 shows a flow chart of a named entity recognition method involving vector fusion prediction provided by an embodiment of the present application. As shown in FIG7 , after data preprocessing of the second natural language text, such as word segmentation, a plurality of words to be recognized in the second natural language text are obtained. Subsequently, each of the plurality of words to be recognized is mapped to a dynamic word vector and a static word vector. Afterwards, the dynamic word vectors and static word vectors of the plurality of words to be recognized or the fusion vectors of the dynamic word vectors and the static word vectors are input into a finite state machine to obtain type labels of the plurality of words to be recognized output by the finite state machine.

[0101] Optionally, the representation of dynamic word vectors and static word vectors can adopt algorithms such as word embedding in relevant existing technologies or improved algorithms to convert indexes or other types of indexes used to represent division units (such as characters or words) of natural language into multidimensional vectors that can be recognized by machines.

[0102] Optionally, the way to convert (map) the words to be recognized into static word vectors can be the word2vec algorithm based on local context prediction and / or the GloVe regression algorithm based on explicit global co-occurrence information, the essence of which is to aggregate the co-occurrence context information of a word in the entire text to be recognized into the vector representation of the word, and finally form a dictionary containing words and their corresponding vector representations. In other embodiments of the present application, during the mapping process, the corresponding static word vector can be obtained by querying the words to be recognized in the word vector library or dictionary. Additionally, when the word to be queried does not exist in the dictionary, it can be filled with an all-zero vector. Among them, the word vector library or dictionary contains multiple words and the static word vector corresponding to each word in the multiple words.

[0103] Optionally, the method of converting (mapping) the words to be recognized into dynamic word vectors can be based on a pre-trained dynamic word vector model such as ELMo, BERT or convolutional neural network (CNN).

[0104] Optionally, the finite state machine can annotate the type label of each word to be recognized based on the fusion vector result of the static word vector and the dynamic word vector. Exemplarily, the vector fusion method in the embodiment of the present application can be weighted fusion, fusion based on feature selection, etc. For example, the result of the fusion vector can be obtained by adding the static word vector and the dynamic word vector. For another example, the static word vector and the dynamic word vector of each word to be recognized can be weighted averaged according to a certain weight, and the weight can be set according to the different importance of the static word vector and the dynamic word vector. For another example, a static word vector or a dynamic word vector can be selected according to the importance of the feature, and the most important feature vector can be selected through features such as mutual information and correlation coefficient to obtain a more accurate fusion result.

[0105] Figure 8 shows a training process of a named entity recognition model provided by an embodiment of the present application. First, the training corpus is subjected to data preprocessing such as word segmentation to obtain a plurality of words to be recognized in the training corpus. Secondly, the dynamic word vector model and the static word vector model required for the word vector mapping process of this application are obtained by using a pre-trained word vector model or calling a trained word vector model. In other embodiments of the present application, a static word vector library or dictionary obtained by a static word vector model can be used, wherein the static word vector library or dictionary includes words and static word vectors corresponding to the words. Then, according to the dynamic word vector and static word vector of each word in the training corpus and the type label of each word in the training expectation, a finite state machine is trained to obtain a named entity recognition model.

[0106] In the process of training the finite state machine, after the dynamic word vector and the static word vector or the fusion vector of the dynamic word vector and the static word vector are input into the finite state machine, the present application can further perform feature transformation based on the context and the state of the state machine for the current word to be recognized, or in other words, the fused word vector is further adjusted based on the context and the state machine state. Ultimately, the transformed features or the transformed word vectors will be used to predict the type labels of the words to be recognized, and the predicted type labels will be compared with the type labels marked in the corpus, thereby training the finite state machine and the feature transformation process.

[0107] For example, if the training corpus "Tomorrow, May 1st is Labor Day" contains the words "May 1st" and "Labor Day" to be recognized and the annotated type labels are "date-word" and "holiday-word" respectively, then this application can obtain the dynamic word vector and static word vector of the above two words to be recognized through the dynamic word vector model and the static word vector model, and then obtain the fused word vector. After the fused word vector of the above two words to be recognized is input into the finite state machine, the finite state machine will further determine the adjusted fused word vector based on the context of the above two words to be recognized and the state of the finite state machine at this time. For example, the finite state machine predicts that the type label of the word to be recognized "May 1st" is "date-word". Then, when the finite state machine predicts the type label of the word to be recognized "Labor Day", it will adjust the value of the fused word vector of the word to be recognized "Labor Day" based on the correlation between "Labor Day" and "May 1st" and the state of the finite state machine at this time (such as the state at this time can be "date-word" of "May 1st" or "non-entity" of "yes", which can be determined according to the word segmentation method) and use the adjusted fused word vector to predict the type label of the word to be recognized "Labor Day". If the predicted type label of the word to be recognized "Labor Day" is not the marked "holiday-word", the finite state machine can adjust the value of the fused word vector of the word to be recognized "Labor Day" again until the prediction result is the same as the marked result. Finally, a finite state machine or named entity recognition model that meets the prediction accuracy is obtained.

[0108] Figure 9 shows the recognition process of the named entity recognition model provided by an embodiment of the present application. First, data preprocessing is performed on the second natural language text to obtain a plurality of words to be recognized in the second natural language text. Secondly, the dynamic word vector and the static word vector of each word to be recognized mapping in the second natural language text are determined based on the dynamic word vector model, the static word vector model or the static word vector library, and then the fusion vector of the dynamic word vector and the static word vector of each word to be recognized mapping is determined. Then, the fusion vector of each word to be recognized mapping in the second natural language text is input into a finite state machine, which can adjust the fusion vector of each word to be recognized mapping based on the feature transformation process described in the training process, and finally determine the type label of each word to be recognized based on the adjusted fusion vector.

[0109] For ease of understanding, Figure 10 shows a schematic diagram of a named entity recognition process provided by an embodiment of the present application. As shown in Figure 10, the second natural language text includes the sentence "Company A is headquartered in Shenzhen, China." After data preprocessing, the words to be recognized are "Company A", "Headquarters", "Located in", "China" and "Shenzhen". Figure 10 takes the recognition process of the word "Shenzhen" to be recognized as an example, where "Company A" and "China" are both named entities and are stored in the entity stack (that is, all recognized named entities are saved), and the type labels are "Company-Word" and "Place Name-Start" respectively.

[0110] As shown in Figure 10, the static word vector of the word to be identified "Shenzhen" is [0.95, 0.44...] according to the static word vector model or the static word vector library, and the dynamic word vector of the word to be identified "Shenzhen" is [0.36, 0.48...] according to the dynamic word vector model. The dynamic word vector model can determine the dynamic word vector of "Shenzhen" based on the association between the two identified named entities "Company A" and "China" and the word to be identified "Shenzhen".

[0111] Furthermore, the dynamic word vector [0.36, 0.48…] and the static word vector [0.95, 0.44…] of the word to be recognized "Shenzhen" can be fused (e.g., directly added and averaged) to obtain the fused vector [0.66, 0.46…], which is then input into the finite state machine. During the finite state machine's prediction process, the finite state machine can adjust the fused vector of the word to be recognized "Shenzhen" based on the association between the two recognized named entities "Company A" and "China" and the word to be recognized "Shenzhen" and the current state of the finite state machine (i.e., the type label "Place Name-Origin" corresponding to the word to be recognized "China"). The final adjusted fused vector is, for example, [0.64, 0.45…].

[0112] As shown in the prediction results of Figure 10, according to the fusion vector [0.64, 0.45...], the score or probability of the type label of "Shenzhen" being "place name-end" is the highest, and then the finite state machine determines that the type label of "Shenzhen" is "place name-end". Among them, the finite state machine automatically filters out inapplicable type labels based on the type label of the word "China" to be identified. For example, after the finite state machine determines that the state or type label of "China" is "place name-start", it can automatically process states such as "person name-start". In other words, the state of the finite state machine corresponding to "Shenzhen" or the corresponding type label cannot be the above-mentioned "person name-start" etc. In the named entity recognition method provided in the embodiment of the present application, while method 300 greatly reduces the amount of tasks required for named entity recognition, method 600 can ensure that the accuracy of the named entity recognition process is on par with the accuracy of the named entity recognition method based on the neural network. However, in the text screening process of method 300, the classification model may make misjudgments, that is, sentences with named entities may be misclassified as sentences without named entities. Therefore, in view of this, the present application proposes another named entity recognition method based on method 300, in which the re-matching and re-recognition processes are added to reduce omissions in the named entity recognition results. Figure 11 shows a flowchart of the named entity recognition method, and Figure 12 shows a schematic block diagram of the named entity recognition method 1200. Method 1200 includes steps 1210 and 1220, and occurs after step 330 of method 300, that is, after the first named entity result is obtained.

[0113] Step 1210: Based on the first named entity recognition result, determine one or more missing sentences from the one or more sentences that do not contain named entities.

[0114] As shown in Figure 11, after the finite state machine or named entity recognition model outputs the first named entity recognition result, in order to prevent the existence of missed named entities, the present application matches the first named entity result with the filtered text sentences in the first natural language text (or, the classification results filtered in step 320 are sentences that do not contain named entities), for example, using a rule matching algorithm to match, to determine whether there are text sentences that have been incorrectly filtered.

[0115] For ease of understanding, Figure 13 shows a schematic diagram of the rematching and recognition process provided by an embodiment of the present application. As shown in Figure 13, the sentence "Beijing's C Tech Company is a state-owned enterprise." is classified by the classification model as a sentence without a named entity during the named entity recognition process shown in Figure 3 and is therefore filtered out. Furthermore, the first named entity recognition results are "A Company," "China," "Shenzhen," "Beijing," and "B Power Company, Ltd." for the remaining two sentences shown in Figure 13.

[0116] However, in the process of re-matching or rule matching the first named entity recognition result with the filtered sentence, it is determined that the filtered sentence "Beijing's C Tech Company is a state-owned enterprise." includes the named entity "Beijing" in the first named entity recognition result. Therefore, the filtered sentence "Beijing's C Tech Company is a state-owned enterprise." is an omitted sentence that was incorrectly filtered in method 300.

[0117] Step 1220: Perform named entity recognition on the one or more missing sentences to obtain a second named entity recognition result.

[0118] Then, as shown in FIG11 , if there are missing sentences containing named entities that are incorrectly filtered, the present application re-inputs these missing sentences into the finite state machine to obtain a corrected second named entity recognition result. Furthermore, the present application can also match the corrected second named entity recognition result with the filtered text sentences in the first natural language text again, or in other words, based on the corrected second named entity recognition result, perform the re-matching and re-recognition process again until no new named entities appear in the re-matching process, or when the named entity recognition result before re-matching is the same as the named entity recognition result after re-matching, the final accurate named entity recognition result without missing named entities is output.

[0119] For example, the embodiment of the present application re-inputs the missed sentences that were incorrectly filtered, such as "Beijing's C Tech Company is a state-owned enterprise." as shown in Figure 13, into the finite state machine. As shown in Figure 13, after the re-recognition process, "C Tech Company" is newly added to the named entity recognition result, that is, the second named entity recognition result includes the newly added named entity "C Tech Company". Subsequently, the embodiment of the present application can re-match and / or re-recognize the second named entity recognition result containing the newly added named entity with the filtered sentences in the second natural language text. And so on, until there are no newly added named entities, the named entity recognition process is stopped, and the final accurate named entity recognition result is output.

[0120] Optionally, in some other embodiments of the present application, the named entity recognition process of the present application can also be combined with the existing rule matching process to improve the accuracy of the named entity recognition results. Figure 14 shows a schematic flow chart of a named entity recognition method 1400 that combines the named entity recognition process and the rule matching process of the present application. As shown in Figure 14, since the execution time of the rule matching process is much shorter than the recognition process of the named entity recognition algorithm, the rule matching process can be directly performed on the first natural language text. Optionally, before the rule matching process, the user-configured rule library or the internal preset matching rules can be initialized.

[0121] Furthermore, the first named entity recognition result output by the finite state machine is combined with the named entity matching result of the rule matching process to obtain a named entity merge result. Furthermore, a rematching and re-recognition process based on the named entity merge result can be performed. For example, after rematching the named entity merge result, the error-filtered sentence can be re-input into the named entity recognition model and / or rule matching can be performed again. In this way, the newly added rule matching process can improve the accuracy of the named entity recognition process without affecting the execution time and efficiency of the named entity recognition model.

[0122] The method for named entity recognition according to an embodiment of the present application is described in detail above in conjunction with Figures 1 to 14 . The apparatus for named entity recognition according to an embodiment of the present application is described in detail below in conjunction with Figures 15 and 16 . It should be noted that the apparatus for named entity recognition shown in Figures 15 and 16 can implement each step of the above method, and for the sake of brevity, they will not be described here in detail.

[0123] Figure 15 is a schematic diagram of an apparatus for named entity recognition according to an embodiment of the present application. Apparatus 1500 shown in Figure 15 can implement the steps in Figures 3, 6, 12, and 14, or in other words, can implement the steps in the embodiments. Apparatus 1500 shown in Figure 15 includes an acquisition module 1510 and a processing module 1520.

[0124] The acquisition module 1510 is configured to: acquire a first natural language text;

[0125] Processing module 1520 is configured to: based on the first natural language text and the classification model, filter one or more sentences in the first natural language text that do not contain named entities to obtain a second natural language text, the second natural language text including the one or more sentences in the first natural language text that contain named entities, the classification model being configured to classify sentences in the first natural language text as containing named entities or not containing named entities;

[0126] The processing module 1520 is further configured to perform named entity recognition on the second natural language text to obtain a first named entity recognition result.

[0127] In an optional embodiment, the acquisition module 1510 and the processing module 1520 may be a processor 1620 , and the apparatus may further include an input / output interface 1630 and a memory 1610 , as specifically shown in FIG. 16 .

[0128] FIG16 is a schematic block diagram of an apparatus according to another embodiment of the present application. The apparatus 1600 shown in FIG16 may include a memory 1610, a processor 1620, and an input / output interface 1630. The memory 1610, the processor 1620, and the input / output interface 1630 are connected via a communication connection. The memory 1610 is used to store program instructions, and the processor 1620 is used to execute the program instructions stored in the memory 1610 to control the input / output interface 1630 to receive input data and information and output data such as operation results. The data and information received by the input / output interface 1630 may be stored on a hard disk.

[0129] It should be understood that in the embodiment of the present application, the processor 1620 can adopt a general central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), or one or more integrated circuits to execute relevant programs to implement the technical solutions provided in the embodiment of the present application.

[0130] The memory 1610 may include a read-only memory and a random access memory, and provides instructions and data to the processor 1620. A portion of the processor 1620 may also include a non-volatile random access memory. For example, the processor 1620 may also store information about the device type.

[0131] During implementation, each step of the above method can be completed by an integrated logic circuit of the hardware in the processor 1620 or by instructions in the form of software. The method disclosed in conjunction with the embodiments of the present application can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor. The software module can be located in a mature storage medium in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 1610, and the processor 1620 reads the information in the memory 1610 and completes the steps of the above method in combination with its hardware. To avoid repetition, it will not be described in detail here.

[0132] It should be understood that in the embodiments of the present application, the processor may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0133] It should also be understood that in the embodiments of the present application, the hard disk drive (HDD) serves as one of the storage media for the primary site and the backup site, and can be a solid state disk (SSD), a mechanical hard disk (mechanical hard disk), a hybrid hard disk (SSHD), etc.

[0134] It should also be understood that in the embodiments of the present application, the memory may include read-only memory and random access memory, and provide instructions and data to the processor. A portion of the processor may also include non-volatile random access memory. For example, the processor may also store device type information.

[0135] It should be understood that in the embodiments of the present application, "B corresponding to A" means that B is associated with A and B can be determined based on A. However, it should also be understood that determining B based on A does not mean determining B based solely on A, but B can also be determined based on A and / or other information.

[0136] It should be understood that the term "and / or" in this document simply describes a relationship between related objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this document generally indicates that the related objects are in an "or" relationship.

[0137] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0138] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the modules is merely a logical function division. In actual implementation, there may be other division methods, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0139] The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules may be selected to achieve the purpose of the present embodiment according to actual needs.

[0140] In addition, each functional module in each embodiment of the present application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0141] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that can be read by a computer or a data storage device such as a server or data center that includes one or more available media integrated therein. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a digital versatile disc (DVD)), or a semiconductor medium (eg, a solid state disk (SSD)).

[0142] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A method for named entity recognition, characterized in that: include: Obtaining a first natural language text; Based on the first natural language text and the classification model, filtering one or more sentences in the first natural language text that do not contain named entities to obtain a second natural language text, the second natural language text including the one or more sentences in the first natural language text that contain named entities, the classification model being used to classify sentences in the first natural language text as containing named entities or not containing named entities; Perform named entity recognition on the second natural language text to obtain a first named entity recognition result.

2. The method according to claim 1, characterized in that The method is applied to a computing platform that executes computing tasks through a central processing unit (CPU).

3. The method according to claim 1 or 2, characterized in that The classification model determines whether a sentence contains a named entity based on the word frequency and word order of the words in the sentence.

4. The method according to any one of claims 1 to 3, characterized in that The performing named entity recognition on the second natural language text to obtain a first named entity recognition result includes: Obtaining a plurality of words to be recognized in the second natural language text; According to the plurality of words to be recognized and a finite state machine, obtaining type labels of the plurality of words to be recognized output by the finite state machine, wherein the type labels are used to indicate whether the words to be recognized are named entities; The first named entity recognition result is determined according to the type labels of the multiple words to be recognized.

5. The method according to claim 4, characterized in that The step of obtaining type labels of the plurality of words to be recognized output by the finite state machine according to the plurality of words to be recognized and the finite state machine includes: Determining a dynamic word vector and a static word vector mapped to each of the multiple words to be recognized, wherein the static word vector mapped to each word to be recognized in different context sentences has the same vector representation, and the dynamic word vector mapped to each word to be recognized in different context sentences has a different vector representation; According to the dynamic word vectors and static word vectors mapped to the multiple words to be recognized and a finite state machine, the type labels of the multiple words to be recognized output by the finite state machine are obtained, and the input of the finite state machine is a fusion word vector of the dynamic word vector and the static word vector mapped to each word to be recognized in the multiple words to be recognized.

6. The method according to claim 5, characterized in that The static word vector mapped to each word to be recognized is determined based on a preset word vector library, which includes static word vectors mapped to multiple words, and the multiple words include the multiple words to be recognized.

7. The method according to claim 5, characterized in that The static word vector of each word to be recognized is determined based on the pre-trained static word vector model.

8. The method according to any one of claims 5 to 7, characterized in that The dynamic word vector of each word to be recognized is determined based on the pre-trained dynamic word vector model.

9. The method according to any one of claims 4 to 8, characterized in that The type tag is also used to indicate the meaning of the word to be recognized and / or the position of the word to be recognized in the named entity.

10. The method according to any one of claims 1 to 9, characterized in that After performing named entity recognition on the second natural language text to obtain a first named entity recognition result, the method further includes: determining, based on the first named entity recognition result, one or more missing sentences from the one or more sentences not containing named entities, wherein the first named entity recognition result includes one or more named entities in the second natural language text, and the one or more missing sentences include at least one of the one or more named entities; Perform named entity recognition on the one or more missing sentences to obtain a second named entity recognition result.

11. A device for named entity recognition, characterized in that: include: An acquisition module, configured to: acquire a first natural language text; a processing module configured to: based on the first natural language text and a classification model, filter one or more sentences in the first natural language text that do not contain named entities to obtain a second natural language text, wherein the second natural language text includes the one or more sentences in the first natural language text that contain named entities, and the classification model is configured to classify sentences in the first natural language text as containing named entities or not containing named entities; The processing module is further configured to perform named entity recognition on the second natural language text to obtain a first named entity recognition result.

12. The device according to claim 11, characterized in that The acquisition module is specifically configured to: acquire a plurality of words to be recognized in the second natural language text; The processing module is specifically configured to: obtain type labels of the multiple words to be recognized output by the finite state machine according to the multiple words to be recognized and the finite state machine, wherein the type labels are used to indicate whether the words to be recognized are named entities; The first named entity recognition result is determined according to the type labels of the multiple words to be recognized.

13. The device according to claim 12, characterized in that The processing module is specifically used for: Determining a dynamic word vector and a static word vector mapped to each of the multiple words to be recognized, wherein the static word vector mapped to each word to be recognized in different context sentences has the same vector representation, and the dynamic word vector mapped to each word to be recognized in different context sentences has a different vector representation; According to the dynamic word vectors and static word vectors mapped to the multiple words to be recognized and a finite state machine, the type labels of the multiple words to be recognized output by the finite state machine are obtained, and the input of the finite state machine is a fusion word vector of the dynamic word vector and the static word vector mapped to each word to be recognized in the multiple words to be recognized.

14. The device according to any one of claims 11 to 13, characterized in that After the processing module performs named entity recognition on the second natural language text to obtain a first named entity recognition result, the processing module is further configured to: determining, based on the first named entity recognition result, one or more missing sentences from the one or more sentences not containing named entities, wherein the first named entity recognition result includes one or more named entities in the second natural language text, and the one or more missing sentences include at least one of the one or more named entities; Perform named entity recognition on the one or more missing sentences to obtain a second named entity recognition result.

15. A named entity recognition device, characterized in that include: A processor, wherein the processor is coupled to a memory, and is configured to read and execute instructions and / or program codes in the memory to perform the method according to any one of claims 1 to 10.

16. A chip system, characterized in that: include: A logic circuit, the logic circuit being configured to be coupled to an input / output interface and to transmit data via the input / output interface, so as to execute the method according to any one of claims 1 to 10.

17. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is run on a computer, the computer is caused to perform the method according to any one of claims 1 to 10.

18. A computer program product, characterized in that The method comprises a computer program code, which, when executed, implements the method according to any one of claims 1 to 10.

Images