Entity type identification for named entity recognition systems

Abstract

Methods, apparatuses, and systems are provided for entity type recognition and / or disambiguation of entities within a text corpus, the method comprising: receiving one or more entity results, each entity result including data representing an identified entity and a location of the identified entity within the text corpus; identifying an entity type for each entity in the received entity results by inputting text in the text corpus associated with the location of the each entity to a trained entity type (ET) model, the trained ET model configured to predict or extract an entity type for the each entity from the text corpus; and outputting data representing the identified entity type for each entity in the received entity results.

Description

Technical Field

[0001] This application relates to a system and method for performing entity type identification in a named entity recognition system when identifying entities from a large dataset, such as a text corpus. Background Technology

[0002] Entity identification within large-scale datasets, such as data or text corpora in one or more domains, for example, bioinformatics or chemistry information literature (e.g., publications stored by various sources, such as PubMed, as instances, but not limited to) as instances, is important for identifying entity types, entities of interest, and / or corresponding entity relationships between them. Large-scale datasets or data / text corpora can include or represent any information or data from one or more data sources, content sources, content providers, etc., and can contain, but are not limited to, unstructured data / text, structured data / text, text content, articles, publications, documents, text, emails, images and / or videos as instances, or any other information or data that may contain a large amount of information. This data may be compiled, generated by, and / or stored with or with one or more sources, content sources / providers, or multiple sources (e.g., PubMed, MEDLINE, Wikipedia), and can be used to form large-scale datasets or data / text corpora from which entities, entity types, and relationships of interest can be extracted. Such large-scale datasets or data / text corpora may contain data or information from one or more data sources, each of which provides data representing multiple unstructured and / or structured text / documents, articles, or documents, etc. Although most documents, articles, or documents from publishers, content providers / sources have specific document formats / structures—for example, PubMed documents are stored as XML with information about authors, journals, publication dates, and chapters and paragraphs within the document—such documents can be considered part of a data / text corpus. For simplicity, large-scale datasets or data / text corpora are referred to herein as examples, but not limited to, text corpora.

[0003] Traditional Named Entity Recognition (NER) systems can be used to identify and extract entities, entity types, and / or the location of entities / entity types within a text corpus. These NER systems use manually curated entity dictionaries based on multiple sources of the text corpus. Each entity dictionary can store entity names for multiple specific entity types and is used in conjunction with text matching techniques to identify and extract the name and type of "entity" from a text or document corpus. For example, in a bioinformatics context, an entity named "leukemia" with the "disease" type can be identified from the text within a corpus containing, but not limited to, PubMed / MEDLINE / Wikipedia publications associated with "leukemia" as instances. For these NER systems to function, the entity dictionaries need to be continuously updated, curated, and annotated manually to reliably match and identify known entities within different parts of the text corpus. As the subjects of literary or scientific research increase and the resulting text corpora grow, such NER systems are rarely up-to-date because they cannot accommodate the latest, created, or discovered entities; or recognize or adapt to any non-standardized or alternative uses of current entity terms and / or entity types. This can lead to unfavorable, incorrect, or ambiguous entity results for downstream processes that rely on NER.

[0004] Furthermore, NER systems that typically use curated entity dictionaries to identify and extract entities from text corpora are generally unaffected by context and rely heavily on the correct curation and updating of the entity dictionary. However, entity results can be ambiguous when entities of different types have the same name in different documents. This means that when a string containing a specific entity and entity type in the text actually refers to the same named entity of a different entity type or subtype, such NER systems may misidentify the string. Alternatively, when ambiguity arises, such NER systems may default to a specific entity type, hoping that the most common use of that entity type is correct. Therefore, entity results from dictionary-based NER systems where the identified entities may have more than one entity type may select the wrong entity type for the identified entities. For example, the chemical element silver can be simply referred to as "Ag," which can be defined as a chemical type entity, while a protein pathway antigen can also be simply referred to as "Ag," which can be defined as a protein type entity. Therefore, the classic / traditional NER system may incorrectly identify the string "Ag" in some text as silver—a chemical type entity, when in fact this string "Ag" may be associated with an antigen—a protein type entity.

[0005] While machine learning (ML)-based NER systems take into account the context of entities, these systems rely on the correct identification of entities and entity types to generate extensive training datasets, enabling ML techniques to produce reliable or robust NER models for identifying entities and / or entity types. Similarly, generating extensive training datasets often relies on dictionary-based NER systems and manual grooming and annotation. Even though this allows for rapid generation of training datasets, using such dictionary-based NER systems with manual grooming means that the training dataset being generated and the NER model being created are outdated. This increases the risk that the NER model may fail to adapt to or correctly identify the context of recently created or discovered entities and / or entity types; and / or fail to recognize the context of current entity terms and / or entity types or adapt to any non-normalized or alternative uses of current entity terms and / or entity types, etc. This further leads to unfavorable, incorrect, or ambiguous entity results for downstream processes that rely on NER.

[0006] Furthermore, even though NER systems using machine learning (ML) techniques are thought to provide improved classification of entities and / or entity types by taking into account the context of entities within a document through a training dataset containing text surrounding instances of the name of each known entity, this specificity often leads to overfitting of the learned entities and entity types, as well as biased ML-based NER models. That is, ML-based NER systems learn the most common and prevalent entity type for a given entity name and typically bias their predictions towards this entity type. Since entity results may be fed back as additional training data to update the ML-based NER system, this further amplifies the overfitting of one or more NER models, generating inaccurate and / or biased entity recognition results, as the system may fail to properly learn to adapt to the context of the text surrounding entities it encounters in updated text corpora. In effect, the predictions ultimately provided by such systems are more akin to simple probability estimation or dictionary-based NER systems.

[0007] There is a need for a more efficient and robust entity and entity type identification and / or disambiguation system for use with ML-based NER systems. This system should avoid the biases that overfitting can cause in predicting and / or identifying entities and entity types from text corpora. Therefore, such a system could provide more accurate entity / entity type predictions.

[0008] The embodiments described below are not limited to implementations that address any or all of the drawbacks of the known methods described above. Summary of the Invention

[0009] The purpose of providing this summary is to introduce, in a simplified form, a series of concepts further described below in the detailed description. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to define the scope of the claimed subject matter; various variations and alternative features that facilitate the work of this invention and / or achieve substantially similar technical effects should be considered to fall within the scope of the invention disclosed herein.

[0010] This disclosure provides an entity type (ET) recognition or disambiguation system based on machine learning (ML) techniques. This system can minimize or avoid overfitting of named entity recognition (NER) systems using ML-based techniques, while improving the accuracy of the resulting entity and / or entity type recognition. The NER system is coupled to an ET recognition / disambiguation system, wherein the NER system outputs a set of entity results for further processing by the ET recognition / disambiguation system using an ET recognition model. The ET recognition model is configured to predict and identify the most probable entity type associated with one or more entities from the received set of entity results using relevant portions of text from a corpus of text associated with the entity results.

[0011] For example, ML techniques are used to generate ET models (ET recognition models or ET models) configured to supplement NER systems used on text corpora. ML techniques are trained to generate ET models that identify the types of text that appear before and after entities, such as entity names or strings, within a portion of text (e.g., a document), and provide the most probable entity type among several different entity types associated with said entity. In this way, the ET model is configured to contextualize the occurrence of entity names, thereby better predicting the entity type of each entity. This information can then be further used by the NER system to help more accurately predict the entity type of entities identified within the text corpus.

[0012] In a first aspect, this disclosure provides a computer-implemented method for entity type recognition of entities within a text corpus, the method comprising: receiving one or more entity results, each entity result including data representing an identified entity and the location of the identified entity within the text corpus; identifying the entity type of each entity in the received entity results by inputting text in the text corpus associated with the location of each entity into a trained entity type (ET) model, the trained ET model being configured to predict or extract the entity type of each entity from the text corpus; and outputting data representing the identified entity type of each entity in the received entity results.

[0013] Preferably, the input text associated with the position of each entity further includes inputting text from the text corpus before and after the position of each entity.

[0014] Preferably, receiving the one or more entity results further includes receiving the one or more entity results generated by a Named Entity Recognition (NER) system.

[0015] Preferably, each entity result, which includes data representing the identified entity and its position within the text corpus, comprises one or more of the following groups: an indication of the identified entity and its position within the text corpus; the identified entity and a portion of text from the text corpus surrounding the identified entity; the identified entity, a first portion of text preceding the position of the identified entity, and a second portion of text following the position of the identified entity; a portion of text surrounding the position of the identified entity; and a first portion of text preceding the position of the identified entity and a second portion of text following the position of the identified entity.

[0016] Preferably, the computer-implemented method further includes: receiving one or more entity results from a NER system, wherein each entity result includes a first text portion representing the text corpus preceding the position of the identified entity and a second text portion representing the text corpus following the position of the identified entity; identifying the entity type of each entity in the received entity results by inputting the first and second text portions associated with the identified entity in the text corpus into the trained entity type ET model, the trained ET model being configured to predict or extract the entity type of each entity from the text corpus; and sending data representing the identified entity type of each entity in the received entity results to the NER system.

[0017] Preferably, the computer-implemented method further includes: receiving one or more entity results from a NER system, wherein each entity result contains data representing an identified entity, an identified entity type, and the location of the identified entity within the text corpus; identifying an ambiguous entity type among the multiple entity results if multiple entity results associated with a relevant text portion represent the same entity, wherein at least one of the multiple entity results has a different entity type from another of the multiple entity results; for each of the multiple entity results, inputting text in the text corpus associated with the location of each entity into the trained ET model, the trained ET model being configured to predict or extract the entity type of each entity from the text corpus; and aggregating the predicted or extracted entity types of the multiple entity results associated with the relevant text portion to output an entity type representing the multiple entity results.

[0018] Preferably, the relevant text portion is located in a document from the text corpus, and the predicted or extracted entity type of the aggregated entity results for the document includes aggregating the prediction of each entity in the aggregated entity results to form an overall prediction of the entity type of the entity in the aggregated entity results.

[0019] Preferably, the computer implementation method further includes generating or updating the ET model by training machine learning (ML) techniques to predict or extract entity types from the text corpus based on the one or more entity results.

[0020] Preferably, the ML technique includes at least one ML technique from the group consisting of: neural networks; recurrent neural networks; feedforward neural networks; convolutional neural networks; long short-term memory (LSTM) neural networks; bidirectional LSTM neural networks; LSTM-based conditional random field (CRF) neural networks; and any other ML technique for predicting entity types from a text corpus based on the one or more entity results.

[0021] Preferably, the computer-implemented method further includes updating the ET model based on the identified entity type.

[0022] Preferably, the computer implementation method further includes training the ML technique based on a labeled dataset to generate or update the ET model, wherein the labeled dataset includes a plurality of labeled data items, wherein each labeled data item includes a text portion from the text corpus that is associated with an entity and labeled with an entity type.

[0023] Preferably, the text portion associated with an entity from the text corpus includes a first text portion, a second text portion representing the entity, and a third text portion following the entity.

[0024] Preferably, each tagged data item includes the first and third text portions surrounding the second text portion representing the entity.

[0025] Preferably, training the ML technique to generate or update the ET model further comprises, for each iteration, iteratively training the ML technique based on the labeled dataset to generate or update the ET model by: comparing the labeled data items with the corresponding predicted or extracted entity types output from the ET model; and updating the ET model based on the comparison, using a loss function associated with the ML technique.

[0026] Preferably, the ET model is based on a neural network that includes forward and backward hidden states configured to represent a first text portion preceding the position of the entity and a second text portion following the position of the entity, wherein the forward and backward hidden states are concatenated to generate entity type representations of the first and second text portions.

[0027] Preferably, the ET model further includes a first part and a second part of the ET model, the first and second parts of the ET model being configured to predict or extract data representing entity types from the text corpus based on the one or more entity results, the method further comprising: segmenting a text portion associated with an entity in the entity results into a first text portion preceding the position of the entity and a second text portion following the position of the entity; inputting the first text portion into the first part of the ET model to predict data representing a first entity type estimate; inputting the second text portion into the second part of the ET model to predict data representing a second entity type estimate; and combining the data representing the first and second entity type estimates to form an identified entity type associated with the entity.

[0028] Preferably, the data representing the first entity type estimate includes a first N-dimensional vector, and the data representing the second entity type estimate includes a second N-dimensional vector, wherein combining the data representing the first and second entity types further includes concatenating the first and second N-dimensional vectors and reducing the dimension of the resulting vector to the number of potential entity types, wherein each vector element of the resulting vector corresponds to a different entity type, and the value of each vector element represents the probability that the entity is the specific entity type.

[0029] Preferably, the ET model is generated from ML technology based on a bidirectional LSTM neural network.

[0030] Preferably, the model parameters of the ET model are jointly trained based on multiple labeled data items, each labeled data item including data representing a first text portion before the entity's position and labeled with an entity type associated with the entity, and a second text portion after the entity's position and labeled with an entity type associated with the entity.

[0031] In a second aspect, this disclosure provides machine learning (ML) models obtained from computer implementation methods as claimed herein and / or as described herein, in accordance with the first aspect, combinations thereof, and / or modifications thereof.

[0032] In a third aspect, this disclosure provides an apparatus comprising: a receiver configured to receive one or more entity results, each entity result including data representing the location of an entity and an identified entity within a text corpus; an entity type identification module including an entity type model configured to predict or extract an entity type of each of the one or more entities in the received entity results based on text associated with the identified entity in the text corpus; and a transmitter configured to output data representing the predicted or identified entity type of each entity in the received entity results.

[0033] Preferably, the entity type model is configured to receive text associated with the position of each entity, wherein the text associated with the position of each entity includes a portion of text in the text corpus preceding the position of each entity and a portion of text following the position of each entity.

[0034] Preferably, the device is further configured to implement a computer-based method according to the first aspect, combinations thereof, and / or modifications thereof, as claimed herein, and / or as described herein.

[0035] In a fourth aspect, this disclosure provides an apparatus including a processor, a memory unit, and a communication interface, wherein the processor is connected to the memory unit and the communication interface, and wherein the processor and memory are configured to implement computer implementation methods according to the first aspect, combinations thereof, and / or modifications thereof, as claimed herein, and / or as described herein.

[0036] In a fifth aspect, this disclosure provides a system comprising: a named entity recognition (NER) system for generating entity results from a text corpus, each entity result including data representing an identified entity and the location of the identified entity within the text corpus; and a device, coupled to the NER system, configured to receive one or more entity results from the NER system, according to any of the third or fourth aspects, combinations and / or modifications thereof, as claimed herein and / or as described herein.

[0037] In a sixth aspect, this disclosure provides a computer-readable medium including data or instruction code that, when executed on a processor, causes the processor to implement a computer-implemented method according to the first aspect, combinations thereof, and / or modifications thereof, as claimed herein, and / or as described herein.

[0038] In a seventh aspect, this disclosure provides a tangible computer-readable medium including data or instruction code that, when executed on a processor, causes the processor to implement a computer-implemented method according to the first aspect, combinations thereof, and / or modifications thereof, as claimed herein, and / or as described herein.

[0039] In an eighth aspect, this disclosure provides an ET model obtained from computer implementation methods according to the first aspect, combinations therewith and / or modifications thereof, as claimed herein and / or as described herein.

[0040] Preferably, according to any one, combination and / or modification of the first, second, third, fourth, fifth, sixth, seventh and / or eighth aspects, as claimed herein and / or as described herein, a computer-implemented method, apparatus, system or computer-readable medium, wherein the entity includes entity data associated with entity types from the group consisting of: genes; diseases; compounds / drugs; proteins; chemicals, organs, organisms; or any other entity types associated with bioinformatics or cheminformatics, etc.

[0041] The methods described herein can be executed by software in a machine-readable form on a tangible storage medium, such as a computer program, which includes computer program code components adapted to perform all steps of any of the methods described herein when the program is run on a computer and when the computer program can be embodied on a computer-readable medium. Examples of tangible (or non-transitory) storage media include disks, thumb drives, memory cards, etc., and do not include propagated signals. The software can be adapted to execute on a parallel or serial processor, such that the method steps can be executed in any suitable order or simultaneously.

[0042] This application confirms that firmware and software can be valuable, separately tradable commodities. It aims to cover software that runs on or controls “dumb” or standard hardware to achieve desired functionality. It also aims to cover software that “describes” or defines the configuration of hardware, such as HDL (Hardware Description Language) software, such as that used to design silicon chips or configure general-purpose configurable chips to achieve desired functionality.

[0043] As will be apparent to those skilled in the art, preferred features may be suitably combined and may be incorporated into any aspect of the invention. Attached Figure Description

[0044] Embodiments of the invention will be described by way of example only with reference to the following figures, in which:

[0045] Figure 1a This is a schematic diagram illustrating an example entity type recognition system according to the present invention;

[0046] Figure 1b This is a schematic diagram illustrating another example entity type (ET) identification system according to the present invention;

[0047] Figure 1c This is a schematic diagram illustrating an example ET recognition system for entity type disambiguation according to the present invention;

[0048] Figure 1d This is a flowchart illustrating an example process for performing ET identification according to the present invention;

[0049] Figure 1e This is a flowchart illustrating another example process for ET identification for entity type disambiguation according to the present invention;

[0050] Figure 1f This is a flowchart illustrating an example process of ET aggregation during ET identification according to the present invention;

[0051] Figure 2a This is a schematic diagram illustrating an example named entity recognition (NER) system based on an entity dictionary for generating entity results from a text corpus;

[0052] Figure 2b This is a schematic diagram illustrating an example NER system based on a machine learning model for generating entity results from a text corpus;

[0053] Figure 3a This is a schematic diagram illustrating an example ET machine learning (ET-ML) system according to the present invention when trained for an ET recognition system;

[0054] Figure 3b This illustrates the application of the present invention in training for an ET identification system. Figure 3aA schematic diagram of an example ET-ML system;

[0055] Figure 3c This is a schematic diagram illustrating an ET-ML model used with an ET-ML system according to the present invention;

[0056] Figure 3d This illustrates a system for training an ET recognition system according to the present invention. Figures 3a to 3c A flowchart of the process of any of the ET-ML models;

[0057] Figure 3e This illustrates the application of the present invention in training for an ET identification system. Figure 3d A schematic diagram of an example of the ET-ML model;

[0058] Figure 4 This illustrates the basis according to the present invention. Figures 3a-3e Another example of the ET-ML model is a schematic diagram of an ET recognition system;

[0059] Figure 5 This is a schematic diagram illustrating an example ET-ML model based on a bidirectional long short-term memory neural network for an ET identification system according to the present invention;

[0060] Figure 6a This is a schematic diagram illustrating a computing system / device according to the present invention; and

[0061] Figure 6b This is a schematic diagram illustrating the system according to the present invention.

[0062] Common reference numerals are used in all figures to indicate similar features. Detailed Implementation

[0063] The following description of embodiments of the invention is by way of example only. These examples represent the best mode of practicing the invention as currently known to the applicant, but they are not the only ways to implement the invention. The description illustrates the function of the examples and the sequence of steps for constructing and operating the examples. However, the same or equivalent functions and sequences can be accomplished by different examples.

[0064] For example, entity recognition within data or text in bioinformatics or cheminformatics literature is crucial for identifying and extracting entities of interest and / or entity types, corresponding entities, and relationships between them. Entity recognition information can be used in subsequent downstream processes, such as relationship extraction and generating bioinformatics and / or cheminformatics models from labeled training data based on identified entities, entity types, and relationships between entities using machine learning techniques.

[0065] Researchers and scientists are constantly adding to the human knowledge system across numerous fields and scientific domains (e.g., bioinformatics and / or cheminformatics), resulting in a vast, exponentially growing corpus of text / document data (or large-scale datasets). Examples include, but are not limited to, structured / unstructured text, documents, articles, publications, research-related text and / or documents; conference and journal papers; published patents / patent applications; news articles; white papers; website / blog posts; and / or any other source of useful information. Given the sheer volume of bioinformatics and / or cheminformatics literature, Named Entity Recognition (NER) systems are commonly used in text mining and other fields to identify and extract identified entities, entity types, and / or the location of entities and / or entity types within text. These NER systems typically use manually annotated entity dictionaries to identify entities within text. In medical sciences such as bioinformatics and / or cheminformatics, entity types can be a range of categories, such as genes, diseases, chemicals, proteins, cell lines, etc. As research in these fields progresses, the amount of unstructured data, text, and / or documents being generated is constantly increasing, thus the number of entities and / or entity types is also increasing. Consequently, there are increasingly more entities that may have the same name or string but belong to several different entity types. This leads to ambiguity in identifying entities and / or entity types from the ever-growing text / data corpora.

[0066] While some ML-based NER systems can provide improved entity classification by examining the text surrounding each instance of an entity name to account for the context of entities within the text / document, this can lead to overfitting of the learned entity types. In this case, the ML-based NER system learns the most common entity type for a given entity name and biases its predictions towards this type. Given that this can feed back into the ML-based NER system, this can result in inaccurate entity recognition results because the system may fail to properly learn the context of the text surrounding the entities. In effect, the predictions provided by such systems are more akin to simple probability estimates.

[0067] This invention provides a solution to the aforementioned key problems by offering a more robust ML-based Entity Type (ET) recognition or disambiguation system that minimizes or avoids overfitting of ML-based NER systems while improving the accuracy of entity and / or entity type recognition generated therefrom. The invention aims to overcome these key problems through a sensible combination or coupling of NER systems with ET recognition / disambiguation systems, wherein the NER system outputs a set of entity results for further processing by the ET recognition / disambiguation system using an ET recognition model. The ET recognition model is configured to predict and identify the most probable entity type associated with one or more entities from the received set of entity results using relevant portions of text from a corpus of text associated with the entity results.

[0068] Specifically, ML techniques are used to generate an ET model (ET recognition model or ET model) configured to supplement a NER system used on a text corpus. The ML techniques are trained to generate an ET model that identifies the types of text that appear before and after entities, such as entity names or strings, within a portion of text (e.g., a document), and provides the most probable entity type among several different entity types associated with said entity. In this way, the ET model is configured to contextualize the occurrence of entity names, thereby enabling better prediction of the entity type for each entity. This information can then be further used by the NER system to help more accurately predict the entity type of entities identified within the text corpus.

[0069] An ET model or ET recognition model is trained and generated using one or more ML techniques using a labeled training dataset containing multiple labeled training data elements. Each labeled training data element is associated with one of a plurality of entities and contains data representing a text portion associated with said entity and labeled with the entity type associated with said entity. For each labeled training data element associated with an entity, the text portion consists of text or characters surrounding the string associated with said entity.

[0070] For example, a text portion labeled with an entity type tag may contain: a) text or characters preceding the text or string associated with the entity; and b) text or characters following the text or string associated with the entity; but not the text or string associated with the entity. In another instance, a text portion containing a named entity and labeled with an entity type tag may contain surrounding text of the named entity, wherein the surrounding text is segmented into text appearing "to the left" and "to the right" of the named entity, where an ML technique or system is applied to the result. The surrounding text of the text portion may not contain the entity name or string of the named entity, such that the ML technique or system does not consider the named entity during training. It has been found that omitting the text or string associated with the entity (or named entity) in each training data element further minimizes or avoids the ML technique generating an overfitted entity type recognition model, thereby further reducing the bias of the entity type recognition model. It has been found that this improves the accuracy of predicting the correct entity type of entities from the text corpus of input to the resulting ET recognition model.

[0071] ML techniques are used to train and generate one or more trained models or classifiers with the same or similar output targets associated with input data. ML techniques may include or represent one or more computational methods or combinations thereof that can be used to generate analytical models, classifiers, and / or algorithms that help solve complex problems, such as, but not limited to, efficient and accurate large-scale data mining, involving text mining, prediction, analysis, and extraction of entities and / or entity types and entity relationships from large-scale datasets (e.g., corpora or collections of text / documents or unstructured data) to generate large-scale labeled training datasets for ML techniques, thereby generating further trained models for downstream processes. Downstream processes may include, but are not limited to, training models or classifiers and / or analytical models related to complex processes and / or compounds; and classification of input data related to one or more relationships. With properly labeled training datasets in the fields of cheminformatics and / or bioinformatics, ML techniques can be used to generate further trained models, classifiers, and / or analytical models for use in downstream processes, such as, but not limited to, drug discovery, identification, and optimization, as well as other relevant biomedical products, treatments, analyses, and / or modeling in the fields of informatics, cheminformatics, and / or bioinformatics.

[0072] As described herein, examples of ML techniques that may be used by this invention may include or be based on, by way of example only, but not limited to, any ML technique or algorithm / method that can be trained on labeled and / or unlabeled datasets to generate models or classifiers associated with labeled and / or unlabeled datasets, one or more supervised ML techniques, semi-supervised ML techniques, unsupervised ML techniques, linear and / or nonlinear ML techniques, ML techniques associated with classification, ML techniques associated with regression, and so on, and / or combinations thereof. Some examples of ML techniques may include or be based on, by way of example only, but not limited to, one or more of the following: active learning, multi-task learning, transfer learning, neural message parsing, one-time learning, dimensionality reduction, decision tree learning, association rule learning, similarity learning, data mining algorithms / methods, artificial neural networks (NN), deep NN, deep learning, deep learning ANN, inductive logic programming, support vector machines (SVM), sparse dictionary learning, clustering, Bayesian networks, reinforcement learning, representation learning, similarity and metric learning, sparse dictionary learning, genetic algorithms, rule-based machine learning, learning classifier systems, and / or one or more combinations thereof.

[0073] Examples of supervised machine learning techniques may include or be based on, but are not limited to, ANNs, DNNs, association rule learning algorithms, and prior algorithms. Algorithms, case-based reasoning, Gaussian process regression, generative expression programming, group methods for data processing (GMDH), inductive logic programming, instance-based learning, lazy learning, learning automata, learning vector quantization, logistic model trees, minimum message length (decision trees, decision graphs, etc.), nearest neighbor algorithms, analogy modeling, possibly approximate correct learning (PAC) learning, chain wave descent rules, knowledge acquisition methods, symbolic machine learning algorithms, support vector machines, random forests, classifier ensembles, guided aggregation (BAGGING), boosting (meta-algorithms), ordinal classification, information fuzzy networks (IFN), conditional random fields, analysis of variance, quadratic classifiers, k-nearest neighbors, boosting, sprint, Bayesian networks, Naive Bayes, Hidden Markov Models (HMMs), Hierarchical Hidden Markov Models (HHMMs), and any other ML techniques or tasks that can infer functions or generate models from labeled training data.

[0074] Examples of unsupervised ML techniques may include or be based on, by way of example only, but not limited to, the Expectation-Maximization (EM) algorithm, vector quantization, generated topographic mapping, Information Bottleneck (IB) methods, and any other ML technique or ML task capable of inferring descriptions of hidden structures and / or unlabeled data and / or labels in a training dataset that ignores labels. Examples of semi-supervised ML techniques may include or be based on, by way of example only, but not limited to, one or more of the following: active learning, generative models, low-density segregation, graph-based methods, co-training, transduction, or any other ML technique, task, or category of supervised ML techniques capable of training using both unlabeled and labeled datasets (e.g., a training dataset may typically contain a small amount of labeled training data combined with a large amount of unlabeled data).

[0075] Examples of Artificial Neural Network (ANN) ML techniques may include or be based on, but are not limited to, one or more of the following: Artificial Neural Networks, Feedforward Neural Networks, Recurrent Neural Networks (RNNs), Convolutional Neural Networks (CNNs), Autoencoders, Long Short-Term Memory (LSTM), LSTM Conditional Random Fields (CRFs), Bidirectional LSTM, Bidirectional LSTM-CRF; Recursive Artificial Neural Networks, Extreme Learning Machines, Logical Learning Machines, Self-Organizing Maps, and other ANN ML techniques or connectionic systems / computing systems inspired by biological neural networks that constitute the animal brain and capable of learning or generating models based on labeled and / or unlabeled datasets. Examples of Deep Learning ML techniques may include or be based on, but are not limited to, one or more of the following: Deep Belief Networks, Deep Boltzmann Machines, DNNs, Deep CNNs, Deep RNNs, Hierarchical Temporal Memory, Deep Boltzmann Machines (DBMs), Stacked Autoencoders, and / or any other ML techniques capable of learning or generating models based on learning data representations from labeled and / or unlabeled datasets.

[0076] For simplicity and as an example only, this invention can be described using ML techniques (also known as deep learning techniques) capable of learning from labeled training datasets, and can be based on any ML technique or algorithm from the following group: neural networks; recurrent neural networks; feedforward neural networks; convolutional neural networks; long short-term memory (LSTM) neural networks; bidirectional neural networks; bidirectional LSTM neural networks; neural networks combined with conditional random fields (CRFs); LSTM-CRF based neural networks; or combinations thereof; and / or any other ML techniques suitable for training ET models or ET ML models to predict, identify, and / or classify entity types of one or more entities and corresponding text portions of text corpora / documents, etc. Once trained, the trained ET model is configured to predict, identify, and / or classify the entity type of a matching entity within one or more text portions associated with a matching entity.

[0077] Figure 1a This is a schematic diagram illustrating an example entity type recognition system 100 according to the present invention. The ET recognition system 100 includes, for example, a large-scale dataset of a text corpus 102 coupled to an ET recognition device 104, an entity result module 106 coupled to the ET recognition device 104, and an ET recognition result module 108 coupled to the ET recognition device 104.

[0078] The large-scale dataset or text corpus 102 may contain one or more data sources, each of which may contain data representing multiple unstructured and / or structured texts / documents, etc. In this example, dataset 102 may be referred to as text / document corpus 102, and preferably may contain unstructured text or documents. For example, text / document corpus 102 may contain datasets from one or more data sources, such as, but not limited to, PubMed publications, MEDLINE publications, or any journal or article publications in the fields of bioinformatics and / or cheminformatics. Although the examples provided herein include text / document corpora in the fields of bioinformatics and / or cheminformatics, this is merely an example, and the invention is not limited thereto. Those skilled in the art will understand that a text / document corpus may contain any data representing texts / documents in any research or scientific field and available for text mining or data mining techniques to perform entity and entity type identification, etc., from the text / document corpus.

[0079] The entity results module 106 may be: a) a portion of a Named Entity Recognition (NER) system or another data processing system that receives a set of entity results relating to an identified entity within a text / document corpus, etc.; or b) a portion of an ET recognition device configured to receive a set of entity results relating to an identified entity from a NER system or another data processing system. In any case, the entity results module 106 receives one or more entity results, each entity result containing data indicating the identified entity and its location within a text / document corpus (e.g., a text or data corpus). These may have been identified by, for example, a NER system.

[0080] For example, the set of entity results may include data representing one or more identified entities, one or more entity types associated with each identified entity, and text portions and / or indications of the location of the entity within text portions in a text / document corpus. The indication of the entity's location within text portions in the text / document corpus can be used to enable the ET recognition device to locate the text portions surrounding the entity in the text / document corpus 102 for each entity result.

[0081] An ET recognition device may include one or more ET recognition models 104a or ET models configured to predict and / or recognize the entity type of each entity in a received entity result using a text / document corpus 102. For example, a text portion in the text corpus associated with the location of each entity may be input to an ET model configured to predict or extract the entity type of each entity from the text corpus. Before using the ET model to predict the entity type for each entity result, the ET model is trained and configured to predict or extract the entity type of each entity in the entity result from relevant text portions, etc., in the text / document corpus. ET recognition device 106 may output data representing the recognized entity type of each entity in the received entity result. An entity recognition result module 108 may be configured to update the entity result to include data representing the predicted or recognized entity type of each entity in the received entity result. Entity recognition result module 108 may be part of ET recognition device 104.

[0082] During the training of ET model 104a, ET device 104 can be configured for each identified entity in the entity results to retrieve and input text associated with the location of each identified entity, the input text comprising text portions from a text corpus surrounding the entity, such as text portions in the text corpus preceding and following the location of each identified entity. Each entity result can be structured to include data representing the entity and data representing text portions within the text corpus associated with the identified entity. For example, each entity result may include data representing the identified entity and text portions representing the associated identified entity based on at least one or more of the following groups: an indication of the identified entity and its location within a text / unstructured text corpus, etc.; the identified entity and text portions surrounding it from a text / unstructured text corpus, etc.; the identified entity, a first text portion preceding its location, and a second text portion following its location; text portions surrounding its location; and a first text portion preceding its location and a second text portion following its location; or text portions containing the identified entity; and / or any other suitable data representing the identified entity and text portions surrounding it. Entity results may include data representing the location of the identified entity within a text portion of a text corpus, or may include data representing the text portion containing the identified entity and the location of the identified entity within that text portion.

[0083] ET device 104 can be configured to train, generate, or update one or more of ET models 104a-104n by training corresponding ML techniques for predicting or extracting entity types from a text corpus based on the one or more entity results. To train the ML techniques to generate or update the ET models, a labeled training dataset containing multiple labeled data items can be used. Each labeled data item may contain data representing a text portion from the text corpus that is associated with a known or previously identified entity and is labeled with an entity type tag. Each entity type tag may come from a set of entity type tags associated with one or more entity types. Entity type tags may contain data representing or identifying a specific entity type. As an example, a text portion from the text corpus associated with a known entity may be contained within a first text portion preceding a second text portion representing the entity and a third text portion following the entity. That is, each labeled data item contains first and third text portions surrounding a second text portion representing the entity. Training ML techniques to generate or update ET models may involve iteratively training ML techniques based on labeled datasets to generate or update ET models, wherein the first and third text portions of each labeled data item are input and processed by ML techniques, wherein for each iteration: the entity type label associated with the labeled data item is compared with the corresponding predicted or extracted entity type output from the one or more ET models 104a-104n, wherein the model parameters defining each of the one or more ET models 104a-104n are updated based on the comparison results (e.g., one or more loss functions associated with the ML techniques may be computed to update the model parameters defining the ET models 104a-104n).

[0084] Once the ET model is trained, the ET device 104 can receive entity results to identify entity types. These identified entity types can be used to disambiguate and / or correct any entity types that the NER system previously attempted to predict relative to the identified entities in the entity results. The ET identification device 104 can input text portions that do not contain strings associated with the identified entity into one or more ET models for the identified entity. Alternatively or additionally, the ET identification device 104 can input text portions and the position of the identified entity within the text portions into one or more ET models, wherein the ET model can process text portions before and after the string (or text portion) describing the identified entity. The ET model 104a can process text portions surrounding the identified entity, wherein the ET model 104a ignores or does not include the identified entity in the text portions to predict or identify the entity type associated with the identified entity.

[0085] Figure 1b This illustrates the basis according to the present invention. Figure 1aAnother example of the ET recognition system 100 is illustrated in the diagram of the ET recognition system 110. In this example, the NER system 112 includes functionality such as an entity results module 106. The NER system 112 can be configured to identify entities and / or entity types from the text corpus 102 and entity results, as shown in the reference. Figure 1a As described, the entity results may include data indicating the identified entities and the location of each entity within the text corpus (e.g., the text portion associated with the entity, or the location of the entity within a text portion of the text corpus, etc.). The entity results module 106 may send a set of entity results to an entity type (ET) device 104, which is configured to process the entity results using one or more ET models 104a-104n to predict and / or identify the entity type of each entity in the entity results. The ET device 104 may output entity type identification results via an entity identification results module 108, which includes data indicating: each entity in the set of entity results, an indication of the text portion associated with each entity (or the location of the text portion within the text corpus 102), and the predicted / identified entity type of each entity.

[0086] For example, ET device 104 may be configured to receive one or more entity results or a set of entity results from NER system 112. Each entity result contains data representing a first text portion preceding the location of the identified entity within the text corpus and a second text portion following the location of the identified entity. Alternatively, each entity result may contain data representing the identified entity and the text portion containing the identified entity and / or an indication of the entity's location within the text corpus. Those skilled in the art will understand that entity results may be available to ET device 104 for identifying and / or retrieving the text portion surrounding each identified entity in the entity results, or may be provided as needed for the application. ET device 104 may be configured to extract or process each entity result containing the identified entity to determine and / or retrieve (e.g., from the entity results or from the text corpus) the text surrounding the identified entity for each identified entity in the entity results. For example, ET device 104 can retrieve, for each identified entity within a received set of entity results: a) a first text portion in text corpus 102 preceding and adjacent to the text or string representing the identified entity, and b) a second text portion in text corpus 102 following and adjacent to the text or string representing the identified entity. ET device 104 is configured to identify the entity type of each identified entity in the received entity results by inputting the first and second text portions in text corpus 102 associated with the identified entity into one or more trained ET models 104a-104n.

[0087] The model parameters of ET models 104a-104n are trained and configured to predict, identify, and / or extract the entity type of each identified entity from the text corpus 102 based on the context of text portions containing the identified entities. The training dataset used to train ET models 104a-104n may be based on multiple labeled training data items containing data representing multiple known entities, where each known entity is labeled with a known entity type tag and associated with a corresponding text portion from the text corpus containing the known entity. During training, ML techniques generate and / or update the model parameters of the ET models by inputting data representing a first text portion preceding the text or string representing the known entity and a second text portion following the text or string representing the known entity for each known entity and entity type tag input. The lengths of the first and second text portions may vary depending on the position of the known entity within the sentence and / or paragraph of the text portion associated with the known entity.

[0088] It should be noted that during or after training, the string of the identified entity in the text portion associated with the identified entity is not provided as input to the ET models 104a-104n. This allows the ET models 104a-104n to determine the entity type of the identified / known entity based on the context of the text portion associated with the identified / known entity, while minimizing the possibility that the model parameters of the one or more ET models 104a-104n are biased towards known entities and associated entity types or overfit to them. This allows the ET models 104a-104n to be more likely to correctly predict the entity type when encountering unknown text portions and / or new contexts where the identified entity can be used, in which case the NER system 112 may fail to determine it correctly.

[0089] Based on ET models 104a-104n, which process the first and second text portions associated with each identified entity, ET device 104 can output data representing the predicted / identified entity type of each identified entity. Entity recognition result module 1087 can send to NER system 112 an indication of the identified entity type of each entity in the received entity results. Alternatively or additionally, entity recognition results can be sent to NER system 112 for further processing. For example, NER system 112 can use entity recognition results to correct or update the set of entity results initially derived from text corpus 102. Alternatively or additionally, NER system 112 can be configured to update or correct the NER technology used for identifying entity types within NER system 112 while processing the text corpus. For example, NER system 112 may include one or more NER models based on ML technology, configured to predict / identify entities within the text corpus and / or configured to predict / identify the entity type of the identified entities, wherein entity recognition results can be used to update NER models and / or generate new NER models for predicting / identifying the entity type of the identified entities. Alternatively or concurrently, NER system 112 may use text / pattern matching-based NER technology and / or entity dictionaries (e.g., each entity dictionary contains multiple entities associated with a specific entity type), wherein the entity dictionary and / or entity type recognition technology may be updated based on the entity type recognition results.

[0090] Figure 1c It shows based on Figure 1a and 1b A schematic diagram of another example of an ET recognition system 120 for entity type disambiguation, used in the ET recognition system 100. In this example, the NER system 112 is configured to identify entities from a text corpus in the fields of bioinformatics / cheminformatics. The NER system 112 may use multiple entity dictionaries, each containing entities of a specific entity type, and the system also uses pattern / text matching techniques for identifying entities in the text corpus 102 and the entity type of each identified entity. Each entity type may be represented by one entity type tag from a set of entity type tags. In this example, the set of entity types may include, but is not limited to, chemical entity type 122a, disease entity type 122b, gene entity type 122c, and other entity types 122d, as examples only. Each of these entity types may have one entity type tag from a set of entity type tags.

[0091] Chemical entity type 122a is used to label and identify entities identified as chemicals in the text corpus. Disease entity type 122b is used to label and identify entities identified as diseases in the text corpus. Gene entity type 122b is used to label and identify entities identified as genes in the text corpus. Other entity types 122b are used to label and identify entities in the text corpus that have not been considered or identified as chemical, disease, or gene entity types. These may include ambiguous entities that the NER system 112 cannot determine belong to the chemical, disease, or gene entity type. For example, researchers may sometimes use the same entity name in different contexts when describing genes, chemicals, and / or diseases. This can lead to misidentification of the entity type of the identified entity, which the NER system 112 may not be able to handle. Although various selection strategies may exist for the NER system 112 to select entity types, these may lead to incorrect or incomplete identification of entities, which may affect the accuracy of downstream processes that rely on the correct identification of entities and entity types in the text corpus. For example, when NER system 112 cannot determine the entity type of an entity, the selection strategy could be to make a default selection by choosing the entity type from the ordered list of entity types 122a-122d: 1. Chemical; 2. Disease; 3. Gene; 4. Other. As another example, the selection strategy where NER system 112 does not provide entity type labeling could be very simple: that is, to label the entity as another entity type.

[0092] When the NER system 112 identifies entities within a text corpus that belong to one or more of these entity types 122a-122d, it can record and / or store in a set of entity results 124 data representing the entity and the most likely entity type to which the entity belongs (e.g., gene 126a, chemistry 126b, disease 126c, or others), as well as data representing the location of the text portion containing the entity or the text portion containing the entity in the text corpus. Entity results module 106 or other functionalities of the NER system 112 can be configured to send the set of entity results or portions / selections of the set of entity results (e.g., those entities with ambiguous entity types or entity types of other entity types) to the ET recognition device 104 for further processing / analysis.

[0093] ET recognition device 104 receives one or more entity results from NER system 112. As described, each entity result may contain data representing the identified entity, the identified entity type, and the location of the identified entity within a text corpus. Alternatively, each entity result may contain an indication of the identified entity and its location within the text corpus, or data containing a portion of text in the text corpus containing the identified entity. The ET recognition device is configured to input text from the text corpus associated with the location of each entity into a trained ET model for each entity result. The ET model is trained and configured to predict or extract the entity type of each entity from the text corpus. The text input for an identified entity or entity result may be text surrounding the identified entity and may not contain text or strings representing the identified entity. The ET model may output an entity type recognition result for each identified entity from the received set of entity results.

[0094] Furthermore, the NER system 112 and / or the ET recognition device 104 can also be configured to identify ambiguous entity types among the multiple entity results if multiple entity results associated with a relevant text portion represent the same entity, wherein at least one of the multiple entity results has a different entity type from another of the multiple entity results. This can indicate ambiguity in the entity type assigned to the identified entity. For each of the multiple entity results, a text portion in the text corpus associated with the location of each entity can be input into a trained ET model of the ET recognition device 104, which is trained or configured to predict or extract the entity type of each entity from the text corpus. Similarly, the text input for the identified entity or entity result can be text surrounding the identified entity, and the input text may not contain text or strings representing the identified entity.

[0095] The ET recognition device 104 or ET model can aggregate the predicted or extracted entity types of the multiple entity results associated with relevant text portions to output an entity type representing the multiple entity results. Furthermore, the relevant text portions associated with the multiple entity results may be located in the same document or the same text body from a text corpus. Therefore, the ET recognition device 104 can be further configured to aggregate the predicted or extracted entity types of the multiple entity results in the document / text body by aggregating the predictions of each entity in the multiple entity results together to form an overall prediction of the entity types of the entities in the multiple entity results. The entity types representing the multiple entity results can be output to the entity recognition result module 108, which can send the entity type recognition result to the NER system 112 to update the corresponding identified entities and / or a set of entity results.

[0096] Figure 1dThis is a flowchart illustrating an example process 130 for performing ET identification within a text corpus according to the present invention. Process 130 may include the following steps: In step 132, receiving one or more entity results, each entity result containing data representing the identified entity and the location of the identified entity within the text corpus. In step 134, identifying the entity type of each entity in the received entity results by inputting text associated with the location of each entity in the text corpus into a trained ET model, the trained ET model being configured to predict or extract the entity type of each entity from an unstructured text corpus. For example, the input text associated with the location of each entity may include text in the input text corpus preceding and following the location of each entity. The input text may not contain text or strings representing entities. In step 136, the ET model may output an entity type identification result containing data representing the identified entity type of each entity in the received entity results. This may include sending the entity type identification result with the identified entity type of each entity in the received entity results to a NER system or other systems that can use entities and identified entity types.

[0097] Figure 1eThis is a flowchart illustrating another example process 140 for ET identification for entity type disambiguation according to the present invention. Process 140 may include at least the following steps: In step 142, receiving one or more entity results from a NER system or other system, wherein each entity result contains data representing the identified entity, the identified entity type, and / or the position of the identified entity within a text corpus. In step 144, identifying and disambiguating the entity type associated with the entity results based on the text corpus. This may include identifying an ambiguous entity type among the multiple entity results if multiple entity results associated with a relevant text portion represent the same entity, wherein at least one of the multiple entity results has a different entity type from another of the multiple entity results. Identifying the entity type may further include, for each of the multiple entity results, inputting text in the text corpus associated with the position of each entity into a trained ET model, the ET model being configured to predict or extract the entity type of each entity from the text corpus. The input text associated with the position of each entity further includes text in the input text corpus preceding and following the position of each entity. The predicted or extracted entity types of the multiple entity results associated with the relevant text portions are aggregated to output an entity type representing the multiple entity results. In step 146, the ET model may output an entity type identification result containing data representing the identified entity type of each entity in the received entity results. This may include sending the entity type identification result with the identified entity type of each entity in the received entity results to a NER system or other systems that can use entities and identified entity types.

[0098] Figure 1f It is shown that it is used in Figure 1e The flowchart illustrates an example process 150 for entity type aggregation during ET identification, as outlined in process 140. Process 150 can be performed when multiple identical entities exist within a text portion, as some of these entities may have different entity types, thus the entity types of the multiple entities may be ambiguous. Process 150 may include the following step: In step 152, identifying and / or recognizing instances of entities with ambiguous entity types in the text portion or body. For example, multiple entity results relating to the same identified entity associated with the text body or related text portion may have different entity types. This could be an indication that the identified entity has an ambiguous entity type for the text body. If the multiple entity results represent the same entity, and at least one of the multiple entity results has a different entity type than another of the multiple entity results, then these multiple entity results may be ambiguous and require disambiguation. Figures 1a to 1eAs described in any of the above, the entity type of each of the plurality of entity results can be identified by inputting text associated with the location of each entity in a text corpus into a trained ET model for each of the plurality of entity results, the trained ET model being trained and configured to predict or extract the entity type of each entity from the text corpus. In step 154, the identified entity types can be aggregated to determine the overall entity type of the plurality of entity results. For example, the predicted or extracted entity types of the plurality of entity results associated with relevant text portions can be aggregated to output an entity type representing the plurality of entity results. In step 156, when the relevant text portion or text body is located in the same document from the text corpus, the aggregated predicted or extracted entity types of the plurality of entity results of the document may further include predicting the entity type of the entity instance in the plurality of entity results by aggregating the predictions of each entity in the plurality of entity results to form an overall prediction of the entity type of the entity in the plurality of entity results.

[0099] Figure 2a This is a schematic diagram illustrating an example Named Entity Recognition (NER) system 200, in which a dictionary-based NER system 202 uses an entity dictionary to generate entity results 204a or 204b from a text corpus 102. The entity dictionary, when input into the dictionary-based NER system 202, is used to identify entities within the text / document corpus 102 and their corresponding entity types, where text matching techniques can be applied to the text corpus 102 based on the entity dictionary. The text matching techniques can be conventional text matching techniques used to identify potential entities and entity types, where a first set of entity results 204a or 204b is generated (e.g., a list of entities, corresponding entity types, and indications of entity locations within the text of the text corpus).

[0100] Processing of the text corpus 102 can be implemented using text matching techniques to identify any entity from each entity dictionary that matches phrases, character portions, and / or strings in text portions or text documents from the text / document corpus 102. Those strings that match entities within the entity dictionary are stored as identified entities in a set of entity results. Each identified entity within the set of entity results may contain data representing: the identified entity, the entity type of the identified entity, an indication of the location of the identified entity within the text or text document, and / or optionally, a document identifier from the text / document corpus 102 in which the entity appears.

[0101] The set of entity results may include multiple entities of the same type but appearing in different locations within text portions or documents and / or in different text portions and / or documents of the text / document corpus 102. The set of entity results may be represented by any suitable data structure, such as, but not limited to, data representing entity lists and / or entity tables, where each row represents an identified entity, each column represents information associated with the identified entity, including, but not limited to, strings or text portions representing the identified entity, tags or data indicating the entity type of the identified entity, and / or indications of the location of the identified entity within the text corpus 102. The list or table may be represented as a data structure or multiple records representing each identified entity in a database or library.

[0102] NER system 202 may contain multiple entity dictionaries, each representing entities of a specific entity type (e.g., a drug entity dictionary, a chemical entity dictionary, a gene entity dictionary, a disease / symptom entity dictionary, etc.) and populated with known or manually curated entities associated with said entity type. Entity types of interest from the fields of bioinformatics and / or cheminformatics may include, but are not limited to, drugs, proteins, genes, chemicals, tissues / cells, cell lines, treatments, diseases / symptoms, or any other entity types associated with bioinformatics and / or cheminformatics, as examples only. For simplicity, the examples provided herein are based on the fields of bioinformatics and / or cheminformatics. Although entity types of interest from the fields of bioinformatics and / or cheminformatics are described, this is merely for simplicity and as an example only. Those skilled in the art should understand that the invention is not limited to entity types from the fields of bioinformatics and / or cheminformatics and / or their applications; in fact, the invention can be applied to any entity type from any other field and any other suitable application, and can be applied as needed.

[0103] Each entity dictionary is typically manually labeled with entities of a specific entity type. Clearly, as ongoing research generates unstructured text—such as articles, reports, documents, test results, etc.—in every domain associated with each entity type, the entity dictionary can quickly become outdated. Therefore, the text corpus 102 can be continuously or periodically updated with the latest generated research. This also means that the entity dictionary of the dictionary-based NER system 202 must be updated periodically to ensure it has the opportunity to correctly identify entities and / or entity types from the text corpus 102. However, such systems heavily rely on up-to-date entity dictionaries, which is often impossible given the vast amount of new documents / text added to human knowledge across various scientific fields (e.g., bioinformatics and / or cheminformatics). Therefore, such dictionary-based NER systems cannot avoid inaccuracies in entity results.

[0104] A dictionary-based NER system 202 processes a text / document corpus 102 to generate a first set of entity result lists 206. This set of entity results can be verified and provided as a list or table of data representing the identified entities, the identified entity types, and / or the positions of the identified entities / entity types within the text / documents of the text / document corpus. For example, each piece of text or document can be considered a string or an array of characters, where the position of an entity can be based on the indices of the first and last characters of the text portion identified as representing the entity.

[0105] For example, a document or text document from text corpus 102 may contain, but is not limited to, 5000 characters, including spaces, where each consecutive character is indexed by a consecutive index from 1 to 5000. The document may be in the field of bioinformatics / cheminformatics and may contain a phrase or sentence from a text document in text corpus 102 that begins with character index 100, such as “paracetamol is used to treat headaches.” This sentence has two entities: “paracetamol” of the drug entity type and “headache” of the disease / symptom entity type. Therefore, the dictionary-based NER system 202 can use a drug entity dictionary 106a containing a known list of drug entity types such as “paracetamol” and text matching technology 204 to identify that this text statement contains the entity “paracetamol” of the drug entity type. The entity “paracetamol” has a position in the text document defined by the start character index 100 to the end character index 110. The dictionary-based NER system 202 can also use a disease / symptom entity dictionary 106d and text matching technology 204 to identify that the text statement also contains the entity "headache" of the disease / symptom entity type, which has a position defined by the start character index 131 and end character index 138 of this text document. This is performed throughout the text document and the text document execution in the text corpus 102 to identify entities, entity types, and / or the position of entities within the text corpus based on the entity dictionary of the NER system 202.

[0106] As previously described, the set of entity results 204a or 204b may contain data representing: the identified entity and its corresponding identified entity type, the location of the identified entity within the text, and / or optionally, data for identifying which part of the text / document corpus the identified entity appears in, the document's text document identifier, or links, etc. For example, each entity result in the set of entity results 204a may be based on a data structure in the format {<entity>; <location>}, where <entity> includes data representing the identified entity, and <location> includes data indicating the entity's location within the text corpus 102. In another example, each entity result in the set of entity results may be based on a data structure in the format {<entity>; <entity type>; <location>}, where <entity> includes data representing the identified entity; <entity type> includes data representing the entity type of the identified entity (e.g., an entity type tag); and <location> includes an indication of the entity's location within the text corpus 102. <loc>The data can be represented in any way suitable for recognizing text portions associated with the identified entities within the text corpus 102. For example, entity results in a set of entity results can be based on a data structure with the format {<entity>, <entity type>, <position>}, where <position> contains data representing {<position-start character index>:<position-end character index>, <text document ID>}. For instance, the above "paracetamol / headache" statement can be represented using rows and columns containing the following data in entity result list 204b:

[0107] <Entity> <Entity type> <Start character index>: <End character index> <Document ID> paracetamol drug 100：110 102i Headache Disease / Symptoms 131：138 102i ... ... ... ...

[0108] Figure 2b This is a schematic diagram illustrating another example NER system 210, in which the NER-ML system 212 uses one or more machine learning techniques to create one or more NER models to generate entity results 214a or 214b from a text corpus 102 or a corpus of unstructured text, etc. To apply one or more ML techniques to create a trained NER model, a large amount of training data is typically required, ensuring the model is robust enough to identify and extract entities, entity types, and / or relations from the text corpus 102. The ML techniques can be trained using a labeled training dataset associated with entities and / or entity types, etc., which can be generated manually, semi-automatically, or automatically. The labeled training dataset may contain multiple training data instances or a set of labeled training data instances. Each labeled training data instance may contain data representing a text portion containing one or more known entities and an indication of the position of each of the one or more known entities within the text portion. Each of the one or more entities in each labeled training data instance may also be labeled with a known entity type tag indicating the entity type of each of those one or more known entities.

[0109] For example, each training data instance may contain data representing a specific text portion containing a known entity (e.g., a sentence containing the known entity of interest) and an indication of the location of said known entity within said specific text portion. The training data instances may also be labeled with entity type tags based on the types of known entities contained in said specific text portion. ML techniques use labeled training datasets to generate NER ML models based on labeled training datasets, wherein the trained NER ML models are configured to predict, identify, and / or extract one or more entities and / or entity types associated with the labeled training dataset from a text corpus 102. It should be understood that the NER model may be configured to generate a set of entity results 214a and / or 214b, etc., as exemplified but not limited to. Figure 2a As described. It should be noted that the accuracy and robustness of any NER model depends on the design / type of one or more ML techniques used to generate the NER model, but more importantly, on the size, diversity, and accuracy of the labeled training dataset.

[0110] For example, NER systems using ML techniques and NERML models can take into account the entity context within a document. However, this is limited by the scope of the labeled training dataset and the instances in which entities are used. Specific text instances of the name and context of each known entity can provide predictions, recognition, and / or classification of entities and / or entity types for a specific context defined in the labeled training dataset. However, such specificity often leads to overfitting of learned entities and entity types, and biased ML-based NER models. That is, ML-based NER systems learn the most common and prevalent entity type for a given entity name and typically bias their predictions towards this entity type.

[0111] As the amount of human knowledge and discoveries, texts, documents, research, etc., continuously or periodically generated within the scientific and / or research community and feedable into text corpora increases—not limited to examples—references... Figure 2a and 2b The described dictionary-based NER system 202 and ML-based NER system 204 face the risk of becoming obsolete almost immediately upon implementation or training; this leads to ambiguous and / or inaccurate entity results, where the identified entities have incorrect or ambiguous entity types that each system 202 or 204 cannot resolve correctly or adequately. This requires manually tidying up, checking, and validating all entity results, which is extremely expensive in terms of time and resources. References Figures 1a-1e The ET recognition system according to the invention described herein is configured to process a set of entity results and identify entity types by taking into account the context in which the identified entity or entity appears in a text corpus while minimizing all biases of known entities and entity types.

[0112] Figure 3a This is a schematic diagram illustrating an example trained ET recognition system 300 according to the present invention for training machine learning (ML) techniques to generate a trained ET model for use with a NER system. The trained ET recognition system 300 includes a training dataset module 302, an ML technique module 304, and an update module 306. The training dataset module 302 is configured to provide the ML technique module 304 with labeled training datasets associated with known entities, each known entity being labeled with an entity type tag. For example, the labeled training dataset may contain multiple labeled training data instances for each of a plurality of entities, wherein each labeled training data instance contains data representing a text portion containing a known entity, the position of the known entity within the text portion, and entity type tags. As another example, the labeled training dataset X may contain multiple labeled training data instances {x1, ... x2}. i , ..., x T }, 1≤i≤T, where T is the number of labeled training data instances in the labeled training dataset X. Each labeled training data instance x i It may include a text portion representing a known entity, the position of the known entity within the text portion, and a set of binary or non-binary tokens. Entity type marker variable The data, where each binary or non-binary token represents an entity type, where L≥1 is the number of entity types to be modeled.

[0113] ML technology module 304 is configured to train one or more ML technologies using a labeled training dataset to generate one or more ET models to predict, identify, and / or classify data from a reference dataset. Figures 1a-2b The described NER system generates a set of entity results, identifying the entity types of the identified entities. The ET model is configured to output entity type identification result data representing an indication of the entity type of each identified entity. Update module 306 is configured to receive entity type identification results / labels associated with the labeled training dataset and determine, based on the labeled training dataset, whether the ET model is sufficiently trained or requires further training iterations. If further training is required, then ML technique module 304 is configured to update the model parameters of the ET model based on a loss function, etc., associated with the corresponding ML technique, and continue training the ET model based on the labeled training dataset from training dataset module 302. If further training is not required, then given a portion of text from a text corpus, etc., containing the identified entities, the ET model can be considered trained in predicting, identifying, and / or classifying the entity types of the identified entities. If the ET model is trained on the text portions surrounding known entities, then overfitting can be mitigated. In some instances, this involves excluding known entities from the text portions surrounding known entities to avoid or reduce / minimize overfitting and / or bias of the ET model associated with each known entity.

[0114] Figure 3b This illustrates the application of the present invention in training for an ET identification system. Figure 3a A schematic diagram of an example ET-ML system 310. The corpus module 312 can store and / or maintain a corpus of text / documents or unstructured text, which can be processed by a NER system for predicting, identifying, and / or classifying entities and entity types from the text corpus. The text corpus can be updated continuously, intermittently, or periodically as needed by the application. The corpus module 312 can input a text corpus, or even a subset of the text corpus used for processing, into the NER system (e.g., Figure 2a NER system 200 or 210 (or 2b NER system 200 or 210) can output a set of entity results, as described herein, wherein each entity result contains at least the following data: the identified entity and the position of the identified entity within a text corpus or a text portion from a text corpus containing the identified entity (e.g., a text portion containing the identified entity plus a first text portion preceding the string or text representing the identified entity and a second text portion following the identified entity). Entity results may also contain entity types, etc. NER system 200 or 210 can provide one or more entity results from the set of entity results to ET recognition system 316, which includes one or more trained ET models. These ET models can each output a set of entity type identification results and / or tagging data representing each identified entity in the received entity results, which can be checked by entity result module 318 from each ET model. Therefore, entity result module 318 can output a set of entity type identification results for use by NER system 200 or 210. This set of entity type identification results can be used for reference. Figures 1a-2b The NER systems described herein, or those described in this document, are used to update a set of entity results for entity types. These results can also be used to update the entity dictionary and / or NER model of NER system 200 or 210, etc. Figure 3a The labeled training dataset described for training the ET model, and / or references Figure 2b The labeled training dataset described is used to train the NER ML model, etc.

[0115] Furthermore, as a supplement or alternative to the entity type of each entity in the entity identification results, the ET identification system 216 and / or the ET model used therein may be configured to determine whether multiple instances of the identified entity within the same text document or text section from a text corpus have an ambiguous entity type. The ET identification system 310 may include an aggregation module 319 for identifying such ambiguous entity types by, for example, identifying the entity type of each of the multiple instances of the identified entity using an ET model and aggregating, merging, or combining the identified entity types of the multiple instances of the identified entity to form an entity type covering the multiple instances. For example, the ET model may output a probabilistic entity type vector for each instance of the identified entity, containing a set of entity types, where each element of the probabilistic entity type vector represents a specific entity type. The probabilistic entity type vectors of the multiple instances of the identified entity may be averaged or combined into a weighted linear combination to form a single probabilistic entity type vector representing the entity type of the multiple instances. That is, the entity type corresponding to the element of the probabilistic entity type vector with the highest probability can be used to form the entity type of each of the multiple instances of the identified entity.

[0116] Figure 3c The invention is illustrated by way of example only, but not limited to, reference. Figures 1a-3b A schematic diagram of an ET-ML model 320 (or ET model) used in conjunction with the ET recognition system described herein. The ET-ML model 320 can be used to identify entity types based on the identified entity and text surrounding the identified entity from a text corpus. This can also be used to disambiguate one or more entity types of the identified entity from a set of entity results output from NER system 200 or 210. The ET-ML model 320 is shown receiving data representing an input text portion 322 associated with the identified entity from a text corpus. The text portion 322 contains the text or string of the identified entity and characters of the surrounding text, which may be in the form of [left input text][entity name][right input text]. That is, the input text portion 322 contains data representing a first input text portion preceding the identified entity (e.g., [left input text]), data representing the identified entity (e.g., [entity name]), and data representing a second input text portion following the identified entity (e.g., [right input text]). The text portion surrounding the identified entity is processed by the ET-ML model 320, which is configured to output a prediction of the entity type of the identified entity.

[0117] Specifically, the ET-ML model 320 receives an input text portion 322, which may be an input text string 322, including: a first text portion, which may be a left input text string (e.g., [left input text]); an identified entity name or a string of identified entity names in the text portion 322 (e.g., [entity name]); and a second text portion, which may be a right input text string (e.g., [right input text]). The text string 322 is segmented by a text segmenter 324 into separate left text components 326a and right text components 326b, wherein the entity names themselves are discarded. The left text components 326a and right text components 326b may be encoded as or embedded with data representing the left text components 326a and right text components 326b for use in the first ML module 328a and the second ML module 328b, respectively.

[0118] The first ML module 328a operates on data representing the left input text 326a to generate a first entity type vector embedding representation 330a based on the left input text segment 326a. The first entity type vector embedding representation 330a can be an N-dimensional vector in an N-dimensional vector space, where N > 1. The second ML module 328b operates on data representing the right input text 326b to generate a second entity type vector embedding representation 330b based on the right input text segment 326b. The second entity type vector embedding representation 330b can also be an N-dimensional vector in an N-dimensional vector space, where N > 1. The entity type vector combiner 332 combines the first entity type vector embedding representation 330a and the second entity type vector embedding representation 330b to form a total entity type prediction vector 334 of the entity type of the identified entity (e.g., [entity name]). The entity type prediction vector 334 can be an L-dimensional vector in an L-dimensional vector space, representing a set of binary or non-binary entity type tags. The probability of L ≥ 1 is the number of entity types to be modeled by the ET recognition model 320. The l-th element of the entity type prediction vector 334 can represent the entity type label of the identified entity. The probability or likelihood.

[0119] The first ML module 328a contains a first part of the ET ML model 320 (or the first ET ML model) configured and trained using a labeled training dataset by a first ML technique. The second ML module 328b contains a second part of the ET ML model 320, also configured and trained using a labeled training dataset by a second ML technique. The first and second ML techniques can be the same or different ML techniques. The first and second ML techniques can be jointly trained to generate the first and second parts of the ET ML model 320. The labeled training dataset X can contain multiple labeled training data instances {x1, ... x2}. i , ..., x T }, 1≤i≤T, where T is the number of labeled training data instances in the labeled training dataset X. Each labeled training data instance x i It may contain data representing: text portions containing known entities (e.g., [left input text][entity name][right input text]), and the position of the known entities within the text portions (e.g., [entity name]), wherein labeled training data instances are labeled with a set of binary or non-binary entity types. Entity type tag The set of entity type tags, each tag representing an entity type, wherein L≥1 is the number of entity types to be modeled by ET model 320.

[0120] The first ML module 328a and the second ML module 3428b can be jointly trained based on a repetitive labeled training dataset. The first ML module 328a outputs a first entity type vector embedding representation 330a, which can be an N-dimensional vector in an N-dimensional vector space, where N > 1. The second ML module 328b outputs a second entity type vector embedding representation 330b, which can be an N-dimensional vector in an N-dimensional vector space, where N > 1. The first entity type vector 330a and the second entity type vector 330b are combined to form an entity type prediction vector 334. The entity type prediction vector 334 can be an L-dimensional vector in an L-dimensional vector space, representing a set of binary or non-binary entity type labels. The probability is given by L, where L≥1 is the number of entity types to be modeled by the ET recognition model 320. The model parameters of the first and second parts of the ET ML model 320 are updated based on a comparison between the estimated output of the predicted entity type vector 334 and the corresponding entity type labels of the labeled training data instances for each labeled training data instance, or the generated error vector. For example, the loss function corresponding to the ML technique can be used to update the model parameters of the first and second parts of the ET ML model 320 at each iteration of the labeled training dataset until the first and second parts of the ET ML model 320 are properly trained (e.g., when the number of incorrectly predicted entity types is below a prediction error threshold).

[0121] An entity type vector combiner 332 for combining embedding vectors 330a and 330b to generate an L-dimensional predicted entity type vector 334 may include the following steps: concatenating a first vector entity type embedding 330a and a second vector entity type embedding 330b output from a first ML module 328a and a second ML module 328, respectively, based on left input text 326a and right input text 326b; and reducing the dimension of the resulting vector to the number of potential entity types (or categories) in a set of predefined entity types (e.g., drug entity type, gene entity type, chemical entity type, protein entity type, other entity types). The output predicted entity type vector 334 may consist of probability vectors representing the likelihood that an entity is one of L different entity types, where each vector element corresponds to the set of entity types (e.g., a set of entity type labels). Different entity types in the text.

[0122] The ET model 320 may be based on a neural network, wherein the first ET module 328a and the second ET module 328b may be based on joint ML neural network technology. A first portion of the ET model 320 of the first ML module 328a may be regarded as a first ML model. A second portion of the ET model 320 of the second ML module 328b may be regarded as a second ML model. The first ML model of the first ML module 328a (or the first portion of the ET model 320) may include a forward hidden state of a neural network configured to represent a first text portion 326a (e.g., left-hand input text) preceding the location of the identified entity. The second ML model (or the second part of the ET ML model 320) of the second ML module 328b may include a backward hidden state of a neural network configured to represent a second text portion 326b (e.g., (right input text)) following the location of the identified entity. The forward and backward hidden states may be concatenated to generate entity types representing the first text portion 326a and the second text portion 326b. For example, the first ML module 328a and the second ML module 328b operating on the left input text 326a and the right input text 326b may consist of joint ML techniques, such as a bidirectional long short-term memory (LSTM) ML system. This system may be trained on text surrounding several pre-labeled, explicitly known entities with known entity types to learn the entity types of the left and right input texts that co-occur with each entity type, wherein the bidirectional LSTM ML system can then classify the identified entities according to these entity types.

[0123] A labeled training dataset X can contain multiple labeled training data instances {x1, ... x2}. i , ..., x T }, 1≤i≤T, where T is the number of labeled training data instances in the labeled training dataset X. Each labeled training data instance x i It may contain data representing a text portion containing the entity and marked with at least one of the following: 1) an indication of the entity's position within the text portion; and 2) data from a set of binary or non-binary markers. Entity type marker variable Each binary or non-binary token represents an entity type, where L≥1 is the number of entity types to be modeled.

[0124] Figure 3d This illustrates the use of the present invention for training. Figure 3c The flowchart of the training process of ET model 320 340. (See also: regarding...) Figure 3c As described, the labeled training dataset X342 can be used to train the ET ML model 320 to predict the entity type of identified entities. The ET ML model 320 can be trained by providing it with the labeled training dataset X342, which includes multiple text portions 342a-342n, each of which surrounds a known entity among multiple entities, where each known entity has a corresponding known entity type 344a-344n. The set of text portions 342a-342n surrounding the known entities is input from the labeled training dataset X342 into the ET ML model 320, which is configured to predict a set of entity types 334a-334n for each of the text portions 342a-342n. The predicted entity types 334a-334n are compared with the known entity types 344a-344n 346, wherein the comparison is fed back 348 to the ET ML model 320 to update the model parameters according to the ML technique used to generate the ET ML model 320.

[0125] Specifically, the labeled training dataset X342 can be based on a provided list of known, pre-validated entity types 344a-344n and text 342a-342n surrounding entity names or known entities in the original document. This can be pre-generated by a NER system programmed to return only explicit entity names and entity types. Alternatively or additionally, the labeled training dataset X342 can be initially prepared manually. Each text portion 344a-344n is fed into the ET ML model 320, which outputs a predicted entity type 334a-334n for each text portion 344a-344n surrounding a known entity in the input. The ET ML model 320 only considers the text portions 344a-344n surrounding known entities, excluding text or strings representing known entities. A comparison is then made between the known (validated) entity types 344a-344n and the predicted entity types 334a-334n. The results of these comparisons are fed back to the ET ML model 320, which can then be used to train the prediction system. For example, if the ET ML model 320 is based on a neural network, the output of the comparisons can be a loss function used for backpropagation through the neural network.

[0126] Figure 3e This illustrates the use of the invention based on a reference. Figures 1a-3d A schematic diagram illustrating example inputs and outputs of an ET ML system 350 for any ET model, ET model 354. The ET ML system 350 includes an ET ML model 354, as described in reference... Figures 1a-3d The description describes obtaining a text portion 352 from a text corpus as input, processing the text portion 352 to output an entity type prediction vector 356. In this example, the input text portion 352, in bold, contains the text: "Western blot analysis, Uhlik et al. (2003) found that CCM2, which they called OSM, was expressed in the majority", where the string "CCM2" is the identified entity. The output predicted entity type vector 356 can be written as a row vector PE = (p1, p2, p3, p4), where the values ​​p1, p2, p3, and p4 represent the probability that the identified entity "CCM2" is one of the four entity types: "Chemistry", "Gene", "Disease", and "Other". In this case, p1 is the probability or likelihood that the identified entity is of the "chemical" entity type; p2 is the probability or likelihood that the identified entity is of the "gene" entity type; p3 is the probability or likelihood that the identified entity is of the "disease" entity type; and p4 is the probability or likelihood that the identified entity is of the "other" entity type.

[0127] In this example, the input text 352 for the ET ML model 354 includes several characters flanking the named or identified entity "CCM2," which can be excerpts from the larger text provided by the NER system. The text 352 is fed into the ET ML model 356, and entity type prediction is performed to produce a predicted entity type vector 356. For example, the predicted entity type vector 356 is a vector corresponding to the probabilities of different entity types: "Chemistry," "Gene," "Disease," or "Other." Based on the categories or entity types learned by the ET ML model 356 from the labeled training dataset 342, the probabilities can correspond to the entity types "Chemistry," "Gene," "Disease," or "Other." In this example, the given probabilities for these entity types are 0.1, 0.6, 0.3, and 0.0, respectively, meaning that the predicted entity type for the identified entity "CCM2" is the entity type "Gene," as it has the highest probability or likelihood of 0.6. Essentially, the entity type is indicated in the predicted entity type vector 356 using the vector element with the highest value.

[0128] Figure 4 This is an illustration based on the present invention. Figures 3a-3e A schematic diagram of another example of the described ET-ML model, an ET recognition system 400. The ET recognition system 400 includes a document 402 from a text corpus as input to a NER system 404, which includes an NER prediction module 404a and an entity result 404b. The NER prediction module 404a outputs the entity result 404b based on the input text document 402. The entity result 404b contains data representing the text portion of document 402 from the text corpus containing the identified entity (e.g., [entity]). The entity result 404b may contain data representing the identified entity, the position of the string of the identified entity, and indications of the text portion 404b around the identified entity selected from document 402. In this example, the entity result includes the text portion 404b of document 402 from the text corpus surrounding the identified entity (e.g., [entity]). The entity result 404b may be input to an ET ML model 406, in which case the ET ML model 406 may be based on, but is not limited to, the text portion 404b from the text corpus. Figure 3c And / or 3D ET ML model 320. Although ET ML model 406 is based on Figure 3c And / or a 3D ET ML model 320, but those skilled in the art should understand that the ET ML model 406 may be based on, but is not limited to, a reference only. Figures 1a to 3e And / or one or more ET models described in 5, their modifications, combinations thereof, etc. ET ML model 406 includes a text segmenter 406a that extracts a left text portion 406b (or a first text portion before the identified entity) and a right text portion 406c (or a second text portion after the identified entity) adjacent to the identified entity, wherein neither text portion 406b nor 406c contains the identified entity. The left and right text portions are input to a prediction module 406d that may contain one or more ET ML models (not shown), which are based on, but are not limited to, reference models. Figures 1a to 3e And / or one or more ET models described in 5, their modifications, combinations thereof, etc. The prediction module 406d outputs corresponding entity type vector embeddings, which, when combined, output a predicted entity type vector 406e representing the likelihood or probability of an entity type. This is fed back to the NER system 404 to update the entity types in the entity result 404b. The NER system 404 outputs a list 408 of identified entities and their corresponding entity types, which may have been updated using the identified entity types from the ET ML model 406.

[0129] In another example, input text 404b is provided by NER system 404 to operate on text document 402 to identify and extract entity names and entity types. In cases where the entity type is ambiguous, NER system 404 may provide the identified entity and surrounding text 404b to ET ML model 406 to perform disambiguation or identify the most probable entity type of the identified entity in text 404b. That is, when NER system 404 processes text document 402, if it encounters an entity with an ambiguous entity type or only needs to identify the entity type of the identified entity, then NER system 404 may extract the text 404b surrounding the identified entity for input to ET ML model 406. ET ML model 406 extracts the left-hand text 406b and right-hand text 406c of the identified entity name from input text 404b (e.g., the size of the left-hand or right-hand text can be a predefined number: characters; words, sentences, and / or paragraphs, etc.), where the identified entity name is discarded. These text portions 404b and 404c are fed into prediction module 406d, which contains one or more ET ML models trained and configured to predict the entity type of the identified entity based on text portions surrounding but not containing the identified entity. Then, similar to, but not limited to, examples... Figures 1a-3e The described method involves predicting the entity type of the identified entities in prediction module 406d. This produces a final prediction of the entity type 406e, which is then fed back to NER system 404 via path 407 for inclusion in the final output 408 of NER system 404. NER system 404 then delivers the entity names and corresponding entity types contained in the input text body 402 as output.

[0130] Optionally, if the same ambiguous entity is encountered multiple times throughout a single document 402, the ET ML model can be configured to aggregate the predictions 406e of each instance of the identified entity in text 402 to form an overall prediction of the entity type of the identified entity. It is assumed that within a single document 402, a single identified entity name should always refer to the same entity type. This can be achieved by concatenating the individual predicted entity type vectors and using a normalization function (e.g., softmax) to output the final document-level predicted entity type vector for the single identified entity.

[0131] Figure 5 This is a schematic diagram illustrating an example bidirectional LSTM neural network 500 as an example ET-ML model according to the present invention, serving as an ET recognition system. The bidirectional LSTM neural network 500 can be used in ET-ML model 320, as shown in the diagram. Figure 3c The ML modules 328a and 328b, which operate on the left input text portion 326a and the right input text portion 326b, can be composed of a bidirectional LSTM neural network 500. (Reference) Figure 5 The bidirectional LSTM neural network 500 operates on the input text string segmented into words. In this example, the bidirectional LSTM neural network 500 is fed a left or right input text string 502 segmented into individual words. In this example, the input... Figure 3e The text portion 352 shows only the words "OSM" 502a, "was" 502b, and "expressed" 502c, but it is assumed that all words in the text portion 352 are input as either the left or right input text string 502, excluding the entity name "CCM2". Each word in the text string 502 is passed to two LSTM systems 504 and 506 running in parallel. The first LSTM system 504 (e.g., the first part of an ET-ML model or the first ET-ML model) operates in the forward direction of the input text string 502, and the second LSTM system 506 (e.g., the second part of an ET-ML model or the second ET-ML model) operates in the backward direction of the input text string 502.

[0132] The first LSTM system 504 includes at least one hidden layer containing multiple LSTM units 504a-504b connected to the outputs of each input word 502a-502c and adjacent previous LSTM units 504a-504m. The second LSTM system 506 includes at least one hidden layer containing multiple LSTM units 506a-506o connected to the outputs of each input word 502a-502c and adjacent subsequent LSTM units 506a-506o. Because the LSTM systems have 'memory' of previous input values, the first LSTM system 504 (or the forward LSTM system) provides the output of the final word in the text string 502 based on the words and all previous words in the text string 502, while the second LSTM system 506 (or the backward LSTM system) provides the output based on the first word in the text string 502 and each subsequent word in the text string 502. In this way, as one would expect from a single forward LSTM, bias towards the last word in the string is eliminated. Forward LSTM 504 and backward LSTM 506 output prediction vectors from the last and first words of the final LSTM units 504m and 506o in the text string 502, respectively. These predictions are then aggregated (e.g., by concatenation) together 508 to produce a reference vector, as shown in the example only, but not limited to. Figures 3a-3e The predicted entity type vector embedding of the entity type described in section 4.

[0133] The bidirectional LSTM neural network 500 is trained on a labeled training dataset, and is used as an example but not limited to this. Figures 1a to 4 The training of the ET ML model is described below. For example, a labeled training dataset may contain multiple labeled training data instances, where each labeled training data instance contains data representing text portions associated with a known entity and labeled with a known entity type. A bidirectional LSTM neural network 500 receives each labeled training data instance as an input text string 502, which contains text portions surrounding the text corresponding to the known entity, but does not contain text corresponding to the known entity. Therefore, the bidirectional LSTM neural network 500 avoids overfitting entity names to entity types and learns how to identify entity types based on the context in which the entity appears within the text portions.

[0134] A bidirectional LSTM neural network 500 combines the outputs of a first LSTM network 504 and a second LSTM network 506 (forward and backward LSTM networks) to generate a predicted entity type vector for each labeled training data instance. The bidirectional LSTM neural network 500 also performs comparisons between the known (validated) entity types of the labeled training data instances and the output predicted entity type vectors. The results of these comparisons are fed back to the bidirectional LSTM neural network 500 to update the model parameters of the first and second LSTM networks and / or any other input / output embedding / encoding neural networks that may be used before the data representing the input text string 502 is input into the first and second LSTM networks. The output of the comparisons may be a loss function used for backpropagation through the one or more neural networks including the first LSTM 504 and the second LSTM 506 and / or any other input / output embedding / encoding neural networks.

[0135] Figure 6a This is a schematic diagram illustrating an example computing system 600 having a computing device 602. This system can be used to implement one or more of the following aspects: an ET recognition system for processing a text corpus to generate a set of entity results containing data representing identified entities, the position of the identified entities in a text portion of the text corpus, and / or text portions surrounding the identified entities; and / or for processing said set of entity results according to the invention and / or based on a reference. Figure 1a-5 The described processes, methods, systems, and / or devices identify, predict, and / or extract entity types from an ET ML model for each entity outcome. Computing device 602 includes one or more processor units 604, a memory unit 606, and a communication interface 608, wherein the one or more processor units 604 are connected to the memory unit 606 and the communication interface 608. The communication interface 608 can connect computing device 602 to one or more databases or other processing systems or computing devices for implementing the invention as described herein via a communication network 610. The memory unit 606 can store one or more program instructions, code, or components, such as, but not limited to, an operating system 606a for operating computing device 602 and a data storage device 606b for storing additional data and / or other program instructions, code, and / or components associated with: implementing the aforementioned functionality and / or one or more functions or as described herein and / or referenced. Figures 1a to 5 One or more associated functionalities in at least one of the described devices, mechanisms, and / or systems / platforms / architectures, methods, and / or processes.

[0136] Another aspect of the invention may include one or more devices and / or apparatuses, said devices and / or apparatuses comprising a communication interface, a memory unit, and a processor unit connected to the communication interface and the memory unit, wherein the processor unit, the memory unit, and the communication interface are configured to perform or implement as referenced. Figures 1a to 5 The systems, apparatus, methods and / or processes or combinations thereof described herein.

[0137] Figure 6b This is a schematic diagram illustrating a system 620 according to the invention for performing ET recognition on a set of entity results generated by a NER system 622 processing a text / document corpus to identify one or more entities. System 620 includes one or more NER systems 622, one or more ET ML models 624, and an entity type result / recognition module / apparatus 626. The NER system 622 may include, but is not limited to, one or more entity dictionary-based NER systems, one or more ML-based NER systems, modifications thereof, combinations thereof, and / or systems as described herein, which process a text corpus to generate a set of entity results, identify one or more entities of interest, text portions from the text corpus surrounding the identified entities of interest, and / or the position of the entities of interest within the text portions, etc. The ET ML model 624 may include, but is not limited to, an ET model trained using ML techniques based on a labeled training dataset for each known entity, containing text portions surrounding the known entity and known entity type tags. ML techniques process each text portion that does not contain a known entity to generate, update, and / or adjust the model parameters of the ET ML model 624. Once trained, the ET ML model 624 is configured to predict and / or identify the entity type of the identified entity based on the text portion surrounding but not containing the identified entity. The identified entity and / or the text portion surrounding the identified entity can be provided from a set of entity results output by the NER system 622. The entity type result / identification module / device 626 may include, as an example only but not limited to, an analysis / comparison mechanism for identifying entity types from multiple sets of entity type results output by the ET ML model 624, wherein the identified entity types can be verified automatically, semi-automatically, or manually. The NER system 622, the ET ML model 624, and the entity type result / identification module / device 626 may be associated with and / or as described herein and / or as referenced in this invention, its modifications, and / or as described herein and / or as referenced. Figures 1a-6a The described methods, processes, attention devices, and / or systems are configured to provide efficiency and improved accuracy in predicting / identifying entity types from identified entities in a text corpus. They can be used to generate suitable training datasets to train ET ML models, NER ML models, and / or to train downstream ML processes, models, classifiers, etc.

[0138] In other aspects, the ET identification device according to the invention may include one or more processors, a memory, and / or a communication interface, said one or more processors being connected to the memory and / or the communication interface, said one or more processors being configured to implement one or more of processes 130, 140, 150, and 340 and / or one or more of devices / systems / models 100, 110, 120, 200, 210, 300, 310, 320, 350, 400, 500, 600, and 610 and / or an ET identification system, an ET ML model or ET model, a dictionary-based NER system, an ML-based NER system, a classifier, and / or a reference. Figures 1a to 5 Any methods / processes, steps of such processes, combinations thereof, or modifications thereof described in any one or more of b. Furthermore, one or more of processes 130, 140, 150, and 340 and / or one or more of devices / systems / models 100, 110, 120, 200, 210, 300, 310, 320, 350, 400, 500, 600, and 610 and / or ET identification systems, ET ML models or ET models, dictionary-based NER systems, ML-based NER systems, classifiers, and / or references. Figures 1a to 6b Any one or more of the methods / processes described herein, the steps of such processes, combinations thereof, and modifications thereof may be implemented in hardware and / or software.

[0139] In the embodiments described above, the server may comprise a single server or a network of servers. In some instances, the functionality of the server may be provided by a network of servers distributed across geographical regions (such as a globally distributed network of servers), and a user can connect to the appropriate one in the server network based on the user's location.

[0140] For clarity, the above description has referred to embodiments of the invention discussed with reference to a single user. It should be understood that, in practice, the system can be shared by multiple users and may be shared by a very large number of users simultaneously.

[0141] The embodiments described above are fully automated. In some instances, the system user or operator may manually guide the execution of some steps of the method.

[0142] In the embodiments described in this invention, the system can be implemented as any form of computing and / or electronic device. Such a device may include one or more processors, which may be microprocessors, controllers, or any other suitable type of processor for processing computer-executable instructions to control the operation of the device to acquire and record routing information. In some instances, such as when using a system-on-a-chip architecture, the processor may include one or more fixed functional blocks (also referred to as accelerators) that implement part of the method in hardware (rather than software or firmware). Platform software including an operating system or any other suitable platform software may be provided at the computing-based device to enable application software to execute on said device.

[0143] The various functions described herein can be implemented in hardware, software, or any combination thereof. If implemented in software, the functions can be stored as one or more instructions or codes on or transmitted through a computer-readable medium. A computer-readable medium can include, for example, a computationally readable storage medium. A computer-readable storage medium can include volatile or non-volatile, removable or non-removable media implemented in any method or technique for storing information such as computer-readable instructions, data structures, program modules, or other data. A computer-readable storage medium can be any available storage medium that can be accessed by a computer. By way of example, and not limitation, such a computer-readable storage medium can include RAM, ROM, EEPROM, flash memory or other memory devices, CD-ROM or other optical disc storage devices, magnetic disk storage devices or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. As used herein, disks and platters include CDs, laser discs, optical discs, DVDs, floppy disks, and Blu-ray discs (BD). Furthermore, transmitted signals are not included within the scope of a computer-readable storage medium. Computer-readable media also include communication media, which include any medium that facilitates the transfer of computer programs from one place to another. For example, a connection can be a communication medium. For instance, if software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of a communication medium. Combinations of the foregoing should also be included within the scope of computer-readable media.

[0144] Alternatively or additionally, the functionality described herein may be performed at least in part by one or more hardware logic components. For example, and not as a limitation, the hardware logic components that may be used may include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), etc.

[0145] Although shown as a single system, it should be understood that computing devices can be distributed systems. Thus, for example, several devices can communicate via a network connection and collaboratively perform tasks described as being performed by computing devices.

[0146] Although presented as a local device, it should be understood that the computing device can be located remotely and accessed via a network or other communication link (e.g., using a communication interface).

[0147] The term "computer" is used herein to refer to any device that has processing power that enables it to execute instructions. Those skilled in the art will recognize that such processing power is incorporated into many different devices, and therefore the term "computer" includes PCs, servers, mobile phones, personal digital assistants, and many other devices.

[0148] Those skilled in the art will recognize that storage devices used to store program instructions can be distributed across a network. For example, a remote computer can store processing instances described as software. A local or terminal computer can access a remote computer and download part or all of the software to run the program. Alternatively, a local computer can download software on demand or execute some software instructions at a local terminal and some software instructions at a remote computer (or computer network). Those skilled in the art will also recognize that, by using conventional techniques known to them, all or part of the software instructions can be executed by dedicated circuitry (such as DSPs, programmable logic arrays, etc.).

[0149] It should be understood that the benefits and advantages described above may relate to one embodiment or several embodiments. The embodiments are not limited to those that solve any or all of the stated problems or have any or all of the stated benefits and advantages. Variations should be considered to be included within the scope of this invention.

[0150] Any reference to "a" means one or more of those items. The term "comprising" is used herein to mean including the identified method step or element, but such step or element is not an exclusive list and the method or apparatus may contain additional steps or elements.

[0151] As used herein, the terms "component" and "system" are intended to cover a computer-readable data storage device configured with computer-executable instructions that enable certain functionalities to be executed when performed by a processor. Computer-executable instructions may include routines, functions, etc. It should also be understood that a component or system may be localized on a single device or distributed across several devices.

[0152] Furthermore, as used herein, the term "exemplary" is intended to mean "serving as an illustration or example of something".

[0153] Furthermore, with respect to the use of the term "comprising" in the detailed description or claims, this term is intended to be open-ended in a manner similar to the term "including," since "comprising" is interpreted as a transitional word in the claims when used.

[0154] The accompanying figures illustrate an exemplary method. While the method is shown and described as a series of actions performed in a specific sequence, it should be understood and appreciated that the method is not limited by the order of the sequence. For example, some actions may occur in a different order than that described herein. Furthermore, one action may occur simultaneously with another. Moreover, in some instances, not all actions are required to implement the method described herein.

[0155] Furthermore, the actions described herein may include computer-executable instructions that can be implemented by one or more processors and / or stored on one or more computer-readable media. Computer-executable instructions may include routines, subroutines, programs, threads of execution, and / or the like. Further still, the results of the actions of the method may be stored in a computer-readable medium, displayed on a display device, and / or the like.

[0156] The order of steps in the methods described herein is exemplary; however, these steps may be performed in any suitable order or simultaneously, where appropriate. Furthermore, steps may be added or substituted, or individual steps may be removed from any method, without departing from the scope of the subject matter described herein. Aspects of any instance described herein may be combined with aspects of any other instance described to form further instances without loss of effect.

[0157] It should be understood that the above description of the preferred embodiments is given by way of example only and various modifications can be made by those skilled in the art. The content described above includes examples of one or more embodiments. Of course, it is not possible to describe every conceivable modification and alteration to the above-mentioned apparatus or method for the purpose of describing the aspects mentioned above, but those skilled in the art will recognize that many other modifications and arrangements of the aspects are possible. Therefore, the described aspects are intended to cover all such changes, modifications, and variations that fall within the scope of the appended claims.< / loc>

Claims

1. A computer-implemented method for entity type identification of entities within a text corpus, the method comprising: receiving one or more entity results, each entity result comprising data representing an identified entity and a location of the identified entity within the text corpus; identifying an entity type for each entity in the received entity results by inputting text in the text corpus associated with the location of the each entity to a trained entity type (ET) model, the trained ET model configured to predict or extract an entity type for the each entity from the text associated with the location of the each entity; and outputting data representing the identified entity type for each entity in the received entity results; wherein the text associated with the location of the each entity input to the ET model comprises text in the text corpus before and after the location of the each entity and does not include the text or string representing the entity.

2. The computer-implemented method of claim 1, wherein an entity comprises entity data associated with an entity type from the group of: gene; disease; compound / drug; protein; chemical, organ, organism.

3. The computer-implemented method of claim 1, wherein receiving the one or more entity results further comprises receiving the one or more entity results generated by a named entity recognition (NER) system.

4. The computer-implemented method of claim 1, wherein each entity result comprising data representing an identified entity and a location of the identified entity within the text corpus contains one or more from the group of: an indication of the identified entity and the location of the identified entity within the text corpus; the identified entity and a portion of text from the text corpus surrounding the identified entity; the identified entity, a first portion of text before the location of the identified entity, and a second portion of text after the location of the identified entity; a portion of text surrounding the location of the identified entity; and a first portion of text before the location of the identified entity and a second portion of text after the location of the identified entity.

5. The computer-implemented method of claim 1, further comprising: receiving one or more entity results from a NER system, wherein each entity result contains data representing a first portion of text before the location of an identified entity and a second portion of text after the location of the identified entity within the text corpus; identifying an entity type for each entity in the received entity results by inputting the first and second portions of text in the text corpus associated with the identified entity to the trained entity type (ET) model, the trained ET model configured to predict or extract an entity type for the each entity from the text corpus; and sending data representing the identified entity type for each entity in the received entity results to the NER system. ​ 6. The computer-implemented method of claim 1, further comprising: receiving one or more entity results from a NER system, wherein each entity result includes data representing an identified entity, an identified entity type, and a location of the identified entity within the corpus of text; identifying an ambiguous entity type between a plurality of entity results associated with a relevant text portion, wherein at least one of the plurality of entity results has a different entity type than another of the plurality of entity results; for each of the plurality of entity results, inputting text from the corpus of text associated with the location of the each entity to the trained ET model configured to predict or extract an entity type of the each entity from the corpus of text; and aggregating the predicted or extracted entity types of the plurality of entity results associated with the relevant text portion to output an entity type representing the plurality of entity results.

7. The computer-implemented method of claim 6, wherein the relevant text portion is located in a document from the corpus of text, and aggregating the predicted or extracted entity types of the plurality of entity results for the document comprises aggregating predictions of each entity in the plurality of entity results to form an overall prediction of the entity type of the entity in the plurality of entity results.

8. The computer-implemented method of claim 1, further comprising generating or updating the ET model by training a machine learning (ML) technique to predict or extract an entity type from the corpus of text based on the one or more entity results.

9. The computer-implemented method of claim 8, wherein the ML technique comprises at least one ML technique from the following group: a neural network; a recurrent neural network; a feedforward neural network; a convolutional neural network; a long short-term memory (LSTM) neural network; a bidirectional LSTM neural network; a neural network based on LSTM conditional random fields (CRFs); and any other ML technique for predicting an entity type from a corpus of text based on one or more entity results.

10. The computer-implemented method of claim 8, further comprising updating the ET model based on the identified entity type.

11. The computer-implemented method of claim 10, further comprising training the ML technique to generate or update the ET model based on a labeled dataset comprising a plurality of labeled data items, wherein each labeled data item comprises a portion of text from the corpus of text associated with an entity and annotated with an entity type label.

12. The computer-implemented method of claim 11, wherein the portion of text from the corpus of text associated with an entity includes a first portion of text, a second portion of text representing the entity, and a third portion of text following the entity.

13. The computer-implemented method of claim 12, wherein each labeled data item comprises the first and third text portions around the second text portion representing the entity.

14. The computer-implemented method of claim 11, wherein training the ML technique to generate or update the ET model further comprises iteratively training the ML technique to generate or update the ET model based on the labeled data set for each iteration by: comparing the labeled data item and a corresponding predicted or extracted entity type output from the ET model; updating the ET model based on the comparison based on a loss function associated with the ML technique.

15. The computer-implemented method of claim 1, wherein the ET model is based on a neural network comprising forward and backward hidden states configured to represent first and second text portions before and after the location of the entity, respectively, wherein the forward and backward hidden states are concatenated together to generate an entity type representation of the first and second text portions.

16. The computer-implemented method of claim 1, the ET model further comprising a first portion of the ET model and a second portion of the ET model, the first and second portions of the ET model configured to predict or extract data representing an entity type from the text corpus based on the one or more entity results, the method further comprising: segmenting a text portion associated with an entity of an entity result into a first text portion before the location of the entity and a second text portion after the location of the entity; inputting the first text portion to the first portion of the ET model to predict data representing a first entity type estimate; inputting the second text portion to the second portion of the ET model to predict data representing a second entity type estimate; and combining the data representing the first and second entity type estimates to form an identified entity type associated with the entity.

17. The computer-implemented method of claim 16, wherein the data representing the first entity type estimate comprises a first N-dimensional vector and the data representing the second entity type estimate comprises a second N-dimensional vector, wherein combining the data representing the first and second entity types further comprises concatenating the first and second N-dimensional vectors and reducing a dimension of the resulting vector to a number of potential entity types, wherein each vector element of the resulting vector corresponds to a different entity type, a value of each vector element representing a likelihood that the entity is a particular entity type.

18. The computer-implemented method of claim 1, wherein the ET model is generated from the ML technique based on a bidirectional LSTM neural network. ​ 19. The computer-implemented method of claim 1, wherein model parameters of the ET model are jointly trained based on a plurality of labeled data items, each labeled data item comprising data representing a first text portion preceding a location of an entity and annotated with an entity type label associated with the entity, a second text portion following the location of the entity and annotated with the entity type label associated with the entity.

20. A machine learning, ML, model obtained from the computer-implemented method of any one of claims 1 to 19.

21. A device for entity type identification, comprising an entity type identification module configured to: receive one or more entity results, each entity result comprising data representing an identified entity and a location of the identified entity within a text corpus; identify an entity type of each entity in the received entity results by inputting text associated with the location of the each entity in the text corpus to a trained entity type, ET, model configured to predict or extract an entity type of the each entity from the text associated with the location of the each entity; and output data representing the identified entity type of each entity in the received entity results; wherein the text associated with the location of the each entity input to the ET model comprises text in the text corpus preceding and following the location of the each entity and does not include the text or string representing the entity.

22. A device for entity type identification, comprising a processor, a memory unit, and a communication interface, wherein the processor is connected to the memory unit and the communication interface, wherein the processor and memory are configured to implement the computer-implemented method of any one of claims 1 to 19.

23. A system for entity type identification, comprising: a named entity recognition, NER, system for generating entity results from a text corpus, each entity result comprising data representing an identified entity and a location of the identified entity within the text corpus; and the device of claim 21, the device coupled to the NER system, the device configured to receive one or more entity results from the NER system.

24. A computer readable medium comprising data or instruction code that, when executed on a processor, causes the processor to implement the computer-implemented method of any one of claims 1 to 19.

Images