A system for retrieving natural language documents

Abstract

The present invention provides a natural language search system and method. The system includes a digital data storage means for storing a plurality of blocks of natural language and a data graph corresponding to the blocks. A first data processing means is adapted to convert the blocks into the graph stored in the storage means. The graph includes a plurality of nodes having natural language units extracted from the blocks as node values. A second data processing means is adapted to run a machine learning algorithm that traverses the graph and reads the node values ​​to form a trained machine learning model based on the node structure of the graph and the node values ​​of the graph. A third data processing means is adapted to read the fresh graph and use the model to determine a subset of the blocks of natural language based on the fresh graph.

Description

Technical Field

[0001] Field of the Invention The present invention relates to natural language processing. In particular, the present invention relates to machine learning-based systems and methods, such as neural network-based, for searching, comparing, or analyzing documents containing natural language. The documents may be technical or scientific documents. In particular, the documents may be patent documents.

Background Art

[0002] Background of the Invention The comparison of documented technical concepts is required in many fields of business, industry, economy, and culture. A specific example is the examination of patent applications. The purpose of this examination is to determine whether the technical concept defined in the claims of a patent application meaningfully covers another technical concept defined in another document.

[0003] Currently, although the number of search tools for finding individual documents has increased, the analysis and comparison of concepts disclosed in documents are mostly manual tasks that rely on human inference of the meaning of words, sentences, and larger entities.

[0004] Scientific research on natural language processing has led to the development of tools for automatically analyzing language by computers. These tools can be used for text tokenization, part-of-speech tagging, entity recognition, identification of dependencies between words and entities, etc.

[0005] In addition, automatic analysis of patents aimed at extracting key concepts from patent documents and for purposes such as text summarization and analysis of technological trends is also being scientifically conducted.

[0006] In recent years, word embeddings using multidimensional word vectors have become an important tool for mapping the meaning of words into a numerical format that can be processed by computers. This approach can be used in neural networks such as recurrent neural networks, enabling computers to understand the content of documents more deeply.

[0007] Patent searches have traditionally been conducted using keyword searches. This involves defining appropriate keywords, their synonyms, and variations, and then creating a Boolean search strategy. This is time-consuming and requires specialized knowledge. More recently, semantic searches have been developed. These are more ambiguous and sometimes utilize artificial intelligence technology. Semantic searches are useful for quickly finding a large number of documents that have some relevance to concepts discussed in other documents. However, their ability to actually assess novelty—for example, finding documents disclosing specific content that corresponds to a general concept defined in a patent claim—is relatively limited.

[0008] In summary, there are technologies suitable for general searching, extracting core concepts from text, and summarizing text. However, they are not suitable for detailed comparisons between concepts disclosed in different documents contained in large datasets, which are important for patent novelty searches and other technical comparisons.

[0009] In particular, improvements in text analysis and comparison technologies are needed to realize more efficient search and novelty evaluation tools. [Overview of the Initiative]

[0010] The object of the present invention is to provide a novel system and method that solves at least some of the above-mentioned problems and improves the accuracy of technical searches. Specifically, the object is to provide a solution that can perform targeted searches while better considering the technical relationships between concepts in documents.

[0011] In particular, the aim is to provide systems and methods for improved patent searching and automated novelty assessment.

[0012] According to one embodiment, the present invention provides a natural language retrieval system comprising digital data storage means for storing a plurality of blocks of natural language and a data graph corresponding to the blocks. A first data processing means is also provided, adapted to convert the blocks into the graph stored in the storage means. The graph includes a plurality of nodes, preferably consecutive nodes, each containing natural language units extracted from the blocks as a node value or part thereof. A second data processing means is also provided, which executes a machine learning algorithm capable of traveling through the graph and reading node values ​​to form a trained machine learning model based on the node structure of the graph and the node values ​​of the graph. A third data processing means is also provided, adapted to read a fresh graph or fresh blocks of natural language converted into a fresh graph and to utilize the machine learning model to determine a subset of the blocks of natural language based on the fresh graph.

[0013] The present invention also relates to a method adapted to read blocks of natural language and to perform the functions of first, second, and third data processing means.

[0014] In one embodiment, the present invention provides a system and method for searching patent documents, the method comprising reading a plurality of patent documents, each containing a specification and claims, and converting the specification and claims into a specification graph and a claims graph, respectively. The graph comprises a plurality of nodes, each having a first natural language unit extracted from the specification or claims as a node value, and a plurality of edges between the nodes, the edges being determined based on at least one second natural language unit extracted from the specification or claims. The method comprises training a machine learning model using a machine learning algorithm that can travel through the graph according to the edges, and using the node values ​​to form the trained machine learning model using a plurality of different sets of the specification and claims graphs as training data. The method also comprises reading a fresh graph or a block of text converted into a fresh graph, and using the trained machine learning model to determine a subset of the patent documents based on the fresh graph.

[0015] The graph may be a tree-like recursive graph in particular, where there are meronym relationships between the node values ​​of at least some consecutive nodes.

[0016] This method and system are preferably neural network-based, and thereby the machine learning model is a neural network model.

[0017] More specifically, the present invention is characterized by the features described in the independent claims.

[0018] The present invention offers significant advantages. Compared to keyword-based searches, the graph-based neural network approach of the present invention has the advantage of considering the actual technical relationships between concepts within a document, rather than relying solely on the textual content of words or, optionally, other traditional criteria such as word proximity. For this reason, this approach is particularly suitable for patent searches and other applications where technical content is more important than precise wording or document style. This enables more accurate technical searches.

[0019] Compared to so-called semantic search using text-based linear neural network models, graph-based approaches can better consider the actual technical content of a document. Furthermore, lightweight graphs can be processed with far less computation than full text. This allows for the use of more training data, shortens the development and learning cycle, and enables more accurate searches. It also reduces actual search time.

[0020] This approach is compatible with the use of real-world training data, such as novelty search data and citation data provided by patent authorities and patent applicants. Furthermore, as will be detailed later, this approach also allows for advanced learning schemes such as data augmentation.

[0021] Real-world test data has shown that combining a condensed and simplified graphical representation of patent text with real-world training data can yield relatively high search accuracy and high computational learning efficiency.

[0022] The dependent claims are directed to selected embodiments of the present invention.

[0023] Next, selected embodiments of the present invention and their advantages will be described in more detail with reference to the accompanying drawings. [Brief explanation of the drawing]

[0024] [Figure 1A]Figure 1A is a block diagram of an exemplary search system at a general level. [Figure 1B] Figure 1B is a block diagram of a more detailed embodiment of a search system, including a pipeline of a neural network-based search engine and its trainer. [Figure 1C] Figure 1C is a block diagram of a patent search system according to an embodiment. [Figure 2A] Figure 2A is a block diagram of an exemplary nested graph having only meronym / holonym relationships. [Figure 2B] Figure 2B is a block diagram of an exemplary nested graph having meronym / holonym relationships and hyponym / hypernym relationships. [Figure 3] Figure 3 is a flowchart of an exemplary graph analysis algorithm. [Figure 4A] Figure 4A is a block diagram showing the learning of a patent search neural network using patent search / citation data as training data. [Figure 4B] Figure 4B is a block diagram showing the learning of a neural network using pairs of graphs of claims and specifications described in the same patent document as training data. [Figure 4C] Figure 4C is a block diagram of the training of a neural network using a set of claim graphs extended as training data. [Figure 5] Figure 5 shows the functionality of an exemplary graph feeding user interface according to an embodiment.

Best Mode for Carrying Out the Invention

[0025] definition In this specification, “natural language unit” means a chunk of text, or a vector representation of a chunk of text after embedding. A chunk can be a sub-concept of a single word or multiple words that appear one or more times in the original text stored in a computer-readable format. Natural language units can be represented numerically as a set of character values ​​(commonly known in computer science as a “string”), as a multidimensional vector value, or as a reference to such a value.

[0026] A "natural language block" is a data instance containing one or more complete or incomplete sentences in a language such as English, which is a linguistically meaningful combination of natural language units. Natural language blocks can be represented, for example, as a single string, stored in a file in the file system, and / or displayed to the user through a user interface.

[0027] A “document” is a machine-readable entity containing natural language content that is associated with a machine-readable document identifier that is unique to other documents in the system.

[0028] "Patent document" refers to the natural language content of a patent application or granted patent. In this system, patent documents are associated with a publication number and / or other machine-readable unique document identifier assigned by an accredited patent authority such as the EPO, WIPO, USPTO, or other national or regional patent offices. "Claim" refers to the essential content of the claims in a patent document, particularly the independent claims. "Specification" refers to the content of a patent document, including at least a portion of the description of the patent document. The specification may also cover other parts of the patent document, such as the abstract and claims. Claims and specification are examples of blocks of natural language.

[0029] In this specification, a “claim” is defined as any block of natural language that the European Patent Office considers to be a claim on the effective date of this patent application. In particular, a “claim” is a computer-identifiable block of natural language document that is identified therein by a machine-readable integer number, for example, in string form prior to the block and / or as part of related information in a markup file format such as XML or HTML.

[0030] "Specification" is defined as a computer-identifiable block of natural language that includes at least one non-claim portion within a patent document containing at least one claim. "Specification" can also be identified by related information in markup file formats such as XML or HTML.

[0031] In this context, "edge relationships" specifically refer to technical relationships extracted from blocks, and / or semantic relationships obtained using the semantics of the natural language unit in question. Specifically, edge relationships are as follows:

[0032] - Meronym relationship (also known as meronim / holonym relationship); Meronym: X is a part of Y; Holonym: Y is a part of X; For example. For example, "wheel" is a meronim of "car".

[0033] -Hyponymic relationship (also known as: hippomym / hypernym relationship); Hyponym: X is subordinate to Y, hypernym: X is superior to Y; Example: "electric car" is the hippomym of "car", or -Synonym relationship: X is the same as Y.

[0034] In some embodiments, edge relationships are defined between successively nested nodes in a recursive graph, each node containing a natural language unit as its node value.

[0035] Furthermore, in addition to the relationships described above, possible technical relationships include thematic relationships, which refer to the role that one sub-concept of a text plays for one or more other sub-concepts. At least some thematic relationships can be defined between consecutively nested units. In one example, the thematic relationship of a parent unit is defined in a child unit. An example of a thematic relationship is the role class "Function." For example, the function of "Handle" could be "to enable the manipulation of an object." Such a thematic relationship can be stored as a child unit of the "Handle" unit, and the "Function" role is associated with the child unit. Thematic relationships can also be generic relationships that do not have a predefined class (or have a general class such as "Relationship"), but can be freely defined by the user. For example, a generic relationship between a handle and a cup could be "[Handle] is attached to [Cup] with adhesive." Such thematic relationships can be stored as child units of either the "Handle" unit, the "Cup" unit, or both, preferably referencing each other.

[0036] A relational unit is considered to define a relationship of a particular relational class or subclass if, when executed by a data processing device, it is linked to computer-executable code that generates a block of natural language containing the relationships of that class or subclass.

[0037] A “graph” or “data graph” generally refers to a data instance that follows a non-linear recursive data schema and / or network data schema. This system can simultaneously contain multiple different graphs whose data originates from and / or relates to different sources, while still following the same data schema. Graphs can, in practice, be stored in any suitable text or binary format that allows for the storage of data items as recursive and / or networks. Graphs are, in particular, semantic and / or technical graphs (describing semantic and / or technical relationships between node values), in contrast to syntactic graphs (describing only linguistic relationships between node values). Graphs may also be tree-form graphs. A forest-form graph containing multiple trees is considered a tree-form graph in this specification. In particular, a graph may be a technical tree-form graph.

[0038] A "data schema" is a set of rules that organize data, particularly natural language units and related data (such as information about the technical relationships between units).

[0039] "Nesting" in natural language units refers to the ability of a unit to have one or more children and one or more parents, as determined by the data schema. In some examples, a unit can have one or more children and only one parent. Root units have no parents, and leaf units have no children. Sibling units share the same parent. "Consecutive nesting" refers to nesting between a parent unit and its direct child units.

[0040] A "recursive" nesting or data schema is a nesting or data schema that allows natural language units containing data items to be nested within each other.

[0041] A "natural language token" refers to a word or word chunk within a larger block of natural language. Tokens may contain metadata associated with the word or word chunk, such as part-of-speech (POS) labels and syntactic dependency tags. A "set" of natural language tokens refers specifically to tokens that can be grouped according to predetermined rules or fuzzy logic based on text values, POS labels, dependency tags, or a combination thereof.

[0042] "Data storage means," "processing means," and "user interface means" primarily refer to software means, i.e., computer-executable code (instructions), that are stored in a non-temporary computer-readable medium and, when executed by a processor, are adapted to perform specified functions, namely, the storage of digital data, the manipulation of data by a user, and the processing of data, respectively. All of these components of the system can be performed by software executed by either a local computer or a web server, for example, via a locally installed web browser, supported by appropriate hardware for executing the software components. The methods described herein are methods executed on a computer.

[0043] Description of the selected embodiment The following describes a natural language search system comprising a digital data storage means for storing multiple blocks of natural language and data graphs corresponding to those blocks. The storage means may consist of one or more local or cloud data stores. The stores can be file-based or query language-based.

[0044] The first data processing means is a transformation unit adapted to transform the block into the graph. Each graph contains a plurality of nodes, each containing a natural language unit extracted from the block as a node value. Edges are defined between pairs of nodes and define the technical relationships between them. For example, an edge or part thereof may define a meronme relationship between two nodes.

[0045] In some embodiments, the number of at least some nodes containing values ​​for a particular natural language unit in the graph is smaller than the number of occurrences of that particular natural language unit in the corresponding natural language block. In other words, the graph is a condensed representation of the source text, which can be achieved, for example, using token recognition and matching methods described later. By allowing multiple child nodes for each node, the essential technical (and optionally semantic) content of the text can be preserved in the graph representation. Condensed graphs are also efficient for processing by graph-based neural network algorithms, which can learn the essential content of the text better and faster than from the direct text representation. This approach is particularly powerful in the comparison of technical documents, especially in claim-based patent specification searches and automated assessment of claim novelty.

[0046] In some embodiments, the number of nodes containing a particular natural language unit is 1; that is, there are no duplicate nodes. This may simplify the original content of the text, at least when using a tree-like graph, but the result is a highly efficient, relatively expressive graph that is suitable for patent searching and novelty assessment.

[0047] In some embodiments, the graph is a condensed graph for at least the nouns and noun chunks found in the original text. In particular, the graph may be a condensed graph for noun value nodes arranged according to meronymic relationships. In an average patent document, many noun words appear dozens or even hundreds of times throughout the text. This method makes it possible to compress the content of such documents to a fraction of the original space while making them suitable for machine learning.

[0048] In some embodiments, multiple terms that appear multiple times in at least one original block of natural language appear exactly once in the corresponding graph.

[0049] Condensed graph representations also have the advantage of being able to construct graphs that take synonyms and coreferences (expressions that mean the same thing in a particular context) into account. As a result, even more condensed graphs are obtained. In some embodiments, multiple terms that appear in at least one original block of natural language in at least two different descriptive forms appear exactly once in the corresponding graph.

[0050] The second data processing means is a neural network trainer for executing a neural network algorithm that can iteratively traverse a graph structure and learn from both the internal structure of the graph and its node values, as defined by a loss function that defines the learning objective along with the training data cases. The trainer typically receives a combination of the graph specified by the training algorithm or an extended graph derived therefrom as training data. The trainer outputs a trained neural network model.

[0051] Supervised machine learning methods using this type of graph-based data have proven to be highly effective in identifying technically relevant documents within patent and scientific documents.

[0052] In some embodiments, the storage means is further configured to store reference data that links at least some of the blocks together. The reference data is used by the trainer to derive training data, i.e., to define positive or negative training cases, i.e., combinations of graphs used for training as training samples. The trainer's learning objectives depend on this information.

[0053] The third data processing means is a search engine, which is typically adapted to read fresh graphs or fresh blocks of natural language, usually through a user interface or network interface. If necessary, the blocks are converted into graphs by a conversion unit. The search engine uses a trained neural network model to determine a subset of natural language blocks (or graphs derived therefrom) based on the fresh graph.

[0054] Figure 1A shows one embodiment of the system, which is particularly suitable for searching technical documents such as patent documents and scientific documents. The system includes a document store 10A containing multiple natural language documents. A graph parser 12, which is adapted to read documents from the document store 10A and convert them into a graph format, will be described in more detail later. The converted graphs are stored in the graph store 10B.

[0055] This system includes a neural network trainer unit 14, which receives a set of analyzed graphs from a graph store and some information about their interrelationships as training data. In this case, a document reference data store 10C is provided, which includes citation data and / or novelty search results for documents. The trainer unit 14 runs a graph-based neural network algorithm to generate a neural network model for a neural network-based search engine 16. The engine 16 uses graphs from the graph store 10B as the target search set and uses user data (typically text or graphs) obtained from the user interface 18 as references.

[0056] The search engine 16 may be, for example, a graph-vector search engine trained to find the vector in the graph store 10B that is closest to the vector formed from user data. Alternatively, the search engine 16 may be a classifier search engine, such as a binary classifier search engine, that compares pairs of the user's graph, or vectors derived therefrom, with graphs obtained from the graph store 10B, or vectors derived therefrom.

[0057] Figure 1B shows one embodiment of the system, further comprising a text embedding unit 13 that converts natural language units of a graph into a multidimensional vector format. This is done for the converted graph, the graph from the graph store 10B, and the graph input via the user interface 18. Typically, the vector has at least 100 dimensions, for example, 300 or more.

[0058] In one embodiment, also shown in Figure 1B, the neural network search engine 16 is divided into two parts that form a pipeline. Engine 16 consists of a graph embedding engine that converts a graph into a multidimensional vector format using a model trained (learned) by the graph embedding trainer 14A of the neural network trainer 14 using reference data from the document reference data store 10C, for example. The user's graph is compared in the vector comparison engine 16B with graphs pre-generated by the graph embedding engine 16A. As a result, a narrowed subset of graphs that are closest to the user's graph is found. The subset of graphs is further compared with the user's graph by the graph classifier engine 16C to further narrow down the set of related graphs. The graph classifier engine 16C is trained (learned) by the graph classifier learner 14C using data from the document reference data store 10C as training data, for example. This embodiment is advantageous because, while the comparison of pre-formed vectors by the vector comparison engine 16B is very fast, the graph classification engine can access the detailed data content and structure of the graph and perform accurate comparisons to find differences in the graphs. The graph embedding engine 16A and the vector comparison engine 16B function as efficient pre-filters for the graph classification engine 16C, reducing the amount of data that needs to be processed by the graph classification engine 16C.

[0059] The graph embedding engine can convert a graph into a vector with at least 100 dimensions, preferably 200 dimensions or more, and even 300 dimensions or more.

[0060] The neural network trainer 14 is divided into a graph embedding unit and a graph classification unit, which are trained (learned) using the graph embedding trainer 14A and the graph classification trainer 16C, respectively. The graph embedding trainer 14A forms a neural network-based graph-vector model and aims to form neighbor vectors of graphs that are similar to each other in terms of text content and internal structure. The graph classifier trainer 14B forms a classifier model and can rank pairs of graphs according to the similarity of their text content and internal structure.

[0061] The user data obtained from the user interface 18 is embedded in the embedding unit 13 and then fed to the graph embedding engine for vectorization. Subsequently, the vector comparison engine 16B finds the set of vectors that most closely correspond to the graphs in the graph store 10B. The set of closest graphs is fed to the graph classifier engine 16C, which uses a trained graph classifier model to compare them one by one with the user's graphs in order to obtain an accurate match.

[0062] In some embodiments, the graph embedding engine 16A outputs vectors whose angles are closer to each other in terms of both node content and node structure, learned from reference data using its dependent learning objectives, as trained by the graph embedding trainer 14A. Through training, the angles of the vectors obtained from the reference data can be minimized for positive learning examples (graphs depicting the same concept) and maximized, or at least significantly deviated from zero, for negative learning examples (graphs depicting different concepts).

[0063] Graph vectors can have dimensions of, for example, 200-1000, or 250-600.

[0064] Such supervised machine learning models have been shown to efficiently evaluate the similarity of technical concepts revealed by graphs, and furthermore, to evaluate blocks of natural language derived from those graphs.

[0065] In some embodiments, the graph classifier engine 16C, once trained by the graph classifier learner 14C, outputs a similarity score that is higher the more similar the compared graphs are in terms of both node content and node structure, using learning objectives that depend on the engine. Through training, the similarity scores of positive training examples (graphs depicting the same concept) obtained from the reference data are minimized, and the similarity scores of negative training examples (graphs depicting different concepts) are maximized.

[0066] Cosine similarity is one of the criteria used to express the similarity between graphs and the vectors derived from them.

[0067] It should be noted that the graph classifier trainer 14C or engine 16C is not mandatory, and graph similarity can be directly evaluated based on the angle between vectors embedded by the graph embedding engine. For this purpose, a known fast vector index can be used to find one or more nearby graph vectors for a given fresh graph vector.

[0068] The neural networks used by the trainer 14 and the search engine 16, or by any or both of its subtrainers 14A, 14C, or subengines 16A, 16C, may be recurrent neural networks, particularly those utilizing LSTM (Long Short-Term Memory) units. In the case of a tree-structured graph, the network may be a tree-LSTM network, such as a Child-Sum-Tree-LSTM network. The network may have one or more LSTM layers and one or more network layers. The network may use an attention mechanism to relate parts of the graph to each other internally or externally during model training and / or execution.

[0069] Some further embodiments of the present invention are described below in the context of a patent search system, wherein the documents processed are patent documents. The general embodiments and principles described above are applicable to a patent search system.

[0070] In some embodiments, the system is configured to store natural language documents in a storage means, each containing a first natural language block and a second natural language block distinct from the first natural language block. The trainer can use a plurality of first graphs corresponding to the first block of the first document, and for each first graph, one or more second graphs defined by reference data and at least partially based on the second block of a second document distinct from the first document. In this way, the neural network model learns from the interrelationships between different parts of different documents. Alternatively, the trainer can use a plurality of first graphs corresponding to the first block of the first document, and for each first graph, a second graph at least partially based on the second block of the first document. In this way, the neural network model can learn from the internal relationships of data within a single document. Both of these learning methods can be used individually or in combination by the patent search system described below.

[0071] The condensed graphical representation described above is particularly well-suited to patent search systems, specifically to graphs of claims and specifications, and especially to graphs of specifications.

[0072] Figure 1C shows a system including a patent document store 10A containing a patent document that includes at least a computer-identifiable specification portion and a claims portion. The graph parser 12 is configured to analyze the claims by a claims graph parser 12A and the specification by a specification graph parser 12B. The analyzed graphs are stored separately in the claims and specification graph stores 10B. The text embedding unit 13 prepares the graphs for processing by a neural network.

[0073] The reference data includes publicly available patent applications, patent search and examination data, and citation data between patent documents. In one embodiment, the reference data includes previous patent search results, i.e., information on which previous patent documents are considered to be the basis for the novelty and / or inventive step of a later filed patent application. The reference data is stored in the previous patent search and / or citation data store 10C.

[0074] The neural network trainer 14 uses the analyzed and embedded graphs to form a neural network model specifically trained for the purpose of patent searching. This is achieved by using patent search and / or citation data as input to the trainer 14. Its objective is, for example, to minimize the vector angle between the claim graph of a patent application and the specification graph of a patent document used as a novelty barrier against it, or to maximize the similarity score. By applying this to multiple (typically hundreds of thousands or millions) claims, the model learns to evaluate the novelty of claims with respect to the prior art. This model is then used by the search engine 16 against the user graph obtained via the user interface 18A to find the most likely novelty barriers (bars). The results can be displayed on the search results display interface 18B.

[0075] In the system shown in Figure 1C, a search engine pipeline can be utilized. The engine may be trained on the same or different subsets of training data obtained from previous patent searches and / or citation data store 10C. For example, a set of graphs can be filtered from a complete prior art dataset using a graph embedding engine trained on a large or complete reference dataset, i.e., positive and negative claim / specification pairs. The filtered set of graphs is then classified against the user's graphs in a classification engine, which may be trained on a smaller, e.g., patent class-specific reference dataset, i.e., positive and negative claim / specification pairs, to find graph similarities.

[0076] Next, with reference to Figures 2A and 2B, we will describe a tree-like graph structure that is particularly applicable to patent search systems.

[0077] Figure 2A is a tree-like graph where only meronim relationships are edge relationships. Text unit AD is derived from the root node 10 and is placed in the graph as linearly recursive nodes 10, 12, 14, and 16, and text unit E is a child of node 12, derived from the shown natural language block and placed as child node 18. Here, meronim relationships are detected from the meronim / holonym expressions "comprises", "having", "is contained in", and "includes".

[0078] Figure 2B is another tree-like graph with two different edge relationships, in this example, a meronim relationship (first relationship) and a hyponym relationship (second relationship). Text unit AC is located as linear recursive nodes 10, 12, and 14, which have a meronim relationship. Text unit D is located as child node 26 of parent node 14, which has a hyponym relationship. Text unit E is located as child node 24 of parent node 12, which has a hyponym relationship. Text unit F is located as child node 28 of node 24, which has a meronim relationship. Here, the meronim and hyponym relationships are found in the meronim / holonym expressions "comprises," "having," "such as," and "is for example."

[0079] According to one embodiment, the first data processing means is adapted to transform a block into a graph by first identifying from the block a first set of natural language tokens (e.g., nouns and noun chunks) and a second set of natural language tokens different from the first set of natural language tokens (e.g., meronims and holonym representations). Then, a matcher is executed using the tokens from the first set and the tokens from the second set to form matched pairs of tokens from the first set (e.g., "body comprises member" - "body" and "member"). Finally, the tokens from the first set are placed as nodes in the graph using the matched pairs (e.g., "body" - (meronym edge) - "member").

[0080] In one embodiment, the graph uses at least meronim edges, and each node contains natural language units obtained from the block that have meronim relationships with one another.

[0081] In one embodiment, hyponym edges are used in the graph, and each node contains natural language units that have a hyponym relationship with one another, derived from natural language blocks.

[0082] In one embodiment, edges are used in the graph, and at least one of its nodes contains a reference to one or more nodes in the same graph, and further contains at least one natural language unit derived from each block of natural language (e.g., "is below" [node id:X]). In this way, it is possible to save graph space and achieve expressive data content within the graph while maintaining a simple graph structure such as a tree form.

[0083] In some embodiments, the graph is a tree-like graph whose node values ​​include words or chunks of multiple words derived from the aforementioned blocks of natural language, typically utilizing the part-of-speech and syntactic dependencies of words, or their vectorized forms, by graph transformation units.

[0084] Figure 3 provides a detailed example of how the text-to-graph conversion is performed in the first data processing method. First, in step 31, the text is read, and a first set of natural language tokens, such as nouns, and a second set of natural language tokens, such as tokens indicating meronymity and holonymity (e.g., "comprising"), are detected from the text. This can be done by tokenizing the text in step 32, tagging the tokens with part-of-speech (POS) tags in step 33, and deriving their syntactic dependencies in step 34. Using this data, noun chunks can be determined in step 35, and meronym and holonym representations in step 36. In step 37, pairs of matched noun chunks are formed using the meronym and holonym representations. These pairs of noun chunks can be used to form or subtract edges of meronym relationships in the graph.

[0085] In one embodiment, as shown in step 38, pairs of noun chunks are arranged as a tree-like graph where the meronim is a child of the corresponding holonym. This graph can be saved to the graph store in step 39, as described above, for further use.

[0086] In one embodiment, the graph formation step uses a probabilistic graph model (PGM), such as a Bayesian network, to infer a preferred graph structure. For example, different edge probabilities of the graph can be calculated based on the Bayesian model, and then the most preferred graph form can be calculated using these edge probabilities.

[0087] In one embodiment, the graph formation step consists of inputting tokenized, POS-tagged, and dependency-analyzed text into a neural network-based technical parser. The neural network-based technical parser finds relevant chunks from the text blocks and extracts desired edge relationships, such as meronym relationships and hyponym relationships.

[0088] In one embodiment, the graph is a tree-like graph consisting of edge relationships recursively arranged according to a tree data schema, and is acyclic. This allows the use of efficient tree-like neural network models, whether recurrent or non-recurrent. For example, there is the Tree-LSTM model.

[0089] In another embodiment, the graph is a network graph, allowing for cycles, or edges between branches. This has the advantage of being able to represent complex edge relationships.

[0090] In yet another embodiment, the graph is a forest of linear and / or nonlinear branches with one or more edge lengths. Linear branches have the advantage of avoiding or dramatically simplifying the tree or network construction steps and allowing the neural network to utilize the maximum amount of source data.

[0091] Each model can save the likelihood of edges obtained from the PGM model and use it in the neural network.

[0092] It should be noted that the graph formation method described with reference to Figure 3 can be implemented separately from other methods and systems described in this book to form and preserve the technical content of a document, particularly a condensed technical representation of patent specifications and claims.

[0093] Figures 4A-C illustrate neural network training methods for patent retrieval, and these methods are not mutually exclusive.

[0094] In general cases, the term "patent document" can be replaced with "document" (which has a unique computer-readable identifier among other documents in the system). Also, "claims" can be replaced with "a first computer-identifiable block," and "specification" with "a second computer-identifiable block that is at least partially different from the first block."

[0095] In the embodiment shown in Figure 4A, multiple claim graphs 41A, linked by reference data, and a nearby prior art specification graph 42A corresponding to each claim graph, are used as training data by a neural network trainer 44A. These form positive training cases, demonstrating the achievement of low vector angles or high similarity scores between the graphs. Furthermore, negative training cases, i.e., one or more distant prior art graphs for each claim graph, can be used as part of the training data. High vector angles or low similarity scores between such graphs should be achieved. Negative training cases can be randomly selected, for example, from the entire set of graphs.

[0096] According to one embodiment, in at least one phase of training, such as that performed by the neural network trainer 44A, a set of negative training cases that are harder than the average of all possible negative training cases are selected from a subset of all possible training cases. For example, hard negative training cases may be selected such that both the claim graph and the description graph are from the same patent class (up to a predetermined classification level), or such that the neural network previously failed to correctly classify the description graph as a negative case (with a predetermined confidence level).

[0097] According to one embodiment, which may be carried out independently of other methods and system components described herein, training (learning) of the neural network-based patent search or novelty assessment system is performed by providing a plurality of patent documents, each having a computer-identifiable claim block and a specification block, the specification block including at least a portion of the description of the patent document. The method also includes providing a neural network model and training the neural network model using a training dataset containing data from the patent documents to form a trained neural network model. The training includes using pairs of claim blocks and specification blocks derived from the same patent document as training cases of the training dataset.

[0098] Generally, positive training cases within such documents account for only about 1-25% of the total training cases, with the remainder being training cases such as search reports (citations regarding novelty by examiners).

[0099] The machine learning model of the present invention is typically configured to convert claims and specifications into vectors, and the learning objective of training the model may be to minimize the vector angle between the claim and specification vectors of the same patent document. Alternatively, another learning objective may be to maximize the vector angle between the claim and specification vectors of at least several different patent documents.

[0100] In the embodiment shown in Figure 4B, multiple claim graphs 41A and specification graphs 42A derived from the same patent document are used as training data by a neural network trainer 44B. The "own" specification of a claim typically forms a complete positive training case; that is, the patent document itself technically becomes the ideal novelty barrier for that claim. Thus, the pair of these graphs forms a positive training case, indicating that a low vector angle or high similarity score is achieved between these graphs. In this scenario as well, reference data or negative training cases can be used.

[0101] Tests have shown that simply adding claim and description pairs from the same document to training data based on real-world novelty searches improves prior art classification accuracy by more than 15% when tested with test data pairs also based on real-world novelty searches.

[0102] In a typical case, at least 80%, usually at least 90%, and often 100%, of the machine-readable content (natural language units, especially words) of a claim is contained somewhere in the specification of the same patent document. Thus, the claims and specification of a patent document are linked to each other not only through the recognizable content and the same unique identifiers (e.g., publication numbers), but also through byte-level content.

[0103] According to one embodiment, which may be carried out independently of other methods and system components described herein, training (learning) the neural network-based patent search or novelty assessment engine includes deriving at least one reduced data instance from at least several original claim or specification blocks that partially correspond to the original blocks, and using the reduced data instance together with the original claim or specification blocks as a training case for the training dataset.

[0104] In the embodiment shown in Figure 4C, positive training cases are augmented by forming multiple reduced claim graphs 41C''-41C'''' from the original claim graph 41C'. A reduced claim graph refers to a graph like the following:

[0105] - At least one node will be removed (e.g., phone display sensor → phone display). - At least one node has moved to a different location that is higher (more common) in the branch (e.g., phone-display-sensor → phone-(display, sensor)), and / or - The value of a natural language unit in at least one node is replaced with the value of a more general natural language unit (e.g., phone-display-sensor → electronic device-display-sensor).

[0106] This augmentation scheme allows for the expansion of the neural network's training set, resulting in a more accurate model. It also makes meaningful novelty searches and evaluations of so-called trivial inventions—those using only a few nodes or very general terminology, which are rarely seen in actual patent novelty search data. Data augmentation can be implemented in relation to either embodiment of Figures 4A and 4B, or a combination thereof. Negative training cases can also be used in this scenario.

[0107] Negative training cases can also be extended by deleting, moving, or swapping nodes and their values ​​in the specification graph.

[0108] Tree-like graph structures, such as those based on meronym relationships, are advantageous for augmentation because they can be augmented while maintaining consistent logic by deleting nodes or moving them to higher positions in the tree. In this case, both the original data instances and the reduced data instances are graphs.

[0109] In one embodiment, a reduced graph is a graph from which at least one leaf node has been removed from the original graph or another reduced graph. In one embodiment, all leaf nodes at a certain depth in the graph are removed.

[0110] In particular, natural language blocks can be directly extended by deleting parts of them or partially changing their content to more general terms.

[0111] The number of reduced data instances per original instance can be, for example, 1 to 10,000, and especially 1 to 100. Good training results can be obtained in claim expansion using 2 to 50 expansion graphs.

[0112] In some embodiments, the search engine reads fresh blocks of natural language, such as fresh claims, and converts them into fresh graphs using a converter, or it inputs fresh graphs directly through a user interface. User interfaces suitable for direct graph input are described below.

[0113] Figure 5 shows an example of graph representation and modification on a user interface display element 50. The display element 50 consists of multiple editable data cells AF, whose values ​​are functionally connected to corresponding natural language units in the underlying graph (e.g., corresponding unit AF) and displayed on their respective user interface (UI) data elements 52, 54, 56, 54', 56', and 56''. The UI data elements may be, for example, text fields whose values ​​can be edited by the keyboard after the element is activated. The UI data elements 52, 54, 65, 54', 56', and 56'' are positioned horizontally and vertically on the display element 50, depending on their position in the graph. Here, the horizontal position corresponds to the depth of the unit in the graph.

[0114] The display element 50 can be, for example, a window, frame, or panel of a web browser running a web application, or a graphical user interface window of a standalone program that can run on a computer.

[0115] Furthermore, the user interface includes a shift engine that can move natural language units horizontally (or vertically) on the display elements in response to user input, thereby modifying the graph accordingly. To illustrate this, Figure 5 shows data cell F (element 56'') being shifted one level to the left (arrow 59A). This causes the original element 56'', which was nested under element 54', to disappear, and an element 54'' is formed nested under the higher-level element 52, constituting data cell F (original value). Subsequently, when data element 54' is shifted two levels to the right (arrow 59B), data element 54' and its children are shifted to the right and nested under data element 56 as data element 56'''' and data element 58. Each shift is reflected in a corresponding shift in the nesting level of the underlying graph. In this way, the children of a unit are preserved in the graph even when shifted to different nesting levels in the user interface.

[0116] In some embodiments, a UI data element consists of a natural language helper element that is displayed in relation to an editable data cell to assist the user in entering natural language data. The content of the helper element can be formed using relational units associated with the natural language unit and, optionally, the natural language unit of its parent element.

[0117] Instead of a graph-based user interface like the one shown in Figure 5, a user interface that allows input of block text, such as independent claims, is also acceptable. This text block is fed to a graph parser to obtain a graph that can be used in the next stage of the search system.

Claims

1. In natural language search systems, A digital data storage means (10A, 10B) comprising: Multiple blocks of natural language; a graph of data corresponding to said blocks; a digital data storage means (10A, 10B) for storing the a first data processing means (12) adapted to convert the blocks into the graph stored in the storage means, the graph comprising a plurality of nodes each comprising as node values ​​the natural language units extracted from the blocks; The system further comprises: a second data processing means (14) for executing a machine learning algorithm using positive training data, which are pairs of the blocks of the natural language that show at least a high similarity score, to form a trained machine learning model based on the node structure of the graph and the node values ​​of the graph; and third data processing means (16) adapted to read a new graph or a new block of natural language to be converted into a new graph, and to use the trained machine learning model to determine, based on the new graph, a subset of the natural language blocks stored in the storage means that are similar to the new block of natural language, thereby vectorizing the new graph and each of the graphs of the plurality of blocks of natural language and outputting a similarity score between the graphs.

2. 2. The system of claim 1, wherein the number of at least some nodes in at least some graphs that include a particular natural language unit value is smaller than the number of occurrences of the particular natural language unit value in a corresponding block of natural language.

3. 3. The system according to claim 1 or 2, wherein the first data processing means (12) comprises: identifying a first set of natural language tokens from the blocks and a second set of natural language tokens different from the first set of natural language tokens; executing a matcher utilizing the first set of tokens and the second set of tokens to form matched pairs of tokens in the first set; placing at least a portion of the first set of tokens as successive nodes of the graph using the matched pairs; the system being adapted to transform the blocks into the graph by

4. 3. A system according to claim 1 or 2, wherein the first data processing means (12) is adapted to form a graph comprising a plurality of edges, each of the nodes comprising natural language units derived from the blocks and having meronymous relationships with each other.

5. 3. A system according to claim 1 or 2, wherein the first data processing means (12) is adapted to form a graph comprising a plurality of edges, each of the nodes comprising natural language units derived from the blocks and having a hyponym relationship to one another.

6. 3. The system according to claim 1 or 2, wherein the first data processing means (12) is adapted to form a graph comprising a plurality of edges, at least one of which may contain a reference to one or more nodes in the same graph and additionally to at least one natural language unit derived from a respective block of the natural language.

7. 3. A system according to claim 1 or 2, wherein the graph is a tree-type graph whose node values ​​comprise words or chunks of multiple words, or a vectorized form thereof, derived by the first data processing means from the blocks of natural language using word parts of speech and syntactic dependencies.

8. 3. The system of claim 1 or 2, wherein the first data processing means (12) is adapted to use a probabilistic graph model (PGM) to determine edge probabilities for a graph and to form a graph using the edge probabilities.

9. 3. The system according to claim 1 or 2, wherein the second data processing means (14) is adapted to process a graph-based neural network algorithm such as a Recurrent Neural Network (RNN) graph algorithm, in particular a Long Short-Term Stochastic Model (LSTM) algorithm such as a Tree-LSTM algorithm. a system adapted to execute a memory-based algorithm.

10. 3. The system of claim 1 or 2, wherein the trained machine learning model is adapted to map a graph into multidimensional vectors, and angles between the multidimensional vectors are defined by a node structure of the graph and node values ​​of the graph.

11. 3. The system of claim 1 or 2, wherein the machine learning model is adapted to classify a graph or pair of graphs into two or more classes depending on the node structure of the graph and the node values ​​of the graph.

12. 3. The system according to claim 1 or 2, the storage means is further configured to store reference data linking at least some of the blocks to one another; The machine learning algorithm has a learning goal that depends on the reference data for training the machine learning model.

13. 3. The system according to claim 1, wherein the storage means is configured to store natural language documents each including a first natural language block and a second natural language block.

14. 13. The system of claim 12, wherein the second data processing means (14) is configured to use, in the training, a plurality of first graphs corresponding to first blocks of a first document and, for each first graph, one or more second graphs based at least in part on second blocks of a second document defined by the reference data and different from the first document.

15. 13. The system of claim 12, wherein the second data processing means (14) is configured to use in the training a plurality of first graphs corresponding to first blocks of first documents and, for each first graph, a second graph that is at least partially based on second blocks of the first documents, the second blocks being at least partially different from the first blocks.

16. 3. A system according to claim 1 or 2, wherein the third data processing means (16) is adapted to read the new natural language input as a new graph or as a new block of natural language which is converted into a corresponding graph.

17. 3. The system of claim 1 or 2, which is a patent search system that uses claims and descriptions as the natural language blocks.

18. 1. A computer-implemented method for searching natural language documents, comprising: storing a plurality of blocks of natural language in a digital data store; converting the blocks into a corresponding graph, the graph comprising a plurality of nodes each containing as a node value a natural language unit extracted from the blocks; storing the graph in the digital data store; The method further comprises: running a machine learning algorithm using positive training data, which are pairs of the blocks of the natural language that exhibit at least high similarity scores, to form a trained machine learning model based on the node structure of the graph and the node values ​​of the graph; reading a new graph or a new block of natural language to be converted into a new graph; and using the trained machine learning model to determine a subset of the natural language blocks stored in the digital data store based on the new graph, the subset being similar to the new block of natural language, and vectorizing each of the new graph and the graphs of the natural language blocks, and outputting a similarity score between the graphs.

Images