A method and system for storage and retrieval of knowledge representation data structures

Abstract

The application discloses a storage and retrieval method and system of knowledge representation data structure, comprising: storing each knowledge unit of the knowledge representation data structure as an equivalent equation document, which contains a first expression field and a second expression field; writing the first expression and the second expression with semantic equivalent mapping relationship into the two fields; establishing a special index library for the equivalent equation document, and taking the equivalent equation document as a first-class citizen data type; and managing the equivalent equation document independently of a common text document, wherein the special index library comprises a word segmentation inverted index layer and a feature code accurate index layer. The application can embed an equivalent relationship primitive into an index layer, and originally support equivalent relationship processing; thus, the application realizes integration of query, reasoning and retrieval, discards an independent reasoning module, reduces module interaction, temporary data and network calling overhead, and lowers system complexity. Meanwhile, the application can switch a keyword fuzzy retrieval mode and an expression accurate matching mode as required without an additional storage structure.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method and system for storing and retrieving knowledge representation data structures. Background Technology

[0002] With the widespread application of knowledge graph technology in fields such as industrial manufacturing, medical diagnosis, and legal retrieval, the core challenge facing search engine systems has become how to quickly retrieve semantically matching information from massive amounts of unstructured or semi-structured knowledge text.

[0003] Existing search engine technologies typically include:

[0004] (1) Full-text search engine based on keyword inverted index

[0005] The most mainstream search engine solution in the industry today is to build an inverted index based on word segmentation, mapping each word to a list of document IDs containing that word, and returning results through keyword matching and relevance scoring during queries. Elasticsearch (hereinafter referred to as ES) and other open-source search engines are representative examples of this approach.

[0006] The shortcomings of this implementation scheme include:

[0007] Unable to handle semantic equivalence relations: Inverted indexes rely solely on keyword matching and cannot establish semantic association channels across terms; for example, if a user queries "what is the pH value of hydrochloric acid", even if the document stores "the pH value of acidic substances is less than 7", the system cannot automatically establish a semantic mapping of "hydrochloric acid is an acidic substance". In other words, the inverted index cannot establish a semantic channel of "hydrochloric acid is an acidic substance" based solely on keyword matching, resulting in an empty set being returned.

[0008] Unable to leverage logical equivalence for extended recall: Inverted indexes are single-level mappings from terms to documents and do not support expressing atomic-level equivalence relationships such as "A is equivalent to B".

[0009] (2) Semantic search engine based on OWL ontology

[0010] Its core is to overlay an ontology reasoning layer on top of the document. First, the query terms are mapped to a pre-built OWL ontology class hierarchy. When the query is executed, the reasoning engine expands the inheritance relationship of the ontology (e.g., "hydrochloric acid rdf:type acidic substance") and substitutes the parent class constraints into the query conditions to achieve extended semantic retrieval.

[0011] The main drawbacks of this implementation scheme include:

[0012] The index size is growing rapidly: it is necessary to extract facts from all documents in the database and build external rule indexes. When the data size increases, the time spent on index building is difficult to control.

[0013] Uncontrollable inference latency: Ontology inference requires recursively traversing the tree structure and generating a temporary triplet table. A single retrieval cannot be completed in one I / O operation, making it difficult to meet the needs of high-frequency real-time queries and thus unsuitable for high-frequency real-time query scenarios.

[0014] In summary, existing search engines cannot directly embed "equivalence relation" primitives into inverted indexes and query processes. For knowledge queries involving semantic equivalence, logical implication, and transitive inference relations, they can only rely on external, independent reasoning modules, which leads to problems such as index redundancy, increased I / O overhead, and higher query latency.

[0015] The above problem can be specifically manifested in the following two related sub-problems:

[0016] Sub-problem 1: Traditional search engines based on keyword inverted indexes cannot express and utilize the atomic relationship of "A is equivalent to B" at the document level, causing semantic equivalence query matching to fail.

[0017] Sub-problem 2: Existing semantic search solutions based on ontology reasoning completely separate the reasoning module from the indexing module. The reasoning process requires the generation of a large amount of intermediate temporary data, which is difficult to complete within a compact query pipeline.

[0018] In view of this, the present invention is hereby proposed. Summary of the Invention

[0019] The purpose of this invention is to provide a method and system for storing and retrieving knowledge representation data structures, so as to directly embed equivalence relation semantics into the inverted index and query process, thereby solving the technical problems existing in the prior art.

[0020] The objective of this invention is achieved through the following technical solution:

[0021] A method for storing a knowledge representation data structure, comprising:

[0022] Each knowledge unit of the knowledge representation data structure is stored as an equivalent equation document, which contains a first expression field and a second expression field.

[0023] Write the first expression and the second expression, which have a semantically equivalent mapping relationship, into the first expression field and the second expression field;

[0024] A dedicated index library is established for the equivalence equation documents, and these documents are treated as first-class citizen data types. The equivalence equation documents are managed independently of ordinary text documents. The dedicated index library includes a word segmentation inverted index layer and a feature-based precise index layer, wherein:

[0025] The word segmentation inverted index layer is used to perform standard word segmentation on the strings of the first expression and the second expression respectively, and then map each word after standard word segmentation to the equivalent equation document identifier ID, so as to write it into the word segmentation inverted index table;

[0026] The expression feature code precise index layer is used to write the complete precise feature codes of the first expression and the second expression as dedicated terms into the same inverted index table.

[0027] The equivalent equation document is in JSON format; the encoding formats of the first expression field and the second expression field include:

[0028] For atomic instances, an encoding format is used that directly maps them to literal string constants;

[0029] For structured semantics of compound assertions that need to carry parameter dependencies or logical mappings, a unified encoding format for "predicate identifier (ordered parameter set)" strings is adopted.

[0030] The method also includes a registration class derivation process, which includes:

[0031] The processor extracts the predicate identifier and parameter list containing query parameters from the query expression, and performs a range scan in the precise feature code index segment with the corresponding predicate as a prefix to obtain the set of all rule documents with the same predicate;

[0032] Iterate through each rule document in the rule document set, load the current rule document and extract its rule expression parameter list, and compare the rule parameters in the rule expression parameter list with the query parameters contained in the query expression bit by bit;

[0033] When the comparison reveals a parameter difference between the rule parameters and the query parameters, a registration query registration class is constructed based on the difference, the feature code hash value of the registration query is calculated, and a precise search is performed in the precise feature code index segment.

[0034] When a registration class relationship is hit, variable substitution is performed, replacing the category parameter in the rule with the instance parameter in the query, generating a new derivation document, and writing the derivation document into the equivalent equation document library. The corresponding word segmentation inverted index and precise feature code index are updated until all rule documents have been traversed, and the registration class derivation process is completed.

[0035] When a registration relationship is matched, it also includes:

[0036] The new derivation document is added to the result set as a derivation result, and this process continues until all rule documents have been traversed, at which point all derivation results are returned.

[0037] The precise feature code is stored in the inverted index marked with a specific prefix.

[0038] A retrieval method for knowledge representation data structures, comprising:

[0039] When a user enters a natural language query string, the processor transforms it into a target expression T through word segmentation and regular expression parsing. Then, it searches for the target expression T through the expression feature code precise index layer. If the target expression T is found, the content of another field in the document is directly extracted and returned. If the target expression T is not found, the processor searches for the target expression T in the inverted index through the word segmentation inverted index layer. After the query is completed, the candidate document set obtained by the query is returned.

[0040] After loading the candidate document set, the intermediate expressions in the hit documents are used as springboards to search for equivalent equation documents with equivalent transitive relations in the precise feature code index.

[0041] The final search result is determined based on the candidate document set and the multiple candidate searches contained in the equivalent equation documents with equivalence transitivity relations.

[0042] The process of determining the final search results includes:

[0043] The multiple candidate search results are subjected to multidimensional scoring processing, and the multiple candidate search results are sorted according to the corresponding scoring results. The multidimensional scoring represents the accuracy score of each candidate search result.

[0044] Based on the results of the corresponding sorting process, candidate search results that meet the requirements are selected as the final search results according to predetermined rules.

[0045] The calculation method for the multidimensional score includes:

[0046] Multidimensional score = w1 × exact equivalence hit score + w2 × transitive inference equivalence score + w3 × keyword segmentation hit score + w4 × source confidence score + w5 × timestamp freshness score;

[0047] The weights are configured as follows: w1 > w2 > w3; w4 = the confidence value itself; w5 is calculated based on the time decay factor.

[0048] The exact equivalent hit score is obtained when the target expression exactly matches a certain field in the document. This score indicates that the target expression has the highest weight.

[0049] The transitive derivation equivalence score is determined by the number of derivation steps a target expression takes to match after several transitive derivations. The more derivation steps a target expression takes, the lower the corresponding transitive derivation equivalence score will be.

[0050] The keyword segmentation hit score is obtained when the target expression and the document field are matched for relevance through term frequency-inverse document frequency (TF-IDF) or best match 25BM25.

[0051] If the knowledge record has a confidence score field containing the confidence value of the record, then the confidence value is used as the score.

[0052] The timestamp freshness score is calculated based on the timestamp of the knowledge record and the time decay factor. It is used to characterize the timeliness of the record, and the score decreases as the storage time of the record increases.

[0053] The scores above are pre-set based on actual needs and their relative weighting.

[0054] A processing system, comprising:

[0055] At least one memory for storing one or more programs;

[0056] At least one processor is capable of executing one or more programs stored in the memory, such that when the processor executes one or more programs, the processor can implement the methods described above.

[0057] A readable storage medium for storing a computer program that, when executed by a processor, enables the implementation of the methods described above.

[0058] Compared with existing technologies, the knowledge representation data structure storage and retrieval method and system provided by this invention can embed equivalence relation primitives into the index layer, natively supporting equivalence relation processing; thus, it achieves integrated querying, reasoning, and retrieval, eliminating the need for independent reasoning modules, reducing module interaction, temporary data, and network call overhead, and lowering system complexity. Furthermore, it adopts a two-layer index structure combining word segmentation inverted index and expression feature code precise index, sharing a unified index system, and allowing for on-demand switching between keyword fuzzy search and expression precise matching modes without additional storage structures. It also supports equivalence transitive derivation and category reasoning, automatically completing knowledge expansion and improving query recall and generalization capabilities. In addition, it can be implemented based on mainstream search engines, offering strong compatibility and facilitating engineering implementation and industrial deployment. Attached Figure Description

[0059] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0060] Figure 1 A schematic diagram of the processing flow of the method provided in the embodiments of the present invention;

[0061] Figure 2 This is a schematic diagram of the two-level index construction process provided in an embodiment of the present invention;

[0062] Figure 3 This is a schematic diagram of the query processing and transmission derivation recall process provided in an embodiment of the present invention;

[0063] Figure 4 This is a schematic diagram of the registration class deduction module provided in an embodiment of the present invention. Detailed Implementation

[0064] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the specific content of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments, which do not constitute a limitation of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0065] First, the following explanations are provided for the terms that may be used in this article:

[0066] The term "and / or" means that either or both can be achieved simultaneously. For example, X and / or Y means that it includes both "X" or "Y" as well as the three cases of "X and Y".

[0067] The terms "comprising," "including," "containing," "having," or other similar semantic descriptions should be interpreted as non-exclusive inclusion. For example, including a technical feature element (such as raw material, component, ingredient, carrier, dosage form, material, size, part, component, mechanism, device, step, process, method, reaction conditions, processing conditions, parameter, algorithm, signal, data, product or article of manufacture, etc.) should be interpreted as including not only the expressly listed technical feature element, but also other technical feature elements that are not expressly listed and are well-known in the art.

[0068] The term "composed of" excludes any technical features not expressly listed. When used in a claim, it closes the claim to exclude all technical features other than those expressly listed, except for associated conventional impurities. If the term appears only in a clause of a claim, it limits the claim to the elements expressly listed in that clause; elements recited in other clauses are not excluded from the overall claim.

[0069] Unless otherwise explicitly specified or limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to the internal connection between two components. Those skilled in the art can understand the specific meaning of the above terms in this document according to the specific circumstances.

[0070] When concentration, temperature, pressure, size, or other parameters are expressed as numerical ranges, such ranges should be understood to specifically disclose all ranges formed by any pairing of upper limits, lower limits, or preferred values ​​within that range, regardless of whether the range is explicitly stated; for example, if the numerical range "2 to 8" is stated, then that range should be interpreted to include ranges such as "2 to 7", "2 to 6", "5 to 7", "3 to 4 and 6 to 7", "3 to 5 and 7", "2 and 5 to 7", etc. Unless otherwise stated, the numerical ranges described herein include both their endpoints and all integers and fractions within that range.

[0071] The terms “center,” “longitudinal,” “lateral,” “length,” “width,” “thickness,” “upper,” “lower,” “front,” “back,” “left,” “right,” “vertical,” “horizontal,” “top,” “bottom,” “inner,” “outer,” “clockwise,” and “counterclockwise” indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience and simplification of description and do not imply that the device or component referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this document.

[0072] The implementation scheme of the knowledge representation data structure storage and retrieval method and system provided in this embodiment of the invention mainly includes the following technical processing methods:

[0073] (1) Storage and indexing structure of equivalent equation documents;

[0074] Each knowledge unit is stored as an equivalent equation document (which can be in JSON format, etc.). The document can contain a first expression field and a second expression field. Both fields use a unified predicate identifier (ordered parameter set) string or atomic instance encoding format. A dedicated index can be built for this type of document in the search engine.

[0075] The encoding formats that can be used in the two corresponding fields include:

[0076] For atomic instances, such as entities, values, or simple facts with explicit designations, the processor can directly map them to literal string constants, such as generator number 001.

[0077] For structured semantics, i.e., compound assertions that need to carry parameter dependencies or logical mappings, the processor can encode them as "predicate identifiers (ordered parameter sets)".

[0078] In the above processing, an implementation scheme is adopted that treats "equivalence relation" as a first-class citizen data type in the search engine and manages it separately from ordinary text documents; moreover, the structured encoding format of the corresponding equivalence equation document enables it to be accurately extracted and compared in the index, effectively avoiding the overhead of extracting knowledge from free text a second time.

[0079] (2) A two-layer indexing mechanism based on word segmentation inverted index and feature code precise index;

[0080] When building an index for equivalent equation documents, the processor can perform the following two indexing operations simultaneously:

[0081] The word segmentation inverted index layer is used to perform standard word segmentation processing (such as Chinese word segmentation, lowercase conversion, word stemming, etc.) on the strings of the first and second expressions respectively, and then map each word after standard word segmentation processing to the document ID so as to write it into the word segmentation inverted index table for fuzzy semantic query.

[0082] The expression feature code precise index layer is used to write the complete feature codes (hash values ​​of the complete normalized strings) of the first and second expressions as special terms into the same inverted index table (i.e., the expression feature code precise index table) to ensure that precise equivalence comparison can be completed in a single index lookup.

[0083] In the above process, the two-layer index, consisting of the word segmentation inverted index layer and the expression feature code precise index layer, coexists in the same inverted index engine. During the search process, fuzzy semantic matching and exact equivalent hits share the same query entry point, without the need to switch between two independent storage engines.

[0084] (3) Query expansion and transitive derivation recall driven by equivalence equations;

[0085] (31) The corresponding query expansion process based on equivalence equations can include:

[0086] When a user enters a natural language query string, the processor can first transform it into a target expression T through word segmentation and regular expression parsing. Then, it first searches for the target expression T through the exact matching layer (i.e., the expression feature code exact index layer). If a match is found, the content of another field in the document is directly extracted and returned. If the exact matching layer does not find a match, the processor further performs word segmentation full text retrieval in the inverted index, that is, searches for the target expression T through the word segmentation inverted index layer. After the query is completed, the candidate document set obtained by the query is returned.

[0087] (32) The process of recall based on equivalent equation-driven transitive derivation may include:

[0088] After loading the candidate document set, a transitive inference recall is performed, that is, using the intermediate expression in the hit document as a springboard, the equivalent equation document with equivalent transitive relationship with it is searched in the precise feature code index, which is called the forwarding associated document; thus, the transitivity of "A is equivalent to B, B is equivalent to C" can be utilized to automatically expand the coverage of the search results.

[0089] For example, when the axiom of class implication exists, the system supports robust class matching; that is, when a user queries "pH value of hydrochloric acid", the system will perform variable substitution derivation and automatically match "pH value (acidic substance)" to "pH value (hydrochloric acid)".

[0090] The above processing steps can convert all the reasoning steps in the external reasoning engine into index lookup and secondary document retrieval operations within the database. There is no need to load the full ontology file or generate temporary intermediate reasoning tables. The reasoning process and the query process are deeply integrated into a pipeline.

[0091] (4) A multi-dimensional ranking and scoring mechanism based on equivalent derivation of link weights;

[0092] In this embodiment of the invention, the search results can also be sorted based on multidimensional scoring. The corresponding processor can calculate the multidimensional scores of the search results in the following ways:

[0093] Exact Equivalence Hit Score (Highest Weight): If the target expression exactly matches a certain field in the document, that score is obtained. This score is the highest, indicating that it has the highest weight.

[0094] Transitive derivation equivalence score (medium weight): The target expression is matched after several transitive derivations. The number of derivation steps is inversely proportional to the score. That is, the more derivation steps there are, the lower the corresponding transitive derivation equivalence score will be.

[0095] Keyword segmentation hit score (lower weight): This score is obtained by matching the target expression with the document field through TF-IDF (term frequency-inverse document frequency) or BM25 (best match 25) relevance.

[0096] Source confidence score: If the knowledge record has a confidence auxiliary field (which carries the confidence value of the record), the corresponding confidence value will be included in the score.

[0097] The specific weight scores mentioned above can be preset according to actual needs and their relative weight levels.

[0098] Through the aforementioned multi-dimensional ranking mechanism, exact equivalent results are always ranked before semantically fuzzy matching results, which ensures both the recall rate of semantic expansion and the ranking quality of exact results.

[0099] Compared with existing solutions, the technical solutions adopted in the above embodiments of the present invention have the following significant advantages:

[0100] In terms of handling equivalence relations, traditional inverted index search engines cannot handle equivalence relations, and their ontology semantic search engines need to rely on external ontology reasoning layers. However, in this embodiment of the invention, the support for equivalence equation document primitives is specifically to embed the capability of equivalence equation document primitives into the index layer.

[0101] The embodiments of the present invention can realize the integration of query, reasoning and retrieval, avoiding the system overhead caused by external reasoning: In contrast, existing semantic search systems mostly adopt a dual-system structure of "search engine + ontology inferencer", which requires frequent interaction between multiple modules during querying and also generates a large amount of temporary reasoning data; Specifically, in the embodiments of the present invention, exact matching, full-text retrieval, transitive inference and ranking are all completed within the same search engine, without the need to deploy an independent reasoning system, reducing system complexity and network call overhead;

[0102] This invention provides a dual-level index structure that balances precise reasoning and fuzzy search. Specifically, it allows for the simultaneous construction of a word segmentation inverted index within the same inverted index for natural language keyword retrieval, and a precise index based on expression feature codes for equivalent matching of complete expressions. Compared to traditional solutions, this invention achieves a unified index structure for both precise semantic matching and fuzzy text matching. It also allows for automatic switching between the two retrieval modes during queries and eliminates the need for additional database storage.

[0103] The embodiments of the present invention also support transitive deduction and class reasoning, that is, it can automatically deduce "A=C" using "A=B, B=C", and support the replacement of registered class variables, realizing the automatic expansion from category knowledge to instance knowledge, thereby improving recall and generalization ability;

[0104] The technical solutions provided by the embodiments of the present invention are easy to implement in engineering and industrial deployment. They can be directly implemented based on existing search engines such as Elasticsearch or OpenSearch, and have good compatibility and scalability.

[0105] To facilitate understanding of the embodiments of the present invention, the specific implementation process of the embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0106] The knowledge representation data structure storage and retrieval method and system provided in this invention embodiment, namely a full-text knowledge semantic retrieval system based on equivalent equation documents, has the following overall architecture: Figure 1 As shown, it may include:

[0107] (1) The user query layer, whose processing procedures may include:

[0108] The query request is input in natural language form through the natural language input section; the query request is processed by the word segmentation and parsing section to be divided into semantic units; based on the word segmentation and parsing results, a structured target expression is generated by the target expression generation section, which serves as the unified input for subsequent retrieval;

[0109] (2) The search engine kernel layer, whose implementation structure may include: precise feature code matching, word segmentation inverted full-text retrieval, propagation and inference recall module, multi-dimensional ranking and scoring module and registration inference module, wherein:

[0110] Precise signature matching is used to quickly locate completely equivalent documents by using the signature of the target expression in the precise signature hash index table.

[0111] Inverted full-text search based on word segmentation is used to retrieve semantically relevant documents by searching word segmentation entries in the inverted index table using target expressions.

[0112] The transitive derivation recall module is used to extend the derivation of recalled documents based on the transitivity of equivalence relations (A=B, B=C→A=C) to supplement more equivalent results.

[0113] The registration inference module is used to handle class variable substitution and extended reasoning from category to instance to further improve the generalization of recall.

[0114] The multidimensional ranking and scoring module is used to evaluate the comprehensive BM25 / TF-IDF relevance score, confidence auxiliary field, equivalent matching weight, etc., to rank all recall results and output the final result;

[0115] (3) Index storage layer, the implementation structure of which may include: an equivalent equation document index library, which may include a word segmentation inverted index table (BM25 / TF-IDF) and a precise feature code hash index table, wherein:

[0116] The word segmentation inverted index table (BM25 / TF-IDF) is used to store the mapping relationship between word segmentation entries and document IDs of all equivalent equation documents for fuzzy semantic retrieval.

[0117] The precise feature code hash index table is used to store the mapping relationship between the normalized string hash values ​​(feature codes) of all equivalent equation documents and the document IDs, for precise equivalence matching;

[0118] Both are managed by the equivalent equation document index library, and the underlying data is linked to the original document library of the document storage layer;

[0119] (4) Document storage layer, which includes: JSON Doc Store, which stores the original documents of all equivalent equations in JSON format. After the kernel layer recalls the document ID, it can read the complete document content from the JSON Doc Store and return it to the user.

[0120] The storage structure of the corresponding equivalent equation document includes: In the document storage layer of the search engine, each piece of knowledge is stored as an independent JSON document. The specific implementation structure of the equivalent equation document JSON is shown in Table 1.

[0121] Table 1

[0122]

[0123] In the corresponding equivalent equation document JSON structure, the two expression fields are completely equivalent in physical storage, without distinguishing between subject and object. The JSON document structure allows the index building process to directly extract data precisely by field name, without needing to perform entity recognition from free text.

[0124] In this embodiment of the invention, in Figure 1 The system's index storage layer includes a word segmentation inverted index table and a precise feature code hash index table, both of which are uniformly managed by the "equivalence equation document index library," thus enabling the construction of a two-layer index. The construction process of this two-layer index is as follows: Figure 2 As shown, it may include:

[0125] The system initiates the index building process, preparing to process the new equivalent equation document; it receives the input new equivalent equation document D to obtain the original document data to be indexed;

[0126] Extract two core fields, D.field_1 and D.field_2, from document D, and their corresponding equivalence relation expressions. These expressions will be used by the subsequent two indexing layers (the word segmentation inverted index layer and the precise feature code index layer) to process the data based on these two core fields.

[0127] The word segmentation inverted index layer, used for fuzzy language retrieval, performs the following processes:

[0128] Perform Chinese word segmentation on field_1. For example, “pH value of acidic substances” can be segmented into ["pH value", "acidic", "substances"];

[0129] Perform Chinese word segmentation on field_2, for example, segment “<7” into ["<7","7"];

[0130] Each term is mapped to a document ID (D.id) to establish a "term → document ID" mapping relationship for each word segmentation result. The mapping relationship is then written into the inverted index table (BM25 / TF-IDF index segment) to support subsequent full-text search based on BM25 or TF-IDF.

[0131] The precise feature code index layer, used for precise matching of equivalence relations, may include the following processing steps:

[0132] Calculate the MD5 / SHA256 hash values ​​of the complete normalized strings of field_1 and field_2, and map the two hash values ​​as special terms to the document ID (D.id) to establish the mapping of "hash value → document ID".

[0133] The established mapping relationship is written into the same inverted index table (precise feature code reserved segment) to support subsequent fast equivalent matching.

[0134] Once both index layers are built, the process ends. At this point, the document has both fuzzy semantic retrieval and exact equivalence matching indexing capabilities.

[0135] In the above construction process, to avoid false matches caused by hash collisions, the precise feature code is stored in the inverted index with a specific prefix, such as _exact_HASH_VALUE (precise hash value) as the feature prefix, to ensure that the word segmenter will not split the word twice.

[0136] In this embodiment of the invention, in Figure 1 The search engine kernel layer in the system shown performs query processing and propagation / inference recall processes, as follows: Figure 3 As shown, it may include:

[0137] First, the system receives user query input, performs word segmentation and regular expression processing on the natural language query, and generates a structured target expression T. For example, if the user query input is "what is the pH value of hydrochloric acid", then the corresponding target expression T is T = pH value (hydrochloric acid).

[0138] Next, step 1 is executed to perform exact signature matching. The hash value Hash(T) of the target expression T is calculated, and a search is performed in the exact signature hash index table. It is then determined whether an exact match has been found.

[0139] If an exact match is found, the corresponding document is read directly and the field_1 / field_2 fields are extracted symmetrically. Then, the sorting stage begins, and step 4 is executed.

[0140] If no exact match is found, proceed to step 2 and perform word segmentation inverted full-text search, that is, perform BM25 query in the word segmentation inverted index based on the word segmentation result of the target expression to obtain the initial candidate set C;

[0141] Then, step 3 is executed, which involves performing a transitive inference recall. This involves traversing each document Di in the candidate set C, reading the field_1 and field_2 fields of Di, and taking the other field that does not match the target expression T as the Bridge expression. The hash value Hash(Bridge) of the Bridge expression is calculated and the index is searched to obtain other equivalent documents associated with the Bridge, which are then added to the extended candidate set C'. At the same time, a registration class inference check is performed to determine whether there is a class or implication relationship between the Bridge expression and the target expression T, generating a temporary inference conclusion to further expand the recall results.

[0142] Finally, step 4 is executed: multi-dimensional ranking and scoring. The results are comprehensively ranked based on multiple dimensions such as exact matching score, number of derivation steps, BM25 relevance score and knowledge record confidence. The Top-N (the N highest-scoring results) results are returned to the user, completing the corresponding query and derivation recall process.

[0143] In the above processing steps 1, 2, and 3, the queries all occur within the same search engine's index database. All index segments share the same query entry point, eliminating the need for cross-system RPC. Furthermore, the pass-through-recall process described above can be executed recursively, and the maximum depth can be controlled through configuration options to limit latency.

[0144] In this embodiment of the invention, in Figure 1 The implementation of the multidimensional ranking and scoring algorithm used in the multidimensional ranking and scoring module of the search engine kernel layer in the system shown may include:

[0145] For each candidate document, the processor can calculate the overall score in the following ways:

[0146] The total score for multidimensional ranking is calculated as follows: w1 × exact equivalence hit score + w2 × transitive inference equivalence score + w3 × keyword segmentation hit score + w4 × source confidence score + w5 × timestamp freshness score.

[0147] The weights are configured as follows:

[0148] w1=1.0 (Exact Hit);

[0149] w2 = 0.8^Derivation steps (transitive derivation, the more steps, the lower the weight).

[0150] w3=0.5 (fuzzy semantic matching);

[0151] w4 = the confidence level value itself;

[0152] w5 = calculated based on the time decay factor; The keyword segmentation hit score can be, but is not limited to, the BM25 segmentation relevance score;

[0153] As can be seen from the above implementation of multidimensional ranking and scoring, exact equivalence results are always ranked first, followed by transitive derivation, and pure keyword matching is ranked last; this ensures both the recall rate of semantic expansion and the ranking quality of the results.

[0154] In this embodiment of the invention, when neither exact matching nor full-text search returns satisfactory results, the registration class derivation process is triggered; specifically, in Figure 1 The registration class inference process executed by the registration class inference module of the search engine kernel layer in the system shown is as follows: Figure 4 As shown, it may include:

[0155] First, the processor extracts the predicate identifier and parameter list (i.e., the ordered parameter set containing the query parameters) of the query expression Q, and performs a range scan in the precise feature code index segment with the corresponding predicate as a prefix to obtain the set of all rule documents with the same predicate;

[0156] Then, each rule document in the rule document set is traversed, including:

[0157] When there are no unprocessed rule documents, that is, when all rule documents have been traversed, the derivation result is returned and the processing ends.

[0158] When an unprocessed rule document exists, the current rule document is loaded and its parameter list of rule expressions is extracted. The rule parameters in the parameter list of the rule expression are compared character by character with the query parameters contained in the query expression Q, and it is determined whether there are any parameter differences.

[0159] If no parameter difference item exists, skip this rule document and continue traversing subsequent rule documents;

[0160] If parameter differences exist, a registration query class (query parameters, rule parameters) is constructed based on these differences. The feature code hash of this registration query is calculated, and a precise search is performed in the precise feature code index segment. The search result is used to determine whether a registration class relationship is matched.

[0161] If no registration relationship is found, skip that rule document and continue traversing subsequent rule documents;

[0162] If a registration class relationship is found, variable substitution is performed, replacing the category parameter in the rule with the instance parameter in the query to generate a new derivation document; this derivation document is written into the equivalent equation document library, the word segmentation inverted index and the precise feature code index are updated, and the new derivation document is added to the result set as the derivation result to continue traversing subsequent rule documents;

[0163] After traversing all rule documents, a result set containing all derivation results is returned, completing the registration class derivation process.

[0164] In this embodiment of the invention, an independent inference module mounting scheme can also be used as an alternative to implement "equivalence equation-driven semantic retrieval". That is, a separate inference service is deployed outside of Elasticsearch. When querying, Elasticsearch first returns the initial screening results, and then the inference service loads the ontology file for secondary semantic expansion.

[0165] Through the above description of the embodiments, those skilled in the art can clearly understand that the above embodiments can be implemented by software, or by using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions of the above embodiments can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, mobile hard drive, etc.), including several instructions to cause a computer device (such as a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0166] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims. The information disclosed in the background section is intended only to enhance the understanding of the overall background technology of the present invention and should not be construed as an admission or implication in any way that such information constitutes prior art known to those skilled in the art.

Claims

1. A method for storing a knowledge representation data structure, characterized in that, include: Each knowledge unit of the knowledge representation data structure is stored as an equivalent equation document, which contains a first expression field and a second expression field. Write the first expression and the second expression, which have a semantically equivalent mapping relationship, into the first expression field and the second expression field; A dedicated index library is established for the equivalence equation documents, and these documents are treated as first-class citizen data types. The equivalence equation documents are managed independently of ordinary text documents. The dedicated index library includes a word segmentation inverted index layer and a feature-based precise index layer, wherein: The word segmentation inverted index layer is used to perform standard word segmentation on the strings of the first expression and the second expression respectively, and then map each word after standard word segmentation to the equivalent equation document identifier ID, so as to write it into the word segmentation inverted index table; The expression feature code precise index layer is used to write the complete precise feature codes of the first expression and the second expression as dedicated terms into the same inverted index table.

2. The storage method for the knowledge representation data structure according to claim 1, characterized in that, The equivalent equation document is in JSON format; the encoding formats of the first expression field and the second expression field include: For atomic instances, an encoding format is used that directly maps them to literal string constants; For structured semantics of compound assertions that need to carry parameter dependencies or logical mappings, a unified encoding format for "predicate identifier (ordered parameter set)" strings is adopted.

3. The method for storing knowledge representation data structures according to claim 1 or 2, characterized in that, The method also includes a registration class deduction process, which includes: The processor extracts the predicate identifier and parameter list containing query parameters from the query expression, and performs a range scan in the precise feature code index segment with the corresponding predicate as a prefix to obtain the set of all rule documents with the same predicate; Iterate through each rule document in the rule document set, load the current rule document and extract its rule expression parameter list, and compare the rule parameters in the rule expression parameter list with the query parameters contained in the query expression bit by bit; When the comparison reveals a parameter difference between the rule parameters and the query parameters, a registration query registration class is constructed based on the difference, the feature code hash value of the registration query is calculated, and a precise search is performed in the precise feature code index segment. When a registration class relationship is hit, variable substitution is performed, replacing the category parameter in the rule with the instance parameter in the query, generating a new derivation document, and writing the derivation document into the equivalent equation document library. The corresponding word segmentation inverted index and precise feature code index are updated until all rule documents have been traversed, and the registration class derivation process is completed.

4. The method for storing knowledge representation data structures according to claim 3, characterized in that, When a registration relationship is matched, it also includes: The new derivation document is added to the result set as a derivation result, and this process continues until all rule documents have been traversed, at which point all derivation results are returned.

5. The method for storing knowledge representation data structures according to claim 3, characterized in that, The precise feature code is stored in the inverted index marked with a specific prefix.

6. A retrieval method for a knowledge representation data structure, characterized in that, include: When a user enters a natural language query string, the processor transforms it into the target expression T through word segmentation and regular expression parsing; Then, the target expression T is searched for using the expression feature code in the precise index layer. If a match is found, the content of another field in the document is directly extracted and returned. If no match is found, the processor searches for the target expression T in the inverted index through the tokenized inverted index layer. After the query is completed, the candidate document set obtained by the query is returned. After loading the candidate document set, the intermediate expressions in the hit documents are used as springboards to search for equivalent equation documents with equivalent transitive relations in the precise feature code index. The final search result is determined based on the candidate document set and the multiple candidate searches contained in the equivalent equation documents with equivalence transitivity relations.

7. The retrieval method for the knowledge representation data structure according to claim 6, characterized in that, The process of determining the final search result includes: The multiple candidate search results are subjected to multidimensional scoring processing, and the multiple candidate search results are sorted according to the corresponding scoring results. The multidimensional scoring represents the accuracy score of each candidate search result. Based on the results of the corresponding sorting process, candidate search results that meet the requirements are selected as the final search results according to predetermined rules.

8. The retrieval method for the knowledge representation data structure according to claim 7, characterized in that, The calculation method for the multidimensional score includes: Multidimensional score = w1 × exact equivalence hit score + w2 × transitive inference equivalence score + w3 × keyword segmentation hit score + w4 × source confidence score + w5 × timestamp freshness score; Where w1, w2, w3, w4, and w5 are weighting coefficients, configured as follows: w1 > w2 > w3; w4 = the confidence value itself; w5 is calculated based on the time decay factor. The exact equivalent hit score is obtained when the target expression exactly matches a certain field in the document. This score indicates that the target expression has the highest weight. The transitive derivation equivalence score is determined by the number of derivation steps a target expression takes to match after several transitive derivations. The more derivation steps a target expression takes, the lower the corresponding transitive derivation equivalence score will be. The keyword segmentation hit score is obtained when the target expression and the document field are matched for relevance through term frequency-inverse document frequency (TF-IDF) or best match 25BM25. If the knowledge record has a confidence score field containing the confidence value of the record, then the confidence value is used as the score. The timestamp freshness score is calculated based on the timestamp of the knowledge record and the time decay factor. It is used to characterize the timeliness of the record, and the score decreases as the storage time of the record increases. The scores above are pre-set based on actual needs and their relative weighting.

9. A processing system, characterized in that, include: At least one memory for storing one or more programs; At least one processor is capable of executing one or more programs stored in the memory, such that when the one or more programs are executed by the processor, the processor is able to implement the method according to any one of claims 1 to 8.

10. A readable storage medium for storing a computer program, characterized in that, When the computer program is executed by a processor, it can implement the method described in any one of claims 1 to 8.

Images