Vector database retrieval method and device based on ontology and corpus double-layer structure

Abstract

The invention provides a vector database retrieval method and device based on an ontology and corpus double-layer structure, and relates to the field of database retrieval, and the method comprises the following steps: performing type division on a knowledge resource text to obtain ontology nodes and corpus nodes; the ontology nodes and the corpus nodes are input into a semantic vectorization model for coding, and ontology vectors and corpus vectors are obtained; according to the cosine similarity between the ontology vector and the corpus vector, performing association mapping on the ontology node and the corpus node to obtain a relateds mapping table; obtaining an initial HNSW structure, and inserting the ontology node and the corpus node into the initial HNSW structure according to the relatedds mapping table to obtain an HNSW index structure; a double-layer database is generated through the relateds mapping table and the HNSW index structure; obtaining a query text, and performing ontology matching and corpus matching on the query text through the double-layer database to obtain a final corpus block sequence; and constructing a language model, and inputting the final corpus block sequence into the language model for semantic retrieval to obtain a retrieval answer text.

Description

Technical Field

[0001] This application relates to the field of database retrieval, and in particular to a vector database retrieval method and apparatus based on a two-layer structure of ontology and corpus. Background Technology

[0002] With the explosive growth of knowledge resources, vector database retrieval has become a core means for obtaining accurate information in fields such as education, healthcare, scientific research, and engineering. Its core requirement is to quickly locate semantic information that closely matches the query intent from massive amounts of knowledge resources such as literature, textbooks, experimental reports, and concept definitions, so as to provide users with professional and reliable search results.

[0003] However, existing vector database retrieval technologies suffer from low knowledge resource utilization, insufficient semantic matching accuracy, and low retrieval efficiency, making it difficult to meet the needs for efficient, accurate, and closely related retrieval in scenarios with large-scale knowledge resources. The specific shortcomings are as follows.

[0004] Current technologies fail to perform targeted classification processing on knowledge resource texts. They either break down complete knowledge resources into overly fragmented text segments, resulting in a loss of semantic integrity, or directly use entire text segments as retrieval units, containing a large amount of redundant information. This approach leads to low structuring of knowledge resources, an inability to form a clear index for core semantics, and difficulty in accurately retrieving detailed supporting information. This not only reduces information utilization but also results in redundant search results or missing key information, affecting the reliability of subsequent answer generation.

[0005] Existing technologies typically employ uniform-dimensional vector encoding for various knowledge resource texts, failing to distinguish between core semantics and supporting details. This results in vectors that cannot accurately reflect the semantic features of different types of information. Furthermore, the lack of cross-layer vector alignment mechanisms means that core semantic vectors and supporting information vectors reside in different spaces, leading to weak semantic comparability and difficulty in establishing strong correlations. This makes it prone to mismatches between core questions and supporting corpora during retrieval, resulting in search results that do not fully meet user needs.

[0006] Traditional vector retrieval often employs a single-layer, full-data retrieval model, directly encoding all knowledge resource text into vectors and then matching them through traversal or simple indexing. This approach does not structurally categorize knowledge resources, requiring similarity calculations on all vectors during retrieval, resulting in numerous unnecessary computations and excessively long retrieval times. Especially when the scale of knowledge resources reaches hundreds of thousands or millions, retrieval latency increases significantly, failing to meet real-time retrieval requirements. Summary of the Invention

[0007] To address the aforementioned problems, this invention provides a vector database retrieval method based on a two-layer structure of ontology and corpus, comprising:

[0008] Acquire knowledge resource texts in the target domain, classify the knowledge resource texts by type, and obtain ontology nodes and corpus nodes;

[0009] Construct a semantic vectorization model, input ontology nodes and corpus nodes into the semantic vectorization model for encoding, and obtain ontology vectors and corpus vectors;

[0010] Based on the cosine similarity between ontology vectors and corpus vectors, ontology nodes and corpus nodes are associated and mapped to obtain the related_ids mapping table; the initial HNSW structure is obtained, and ontology nodes and corpus nodes are inserted into the initial HNSW structure according to the related_ids mapping table to obtain the HNSW index structure; a two-layer database is generated through the related_ids mapping table and the HNSW index structure.

[0011] The query text is obtained, and ontology matching and corpus matching are performed on the query text through a two-layer database to obtain the final corpus block sequence.

[0012] A language model is constructed, and the final corpus block sequence is input into the language model for semantic retrieval to obtain the retrieval response text.

[0013] Optionally, the step of classifying the knowledge resource text by type to obtain ontology nodes and corpus nodes includes:

[0014] Knowledge resource texts include abstract problem texts, definition texts, concept texts, literature texts, experimental report texts, and textbook texts;

[0015] Extract atomic problem texts from abstract problem texts, definition texts, and concept texts, and use the atomic problem texts as ontology nodes;

[0016] The literature text, experimental report text, and textbook text are divided into corpus blocks, and the corpus blocks are used as corpus nodes.

[0017] Optionally, the step of inputting ontology nodes and corpus nodes into a semantic vectorization model for encoding to obtain ontology vectors and corpus vectors includes:

[0018] Low-dimensional vectors are obtained by encoding ontology nodes in a low-dimensional manner using a semantic vectorization model.

[0019] High-dimensional vectors are obtained by encoding corpus nodes in a high-dimensional manner using a semantic vectorization model.

[0020] Cross-layer alignment is performed on low-dimensional and high-dimensional vectors to obtain ontology vectors and corpus vectors.

[0021] Optionally, the step of associating ontology nodes and corpus nodes based on the cosine similarity between ontology vectors and corpus vectors to obtain the related_ids mapping table includes:

[0022] Obtain the knowledge graph of the target domain, and calculate the cosine similarity between the ontology vector and the corpus vector based on the knowledge graph.

[0023] Select the corpus vectors with the highest cosine similarity of a preset number as the corpus vectors adjacent to the ontology vector, and take the corpus nodes corresponding to the adjacent corpus vectors as the corpus nodes adjacent to the ontology node.

[0024] Perform association mapping between ontology nodes and adjacent corpus nodes, and construct the related_ids mapping table based on all association mappings.

[0025] Optionally, the step of inserting ontology nodes and corpus nodes into the initial HNSW structure according to the related_ids mapping table to obtain the HNSW index structure includes:

[0026] The initial HNSW structure consists of multiple levels, with the target probability decreasing from top to bottom.

[0027] Set the probability radius, and calculate the probability range of each level based on the probability radius and the target probability;

[0028] Obtain the level probability of the ontology node, and insert the ontology node whose level probability is within the probability range into the corresponding level;

[0029] The adjacent corpus nodes of the ontology node are obtained from the related_ids mapping table, and the adjacent corpus nodes are inserted into each level to obtain the HNSW index structure.

[0030] Optionally, the step of performing ontology matching and corpus matching on the query text through a two-layer database to obtain the final corpus block sequence includes:

[0031] The query vector is obtained by vectorizing the query text using a semantic vectorization model.

[0032] Search the HNSW index structure for ontology nodes that match the query vector;

[0033] Obtain the adjacent corpus nodes corresponding to the ontology node through the related_ids mapping table;

[0034] Calculate the comprehensive score between the query vector, the ontology node, and the adjacent corpus nodes. Sort the adjacent corpus nodes according to the comprehensive score to obtain the final corpus block sequence.

[0035] Optionally, the calculation process of the comprehensive score includes:

[0036] The cosine similarity between the ontology node and the query vector is used as the first score, and the cosine similarity between adjacent corpus nodes and the query vector is used as the second score. The first score and the second score are weighted and summed to obtain the comprehensive score.

[0037] This invention also provides a vector database retrieval device based on a two-layer structure of ontology and corpus, used to implement the aforementioned vector database retrieval method based on a two-layer structure of ontology and corpus. The device includes:

[0038] The node acquisition module is used to acquire knowledge resource texts in the target domain, classify the knowledge resource texts by type, and obtain ontology nodes and corpus nodes.

[0039] The vector acquisition module is used to construct a semantic vectorization model. It inputs ontology nodes and corpus nodes into the semantic vectorization model for encoding to obtain ontology vectors and corpus vectors.

[0040] The two-layer database acquisition module is used to perform association mapping between ontology nodes and corpus nodes based on the cosine similarity between ontology vectors and corpus vectors to obtain the related_ids mapping table; obtain the initial HNSW structure; insert ontology nodes and corpus nodes into the initial HNSW structure according to the related_ids mapping table to obtain the HNSW index structure; and generate a two-layer database through the related_ids mapping table and the HNSW index structure.

[0041] The final corpus block sequence acquisition module is used to acquire query text, and performs ontology matching and corpus matching on the query text through a two-layer database to obtain the final corpus block sequence;

[0042] The retrieval and response text acquisition module is used to build a language model. The final corpus block sequence is input into the language model for semantic retrieval to obtain the retrieval and response text.

[0043] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the vector database retrieval method based on a two-layer structure of ontology and corpus.

[0044] The present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the vector database retrieval method based on a two-layer structure of ontology and corpus.

[0045] The present invention has the following beneficial effects:

[0046] 1. This invention constructs a two-layer database by dividing ontology nodes and corpus nodes, semantic vectorization encoding and cross-layer alignment, related_ids association mapping and HNSW index optimization. Ontology nodes and corpus nodes coexist in the two-layer database, forming a two-layer logical structure. This enables the system to simultaneously achieve hierarchical jumps from semantics to knowledge in a single retrieval, significantly improving the coherence and response speed of the retrieval path. During retrieval, a two-stage retrieval mechanism of ontology matching and corpus matching is adopted. Ontology matching performs fast semantic navigation at the ontology layer to narrow down the candidate range. Corpus matching performs fine-grained search in local corpus clusters to improve the relevance of retrieval results to the query intent.

[0047] 2. Knowledge resource texts are divided into ontology nodes and corpus nodes according to type. Ontology nodes retain the structured index of the core question, while corpus nodes fully store the detailed supporting information, avoiding information fragmentation or redundancy, improving the utilization rate of knowledge resources, and ensuring that subsequent ontology matching and corpus matching strictly rely on the knowledge resources of the target domain, thereby improving the reliability and professionalism of the answers.

[0048] 3. After low-dimensional encoding and high-dimensional encoding of ontology nodes and corpus nodes respectively, cross-layer alignment is used to achieve semantic comparability in the same vector space. Then, an association mapping is constructed based on cosine similarity to ensure strong semantic binding between ontology and corpus, reduce the problem of disconnect between core questions and supporting information, and improve the fit between answer results and query intent.

[0049] 4. The dual-layer database uses a two-tiered partitioning of ontology nodes and corpus nodes. It first uses ontology nodes as the core index to quickly locate the semantic range, and then matches related corpus nodes, avoiding blind retrieval of the entire corpus and reducing unnecessary computation. Combined with the association guidance of the HNSW index structure and the related_ids mapping table, the retrieval range is quickly narrowed from the high-level sparse index, and the low-level precise matching is performed, which greatly reduces the retrieval time and is suitable for large-scale knowledge resource scenarios. Moreover, when adding new knowledge resources, only the node encoding and association mapping need to be supplemented, without the need to reconstruct the overall architecture, which has strong scalability. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 This is a flowchart of a method according to an embodiment of the present invention;

[0052] Figure 2 This is a structural diagram of the device according to an embodiment of the present invention. Detailed Implementation

[0053] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0054] The terminology used in the following embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to include the plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this application refers to and includes any or all possible combinations of one or more of the listed items.

[0055] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature, and in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0056] To enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.

[0057] Reference Figure 1 This invention provides a vector database retrieval method based on a two-layer structure of ontology and corpus, comprising:

[0058] Acquire knowledge resource texts in the target domain, classify the knowledge resource texts by type, and obtain ontology nodes and corpus nodes;

[0059] In some embodiments, the step of classifying the knowledge resource text by type to obtain ontology nodes and corpus nodes includes:

[0060] Knowledge resource texts include abstract problem texts, definition texts, concept texts, literature texts, experimental report texts, and textbook texts;

[0061] In some embodiments, abstract problem text refers to a problem statement in the target domain that does not depend on a specific scenario and focuses on the core logic or principle, such as what is the core difference between supervised learning and unsupervised learning in artificial intelligence;

[0062] Definition texts refer to authoritative explanations or definitions of terms and concepts in a field. For example, machine learning refers to the technology by which computers learn patterns through data and make autonomous decisions.

[0063] Conceptual text refers to the related descriptions that revolve around the core concepts of a field, including the characteristics, attributes, and application scenarios of the concepts. For example, the core feature of deep learning is that it simulates the connection of neurons in the human brain through multi-layer neural networks.

[0064] Literature texts refer to formal documents such as academic papers, research reports, and reviews in the field, covering research background, methods, results, and other content.

[0065] Experimental report texts refer to written materials that record the experimental process, data, and conclusions within a field, including experimental objectives, equipment, procedures, data analysis, etc.

[0066] Textbook texts refer to textbook chapters, lecture notes, etc. used for teaching domain knowledge, and are characterized by being structured and systematic;

[0067] Extract atomic problem texts from abstract problem texts, definition texts, and concept texts, and use the atomic problem texts as ontology nodes;

[0068] In some embodiments, for complex questions in abstract question texts, they are split into a single core question point. For example, how to understand the principle of convolutional neural networks and their application in image recognition can be split into two independent questions: What is the core principle of convolutional neural networks and what is the application logic of convolutional neural networks in image recognition?

[0069] Extract core questions from definition and conceptual texts to form atomic questions. Definition texts should focus on the essence, scope, and characteristics of the term; for example, from "blockchain is a distributed ledger technology," what are the core technical attributes of blockchain? Conceptual texts should focus on the key connections, applicable scenarios, and core value of the concept; for example, from "big data has the characteristics of massive volume, high speed, and diversity," what are the core characteristics of big data?

[0070] The selected atomic questions are uniformly described to ensure that each question contains only one point of doubt, without ambiguity or redundancy, and finally form a set of ontology nodes;

[0071] The document texts, experimental report texts, and textbook texts are segmented into corpus blocks, and the corpus blocks are used as corpus nodes;

[0072] In some embodiments, sentence groups are used as the basic unit to ensure that each corpus block revolves around a single sub-topic. For example, a technical detail description in the experimental methods section of a document can be considered an independent corpus block. A length range for each corpus block is set; if it is too long, it is split according to semantic logic; if it is too short, it is merged with adjacent semantic units. Semantic delimiters in the text are identified using natural language processing tools, combined with manual verification, to complete the corpus segmentation. The segmented corpus blocks are then deduplicated, noise filtered, and labeled with their corresponding text source types, ultimately forming a set of corpus nodes.

[0073] Construct a semantic vectorization model, input ontology nodes and corpus nodes into the semantic vectorization model for encoding, and obtain ontology vectors and corpus vectors;

[0074] In some embodiments, the step of inputting ontology nodes and corpus nodes into a semantic vectorization model for encoding to obtain ontology vectors and corpus vectors includes:

[0075] Low-dimensional vectors are obtained by encoding ontology nodes in a low-dimensional manner using a semantic vectorization model.

[0076] High-dimensional vectors are obtained by encoding corpus nodes in a high-dimensional manner using a semantic vectorization model.

[0077] Cross-layer alignment is performed on low-dimensional and high-dimensional vectors to obtain ontology vectors and corpus vectors.

[0078] In some embodiments, the semantic vectorization model employs a general semantic encoding model, such as Sentence-BERT, E5, or a similar dual-tower semantic model, which can capture the semantic similarity relationship between texts while maintaining cross-domain applicability.

[0079] Low-dimensional vectors primarily represent the abstract semantic relationships between knowledge concepts or atomic problems to improve the efficiency of semantic aggregation and index navigation, while high-dimensional vectors are used to finely express the semantic details and contextual information of specific corpus blocks to ensure the accuracy of knowledge content retrieval.

[0080] Since the low-dimensional vectors of the ontology and the high-dimensional vectors of the corpus are in different vector spaces, semantic similarity cannot be directly calculated. Cross-layer alignment is required to map them to the same space. Ontology nodes and corpus nodes with strong semantic association are selected, and each ontology node is matched with 3-5 corpus nodes, resulting in more than 10,000 positive example pairs. Ontology nodes and corpus nodes with no semantic association are selected, and each positive example pair is matched with 5 negative example pairs, resulting in more than 50,000 negative example pairs. Label 1 represents a positive example, and 0 represents a negative example.

[0081] A two-layer fully connected network is designed as the mapping network to map low-dimensional ontology vectors to a high-dimensional corpus vector space. A contrastive loss is used to train the network, ensuring that the Euclidean distance between the mapped ontology vectors and the high-dimensional corpus vectors in positive pairs is ≤0.3, and the Euclidean distance between negative pairs is ≥1.0, thus ensuring that related semantics are close and unrelated semantics are far apart. The batch size is set to 64, the learning rate is set to 1e-4, the number of training epochs is set to 5, the AdamW optimizer is used, and an early stopping strategy is enabled to avoid model overfitting.

[0082] Input all low-dimensional ontology vectors into the trained mapping network, and output 768-dimensional aligned ontology vectors; the high-dimensional corpus vectors remain unchanged; randomly select 1000 pairs of positive examples and 1000 pairs of negative examples, and calculate the cosine similarity of the aligned vectors. The mean similarity of positive examples should be ≥0.7 and the mean similarity of negative examples should be ≤0.3. If these conditions are not met, supplement the dataset and retrain the mapping network. The aligned ontology vectors are the ontology vectors, and the normalized high-dimensional corpus vectors are the corpus vectors. Since they are in the same 768-dimensional vector space, the semantic relevance can be directly calculated through cosine similarity, providing a foundation for subsequent mapping between ontology and corpus.

[0083] Based on the cosine similarity between ontology vectors and corpus vectors, ontology nodes and corpus nodes are associated and mapped to obtain the related_ids mapping table; the initial HNSW structure is obtained, and ontology nodes and corpus nodes are inserted into the initial HNSW structure according to the related_ids mapping table to obtain the HNSW index structure; a two-layer database is generated through the related_ids mapping table and the HNSW index structure.

[0084] In some embodiments, the step of associating ontology nodes and corpus nodes based on the cosine similarity between ontology vectors and corpus vectors to obtain the related_ids mapping table includes:

[0085] Obtain the knowledge graph of the target domain, and calculate the cosine similarity between the ontology vector and the corpus vector based on the knowledge graph.

[0086] Select the corpus vectors with the highest cosine similarity of a preset number as the corpus vectors adjacent to the ontology vector, and take the corpus nodes corresponding to the adjacent corpus vectors as the corpus nodes adjacent to the ontology node.

[0087] Perform association mapping between ontology nodes and adjacent corpus nodes, and construct the related_ids mapping table based on all association mappings.

[0088] In some embodiments, to reduce invalid computation and improve efficiency, corpus vectors related to ontology nodes are first filtered through a knowledge graph. A node-entity association table is obtained based on the knowledge graph. Taking a single ontology node as an example, if the node ID is Onto_001 and the core entity ID is E001, then corpus nodes containing at least one entity associated with E001 are selected from the node-entity association table. The association rules refer to the graph relationships, including direct association and indirect association. At the same time, corpus nodes that have no association with E001 are excluded, forming a candidate corpus vector pool for the ontology node. Subsequently, only the similarity between the ontology vector and the corpus vectors in the pool is calculated.

[0089] Based on the candidate corpus vector pool, the cosine similarity is calculated using a normalization method. The calculation formula is as follows:

[0090]

[0091] Where A is the ontology vector and B is the corpus vector. For vectors and Cosine similarity between them and Representing vectors respectively and The model;

[0092] To improve efficiency, matrix operations are used for batch calculation. For example, 500 corpus vectors from the candidate corpus vector pool are combined into a 500×768 matrix and multiplied with the ontology vector (1×768) to obtain 500 similarity values ​​at once. After the calculation is completed, the similarity values ​​are associated with the corresponding corpus node IDs to form an ontology-corpus similarity table. The table contains three columns: ontology node ID, corpus node ID, and similarity value. The similarity value ranges from [-1, 1], and the closer the value is to 1, the more similar the semantics.

[0093] The system filters a preset number of corpus nodes based on similarity values. First, the preset number is determined, taking into account the total number of nodes in the domain corpus and subsequent retrieval efficiency. If the total number of domain corpus nodes is in the 100,000 range, the preset number is set to 20 to avoid processing delays; if the total is in the 10,000 range, the preset number is set to 10 to avoid information loss. Finally, optimization is achieved through testing. Three different numbers are selected to test retrieval accuracy, and the number with the highest accuracy and controllable time consumption is chosen as the final value. Next, the ontology-corpus similarity table is sorted in descending order of similarity value, and the top N corpus nodes are taken as the adjacent corpus nodes of that ontology node. If the number of nodes in the candidate corpus vector pool is less than N, all nodes are selected and this situation is recorded for subsequent supplementation of domain corpus data. If the similarity value is ≤0.3, even if the number is sufficient, nodes must be removed to avoid interfering with the retrieval.

[0094] A one-to-one core mapping is used to construct the related_ids mapping table, with the ontology node ID as the unique identifier key and the corresponding list of adjacent corpus node IDs as the associated value. For example, ontology node IDOnto_001 corresponds to the corpus node ID list [Corpus_005, Corpus_012, ..., Corpus_020]. A unified ID format is used, with all node IDs adopting the type_number format to avoid confusion. To ensure the uniqueness of the mapping, each ontology node ID corresponds to only one corpus node ID list. If the same corpus node is selected as an adjacent node by multiple ontology nodes, it can appear repeatedly in different lists.

[0095] In some embodiments, inserting ontology nodes and corpus nodes into the initial HNSW structure according to the related_ids mapping table to obtain the HNSW index structure includes:

[0096] The initial HNSW structure consists of multiple levels, with the target probability decreasing from top to bottom.

[0097] Set the probability radius, and calculate the probability range of each level based on the probability radius and the target probability;

[0098] Obtain the level probability of the ontology node, and insert the ontology node whose level probability is within the probability range into the corresponding level;

[0099] The adjacent corpus nodes of the ontology node are obtained from the related_ids mapping table, and the adjacent corpus nodes are inserted into each level to obtain the HNSW index structure.

[0100] In some embodiments, the initial HNSW structure achieves rapid high-level positioning and accurate low-level matching through a multi-level design. The number of levels and the probability of targets at each level need to be determined in combination with the total number of ontology nodes and retrieval requirements.

[0101] First, determine the number of levels. The core basis is the total number of ontology nodes in the target domain. If the total number of ontology nodes is 10,000-100,000, set 3-4 levels; if the total number is 100,000-1,000,000, set 5-6 levels; if the total number is less than 10,000, only 2 levels are needed to avoid redundant levels increasing retrieval time. Next, set the target probability for each level. This probability represents the expected likelihood of a node being assigned to that level. It should follow the principle of decreasing from top to bottom to ensure that high-level nodes are sparse and low-level nodes are dense, which conforms to the navigation logic of HNSW. Usually, the target probability of the highest level is set to 0.1-0.2, and the target probability of the next level decreases by 1 / 2 to 1 / 3 of the previous level. At the same time, it is necessary to verify that the sum of the target probabilities of all levels does not exceed 0.3 to avoid a single node being assigned to too many levels, increasing storage costs. The target probability of the lowest level is not lower than 0.01 to ensure that there are enough nodes at the lowest level to achieve accurate matching.

[0102] The probability radius is a threshold that defines the probability of a node falling within this range, at which it can be assigned to the corresponding level. A reasonable radius must be set first, and then the probability range of each level should be calculated in combination with the target probability to avoid overlapping ranges or missing retrieval.

[0103] First, set the probability radius, which needs to be adjusted according to the decrease in the target probability. If the decrease in the target probability from high to low levels is large, the probability radius is set to 0.01-0.02; if the decrease is small, the radius is set to 0.005-0.01. The key is to ensure that the probability ranges of adjacent levels do not overlap, while covering the effective probability interval within [0,1]. Then, calculate the probability range of each level using the formula of target probability ± probability radius. If the calculation result exceeds the effective range of probability value [0,1], it is truncated to the boundary value. After calculation, it is necessary to check whether there are gaps in the ranges of adjacent levels. If there is overlap, the probability radius is fine-tuned until all level ranges have neither overlap nor gaps.

[0104] The ontology node needs to be matched to the corresponding level by randomly generating level probabilities and establishing navigation connections within the level. First, the level probabilities of the ontology node are obtained, and an exponential distribution is used for random generation, as this distribution naturally achieves the characteristic of low probability for higher levels and high probability for lower levels, which meets the node distribution requirements of HNSW. When choosing the exponential distribution parameter λ, it needs to be associated with the probability of the highest level target. For example, if the probability of the highest level target is 0.1, λ=10 is chosen. The random.exponential function in Python's numpy library is used to generate the probability value scale=1 / λ. If the generated value is greater than 1, it is regenerated to ensure that the probability value is within the range [0,1].

[0105] Next, hierarchical matching and insertion of ontology nodes are performed. First, the hierarchical probability is compared with the probability range of each level. If the probability falls within a certain level range, the node is assigned to that level. The insertion operation is implemented through the hnswlib library. First, the HNSW index object is initialized, and the vector dimension and distance metric are set. Then, for the ontology node assigned to the target level, a K-nearest neighbor search is performed to determine the navigation neighbors. Finally, bidirectional connections are established between the current node and these 5 neighboring nodes to form a navigation network within the level.

[0106] Adjacent corpus nodes need to be associated with ontology nodes according to the related_ids mapping table, inserted at the same level and connected to form a complete HNSW index structure. First, adjacent corpus nodes are obtained and the insertion level is determined. The related_ids mapping table is used to find the list of adjacent corpus node IDs corresponding to the target ontology node. The insertion level of the corpus node is completely consistent with the corresponding ontology node, ensuring that the related corpus node can be directly located through the ontology node during retrieval, reducing the time spent on cross-level retrieval. After all ontology nodes and adjacent corpus nodes are inserted, each level contains two types of nodes and forms a navigation network through bidirectional connections. Higher-level nodes are sparse and have fewer connections, while lower-level nodes are dense and have more connections. Finally, the index structure is stored as a binary file and associated with the related_ids mapping table, forming the index core of a two-layer database for subsequent query and retrieval.

[0107] The query text is obtained, and ontology matching and corpus matching are performed on the query text through a two-layer database to obtain the final corpus block sequence.

[0108] In some embodiments, the step of performing ontology matching and corpus matching on the query text through a two-layer database to obtain the final corpus block sequence includes:

[0109] The query vector is obtained by vectorizing the query text using a semantic vectorization model.

[0110] In some embodiments, the query text is first subjected to standardized preprocessing, which is completely consistent with the preprocessing of the ontology nodes and corpus nodes. Special symbols in the text are removed, punctuation is standardized, and English text is converted to lowercase. If domain abbreviations are contained, they are converted to full names according to the knowledge graph entity standard to avoid semantic deviations caused by differences in entity representation. Next, length adaptation is performed, referring to the 64 token limit of the ontology nodes. For texts that are too long, the core question part is retained, and for texts that are too short, they are directly retained. Finally, the text is split into tokens using a tokenizer consistent with the ontology nodes, and [CLS] sentence beginning tags and [SEP] sentence ending tags are added to generate an input tensor that the model can recognize.

[0111] The query vector is then encoded, reusing the semantic vectorization model of the previous ontology nodes and the cross-layer alignment mapping network; the preprocessed query text is input into Sentence-BERT to obtain a low-dimensional vector; the low-dimensional vector is then input into the mapping network to output a high-dimensional vector; finally, L2 normalization is performed on the high-dimensional vector to ensure that it is consistent with the normalization state of the ontology vector and the corpus vector.

[0112] Search the HNSW index structure for ontology nodes that match the query vector;

[0113] In some embodiments, following the logic of HNSW's high-level coarse positioning and low-level precise matching, the ontology node most relevant to the query semantics is found through hierarchical search. The specific operation is as follows:

[0114] First, set the search parameters, starting at the highest level of HNSW to ensure rapid narrowing of the search scope; the number of K nearest neighbors in each level increases from high to low according to the level to balance search efficiency and matching comprehensiveness; the minimum threshold of cosine similarity between the ontology node and the query vector is determined through domain testing, and nodes below the threshold are considered semantically irrelevant and are directly filtered.

[0115] Next, a hierarchical search is performed. Taking the HNSW index structure with layers L0 to L3 as an example, during coarse localization, a starting node is randomly selected at the highest layer L3, and its similarity to the query vector is calculated. Then, the neighbor list of this node is traversed, and the top 3 nodes with the highest similarity are selected as the candidate set. The node with the highest similarity in the candidate set is used as the starting node, and the above operation is repeated at layer L2 to further narrow the scope. When the scope shrinks at layer L1, the optimal node in the candidate set of layer L2 is used as the starting node, and a nearest neighbor search with K=5 is performed to select the top 5 nodes with the highest similarity. At this point, the size of the candidate set has been significantly reduced. During precise matching at the lower layer L0, the optimal node in the candidate set of layer L1 is used as the starting node, and a nearest neighbor search with K=8 is performed to calculate the similarity between all neighbor nodes and the query vector. Nodes with similarity greater than or equal to the threshold are selected to form the final list of matching ontology nodes. If the list is empty, the threshold is appropriately lowered and the search is repeated. If no results are found, an error message is displayed indicating that there are no matching knowledge resources.

[0116] Finally, the matching results were optimized by removing duplicate nodes from the list and sorting them in descending order of similarity. If there were nodes with high semantic similarity, only the nodes with higher similarity were retained to avoid obtaining duplicate corpus nodes in the future.

[0117] Obtain the adjacent corpus nodes corresponding to the ontology node through the related_ids mapping table;

[0118] In some embodiments, the matched ontology nodes are associated with the corpus nodes through the previously constructed related_ids mapping table, ensuring that the corpus nodes are the semantic support content of the ontology nodes. The steps are as follows:

[0119] If the final matched ontology node list contains only one node, directly query the list of adjacent corpus node IDs corresponding to that node in the related_ids mapping table to form the initial corpus node list; if the list contains multiple nodes, query the corpus list of each ontology node respectively, merge and delete duplicate IDs to form the merged corpus node list.

[0120] To ensure that the corpus nodes are semantically relevant to the query, the list needs to be filtered twice. First, the cosine similarity between each corpus node vector and the query vector is calculated, and nodes with a similarity of ≥0.3 are filtered out. If the length of the list after filtering is less than the preset number, all nodes are retained. Then, the knowledge graph is used to verify whether the entities associated with the corpus nodes are consistent with the entities in the query text. Nodes associated with irrelevant entities are removed, and finally, a list of adjacent corpus nodes is formed.

[0121] Calculate the comprehensive score between the query vector, the ontology node, and the adjacent corpus nodes. Sort the adjacent corpus nodes according to the comprehensive score to obtain the final corpus block sequence.

[0122] In some embodiments, the calculation process of the comprehensive score includes:

[0123] The cosine similarity between the ontology node and the query vector is used as the first score, and the cosine similarity between adjacent corpus nodes and the query vector is used as the second score. The first score and the second score are weighted and summed to obtain the comprehensive score.

[0124] In some embodiments, the overall score The expression is:

[0125]

[0126] in, The cosine similarity between the ontology node and the query vector is represented. The cosine similarity between adjacent corpus nodes and the query vector is represented. and These are weight parameters;

[0127] The effective corpus nodes are sorted in descending order of their comprehensive scores, with higher scores having higher priority. The sequence length is set according to domain requirements to balance information completeness and reading burden. If the top N nodes after sorting contain semantically highly similar corpus blocks, the node with the higher score is retained and replaced with the next node with a lower score but different semantics to ensure the diversity of sequence content. Finally, the corpus block texts corresponding to the sorted and optimized corpus nodes are arranged in score order to form the final corpus block sequence, which is used by the subsequent language model to generate retrieval response text.

[0128] A language model is constructed, and the final corpus block sequence is input into the language model for semantic retrieval to obtain the retrieval response text.

[0129] In some embodiments, the language model should be selected based on the characteristics of the target domain, deployment conditions, and resource budget. The core is to balance information accuracy, generation efficiency, and professional adaptability. If the target domain is a professional field such as biomedicine or mechanical engineering, or if there are data privacy requirements, open-source models should be prioritized for local deployment. Medium-to-large general-purpose models adapted to Chinese scenarios, such as ChatGLM-3-6B, LLaMA-2-7B, and Qwen-7B, should be selected. The selection criteria are support for instruction following, single answer generation time ≤500ms, and GPU memory usage ≤16GB, and compatibility with ordinary GPU deployment. If local deployment is not required or rapid deployment is desired, commercial large model APIs such as GPT-4oMini, Wenxin Yiyan 4.0, and Tongyi Qianwen 3.0 can be called. These require support for custom context length, response latency ≤1s, and the ability to prevent fabrication of information through instruction constraints.

[0130] Inference parameters are set according to model type and query complexity. The maximum generation length `max_new_tokens` for the open-source localized model is set to 512 or 256, the temperature parameter is set to 0.3-0.5 to reduce randomness, `top_p=0.8` focuses on core information, and sampling generation is enabled to avoid mechanical responses. API calls are set according to the documentation. For example, for GPT-4oMini, `max_tokens=512`, `temperature=0.4`, and `response_format=text` are set, and a fixed random seed is added to ensure consistency of results.

[0131] When executing the inference process, the standardized input is first converted into a format that the model can recognize before inference is started. The open-source model is executed through the model.generate() function, and the API is called through an HTTP request. During the inference process, the generation time is monitored and required to be ≤1 second. If the generated content exceeds the set length, it is automatically truncated. If a statement such as "no relevant information found" appears, it is necessary to check whether the corpus block sequence is empty or lacks information. If necessary, the statement "The current corpus information is insufficient and cannot fully answer your query" is returned.

[0132] During the model inference process, it needs to automatically complete intent matching, weight adaptation and logical integration, identify the core question points of the query, and prioritize extracting information from the corresponding dimensions in the corpus blocks; it should prioritize using the top 3 corpus blocks with high comprehensive scores, and use the top 4-10 as supplements to highlight the key points of the answer; for complex queries, information should be organized by question dimensions, and for simple queries, information should be logically connected into paragraphs by definition + core features + examples to avoid information fragmentation.

[0133] After generating the initial answer, accuracy calibration is performed by comparing the answer with the corpus block sequence to ensure no new or unmentioned information or omissions of key information, achieving 100% corpus-based accuracy. For specialized fields, terminology accuracy is verified using a domain dictionary to avoid errors. Logical inconsistencies within the answer are checked; if any are found, corrections are made based on the corpus block information with the highest overall score. Next, formatting and readability optimization are performed: complex queries must be broken down into bullet points, each corresponding to a question dimension, avoiding large blocks of text; simple queries should be presented as coherent paragraphs, with paragraph length ≤ 3 lines. Duplicate expressions and meaningless modifiers are removed to ensure a smooth and accurate response. Answers should be concise and refined; use consistent punctuation, leave a blank line between paragraphs, and start newlines after punctuation marks to maintain a neat format; when handling exceptions, if the corpus block sequence only contains part of the query-related information, the answer should clearly indicate that it is based on the existing corpus, and if the remaining information is unclear, it is recommended to supplement the query; if the corpus block sequence is empty or all corpus blocks have a similarity of ≤0.3 with the query, return the statement directly: No corpus information related to your query was found, and an answer cannot be generated. Please try adjusting your query keywords; if the answer contains semantic confusion, terminology errors, or other problems, re-execute the inference or replace the Top 5 corpus blocks to regenerate.

[0134] Reference Figure 2 This invention provides a vector database retrieval device 20 based on a two-layer structure of ontology and corpus, used to implement a vector database retrieval method based on a two-layer structure of ontology and corpus. The device includes:

[0135] The node acquisition module 21 is used to acquire knowledge resource texts in the target domain, classify the knowledge resource texts by type, and obtain ontology nodes and corpus nodes.

[0136] The vector acquisition module 22 is used to construct a semantic vectorization model. It inputs ontology nodes and corpus nodes into the semantic vectorization model for encoding to obtain ontology vectors and corpus vectors.

[0137] The two-layer database acquisition module 23 is used to perform association mapping between ontology nodes and corpus nodes based on the cosine similarity between ontology vectors and corpus vectors to obtain the related_ids mapping table; obtain the initial HNSW structure; insert ontology nodes and corpus nodes into the initial HNSW structure according to the related_ids mapping table to obtain the HNSW index structure; and generate a two-layer database through the related_ids mapping table and the HNSW index structure.

[0138] The final corpus block sequence acquisition module 24 is used to acquire the query text, and to obtain the final corpus block sequence by performing ontology matching and corpus matching on the query text through a two-layer database.

[0139] The retrieval response text acquisition module 25 is used to construct a language model. It inputs the final corpus block sequence into the language model for semantic retrieval and obtains the retrieval response text.

[0140] This application provides an electronic device, including a processor and a memory; the memory stores a computer program, wherein the computer program, when executed by the processor, implements the vector database retrieval method based on a dual-layer structure of ontology and corpus as described above.

[0141] Specifically, the processor may include, for example, a general-purpose microprocessor, an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor may also include onboard memory for caching purposes. The processor may be a single processing unit or multiple processing units for performing different actions of the method flow according to embodiments of this application.

[0142] Memory can be any medium capable of containing, storing, transmitting, propagating, or transmitting instructions. For example, memory can include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, instruments, or propagation media. Specific examples of memory include: magnetic storage devices such as magnetic tape or hard disk drives (HDDs); optical storage devices such as optical discs (CD-ROMs); and also random access memory (RAM) or flash memory; and / or wired / wireless communication links.

[0143] This application also provides a computer-readable medium storing a computer program that, when executed by a processor, implements the vector database retrieval method based on a two-layer structure of ontology and corpus described above. This computer-readable medium may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The aforementioned computer-readable medium carries one or more programs, which, when executed, implement the method as described in the embodiments of this application.

[0144] According to embodiments of this application, a computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wired, optical fiber, radio frequency signals, etc., or any suitable combination thereof.

[0145] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments and / or claims of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application. Therefore, the scope of this application should not be limited to the above embodiments, but should be defined not only by the appended claims, but also by their equivalents. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A vector database retrieval method based on a two-layer structure of ontology and corpus, characterized in that, include: Acquire knowledge resource texts in the target domain, classify the knowledge resource texts by type, and obtain ontology nodes and corpus nodes; Construct a semantic vectorization model, input ontology nodes and corpus nodes into the semantic vectorization model for encoding, and obtain ontology vectors and corpus vectors; Based on the cosine similarity between ontology vectors and corpus vectors, ontology nodes and corpus nodes are associated and mapped to obtain the related_ids mapping table; the initial HNSW structure is obtained, and ontology nodes and corpus nodes are inserted into the initial HNSW structure according to the related_ids mapping table to obtain the HNSW index structure; a two-layer database is generated through the related_ids mapping table and the HNSW index structure. The query text is obtained, and ontology matching and corpus matching are performed on the query text through a two-layer database to obtain the final corpus block sequence. A language model is constructed, and the final corpus block sequence is input into the language model for semantic retrieval to obtain the retrieval response text.

2. The vector database retrieval method based on a two-layer structure of ontology and corpus as described in claim 1, characterized in that, The process of classifying knowledge resource texts by type to obtain ontology nodes and corpus nodes includes: Knowledge resource texts include abstract problem texts, definition texts, concept texts, literature texts, experimental report texts, and textbook texts; Atomic problem texts are extracted from abstract problem texts, definition texts, and concept texts, and these atomic problem texts are used as ontology nodes. The literature text, experimental report text, and textbook text are divided into corpus blocks, and the corpus blocks are used as corpus nodes.

3. The vector database retrieval method based on a two-layer structure of ontology and corpus as described in claim 1, characterized in that, The step of inputting ontology nodes and corpus nodes into a semantic vectorization model for encoding to obtain ontology vectors and corpus vectors includes: Low-dimensional vectors are obtained by encoding ontology nodes in a low-dimensional manner using a semantic vectorization model. High-dimensional vectors are obtained by encoding corpus nodes in a high-dimensional manner using a semantic vectorization model. Cross-layer alignment is performed on low-dimensional and high-dimensional vectors to obtain ontology vectors and corpus vectors.

4. The vector database retrieval method based on a two-layer structure of ontology and corpus as described in claim 1, characterized in that, The process of associating ontology nodes and corpus nodes based on the cosine similarity between ontology vectors and corpus vectors to obtain the related_ids mapping table includes: Obtain the knowledge graph of the target domain, and calculate the cosine similarity between the ontology vector and the corpus vector based on the knowledge graph. Select the corpus vectors with the highest cosine similarity of a preset number as the corpus vectors adjacent to the ontology vector, and take the corpus nodes corresponding to the adjacent corpus vectors as the corpus nodes adjacent to the ontology node. Perform association mapping between ontology nodes and adjacent corpus nodes, and construct the related_ids mapping table based on all association mappings.

5. The vector database retrieval method based on a two-layer structure of ontology and corpus as described in claim 1, characterized in that, The step of inserting ontology nodes and corpus nodes into the initial HNSW structure according to the related_ids mapping table to obtain the HNSW index structure includes: The initial HNSW structure consists of multiple levels, with the target probability decreasing from top to bottom. Set the probability radius, and calculate the probability range of each level based on the probability radius and the target probability; Obtain the level probability of the ontology node, and insert the ontology node whose level probability is within the probability range into the corresponding level; The adjacent corpus nodes of the ontology node are obtained from the related_ids mapping table, and the adjacent corpus nodes are inserted into each level to obtain the HNSW index structure.

6. The vector database retrieval method based on a two-layer structure of ontology and corpus as described in claim 1, characterized in that, The process of performing ontology matching and corpus matching on the query text using a two-layer database to obtain the final corpus block sequence includes: The query vector is obtained by vectorizing the query text using a semantic vectorization model. Search the HNSW index structure for ontology nodes that match the query vector; Obtain the adjacent corpus nodes corresponding to the ontology node through the related_ids mapping table; Calculate the comprehensive score between the query vector, the ontology node, and the adjacent corpus nodes. Sort the adjacent corpus nodes according to the comprehensive score to obtain the final corpus block sequence.

7. The vector database retrieval method based on a two-layer structure of ontology and corpus as described in claim 6, characterized in that, The calculation process for the overall score includes: The cosine similarity between the ontology node and the query vector is used as the first score, and the cosine similarity between adjacent corpus nodes and the query vector is used as the second score. The first score and the second score are weighted and summed to obtain the comprehensive score.

8. A vector database retrieval device based on a dual-layer structure of ontology and corpus, used to implement the vector database retrieval method based on a dual-layer structure of ontology and corpus as described in any one of claims 1 to 7, characterized in that, The device includes: The node acquisition module is used to acquire knowledge resource texts in the target domain, classify the knowledge resource texts by type, and obtain ontology nodes and corpus nodes. The vector acquisition module is used to construct a semantic vectorization model. It inputs ontology nodes and corpus nodes into the semantic vectorization model for encoding to obtain ontology vectors and corpus vectors. The two-layer database acquisition module is used to perform association mapping between ontology nodes and corpus nodes based on the cosine similarity between ontology vectors and corpus vectors to obtain the related_ids mapping table; obtain the initial HNSW structure; insert ontology nodes and corpus nodes into the initial HNSW structure according to the related_ids mapping table to obtain the HNSW index structure; and generate a two-layer database through the related_ids mapping table and the HNSW index structure. The final corpus block sequence acquisition module is used to acquire query text, and performs ontology matching and corpus matching on the query text through a two-layer database to obtain the final corpus block sequence; The retrieval and response text acquisition module is used to build a language model. The final corpus block sequence is input into the language model for semantic retrieval to obtain the retrieval and response text.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the vector database retrieval method based on a two-layer structure of ontology and corpus as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the vector database retrieval method based on a two-layer structure of ontology and corpus as described in any one of claims 1 to 7.

Images