Knowledge base construction method and device, electronic equipment and storage medium

Abstract

The application relates to the technical field of knowledge base construction, and provides a knowledge base construction method and device, electronic equipment and a storage medium, the method comprising the following steps: determining semantic boundaries in knowledge data according to a preset segmentation strategy; dividing the knowledge data into multiple logical data units based on the semantic boundaries; dividing each logical data unit into a physical storage unit; constructing a knowledge base comprising a hierarchical index structure, a first semantic representation and a second semantic representation; and the hierarchical index structure comprising a hierarchical mapping relationship between the logical data units and the physical storage units. According to the method, the coarse-grained first semantic representation of the logical data unit level and the fine-grained second semantic representation of the physical storage unit level are constructed, and the hierarchical index structure is combined, so that, when knowledge retrieval is performed, the related logical data unit can be quickly screened out through the first semantic representation, and the most relevant physical storage unit in the unit can be accurately positioned through the second semantic representation, thereby improving the accuracy of knowledge base retrieval.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of knowledge base construction, and in particular to a knowledge base construction method and device, electronic equipment and a storage medium. BACKGROUND

[0002] In the field of artificial intelligence, the wide application of large language models puts forward higher requirements for the construction of knowledge bases. The existing large model knowledge base construction technology has many limitations.

[0003] In the data processing link, the traditional text segmentation method relies on simple rules, such as dividing paragraphs only according to punctuation marks or fixed formats, which cannot accurately capture the semantic logic of the text. For texts with loose structure or complex semantics, the segmentation effect is poor, affecting the effective organization of knowledge. In the aspect of index construction, the single-layer index structure has low retrieval efficiency when processing large-scale knowledge data, making it difficult to quickly locate the required knowledge, and cannot meet the demand of real-time knowledge calling of large language models. SUMMARY

[0004] The present application provides a knowledge base construction method, device, electronic equipment and storage medium to solve the defects in the prior art that in the data processing link, the traditional text segmentation method relies on simple rules, such as dividing paragraphs only according to punctuation marks or fixed formats, which cannot accurately capture the semantic logic of the text. For texts with loose structure or complex semantics, the segmentation effect is poor, affecting the effective organization of knowledge. In the aspect of index construction, the single-layer index structure has low retrieval efficiency when processing large-scale knowledge data, making it difficult to quickly locate the required knowledge, and cannot meet the demand of real-time knowledge calling of large language models.

[0005] The present application provides a knowledge base construction method, comprising the following steps:

[0006] Obtain knowledge data, and determine the semantic boundary in the knowledge data according to a preset segmentation strategy;

[0007] Based on the semantic boundary, the knowledge data is divided into a plurality of logical data units;

[0008] Each of the logical data units is divided into at least one physical storage unit, which is the basic unit of data storage and scheduling;

[0009] A knowledge base including a hierarchical index structure, a first semantic representation and a second semantic representation is constructed; the hierarchical index structure includes a hierarchical mapping relationship between the logical data units and the physical storage units, the first semantic representation is used to represent the content semantics of the logical data units, and the second semantic representation is used to represent the content semantics of the physical storage units.

[0010] According to the knowledge base construction method provided by the application, the hierarchical mapping relationship comprises a global index relationship, a segment table index relationship and a page table index relationship.

[0011] The global index relationship is used for representing the mapping relationship between the knowledge data and the identification of the logical data unit.

[0012] The segment table index relationship is used for representing the mapping relationship between the identification of the logical data unit and the first semantic representation corresponding to the identification of the logical data unit; the first semantic representation is a segment semantic vector corresponding to the logical data unit.

[0013] The page table index relationship is used for representing, in each logical data unit, the mapping relationship between the identification of each physical storage unit of the logical data unit and the second semantic representation corresponding to the physical storage unit; the second semantic representation is a page semantic vector corresponding to the physical storage unit.

[0014] According to the knowledge base construction method provided by the application, the method further comprises:

[0015] Obtaining a domain keyword in a retrieval request of a user;

[0016] Determining a candidate logical data unit based on the global index relationship and the domain keyword;

[0017] Selecting a target logical data unit from the candidate logical data unit based on semantic similarity between a segment semantic feature corresponding to the candidate logical data unit and a retrieval feature of the retrieval request;

[0018] Selecting each first physical storage unit from candidate physical storage units in the target logical data unit based on semantic similarity between the retrieval feature and a page semantic vector corresponding to the candidate physical storage unit;

[0019] Taking each first physical storage unit as a retrieval result of the retrieval request.

[0020] According to the knowledge base construction method provided by the application, the taking each first physical storage unit as a retrieval result of the retrieval request comprises:

[0021] Merging each first physical storage unit in a merging order to obtain a candidate merged storage unit; the merging order is determined based on the order of the first physical storage unit in the logical data unit;

[0022] Adjust the candidate merged storage unit based on semantic breakpoints in the candidate merged storage unit, to obtain a target merged storage unit; the semantic breakpoints include spaces, isolated punctuation marks, and syntactic incoherence position points in the candidate merged storage unit;

[0023] Take the target merged storage unit as a search result of the search request.

[0024] According to the knowledge base construction method provided by the application, the preset segmentation strategy includes a first segmentation strategy, a second segmentation strategy, and a third segmentation strategy;

[0025] The semantic boundaries in the knowledge data are determined according to the preset segmentation strategy, including:

[0026] According to the first segmentation strategy, the semantic boundaries in the knowledge data are determined as semantic mutation point positions at which semantic similarity of adjacent sentences in the knowledge data is lower than a preset threshold value;

[0027] According to the second segmentation strategy, the semantic boundaries in the knowledge data are determined as paragraph end tags; the paragraph end tags are obtained by inputting the knowledge data into a sequence labeling model;

[0028] According to the third segmentation strategy, the semantic boundaries in the knowledge data are determined as positions corresponding to structural features in the knowledge data; the positions corresponding to the structural features are obtained based on display structure rule matching of the knowledge data.

[0029] According to the knowledge base construction method provided by the application, the method further includes:

[0030] Based on access frequencies of the physical storage units, the physical storage units are divided into at least three data levels;

[0031] The physical storage units of the at least three data levels are respectively stored in storage media with different access speeds.

[0032] According to the knowledge base construction method provided by the application, the method further includes:

[0033] When space of a high-speed storage medium in the storage media is insufficient, semantic activity degrees are determined based on a latest access time of the physical storage unit in the high-speed storage medium, a historical search hit frequency, and semantic similarity between a page semantic vector of the physical storage unit and a high-frequency query;

[0034] The physical storage unit corresponding to the lowest semantic activity degree is moved from the high-speed storage medium to a low-speed storage medium, to release available space of the high-speed storage medium.

[0035] According to the knowledge base construction method provided by the application, the method further comprises:

[0036] When the knowledge data changes, at least one of the following operations is performed to update the hierarchical index structure incrementally:

[0037] In response to a deletion operation of a second logical data unit in the knowledge data, the hierarchical index structure is updated according to a first preset operation sequence; the first preset operation sequence is sequentially performing the following operations: deleting the page table index relationship corresponding to the second logical data unit, removing the segment table index relationship corresponding to the second logical data unit, and updating the global index relationship;

[0038] In response to an addition operation of the second logical data unit in the knowledge data, the hierarchical index structure is updated according to a second preset operation sequence; the second preset operation sequence is sequentially performing the following operations: adding the segment table index relationship corresponding to the second logical data unit, updating the global index relationship, and creating the page table index relationship corresponding to the second logical data unit;

[0039] In response to a modification operation of the second logical data unit in the knowledge data, the hierarchical index structure is updated according to a third preset operation sequence; the third preset operation sequence is sequentially performing the following operations: re-determining a second semantic representation corresponding to a second physical storage unit in the second logical data unit to update the page table index relationship, and re-determining a first semantic representation corresponding to the second logical data unit based on second semantic representations of all physical storage units included in the second logical data unit to update the segment table index relationship.

[0040] The application further provides a knowledge base construction device, comprising the following units:

[0041] An acquisition unit is configured to acquire knowledge data and determine semantic boundaries in the knowledge data according to a preset segmentation strategy;

[0042] A division unit is configured to divide the knowledge data into a plurality of logical data units based on the semantic boundaries;

[0043] A segmentation unit is configured to segment each of the logical data units into at least one physical storage unit, and the physical storage unit is a basic unit of data storage and scheduling;

[0044] A construction unit is configured to construct a knowledge base comprising a hierarchical index structure, a first semantic representation and a second semantic representation; the hierarchical index structure comprises a hierarchical mapping relationship between the logical data units and the physical storage units, the first semantic representation is used to represent the content semantics of the logical data units, and the second semantic representation is used to represent the content semantics of the physical storage units.

[0045] The application further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the knowledge base construction method according to any one of the above when executing the program.

[0046] The application further provides a non-transitory computer-readable storage medium, which stores a computer program, wherein the computer program is executable on a processor to implement the knowledge base construction method according to any one of the above.

[0047] The application further provides a computer program product, comprising a computer program, wherein the computer program is executable on a processor to implement the knowledge base construction method according to any one of the above.

[0048] The knowledge base construction method, device, electronic device and storage medium provided by the application acquire knowledge data, and determine semantic boundaries in the knowledge data according to a preset segmentation strategy; based on the semantic boundaries, the knowledge data is divided into a plurality of logical data units; each logical data unit is divided into at least one physical storage unit, and the physical storage unit is a basic unit of data storage and scheduling; a knowledge base comprising a hierarchical index structure, a first semantic representation and a second semantic representation is constructed; the hierarchical index structure comprises a hierarchical mapping relationship between the logical data units and the physical storage units, the first semantic representation is used to represent the content semantics of the logical data units, and the second semantic representation is used to represent the content semantics of the physical storage units. The method constructs a coarse-grained first semantic representation at the logical data unit level and a fine-grained second semantic representation at the physical storage unit level, and combines the hierarchical index structure, so that when knowledge retrieval is performed, the first semantic representation can be used to quickly screen out relevant logical data units, and the second semantic representation can be used to accurately locate the most relevant physical storage unit in the unit, thereby realizing efficient and accurate semantic retrieval from massive knowledge data, overcoming the problem of low efficiency of traditional single-layer index traversal, ensuring the semantic relevance and accuracy of recalled knowledge fragments, and meeting the demand of real-time and accurate knowledge calling of a large language model. BRIEF DESCRIPTION OF DRAWINGS

[0049] In order to more clearly illustrate the technical solutions of the present application or the prior art, the following will briefly introduce the drawings needed in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0050] Figure 1 is a flowchart of the knowledge base construction method provided by the present application.

[0051] Figure 2A functional architecture diagram of a knowledge base construction system based on a segment page type hybrid architecture provided by the present application.

[0052] Figure 3 A structural schematic diagram of a knowledge base construction device provided by the present application.

[0053] Figure 4 A structural schematic diagram of an electronic device provided by the present application. DETAILED DESCRIPTION

[0054] To make the objectives, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below with reference to the drawings in the present application. Obviously, the described embodiments are some embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all the other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the present application.

[0055] The terms "first", "second" and the like in the present application are used to distinguish similar objects, and are not used to describe a particular order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second" and the like are generally of a kind.

[0056] The present application provides a knowledge base construction method, Figure 1 A flowchart of a knowledge base construction method provided by the present application is shown in FIG. 1, which comprises steps 110, 120, 130 and 140. Figure 1

[0057] Step 110: acquiring knowledge data, and determining semantic boundaries in the knowledge data according to a preset segmentation strategy;

[0058] Step 120: dividing the knowledge data into a plurality of logical data units based on the semantic boundaries;

[0059] Step 130: dividing each of the logical data units into at least one physical storage unit, the physical storage unit being a basic unit of data storage and scheduling;

[0060] Step 140: constructing a knowledge base comprising a hierarchical index structure, a first semantic representation and a second semantic representation; the hierarchical index structure comprises a hierarchical mapping relationship between the logical data units and the physical storage units, the first semantic representation is used to represent the content semantics of the logical data units, and the second semantic representation is used to represent the content semantics of the physical storage units.

[0061] ​Specifically, in the following description, for ease of description, a logical data unit can be understood as a unit that logically carries one or more complete semantic topics after semantic segmentation of the original knowledge data, similar to a "paragraph" in a traditional article; while a physical storage unit is a basic physical unit used for computer storage and retrieval scheduling, which is further divided from the aforementioned logical data unit, similar to a "page" in operating system memory management.

[0062] Among related technologies, existing large language model knowledge base construction techniques have many limitations. In terms of data acquisition, the data sources are relatively singular, making it difficult to obtain multi-dimensional and multi-format knowledge data. Furthermore, the data formats from different sources vary greatly, and there is a lack of a unified and effective acquisition and conversion mechanism, resulting in incomplete and non-standard knowledge input.

[0063] First, knowledge data can be acquired, and semantic boundaries within the knowledge data can be determined according to a preset segmentation strategy. Specifically, knowledge data refers to the original materials used to construct a knowledge base, and its sources and formats can be diverse. For example, publicly available web pages (such as HTML format), technical forums, and news reports can be crawled from the Internet, or offline documents such as Word documents, PDF files, and plain text (TXT) files can be obtained locally. This embodiment of the invention does not impose specific limitations on these aspects.

[0064] After acquiring the raw knowledge data, preprocessing is typically required. This preprocessing step aims to improve data quality and lay a solid foundation for subsequent semantic analysis and index building.

[0065] Here, preprocessing operations may include data acquisition, data cleaning, format standardization, and semantic repair, etc., and the embodiments of the present invention do not specifically limit these operations.

[0066] In this process, different parsing methods are used for data of different formats. For example, for HTML format data, the main text can be extracted using regular expressions or HTML parsing libraries (such as BeautifulSoup); for PDF files, optical character recognition (OCR) technology or PDF parsing tools (such as PyPDF2) can be used to convert them into plain text. This embodiment of the invention does not impose specific limitations on these methods.

[0067] Data cleaning is a process of removing noise from text, such as HTML tags, JavaScript, CSS styles, advertising content, special characters, and irrelevant headers and footers.

[0068] Standardization is a process of unifying all text into a standard format, such as using UTF-8 encoding, converting all English letters to lowercase, and standardizing punctuation marks (e.g., unifying full-width punctuation marks into half-width punctuation marks).

[0069] Semantic repair can use pre-trained language models to automatically correct spelling or grammatical errors in knowledge data, such as automatically correcting "large model" to "large template" to improve the semantic accuracy of the text. Here, the pre-trained language model can be a BERT (Bidirectional Encoder Representations from Transformers) model, etc., and this embodiment of the invention does not specifically limit it.

[0070] After preprocessing, it is necessary to determine the semantic boundaries in the knowledge data. Semantic boundaries refer to the points in the knowledge data where the theme shifts or logic breaks. Determining semantic boundaries is crucial for effective segmentation. In this step, the preset segmentation strategy can be a rule-based segmentation strategy based on text structure features, a segmentation strategy based on statistical machine learning models, or a segmentation strategy based on deep learning models; this embodiment of the invention does not specifically limit these methods. The core objective of the preset segmentation strategy is to identify suitable locations in the knowledge data as paragraph dividing points. For example, a simple preset segmentation strategy could be "using two consecutive newline characters as semantic boundaries," while a more complex strategy might be "using a trained neural network model to predict whether each sentence is the end of a paragraph." This strategy can mark multiple semantic boundary points in long pieces of knowledge data.

[0071] After determining the semantic boundaries in knowledge data, the knowledge data can be divided into multiple logical data units based on these boundaries. That is, after determining the semantic boundaries, the knowledge data can be segmented along these boundaries to obtain multiple logical data units. Each logical data unit is a semantically independent and complete fragment. Because the division is based on semantics, the length of these logical data units is usually variable. For example, "Chapter 1 Introduction" in a technical article can be divided into one logical data unit, while "Chapter 2 Related Technologies" can be divided into another logical data unit. A logical data unit can be a collection of one or more paragraphs.

[0072] Furthermore, each logical data unit can be divided into at least one physical storage unit, which is the basic unit for data storage and scheduling. Understandably, to facilitate efficient storage management and fast retrieval scheduling by the computer, variable-length logical data units need to be further divided into standardized physical storage units. The physical storage unit is the basic granularity at which data is stored in storage media (such as hard drives and memory) and scheduled by the retrieval system.

[0073] The segmentation method, which divides each logical data unit into at least one physical storage unit, is typically based on a fixed length. For example, a predefined physical storage unit size can be set, such as 256 bytes, 512 bytes, or 1024 bytes. Then, a logical data unit is sequentially divided into multiple such fixed-size blocks from beginning to end. If the end of a logical data unit is smaller than a complete physical storage unit, it can be formed into a smaller physical storage unit, or padded to the standard size. For example, a logical data unit of 1200 bytes, if the predefined physical storage unit size is 512 bytes, can be divided into three physical storage units: the first two are 512 bytes each, and the last is 176 bytes. This "segmented paging" structure preserves the semantic integrity of the logical data units while achieving standardized management of physical storage units.

[0074] Finally, a knowledge base is constructed that includes a hierarchical index structure, a first semantic representation, and a second semantic representation. The hierarchical index structure includes a hierarchical mapping relationship between logical data units and physical storage units. The first semantic representation is used to represent the content semantics of logical data units, and the second semantic representation is used to represent the content semantics of physical storage units.

[0075] Here, a hierarchical index structure is an index used to organize and associate data units at different levels. At its most basic level, it must be able to record the hierarchical mapping relationship between logical data units and the physical storage units that constitute them. That is, through this index, one can know which physical storage units (e.g., "pages 0-5") constitute a particular logical data unit (e.g., "segment 007"). This hierarchical structure provides the foundation for subsequent multi-level retrieval.

[0076] The first semantic representation can be a high-dimensional vector (also known as an embedding). The first semantic representation aims to capture the core theme and meaning of an entire logical data unit (a semantically complete paragraph or chapter). For example, large language models such as BERT and Sentence-BERT can be used to generate a vector that represents the overall semantics of a logical data unit by taking the entire text content as input.

[0077] The second semantic representation can also be a high-dimensional vector. Since a physical storage unit is only a fragment of logical data, its text content may be incomplete. Therefore, its semantic representation aims to capture the content of this fragment itself. For example, a faster model (such as FastText or Word2Vec's mean pooling) can be used to generate a corresponding second semantic representation for each physical storage unit.

[0078] Through the above steps, the constructed knowledge base not only includes the original knowledge data, but also a complex structure that enables efficient mapping and retrieval from macroscopic semantics (logical data units) to microscopic storage (physical storage units).

[0079] Understandably, by dividing knowledge data into semantically complete logical data units and standardized physical storage units, and establishing semantic representations and hierarchical mapping relationships for these two levels, this method constructs a segmented, page-based hybrid architecture knowledge base. This architecture ensures semantic integrity during logical operations (such as coarse-grained semantic retrieval) while leveraging the efficiency of physical storage units in storage and I / O (Input / Output) scheduling. This lays the foundation for achieving fast and accurate multi-level retrieval, effectively balancing the depth of semantic understanding with system operating efficiency.

[0080] The method provided in this invention acquires knowledge data and determines the semantic boundaries of the knowledge data according to a preset segmentation strategy; based on the semantic boundaries, the knowledge data is divided into multiple logical data units; each logical data unit is divided into at least one physical storage unit, which is the basic unit for data storage and scheduling; a knowledge base is constructed including a hierarchical index structure, a first semantic representation, and a second semantic representation; the hierarchical index structure includes a hierarchical mapping relationship between logical data units and physical storage units, the first semantic representation is used to represent the content semantics of the logical data units, and the second semantic representation is used to represent the content semantics of the physical storage units. This method, by constructing a coarse-grained first semantic representation at the logical data unit level and a fine-grained second semantic representation at the physical storage unit level, combined with a hierarchical index structure, allows for rapid filtering of relevant logical data units during knowledge retrieval using the first semantic representation, followed by precise location of the most relevant physical storage unit within the unit using the second semantic representation. This achieves efficient and accurate semantic retrieval from massive amounts of knowledge data, overcoming the low efficiency of traditional single-layer index traversal and ensuring the semantic relevance and accuracy of recalled knowledge fragments, thus meeting the needs of large language models for real-time and accurate knowledge retrieval.

[0081] Based on the above embodiments, the hierarchical mapping relationship includes global index relationship, segment table index relationship and page table index relationship;

[0082] The global index relationship is used to characterize the mapping relationship between the knowledge data and the identifiers of the logical data units;

[0083] The segment table index relationship is used to represent the mapping relationship between the identifier of the logical data unit and the first semantic representation corresponding to the identifier of the logical data unit; the first semantic representation is the segment semantic vector corresponding to the logical data unit.

[0084] The page table index relationship is used to represent the mapping relationship between the identifier of each physical storage unit of the logical data unit and the second semantic representation corresponding to the physical storage unit within each logical data unit; the second semantic representation is the page semantic vector corresponding to the physical storage unit.

[0085] Specifically, the hierarchical index structure is designed as a three-level index system, including global index relationships, segment table index relationships, and page table index relationships.

[0086] Here, global index relationships are used to represent the mapping between knowledge data and the identifiers of logical data units. Located at the top level of the index structure, the global index relationship's primary function is to perform the highest-level partitioning, typically used to handle knowledge data from multiple domains or sources. Here, knowledge data can be understood as a document, a project, or a specific knowledge domain. For example, in a hybrid knowledge base containing legal, financial, and technical documents, global index relationships can help quickly locate sets of logical data units belonging to a specific domain.

[0087] A concrete example is shown below, where the global index relationship can be a JSON object or a hash table:

[0088] {

[0089] "Legal Knowledge": ["Section Identifier 100", "Section Identifier 101", ..., "Section Identifier 200"],

[0090] "Financial Knowledge": ["Segment ID 300", "Segment ID 301", ..., "Segment ID 450"],

[0091] "Technical Documentation": ["Segment Identifier 500", "Segment Identifier 501", ..., "Segment Identifier 800"]

[0092] }

[0093] In this example, the domain keyword "legal knowledge" is mapped to the identifiers of a series of logical data units (segment identifiers 100 to 200). When a user queries a legally related question, the system can first limit the search scope to these 100 logical data units through a global index, greatly narrowing the initial search range.

[0094] Here, the segment table index relationship is an intermediate layer in the index structure, responsible for managing all logical data units. The segment table index relationship represents the mapping between the identifier of a logical data unit and the corresponding first semantic representation. In practice, it typically also contains other metadata for that logical data unit.

[0095] Specifically, the first semantic representation is the segment semantic vector corresponding to the logical data unit. A segment table can be implemented as a table or an array of objects, where each row / object corresponds to a logical data unit and records the following information:

[0096] Logical data unit identifier (e.g., segment number): a globally unique ID that increments globally.

[0097] Segment semantic vector (i.e., first semantic representation): A vector generated by models such as Sentence-BERT that represents the overall semantics of the logical data unit.

[0098] Other metadata: such as the byte length of the segment, the starting position in the original knowledge data, the data source (e.g., "internal enterprise document.doc", "industry report.pdf"), etc., are not specifically limited in this embodiment of the invention.

[0099] By using the segment table index relationship, the system can quickly find the complete semantic information (segment semantic vector) and other metadata of the logical data unit based on its identifier, thus preparing for segment-level semantic similarity matching.

[0100] Here, the page table index relationship is located at the lowest level of the index structure and is managed within each logical data unit. The page table index relationship is used to represent the mapping relationship between the identifiers of the various physical storage units contained in the logical data unit and the corresponding second semantic representations of these physical storage units within each logical data unit.

[0101] The second semantic representation is the page semantic vector corresponding to the physical storage unit. For each logical data unit, there is a corresponding page table. This page table records information about all physical storage units within it, which may include the physical storage unit identifier, page semantic vector, and other metadata.

[0102] Here, the physical storage unit identifier (e.g., page number) is unique within its logical data unit. The page semantic vector (i.e., the second semantic representation) is a vector generated by models such as FastText that represents the semantics of the text fragment within that physical storage unit.

[0103] Other metadata includes, for example, the starting byte offset of the physical storage unit within its logical data unit (page 1 offset 0, page 2 offset 512), the actual stored text content, or its hash value.

[0104] By using page table indexing, once a specific logical data unit is located, the system can further perform precise, fine-grained semantic matching on all physical storage units within that unit.

[0105] The method provided in this invention constructs a hierarchical mapping relationship including a global index, a segment table index, and a page table index, and associates the identifier, segment semantic vector, and page semantic vector of the logical data unit respectively. During knowledge retrieval, the target logical data unit can be quickly located through the global index, then coarse-grained semantic filtering can be performed using the first semantic representation through the segment table index, and finally, the physical storage unit can be accurately located within the unit using the second semantic representation through the page table index. This achieves hierarchical and efficient retrieval of massive knowledge data, greatly improves the speed and accuracy of semantic queries, overcomes the problems of low traversal efficiency and difficulty in quickly locating knowledge in traditional single-layer index structures, and meets the needs of large language models for real-time knowledge invocation.

[0106] Based on the above embodiments, the method further includes:

[0107] Step 210: Obtain the domain keywords from the user's search request;

[0108] Step 220: Based on the global index relationship and the domain keywords, determine the candidate logical data units;

[0109] Step 230: Based on the semantic similarity between the segment semantic features corresponding to the candidate logical data unit and the retrieval features of the retrieval request, select the target logical data unit from the candidate logical data units;

[0110] Step 240: Based on the semantic similarity between the retrieval features and the page semantic vectors corresponding to the candidate physical storage units in the target logical data unit, select each first physical storage unit from the candidate physical storage units;

[0111] Step 250: Use each of the first physical storage units as the search result of the search request.

[0112] Specifically, the domain keywords in the user's search request are obtained, and candidate logical data units are determined based on the global index relationship and the domain keywords.

[0113] When a user submits a search request (for example, entering the query "How should company financial statements be audited"), the system first parses the request and extracts domain keywords. For example, "finance" can be extracted as a domain keyword from the query above.

[0114] Then, the system performs the first filtering using the global index. By querying the keyword "finance," the global index returns a set of all logical data unit identifiers related to "financial knowledge" (e.g., ["segment identifier 300",..., "segment identifier 450"]). This set is the candidate logical data units. This step, also known as "global index filtering," aims to quickly eliminate a large number of irrelevant knowledge areas, narrowing the search scope to the most likely relevant set of logical data units.

[0115] Then, the target logical data unit can be selected from the candidate logical data units based on the semantic similarity between the segment semantic features corresponding to the candidate logical data unit and the retrieval features of the retrieval request. In this process, firstly, the system needs to convert the user's complete retrieval request ("How should company financial statements be audited") into a vector using the same model as that used to generate the segment semantic vector (such as Sentence-BERT), and this vector is called the retrieval feature.

[0116] Next, the system iterates through all candidate logical data units selected in the previous step. For each candidate logical data unit, the system retrieves its corresponding first semantic representation (i.e., segment semantic vector) from the segment table index relationship. Then, it calculates the semantic similarity between the segment semantic vector and the aforementioned retrieval feature (query vector). The semantic similarity can be calculated using methods such as cosine similarity and Euclidean distance; this embodiment of the invention does not specifically limit the method used.

[0117] Furthermore, all candidate logical data units can be sorted based on the calculated similarity scores, and the group of logical data units with the highest similarity scores can be selected (e.g., Top-N, where N can be set to 5 or 10). These selected logical data units are the target logical data units. This step is "segment-level vector matching," which performs semantic matching at the paragraph (logical unit) level, ensuring that the selected logical data units are highly relevant to the user's query in terms of the overall topic.

[0118] Finally, based on the semantic similarity between the search features and the page semantic vectors corresponding to the candidate physical storage units in the target logical data unit, each first physical storage unit is selected from the candidate physical storage units.

[0119] After identifying the target logical data unit, the retrieval process enters its most refined stage. For each target logical data unit, the page table index is used to obtain all the physical storage units contained within it (these can be considered candidate physical storage units) and their corresponding second semantic representations (i.e., page semantic vectors). Then, the system again uses the user's retrieval features (query vectors) to calculate semantic similarity with the page semantic vectors of these candidate physical storage units. Similarly, semantic similarity can be calculated using methods such as cosine similarity and Euclidean distance, and this embodiment of the invention does not specifically limit the methods used.

[0120] During this process, all physical storage units with similarity scores higher than a certain threshold, or those ranking in the top M positions in terms of similarity score among all target logical data units (e.g., Top-M, where M can be set to 10), are selected. These ultimately selected physical storage units, most relevant to the user's query, are the first physical storage units. This step is "page-level precise retrieval," which performs matching at the physical storage fragment level, accurately locating text fragments containing answer keywords or key semantics.

[0121] Finally, each of the first physical storage units is used as the retrieval result of the retrieval request. These first physical storage units are the basic information fragments in the knowledge base that best match the semantics of the user's query.

[0122] The method provided in this invention combines domain keywords and a hierarchical index structure for retrieval. First, it uses a global index to quickly locate candidate logical data units. Then, it filters out target logical units by comparing the similarity between segment semantic vectors and retrieval requests. Finally, it uses page semantic vectors within the target units to accurately match the most relevant physical storage units as results. This achieves rapid and accurate knowledge location from a massive knowledge base, improving retrieval efficiency and ensuring the semantic relevance of results to requests. It effectively meets the needs of large language models for real-time and accurate knowledge retrieval.

[0123] Based on the above embodiments, step 250 includes:

[0124] Step 251: Merge each of the first physical storage units in a merging order to obtain candidate merged storage units; the merging order is determined based on the order of the first physical storage units in the logical data units;

[0125] Step 252: Based on the semantic breakpoints in the candidate merged storage units, adjust the candidate merged storage units to obtain the target merged storage unit; the semantic breakpoints include spaces, isolated punctuation marks, and syntactic incoherence points in the candidate merged storage units.

[0126] Step 253: Use the target merged storage unit as the retrieval result of the retrieval request.

[0127] Specifically, the first physical storage units are merged according to a merging order to obtain candidate merged storage units. The merging order is determined based on the order of the first physical storage units within the logical data units. In other words, the merging order is determined based on their inherent order within the original logical data units. For example, they can be sorted using page numbers or starting offsets recorded in the page table index. If the retrieved physical storage units come from the same logical data unit, they are sorted in ascending order of their page numbers within that unit; if they come from different logical data units, they can first be sorted by the similarity of the logical data units, and then sorted by page number within each logical data unit.

[0128] After sorting, the text content of these physical storage units is concatenated according to a defined merging order to form one or more longer text blocks, i.e., candidate merged storage units. For example, if "page 3" and "page 4" belonging to the same logical data unit are retrieved, their text content is concatenated in sequence.

[0129] Then, based on the semantic breakpoints in the candidate merged storage units, the candidate merged storage units are adjusted to obtain the target merged storage unit. The semantic breakpoints include spaces, isolated punctuation marks, and syntactic incoherence points in the candidate merged storage units.

[0130] Considering that candidate merged storage units formed by direct concatenation may have semantic breakpoints at the boundaries of physical storage units, a semantic breakpoint refers to a location where sentence structure is incomplete or punctuation is used improperly due to forced paging.

[0131] The semantic breakpoint can be a space at the beginning or end of a sentence, or an isolated punctuation mark. For example, if a physical storage unit ends with a comma and the next physical storage unit begins with a lowercase letter, it is obviously an incomplete sentence.

[0132] Semantic breakpoints can also be points where syntax is disjointed. For example, one physical storage unit ends with "artificial intelligence" while the next begins with "broad development prospects," and the combined result is "the development prospects of artificial intelligence are broad." However, if the splicing point happens to be the end of a complete sentence and the beginning of another sentence, it is necessary to ensure that there are correct punctuation marks (such as periods) and separators between them.

[0133] Here, the adjustment methods for candidate merged storage units can be diverse, based on semantic breakpoints within them. Simpler approaches involve rule-based correction, such as deleting extra spaces at the joints or determining merging based on punctuation. More advanced methods utilize pre-trained language models, taking candidate merged storage units as input for "text polishing" or "sentence repair." The language model, with its strong contextual understanding, can automatically correct grammatical errors, complete or adjust punctuation, and generate more fluent and complete target merged storage units. For example, merging "Artificial Intelligence [End of Page 1], Model Training [Start of Page 2]" into a complete sentence and adjusting punctuation through syntactic analysis.

[0134] Finally, the target merged storage unit is presented as the search result for the retrieval request. After adjustment and repair, the resulting target merged storage unit is coherent and highly readable. This final text block is then presented to the user as the search result.

[0135] The method provided in this invention, by sequentially merging the first physical storage units and adjusting them based on semantic breakpoints, ensures that the final returned retrieval results maintain semantic coherence and structural integrity while effectively avoiding problems such as isolated spaces, punctuation marks, or syntactic breaks that may result from rigid mechanical splicing. This improves the readability and accuracy of the returned knowledge fragments and meets the needs of large language models for high-quality knowledge retrieval.

[0136] Based on the above embodiments, the preset segmentation strategy includes a first segmentation strategy, a second segmentation strategy, and a third segmentation strategy;

[0137] Step 110, which involves determining the semantic boundaries in the knowledge data according to a preset segmentation strategy, includes:

[0138] Step 111: Based on the first segmentation strategy, determine the semantic boundary in the knowledge data as the location of the semantic abrupt change point where the semantic similarity between adjacent sentences in the knowledge data is lower than a preset threshold.

[0139] Step 112: Based on the second segmentation strategy, determine the semantic boundaries in the knowledge data as paragraph end labels; the paragraph end labels are obtained by inputting the knowledge data into the sequence labeling model;

[0140] Step 113: According to the third segmentation strategy, determine the semantic boundary in the knowledge data as the position corresponding to the structural feature in the knowledge data; the position corresponding to the structural feature is obtained based on the explicit structural rule matching of the knowledge data.

[0141] Specifically, determining semantic boundaries is the first step in partitioning knowledge data. The preset segmentation strategies include a first segmentation strategy, a second segmentation strategy, and a third segmentation strategy. That is, this embodiment of the invention provides three specific segmentation strategies that can be selected or combined to adapt to different types and formats of knowledge data.

[0142] Based on the first segmentation strategy, the semantic boundary in the knowledge data can be determined as the location of the semantic abrupt change point where the semantic similarity between adjacent sentences in the knowledge data is lower than a preset threshold.

[0143] In one alternative embodiment, the knowledge data is first segmented into sentences. Then, a sentence encoding model (e.g., the BERT model based on the Transformer architecture) is used to convert each sentence into a high-dimensional semantic vector. Next, the similarity between the semantic vectors of each pair of adjacent sentences is calculated sequentially; a common method is cosine similarity. Finally, a preset threshold is set (e.g., 0.6 or 0.65 based on empirical data). When the calculated similarity between adjacent sentences is lower than this threshold, this location is marked as a semantic abrupt change, i.e., a semantic boundary. This strategy is particularly suitable for processing plain text documents without obvious structural features, such as long essays or novels, as it can intelligently discover logical boundaries based on the content itself.

[0144] Based on the second segmentation strategy, the semantic boundaries in the knowledge data can be determined as paragraph end labels, where the paragraph end labels are obtained by inputting the knowledge data into the sequence labeling model.

[0145] The second segmentation strategy transforms the text segmentation problem into a sequence labeling problem in machine learning. The idea is to treat each sentence (or other text unit) in the text as an element in a sequence and predict a label for each element, which indicates its position in the paragraph.

[0146] In one alternative embodiment, first, a set of labels is defined, such as {B-PARA, I-PARA, E-PARA}, representing the beginning, middle, and end of a paragraph, respectively. Then, labeled training data is prepared, i.e., a series of texts that have been manually segmented into paragraphs. Using this data, a sequence labeling model can be trained, such as a Hidden Markov Model (HMM), a Conditional Random Field (CRF), or a more advanced deep learning model like BiLSTM-CRF (Bidirectional Long Short-Term Memory Conditional Random Field). The input to the sequence labeling model can be word vectors, part-of-speech tags, syntactic structure features, etc.

[0147] After the sequence labeling model is trained, new knowledge data is input into it, and the model predicts a label for each sentence. When a sentence is predicted as an end-of-paragraph label (E-PARA), its ending position is determined as a semantic boundary. This strategy is suitable for loosely structured texts with certain patterns, such as online forum posts and news reports, as it learns segmentation patterns through a data-driven approach.

[0148] Here, based on the third segmentation strategy, the semantic boundaries in the knowledge data can be determined as the positions corresponding to the structural features within the knowledge data. These positions are obtained by matching explicit structural rules within the knowledge data. This strategy is the most direct method, relying on the inherent and explicit structural features of the text to determine the segmentation positions.

[0149] In one optional embodiment, the specific implementation is based on matching the display structure rules of the knowledge data. These rules may be title matching, format matching, and combinations of punctuation and line breaks, etc., and the embodiments of the present invention do not specifically limit them.

[0150] Title matching uses regular expressions to match chapter numbers or title formats, such as matching patterns like "Chapter X", "NN", or "### Title". The position of these titles clearly indicates important semantic boundaries.

[0151] Formatting matching detects specific formatting elements in text, such as consecutive line breaks or paragraph first-line indentation. In many document formats, one or more blank lines are often used to separate different paragraphs.

[0152] Punctuation and line break combinations detect specific combinations of punctuation marks and line breaks, such as the pattern "period + line break," which is a marker for paragraph breaks in many plain texts. This strategy is highly efficient and accurate, and is particularly suitable for well-formatted, highly structured documents, such as academic papers, technical manuals, and legal documents.

[0153] It should be noted that in practical applications, these three strategies can be used individually or in combination. For example, the third strategy can be used first for rapid structured segmentation, and then the first or second strategy can be used for finer-grained semantic segmentation of longer, unsegmented text blocks to achieve the best segmentation results.

[0154] The method provided in this invention integrates a first strategy based on semantic similarity, a second strategy based on sequence labeling models, and a third strategy based on structural rule matching to comprehensively determine the semantic boundaries in knowledge data. This overcomes the problem that single-rule segmentation methods are poorly adaptable to texts with complex semantics, loose structures, or irregular formats. It can accurately capture semantic logical changes and structural features in text, significantly improving the accuracy and robustness of segmentation, and laying a solid foundation for the subsequent construction of a high-quality knowledge base.

[0155] Based on the above embodiments, the method further includes:

[0156] Step 310: Based on the access frequency of each physical storage unit, divide the physical storage unit into at least three data levels;

[0157] Step 320: Store the physical storage units of the at least three data levels in storage media with different access speeds.

[0158] Specifically, to efficiently manage massive amounts of physical storage units, the system can establish a data access frequency statistics mechanism. This mechanism is responsible for continuously and periodically monitoring and recording the frequency of access to each physical storage unit. For example, the system can set an access counter for each physical storage unit, incrementing the counter each time the unit is read or written. Simultaneously, the access frequency can be calculated by combining access timestamps with the number of accesses within a specific time window.

[0159] After determining the access frequency of each physical storage unit, the physical storage unit can be divided into at least three data levels based on the access frequency. In this embodiment, it is preferable to divide the physical storage unit into three data levels, but this is not a limitation, and in other embodiments, there may be four or more levels.

[0160] Specifically, one possible way to divide it is:

[0161] Hot data tier: This includes the physical storage units accessed most frequently. For example, the top 20% of physical storage units based on cumulative access frequency can be classified as the hot data tier.

[0162] Warm data tier: This includes physical storage units with moderate access frequency. For example, the middle 60% of physical storage units after sorting by cumulative access frequency can be classified as warm data tiers.

[0163] Cold data tier: This includes physical storage units with the lowest access frequency. For example, the bottom 20% of physical storage units after sorting by cumulative access frequency can be classified as cold data tier.

[0164] In this way, the "hotness" or "coldness" of data can be accurately quantified and classified.

[0165] Then, at least three data tiers of physical storage units are stored in storage media with different access speeds. After completing the data tier division, in order to achieve a balance between storage cost and access performance, this embodiment adopts a hierarchical storage strategy, storing data of different tiers in storage media with different physical characteristics. It should be understood that the access speed of a storage medium is usually proportional to its cost.

[0166] In one embodiment, physical storage units at the hot data level, which are accessed most frequently and have the highest performance requirements, are stored in the fastest access storage medium, such as RAM (Random Access Memory). To further improve performance, a separate cache can be configured for this data in memory.

[0167] For physical storage units at the warm data level, with moderate access frequency, they can be stored in storage media that offer a balance between access speed and cost, such as solid-state drives (SSDs). To accelerate access to data on SSDs, technologies such as memory-mapped files can be used to reduce I / O overhead.

[0168] For physical storage units at the cold data level, since their access frequency is very low, they can be stored in storage media that are low-cost, large-capacity, but relatively slow in access speed, such as hard disk drives or lower-cost distributed file systems.

[0169] The method provided in this invention can dynamically deploy data to the most suitable hardware storage medium based on the actual data access patterns. This mechanism optimizes the allocation of storage resources, avoiding the indiscriminate storage of all data in expensive high-speed storage devices, thereby significantly reducing hardware costs. Simultaneously, it ensures that frequently accessed data is always retained in high-speed media, achieving extremely fast response times and effectively improving the overall retrieval performance and user experience of the knowledge base, achieving the optimal balance between storage costs and access efficiency. When access to physical storage units stored in low-speed media (such as external storage) is required, the system can automatically load them into high-speed media (such as RAM) for processing.

[0170] In related technologies, storage management lacks analysis of data access characteristics and adopts a uniform storage strategy, resulting in wasted storage resources and low memory utilization.

[0171] Based on the above embodiments, the method further includes:

[0172] Step 410: When the high-speed storage medium space in the storage medium is insufficient, determine the semantic activity based on the recent access time and historical retrieval hit count of the physical storage unit in the high-speed storage medium, as well as the semantic similarity between the page semantic vector of the physical storage unit and the high-frequency query.

[0173] Step 420: Remove the physical storage unit corresponding to the lowest semantic activity from the high-speed storage medium to the low-speed storage medium to free up available space in the high-speed storage medium.

[0174] Specifically, during the operation of the knowledge base, especially when a large number of new query requests cause physical storage units originally stored in low-speed or medium-speed storage media to be frequently moved to high-speed storage media (such as memory), insufficient space may occur in the high-speed storage media. In this case, an efficient page replacement strategy is needed to select one or more "sacrificed" physical storage units from the high-speed storage media and move them out to make room for new data.

[0175] Traditional replacement algorithms, such as Least Recently Used (LRU), rely solely on access time as the only metric, which has limitations in knowledge base applications targeting large language models. This is because it may mistakenly remove a physical storage unit that, while not recently accessed, has extremely high semantic value for frequently asked core questions, thus affecting the hit rate and quality of subsequent searches.

[0176] To address this issue, this invention proposes a page replacement strategy based on semantic activity. This method is triggered when high-speed storage media space is insufficient, and specifically includes:

[0177] First, for each physical storage unit currently stored in high-speed storage media, its semantic activity is determined. Semantic activity is a comprehensive indicator designed to more fully and deeply assess the importance of a physical storage unit to current and future query requests.

[0178] Here, semantic activity can be determined based on the recent access time and historical retrieval hit count of physical storage units in high-speed storage media, as well as the semantic similarity between the page semantic vector of the physical storage unit and the semantics of high-frequency queries. Among these, the recent access time of the physical storage unit is a fundamental indicator, reflecting the principle of temporal locality of data access. Generally, units that have been recently accessed are more likely to be accessed again in the short term.

[0179] Among them, the historical retrieval hit count of a physical storage unit measures the total number of times a physical storage unit has been successfully retrieved as a result of the large model over a period of time. The more times a physical storage unit is hit, the more crucial and valuable its content is in answering user questions. This metric of the historical retrieval hit count of a physical storage unit reflects the unit's historical contribution and importance.

[0180] Here, the page semantic vector is a multi-dimensional vector that can represent the core semantics of a physical storage unit (e.g., a data page) by using a vector embedding model to transform its content.

[0181] The system will compile and maintain a list of frequently asked queries, which are the questions or queries most often submitted by users. Similarly, these frequently asked queries can also be converted into semantic vectors.

[0182] It should be understood that by calculating the similarity between the page semantic vector of a physical storage unit and the semantic vector of high-frequency queries (e.g., using a cosine similarity algorithm), the degree of relevance of the unit's content to current hot topics can be determined. Higher similarity means that the physical storage unit is likely more important for answering future high-frequency queries.

[0183] Then, the system can sum the above three dimensions of indicators according to the preset weights to obtain the final semantic activity score of each physical storage unit.

[0184] Finally, the physical storage unit corresponding to the lowest semantic activity is moved from the high-speed storage medium to the low-speed storage medium to free up available space in the high-speed storage medium.

[0185] In one embodiment, the physical storage unit with the lowest semantic activity score can be selected as the replacement target. A lowest score means that the unit has not only been recently accessed and has a low historical contribution, but its content is also semantically far removed from the current user's key concerns. Removing such a unit from high-speed storage media (e.g., from RAM to SSD) is a more intelligent and efficient decision. To achieve this process efficiently, a priority queue can be maintained to dynamically sort physical storage units in the high-speed storage media according to semantic activity, thereby quickly finding and eliminating the unit with the lowest score.

[0186] The method provided in this invention calculates semantic activity by comprehensively considering the recent access time of physical storage units, the number of historical retrieval hits, and the semantic similarity with high-frequency queries. When high-speed storage space is insufficient, low-activity data is moved to low-speed media based on this indicator, thereby realizing dynamic optimization management of storage space. This effectively improves the utilization efficiency of high-speed media and overall retrieval performance, while ensuring fast access to high-frequency and highly semantically relevant data, thus achieving the best balance between storage cost and system performance under limited hardware resources.

[0187] In related technologies, data updates often employ a full update approach, which incurs high computational overhead and low update efficiency, making it difficult to adapt to scenarios with frequent knowledge base updates. These problems severely restrict the performance of large-scale model knowledge bases, necessitating a more efficient and intelligent construction method and system.

[0188] Based on the above embodiments, the method further includes:

[0189] Step 510: When the knowledge data changes, perform at least one of the following operations to incrementally update the hierarchical index structure:

[0190] Step 511: In response to the deletion operation of the second logical data unit in the knowledge data, update the hierarchical index structure according to a first preset operation order; the first preset operation order is to sequentially delete the page table index relationship corresponding to the second logical data unit, remove the segment table index relationship corresponding to the second logical data unit, and update the global index relationship.

[0191] Step 512: In response to the addition operation of the second logical data unit in the knowledge data, update the hierarchical index structure according to the second preset operation order; the second preset operation order is to sequentially execute the addition of the segment table index relationship corresponding to the second logical data unit, update the global index relationship, and create the page table index relationship corresponding to the second logical data unit.

[0192] Step 513: In response to the modification operation of the second logical data unit in the knowledge data, update the hierarchical index structure according to a third preset operation sequence; the third preset operation sequence is to sequentially perform the re-determination of the second semantic representation corresponding to the second physical storage unit in the second logical data unit to update the page table index relationship, and based on the second semantic representation of all physical storage units included in the second logical data unit, re-determination of the first semantic representation corresponding to the second logical data unit to update the segment table index relationship.

[0193] Specifically, as knowledge is continuously iterated and accumulated, the data in the knowledge base will change, such as being added, deleted, or modified. If the entire knowledge base is rebuilt every time a change occurs, it will result in huge computational overhead. To solve this problem, this embodiment proposes an incremental update mechanism. The core idea of ​​this mechanism is to control the scope of the update to the smallest unit, processing only the data units that have changed and their indexes, thereby significantly reducing system overhead.

[0194] In embodiments of the present invention, to achieve rapid retrieval and location of knowledge data, the system constructs a hierarchical index structure. This hierarchical index structure preferably includes: a global index relationship, a segment table index relationship, and a page table index relationship.

[0195] When knowledge data changes, the system performs corresponding operations to incrementally update the hierarchical index structure. Here, knowledge data changes include at least the addition, deletion, and modification of second logical data units. The second logical data unit can be understood as the logical unit for performing update operations, such as an entire document or a paragraph within a document, which may correspond to a "segment" in the knowledge base.

[0196] When a second logical data unit is deleted, the system updates the hierarchical index structure according to a first preset operation sequence. This sequence is a bottom-up "cascading delete" logic, specifically: sequentially deleting the page table index relationship corresponding to the second logical data unit, removing the segment table index relationship corresponding to the second logical data unit, and updating the global index relationship. This bottom-up deletion sequence ensures the atomicity of data operations and the correctness of the index.

[0197] When a new second logical data unit is added to the knowledge base, the system updates the hierarchical index structure according to a second preset operation sequence. Specifically, this involves sequentially adding a segment table index relationship corresponding to the second logical data unit, updating the global index relationship, and creating a page table index relationship corresponding to the second logical data unit. This sequence ensures that the new data can be smoothly and completely incorporated into the entire index system.

[0198] Besides complete additions and deletions, a more common scenario is modifying partial content of an existing document. This embodiment further discloses an efficient update method for this scenario. When the content of a second logical data unit (e.g., a segment) is modified, the system updates the hierarchical index structure according to a third preset operation sequence. This sequence aims to limit the computational load of the update to a minimum, specifically including:

[0199] Step 1: Sequentially execute the process of re-determining the second semantic representation corresponding to the second physical storage unit in the second logical data unit in order to update the page table index relationship.

[0200] In this embodiment, a "second logical data unit" (such as a segment) is composed of one or more "second physical storage units" (such as a page). Each second physical storage unit corresponds to a "second semantic representation," which is preferably a semantic vector that can represent the core semantic content of the unit.

[0201] When a modification operation occurs, the system first precisely locates which one or more second physical storage units (pages) have had their content changed. Then, the system only reprocesses these physical storage units whose content has changed. Specifically, the system invokes a vector embedding model to calculate the new content of these units, thereby generating entirely new second semantic representations (new page semantic vectors). Subsequently, the system replaces the old semantic representations with this newly generated second semantic representation in the page table index relationship. For other physical storage units whose content did not change during the modification operation, their second semantic representations and page table index records remain unchanged, requiring no calculation, demonstrating the efficiency of incremental updates.

[0202] Step 2: Based on the second semantic representation of all physical storage units included in the second logical data unit, redetermine the first semantic representation corresponding to the second logical data unit in order to update the segment table index relationship.

[0203] To support higher-level semantic retrieval, the second logical data unit (segment) itself also has a higher-level "first semantic representation" (segment semantic vector). This first semantic representation is not generated independently, but is aggregated from the second semantic representations (page vectors) of all the physical storage units (pages) it contains. For example, the first semantic representation of a segment can be the average of all its page vectors, or the result calculated through a more complex aggregation function.

[0204] Since at least one second semantic representation (page vector) was updated in step one, the first semantic representation (segment vector), as the aggregation result, must also be recalculated accordingly. To this end, the system collects all second semantic representations under the second logical data unit (segment)—including those newly generated in step one and those that remain unchanged. Then, based on this complete and up-to-date set of second semantic representations, the system uses a preset aggregation algorithm to recalculate a completely new first semantic representation. Finally, the system uses this new first semantic representation to update its corresponding record in the segment table index relationship.

[0205] The method provided in this invention, by pre-setting differentiated update orders and processing logic for the addition, deletion, and modification operations of knowledge data, only performs incremental updates on the smallest affected logical and physical units when data changes, avoiding the huge overhead of rebuilding the entire index and significantly improving the update efficiency of the knowledge base. At the same time, by strictly following the bottom-up semantic representation update order from the page level to the segment level, the consistency and semantic accuracy of the data within the hierarchical index structure are ensured, thereby dynamically maintaining the timeliness of the knowledge base while ensuring the accuracy of subsequent search results.

[0206] In summary, the method provided by this invention offers several advantages. First, it achieves precise semantic segmentation by employing a complementary approach of structural rules, machine learning, and deep learning to cover the segmentation needs of different text types. Compared to single methods, it improves the accuracy of semantic boundary recognition, ensuring knowledge is organized into logical units for easy human review and machine understanding. Second, it optimizes index retrieval by using a three-level index architecture that decomposes the retrieval process into three stages: domain filtering, segment matching, and page refinement. Combined with semantic vector similarity calculation, this results in a faster average retrieval response time than traditional methods, supporting real-time knowledge retrieval from large models. Third, it optimizes storage resources by using tiered storage for hot and cold data combined with a dynamic page replacement algorithm, improving memory hit rate and reducing external storage access frequency, thus supporting larger-scale knowledge bases under the same computing power. Fourth, it enables low-cost dynamic maintenance by using an incremental update mechanism that only processes changed parts. Combined with index consistency verification, this reduces document update time from minutes to seconds, significantly lowering knowledge base maintenance costs and making it suitable for high-frequency enterprise-level applications. Large-scale model enhancement and adaptation: The global index is deeply adapted to the domain division of the knowledge base and the semantic representation of segment and page vectors, and the retrieval enhancement requirements of the large-scale model, which improves the relevance of knowledge calls, effectively reduces the "illusion" problem, and enhances the accuracy and credibility of the generated content.

[0207] Based on any of the above embodiments Figure 2 This is a functional architecture diagram of the knowledge base construction system based on a segmented paginated hybrid architecture provided by the present invention, such as... Figure 2As shown, the system first acquires and preprocesses raw knowledge data through the data collection and cleaning stages of the knowledge construction module. Then, through data segmentation and pagination operations, it divides the data into logical data units and physical storage units and stores them in the data storage area of ​​the knowledge storage module. When a user initiates a query, the knowledge retrieval module starts a global search, quickly filters relevant logical units through segmented matching, and then accurately locates the physical storage unit through pagination matching and returns the results. If the storage space is insufficient, the system dynamically optimizes the storage hierarchy through a data replacement strategy, thereby realizing a closed-loop management of the entire process of knowledge from structured storage and efficient construction to accurate retrieval.

[0208] The knowledge base construction apparatus provided by the present invention is described below. The knowledge base construction apparatus described below and the knowledge base construction method described above can be referred to in correspondence.

[0209] Based on any of the above embodiments, the present invention provides a knowledge base construction apparatus. Figure 3 This is a schematic diagram of the knowledge base construction device provided by the present invention, as shown below. Figure 3 As shown, the device includes:

[0210] The acquisition unit 310 is used to acquire knowledge data and determine the semantic boundaries in the knowledge data according to a preset segmentation strategy.

[0211] The partitioning unit 320 is used to divide the knowledge data into multiple logical data units based on the semantic boundaries.

[0212] The partitioning unit 330 is used to partition each of the logical data units into at least one physical storage unit, wherein the physical storage unit is the basic unit for data storage and scheduling.

[0213] The construction unit 340 is used to construct a knowledge base including a hierarchical index structure, a first semantic representation, and a second semantic representation; the hierarchical index structure includes a hierarchical mapping relationship between the logical data unit and the physical storage unit, the first semantic representation is used to represent the content semantics of the logical data unit, and the second semantic representation is used to represent the content semantics of the physical storage unit.

[0214] The apparatus provided in this invention acquires knowledge data and determines the semantic boundaries of the knowledge data according to a preset segmentation strategy; based on the semantic boundaries, the knowledge data is divided into multiple logical data units; each logical data unit is divided into at least one physical storage unit, with the physical storage unit being the basic unit for data storage and scheduling; a knowledge base is constructed including a hierarchical index structure, a first semantic representation, and a second semantic representation; the hierarchical index structure includes a hierarchical mapping relationship between logical data units and physical storage units, the first semantic representation is used to represent the content semantics of the logical data units, and the second semantic representation is used to represent the content semantics of the physical storage units. This method, by constructing a coarse-grained first semantic representation at the logical data unit level and a fine-grained second semantic representation at the physical storage unit level, combined with a hierarchical index structure, allows for rapid filtering of relevant logical data units during knowledge retrieval using the first semantic representation, followed by precise location of the most relevant physical storage unit within the unit using the second semantic representation. This achieves efficient and accurate semantic retrieval from massive amounts of knowledge data, overcoming the inefficiency of traditional single-layer index traversal and ensuring the semantic relevance and accuracy of recalled knowledge fragments, thus meeting the needs of large language models for real-time and accurate knowledge retrieval.

[0215] Based on any of the above embodiments, the hierarchical mapping relationship includes a global index relationship, a segment table index relationship, and a page table index relationship;

[0216] The global index relationship is used to characterize the mapping relationship between the knowledge data and the identifiers of the logical data units;

[0217] The segment table index relationship is used to represent the mapping relationship between the identifier of the logical data unit and the first semantic representation corresponding to the identifier of the logical data unit; the first semantic representation is the segment semantic vector corresponding to the logical data unit.

[0218] The page table index relationship is used to represent the mapping relationship between the identifier of each physical storage unit of the logical data unit and the second semantic representation corresponding to the physical storage unit within each logical data unit; the second semantic representation is the page semantic vector corresponding to the physical storage unit.

[0219] Based on any of the above embodiments, a retrieval unit is further included, wherein the retrieval unit specifically includes:

[0220] The domain keyword acquisition unit is used to acquire domain keywords from the user's search request;

[0221] Candidate logical data units are determined based on the global index relationship and the domain keywords.

[0222] The first filtering unit is used to filter out the target logical data unit from the candidate logical data units based on the semantic similarity between the segment semantic features corresponding to the candidate logical data unit and the retrieval features of the retrieval request.

[0223] The second filtering unit is used to filter out each first physical storage unit from the candidate physical storage units based on the semantic similarity between the retrieval features and the page semantic vectors corresponding to the candidate physical storage units in the target logical data unit.

[0224] The retrieval subunit is used to take each of the first physical storage units as the retrieval result of the retrieval request.

[0225] Based on any of the above embodiments, the retrieval subunit is specifically used for:

[0226] Each of the first physical storage units is merged in a merging order to obtain a candidate merged storage unit; the merging order is determined based on the order of the first physical storage units in the logical data unit.

[0227] Based on the semantic breakpoints in the candidate merged storage units, the candidate merged storage units are adjusted to obtain the target merged storage unit; the semantic breakpoints include spaces, isolated punctuation marks, and syntactic incoherence points in the candidate merged storage units.

[0228] The target merged storage unit is used as the retrieval result of the retrieval request.

[0229] Based on any of the above embodiments, the preset segmentation strategy includes a first segmentation strategy, a second segmentation strategy, and a third segmentation strategy;

[0230] The acquisition unit 310 is specifically used for:

[0231] According to the first segmentation strategy, the semantic boundary in the knowledge data is determined as the position of the semantic abrupt change point where the semantic similarity between adjacent sentences in the knowledge data is lower than a preset threshold;

[0232] According to the second segmentation strategy, the semantic boundaries in the knowledge data are determined as paragraph end labels; the paragraph end labels are obtained by inputting the knowledge data into the sequence labeling model;

[0233] According to the third segmentation strategy, the semantic boundary in the knowledge data is determined as the position corresponding to the structural feature in the knowledge data; the position corresponding to the structural feature is obtained based on the explicit structural rule matching of the knowledge data.

[0234] Based on any of the above embodiments, a storage unit is further included, wherein the storage unit is specifically used for:

[0235] Based on the access frequency of each physical storage unit, the physical storage unit is divided into at least three data levels;

[0236] The physical storage units of the at least three data levels are stored in storage media with different access speeds.

[0237] Based on any of the above embodiments, a release unit is further included, wherein the release unit is specifically used for:

[0238] When the high-speed storage medium space in the storage medium is insufficient, the semantic activity is determined based on the recent access time, historical retrieval hit count, and the semantic similarity between the page semantic vector of the physical storage unit in the high-speed storage medium and the high-frequency query.

[0239] The physical storage unit corresponding to the lowest semantic activity is moved from the high-speed storage medium to the low-speed storage medium to free up available space in the high-speed storage medium.

[0240] Based on any of the above embodiments, an incremental update unit is further included, wherein the incremental update unit is specifically used for:

[0241] When the knowledge data changes, perform at least one of the following operations to incrementally update the hierarchical index structure:

[0242] In response to the deletion operation of the second logical data unit in the knowledge data, the hierarchical index structure is updated according to a first preset operation order; the first preset operation order is to sequentially delete the page table index relationship corresponding to the second logical data unit, remove the segment table index relationship corresponding to the second logical data unit, and update the global index relationship.

[0243] In response to the addition of the second logical data unit in the knowledge data, the hierarchical index structure is updated according to a second preset operation order; the second preset operation order is to sequentially perform the addition of the segment table index relationship corresponding to the second logical data unit, update the global index relationship, and create the page table index relationship corresponding to the second logical data unit.

[0244] In response to the modification operation of the second logical data unit in the knowledge data, the hierarchical index structure is updated according to a third preset operation sequence. The third preset operation sequence is to sequentially perform the re-determination of the second semantic representation corresponding to the second physical storage unit in the second logical data unit to update the page table index relationship, and based on the second semantic representation of all physical storage units included in the second logical data unit, to re-determination of the first semantic representation corresponding to the second logical data unit to update the segment table index relationship.

[0245] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 4 As shown, the electronic device may include: a processor 410, a communications interface 420, a memory 430, and a communication bus 440, wherein the processor 410, the communications interface 420, and the memory 430 communicate with each other through the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute a knowledge base construction method, which includes: acquiring knowledge data and determining the semantic boundaries in the knowledge data according to a preset segmentation strategy; dividing the knowledge data into multiple logical data units based on the semantic boundaries; dividing each logical data unit into at least one physical storage unit, the physical storage unit being the basic unit for data storage and scheduling; constructing a knowledge base including a hierarchical index structure, a first semantic representation, and a second semantic representation; the hierarchical index structure includes a hierarchical mapping relationship between the logical data units and the physical storage units, the first semantic representation being used to represent the content semantics of the logical data units, and the second semantic representation being used to represent the content semantics of the physical storage units.

[0246] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0247] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the knowledge base construction method provided by the above methods. The method includes: acquiring knowledge data and determining the semantic boundaries in the knowledge data according to a preset segmentation strategy; dividing the knowledge data into multiple logical data units based on the semantic boundaries; dividing each logical data unit into at least one physical storage unit, wherein the physical storage unit is the basic unit for data storage and scheduling; constructing a knowledge base including a hierarchical index structure, a first semantic representation, and a second semantic representation; wherein the hierarchical index structure includes a hierarchical mapping relationship between the logical data units and the physical storage units, the first semantic representation is used to represent the content semantics of the logical data units, and the second semantic representation is used to represent the content semantics of the physical storage units.

[0248] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements a knowledge base construction method provided by the above methods. The method includes: acquiring knowledge data and determining semantic boundaries in the knowledge data according to a preset segmentation strategy; dividing the knowledge data into multiple logical data units based on the semantic boundaries; dividing each logical data unit into at least one physical storage unit, wherein the physical storage unit is the basic unit for data storage and scheduling; constructing a knowledge base including a hierarchical index structure, a first semantic representation, and a second semantic representation; wherein the hierarchical index structure includes a hierarchical mapping relationship between the logical data units and the physical storage units, the first semantic representation is used to represent the content semantics of the logical data units, and the second semantic representation is used to represent the content semantics of the physical storage units.

[0249] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0250] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0251] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for constructing a knowledge base, characterized in that, include: Acquire knowledge data and determine the semantic boundaries in the knowledge data according to a preset segmentation strategy; the semantic boundaries refer to the locations in the content of the knowledge data where the topic changes or logic is interrupted. Based on the semantic boundaries, the knowledge data is divided into multiple logical data units; Each logical data unit is divided into at least one physical storage unit, which is the basic unit for data storage and scheduling. A knowledge base is constructed, comprising a hierarchical index structure, a first semantic representation, and a second semantic representation; the hierarchical index structure includes a hierarchical mapping relationship between the logical data unit and the physical storage unit, the first semantic representation is used to represent the content semantics of the logical data unit, and the second semantic representation is used to represent the content semantics of the physical storage unit; The hierarchical mapping relationship includes global index relationship, segment table index relationship, and page table index relationship; The global index relationship is used to characterize the mapping relationship between the knowledge data and the identifiers of the logical data units; The segment table index relationship is used to represent the mapping relationship between the identifier of the logical data unit and the first semantic representation corresponding to the identifier of the logical data unit; the first semantic representation is the segment semantic vector corresponding to the logical data unit; the segment table index relationship includes other metadata of the logical data unit, which includes the byte length of the segment, the starting position in the original knowledge data, and the data source; The page table index relationship is used to represent the mapping relationship between the identifier of each physical storage unit of the logical data unit and the second semantic representation corresponding to the physical storage unit within each logical data unit; the second semantic representation is the page semantic vector corresponding to the physical storage unit.

2. The knowledge base construction method according to claim 1, characterized in that, The method further includes: Obtain the domain keywords from the user's search request; Based on the global index relationship and the domain keywords, candidate logical data units are determined; Based on the semantic similarity between the segment semantic features corresponding to the candidate logical data unit and the retrieval features of the retrieval request, the target logical data unit is selected from the candidate logical data units; Based on the semantic similarity between the retrieval features and the page semantic vectors corresponding to the candidate physical storage units in the target logical data unit, each first physical storage unit is selected from the candidate physical storage units; Each of the first physical storage units is used as the search result of the search request.

3. The knowledge base construction method according to claim 2, characterized in that, The step of using each of the first physical storage units as the retrieval result of the retrieval request includes: Each of the first physical storage units is merged in a merging order to obtain a candidate merged storage unit; the merging order is determined based on the order of the first physical storage units in the logical data unit. Based on the semantic breakpoints in the candidate merged storage units, the candidate merged storage units are adjusted to obtain the target merged storage unit; the semantic breakpoints include spaces, isolated punctuation marks, and syntactic incoherence points in the candidate merged storage units. The target merged storage unit is used as the retrieval result of the retrieval request.

4. The knowledge base construction method according to any one of claims 1 to 3, characterized in that, The preset segmentation strategy includes a first segmentation strategy, a second segmentation strategy, and a third segmentation strategy; Determining the semantic boundaries in the knowledge data according to a preset segmentation strategy includes: According to the first segmentation strategy, the semantic boundary in the knowledge data is determined as the position of the semantic abrupt change point where the semantic similarity between adjacent sentences in the knowledge data is lower than a preset threshold; According to the second segmentation strategy, the semantic boundaries in the knowledge data are determined as paragraph end labels; the paragraph end labels are obtained by inputting the knowledge data into the sequence labeling model; According to the third segmentation strategy, the semantic boundary in the knowledge data is determined as the position corresponding to the structural feature in the knowledge data; the position corresponding to the structural feature is obtained based on the explicit structural rule matching of the knowledge data.

5. The knowledge base construction method according to any one of claims 1 to 3, characterized in that, The method further includes: Based on the access frequency of each physical storage unit, the physical storage unit is divided into at least three data levels; The physical storage units of the at least three data levels are stored in storage media with different access speeds.

6. The knowledge base construction method according to claim 5, characterized in that, The method further includes: When the high-speed storage medium space in the storage medium is insufficient, the semantic activity is determined based on the recent access time, historical retrieval hit count, and the semantic similarity between the page semantic vector of the physical storage unit in the high-speed storage medium and the high-frequency query. The physical storage unit corresponding to the lowest semantic activity is moved from the high-speed storage medium to the low-speed storage medium to free up available space in the high-speed storage medium.

7. The knowledge base construction method according to claim 1, characterized in that, The method further includes: When the knowledge data changes, perform at least one of the following operations to incrementally update the hierarchical index structure: In response to the deletion operation of the second logical data unit in the knowledge data, the hierarchical index structure is updated according to a first preset operation order; the first preset operation order is to sequentially delete the page table index relationship corresponding to the second logical data unit, remove the segment table index relationship corresponding to the second logical data unit, and update the global index relationship. In response to the addition of the second logical data unit in the knowledge data, the hierarchical index structure is updated according to a second preset operation order; the second preset operation order is to sequentially perform the addition of the segment table index relationship corresponding to the second logical data unit, update the global index relationship, and create the page table index relationship corresponding to the second logical data unit. In response to the modification operation of the second logical data unit in the knowledge data, the hierarchical index structure is updated according to a third preset operation sequence. The third preset operation sequence is to sequentially perform the re-determination of the second semantic representation corresponding to the second physical storage unit in the second logical data unit to update the page table index relationship, and based on the second semantic representation of all physical storage units included in the second logical data unit, to re-determination of the first semantic representation corresponding to the second logical data unit to update the segment table index relationship.

8. A knowledge base construction device, characterized in that, include: The acquisition unit is used to acquire knowledge data and determine the semantic boundaries in the knowledge data according to a preset segmentation strategy; the semantic boundaries refer to the locations in the content of the knowledge data where the topic changes or logic is interrupted. A partitioning unit is used to divide the knowledge data into multiple logical data units based on the semantic boundaries. A partitioning unit is used to divide each of the logical data units into at least one physical storage unit, wherein the physical storage unit is the basic unit for data storage and scheduling. A construction unit is used to construct a knowledge base including a hierarchical index structure, a first semantic representation, and a second semantic representation; the hierarchical index structure includes a hierarchical mapping relationship between the logical data unit and the physical storage unit, the first semantic representation is used to represent the content semantics of the logical data unit, and the second semantic representation is used to represent the content semantics of the physical storage unit; The hierarchical mapping relationship includes global index relationship, segment table index relationship, and page table index relationship; The global index relationship is used to characterize the mapping relationship between the knowledge data and the identifiers of the logical data units; The segment table index relationship is used to represent the mapping relationship between the identifier of the logical data unit and the first semantic representation corresponding to the identifier of the logical data unit; the first semantic representation is the segment semantic vector corresponding to the logical data unit; the segment table index relationship includes other metadata of the logical data unit, which includes the byte length of the segment, the starting position in the original knowledge data, and the data source; The page table index relationship is used to represent the mapping relationship between the identifier of each physical storage unit of the logical data unit and the second semantic representation corresponding to the physical storage unit within each logical data unit; the second semantic representation is the page semantic vector corresponding to the physical storage unit.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the knowledge base construction method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the knowledge base construction method as described in any one of claims 1 to 7.

Images