Knowledge enhancement retrieval and generation method and system based on semantic association graph

Abstract

The invention provides a knowledge enhancement retrieval and generation method and system based on a semantic association graph, and relates to the technical field of information retrieval. In order to solve the problem that in the prior art, document slice semantics are incomplete and mutually isolated, the method comprises the steps that an initial text unit sequence of a document is obtained, iterative semantic merging is conducted based on a first threshold value, and a set of knowledge slices is generated; calculating association degree based on the group of slices, and constructing a semantic association graph; when user query is responded, after initial slices are obtained through preliminary retrieval, extension retrieval is carried out on the basis of the semantic association graph and a second threshold value, and extension slices are obtained; the initial and extended slices are combined to generate final contextual information. According to the method, the semantic knowledge network is constructed, so that the knowledge island problem is solved, a more comprehensive and deep context can be provided for a large language model, and the retrieval and generation effects are improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, particularly to the fields of natural language processing, information retrieval, and artificial intelligence. Specifically, it relates to a method and system for knowledge enhancement retrieval and generation based on semantic association graphs. Background Technology

[0002] Retrieval-enhanced generation is a key technology in current large language model applications. Its basic process is as follows: the original document is segmented into multiple knowledge fragments and stored in a vector database; when a user query is received, the most relevant fragments are retrieved as context and provided to the large language model along with the original query to generate a more accurate and reliable answer.

[0003] However, in the document segmentation stage, existing technologies mainly employ fixed-size segmentation or recursive segmentation methods based on preset delimiters. These methods are essentially based on the physical boundaries of the text and cannot truly understand the semantic structure of the content. They often result in a complete semantic unit being forcibly split into different fragments, destroying the inherent integrity of knowledge and thus affecting the large language model's understanding of the context, leading to inaccurate generated results.

[0004] To address this issue, some technical solutions propose adaptive text segmentation methods. These methods calculate the semantic relevance of adjacent text units and compare it to a preset threshold to dynamically determine whether to merge them, thereby generating knowledge fragments with more coherent internal semantics. However, even when high-quality knowledge fragments are generated in this way, these fragments remain isolated and linear at a macroscopic level. Deep logical relationships spanning multiple fragments in the document, such as causal, comparative, and general-to-specific relationships, are completely lost. This information silo effect means that when responding to complex queries requiring the integration of multiple aspects of information, the system cannot provide comprehensive and logically deep context, resulting in fragmented search results.

[0005] In addition, other technical solutions employ knowledge graphs for association retrieval in knowledge question answering. However, these solutions typically rely on massive, general-purpose knowledge graphs pre-built from huge amounts of structured or semi-structured data, or require complex entity-relationship extraction processes, making them difficult to apply in a lightweight manner to scenarios involving the construction and retrieval of real-time, dynamic knowledge associations for a single or a small number of specific documents. Summary of the Invention

[0006] The purpose of this application is to provide a knowledge enhancement retrieval method and system based on adaptive slicing and semantic association graphs. It aims to solve the technical problem in the prior art that it is difficult to effectively construct and utilize the macro-level association between slices while ensuring the semantic integrity within the knowledge slice. This results in incomplete and shallow contextual information provided to the large language model in the retrieval enhancement generation application, which ultimately affects the accuracy, comprehensiveness and logic of the generated answer.

[0007] To achieve the above objectives, this application provides a knowledge enhancement retrieval and generation method based on semantic association graphs, characterized by the following steps: Step 1, adaptive semantic segmentation generation: obtaining an initial text unit sequence of the original document; and performing iterative semantic merging on the initial text unit sequence based on a preset first semantic similarity threshold to generate a set of knowledge slices; Step 2, semantic association graph construction: calculating the semantic association degree between any two knowledge slices in the set of knowledge slices based on the set of knowledge slices, and constructing a semantic association graph to represent the association relationship between the knowledge slices based on the semantic association degree; Step 3, enhanced retrieval and context generation: responding to a user query, performing a preliminary retrieval in the set of knowledge slices to obtain an initial slice set semantically related to the user query; based on the semantic association graph and a preset second semantic similarity threshold, obtaining extended slices from the set of knowledge slices whose association degree with the knowledge slices in the initial slice set is higher than the second semantic similarity threshold to form an extended slice set; combining the initial slice set and the extended slice set to generate context information for responding to the user query.

[0008] Optionally, in step one, the iterative semantic merging specifically includes: sequentially processing each text unit in the initial text unit sequence and calculating the semantic similarity between the current text unit to be processed and the knowledge slice currently being constructed; when the semantic similarity is greater than the first semantic similarity threshold, merging the content of the current text unit to be processed to the end of the current knowledge slice; otherwise, using the current text unit to be processed to start a new knowledge slice.

[0009] Optionally, the calculation of semantic similarity and the calculation of semantic association both include: converting the text into a high-dimensional vector using a deep learning-based sentence conversion model; and quantifying the semantic similarity by calculating the cosine similarity between the high-dimensional vectors corresponding to the two texts.

[0010] Optionally, the calculation of semantic similarity and the calculation of semantic association both include: constructing a term frequency-inverse document frequency vector for the text; and quantifying it by calculating the cosine similarity between the term frequency-inverse document frequency vectors corresponding to two texts.

[0011] Optionally, obtaining the initial text unit sequence of the original document includes: physically cutting the original document using preset punctuation marks to generate the initial text unit sequence.

[0012] Optionally, obtaining the initial text unit sequence of the original document includes: identifying structured elements in the original document through a document parser, and taking each structured element as an initial text unit to form the initial text unit sequence.

[0013] Optionally, in step two, constructing the semantic association graph includes: for each knowledge slice, calculating only its semantic association degree with a preset number of other knowledge slices to construct the semantic association graph.

[0014] Optionally, in step three, combining the initial slice set and the extended slice set includes arranging the knowledge slices in the initial slice set and the extended slice set according to their order in the original document to form the context information.

[0015] To achieve the above objectives, this application also provides a knowledge enhancement retrieval and generation system based on a semantic association graph, characterized by comprising: a segmentation generation module, used to obtain an initial text unit sequence of the original document, and perform iterative semantic merging on the initial text unit sequence based on a preset first semantic similarity threshold to generate a set of knowledge slices; an association construction module, used to calculate the semantic association degree between any two knowledge slices in the set of knowledge slices based on the set of knowledge slices, and construct a semantic association graph to represent the association relationship between the knowledge slices based on the semantic association degree; and an enhanced retrieval module, used to respond to user queries, perform preliminary retrieval in the set of knowledge slices to obtain an initial slice set, obtain extended slices with an association degree higher than the second semantic similarity threshold with the knowledge slices in the initial slice set based on the semantic association graph and a preset second semantic similarity threshold to form an extended slice set, and combine the initial slice set and the extended slice set to generate contextual information for responding to the user query.

[0016] Compared with the prior art, this application has the following beneficial effects: First, this application ensures that each generated knowledge slice is a semantically self-consistent unit through iterative semantic merging, thereby avoiding semantic confusion caused by unreasonable segmentation from the source, guaranteeing the integrity and coherence of knowledge, and providing a high-quality contextual foundation for downstream models.

[0017] Secondly, this application connects isolated knowledge fragments into a network by constructing a semantic association graph and performing association expansion during retrieval. This enables the discovery and integration of deep information that is directly and indirectly related to the query, realizing the association and discovery of knowledge, solving the problem of knowledge silos, and making the generated context more comprehensive and in-depth.

[0018] Finally, by generating knowledge units with higher information density and more reasonable length during the preprocessing stage, this application reduces the requirements for the performance of downstream large language models and context windows, thereby improving the overall system's operating efficiency and economy. At the same time, the higher quality context also directly improves the accuracy and logic of the final generated answer, thus enhancing the system's efficiency and effectiveness. Attached Figure Description

[0019] Figure 1 A schematic diagram of the hardware architecture of a knowledge-enhanced retrieval system provided in this application embodiment; Figure 2 A flowchart illustrating a knowledge-enhanced retrieval method provided in this application embodiment; Figure 3 Signaling interaction timing diagram of the enhanced retrieval process provided in the embodiments of this application; Figure 4 A schematic diagram comparing the traditional linear slices provided in the embodiments of this application with the semantic association graph of the present invention.

[0020] The main reference numerals in the accompanying drawings are explained as follows: 100-Processing system; 110-Processor; 120-Memory; 121-Slice generation module; 122-Association construction module; 123-Enhanced retrieval module; 130-Knowledge base; 140-Large language model interface; 150-User terminal; S201-Obtain initial text unit sequence; S202-Iterative semantic merging to generate knowledge slices; S203-Calculate the correlation degree between slices; S204-Construct semantic association graph; S205-Receive user query; S206-Preliminary retrieval to obtain initial slices; S207-Extended retrieval based on semantic association graph; S208-Combination to generate final context information; 501-Traditional linear slice; 502-Knowledge slice node of this invention; 503-Semantic association edge. Detailed Implementation

[0021] To better understand the technical solutions of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be noted that the specific embodiments described herein are only for explaining this application and are not intended to limit this application.

[0022] Example 1 This application provides a knowledge-enhanced retrieval method and system based on adaptive fragmentation and semantic association graphs. In this embodiment, the aim is to transform an unstructured raw document into a structured knowledge network through a series of automated processing steps, and then utilize this network to provide users with high-quality, highly relevant contextual information for queries, thereby improving the accuracy and depth of subsequent answers generated by a large language model.

[0023] Please see Figure 1 This is a schematic diagram of the hardware and software architecture of a processing system 100 provided in an embodiment of this application. As one possible implementation, the processing system 100 can be one or more servers, personal computers, workstations, or embedded devices with considerable computing power. The core hardware of the processing system 100 includes at least one processor 110 and memory 120. The processor 110 serves as the computing core of the system, responsible for executing instructions and processing data; the memory 120 is used to store program instructions and runtime data, and can be random access memory, read-only memory, or other forms of storage media.

[0024] In one embodiment of this application, the memory 120 deploys core functional modules for implementing the technical solution of this application. These modules can be loaded and executed by the processor 110. Specifically, the modules include: a fragmentation generation module 121, used to perform document preprocessing and adaptive semantic fragmentation generation, transforming the original document into a set of knowledge fragments with highly cohesive internal semantics; an association construction module 122, which receives the set of knowledge fragments generated by the fragmentation generation module 121 and constructs a semantic association graph on this basis to represent the macroscopic semantic associations between fragments; and an enhanced retrieval module 123, serving as an online service interface for users, responsible for receiving query requests from user terminals 150, performing preliminary retrieval and extended retrieval based on the semantic association graph, and finally combining them to generate contextual information for processing by a large language model.

[0025] Furthermore, the processing system 100 also interacts with external entities. It communicates with the user terminal 150 via a network interface to receive queries and return results. The system interacts with a knowledge base 130, which can be a vector database, relational database, or file system, to persistently store knowledge slices and semantic relationship graphs generated by the slice generation module 121 and the association construction module 122. The enhanced retrieval module 123 also reads data from the knowledge base 130 in response to queries. Accordingly, the enhanced retrieval module 123 submits the generated context information and user queries to an external or internal large language model for processing via the large language model interface 140, and receives the final answer generated by it.

[0026] Now combine Figure 2The knowledge-enhanced retrieval method provided in this embodiment will be described below. Figure 2 This is a complete flowchart of the method. It can be understood that the method can be divided into three main stages: The first stage is the offline document preprocessing stage, namely adaptive semantic segmentation generation, corresponding to... Figure 2 Steps S201 and S202 in the process. The second stage is also an offline processing stage, namely the semantic association graph construction, corresponding to... Figure 2 Steps S203 and S204 in the process. The third stage is the online query response stage, namely enhanced retrieval and context generation, corresponding to... Figure 2 Steps S205 to S208 in the process.

[0027] The following will use the processing of a plain text document about the "History of Artificial Intelligence" as an example to illustrate the specific execution process of each step.

[0028] In the first stage, in step S201, the segmentation generation module 121 obtains an initial text unit sequence. In this embodiment, the original document is initially segmented using a set of preset punctuation marks (e.g., the Chinese period ".", question mark "?", and the common newline character ""), thereby obtaining a list of strings containing multiple sentences. Accordingly, the list is cleaned, for example, by removing all empty strings containing only whitespace characters, and selectively performing deduplication to eliminate identical consecutive sentences. After this step, the original document is transformed into an ordered initial text unit sequence composed of sentences, which can be stored in an array or list (hereinafter referred to as...). arr )middle.

[0029] Subsequently, in step S202, iterative semantic merging is performed to generate knowledge slices. This step is crucial for ensuring the semantic integrity within each knowledge slice. The slice generation module 121 sets a preset first semantic similarity threshold (hereinafter referred to as...). alpha In this embodiment, the threshold is set to 0.8. In a preferred embodiment, alpha The value can be set between 0.7 and 0.95 to achieve better merging results. Too low a value... alpha An excessively high alpha value may lead to the incorrect merging of semantically unrelated sentences, while an excessively high alpha value may result in overly fragmented segments that fail to form meaningful semantic units.

[0030] At the start of the merge process, an empty list is initialized (hereinafter referred to as...). newarr This is used to store the final generated knowledge slices. The system starts from... arr Starting with the first sentence, each sentence in the initial text unit sequence is processed iteratively. When processing the first sentence, because... newarr If the sentence is empty, it is directly added as a new knowledge slice. newarr The system processes each sentence to be processed, starting with the second sentence, and then performs a step to calculate semantic similarity. Specifically, this embodiment employs deep learning-based techniques, loading a pre-trained sentence transformation model (such as the widely used all-MiniLM-L6-v2 model), which maps text input to a high-dimensional vector space. The system inputs the complete content of the sentence to be processed and the current knowledge slice into the model, obtaining two high-dimensional vectors, and quantifies their semantic similarity by calculating the cosine similarity between these two vectors. The formula for calculating cosine similarity is: in, and These are the vectors corresponding to the two texts.

[0031] After obtaining the semantic similarity, in the next step, it is compared with a threshold. alpha (0.8) is used for comparison. If the calculated similarity is greater than 0.8, then... alpha If the similarity is greater than or equal to the current knowledge slice, it means that the sentence to be processed and the current knowledge slice are highly semantically continuous. The system then performs a merging operation, appending the text content of the sentence to be processed to the end of the current knowledge slice. Conversely, if the calculated similarity is less than or equal to... alpha This indicates that the sentence to be processed has opened up a new semantic topic, so the system performs a new slice creation operation, adding the sentence to be processed as a brand new knowledge slice. newarr At the end of the list, this new slice then becomes the new "current knowledge slice".

[0032] This iterative process continues until the initial sequence of text units is reached. arr All sentences in the file have been processed. Finally, newarr The list contains a set of knowledge slices. Unlike traditional fixed-length segmentation, each knowledge slice here is a semantically consistent and coherent unit. These knowledge slices and their index positions in the original document are stored in knowledge base 130.

[0033] In the second stage, in step S203, the association construction module 122 calculates the association degree between slices. This module reads the set of knowledge slices generated in the previous stage from the knowledge base 130. newarr To construct a knowledge network that reflects the macroscopic relationships between all slices, it is necessary to calculate the semantic relevance between any two different knowledge slices in the set. In this embodiment, the method for calculating the semantic relevance can reuse the similarity calculation method in step S202, that is, it also uses the all-MiniLM-L6-v2 model for vectorization and calculates the cosine similarity.

[0034] Next, in step S204, a semantic relationship graph is constructed. Specifically, an N×N two-dimensional matrix (where N is the total number of knowledge slices, which can be named RelationshipGraph) can be created to store these relationships. Matrix elements (i, j) store the semantic relationship between the i-th knowledge slice and the j-th knowledge slice. All slice pairs (i, j) (where i is not equal to j) are traversed through a double loop, their relationship is calculated, and the matrix is ​​filled. This matrix, or an equivalent graph structure (such as an adjacency list), constitutes the constructed semantic relationship graph, which transforms the original linear, isolated slice sequence into a network structure rich in relational information. Figure 4 As shown, this diagram visually compares the semantic association graph generated by the traditional linear slice 501 with that generated by the embodiment of this application. In the diagram on the right, each knowledge slice is a knowledge slice node 502, and the nodes are connected by semantic association edges 503. The weight of the edge is the semantic association degree between them. This semantic association graph is also stored in the knowledge base 130 for subsequent retrieval.

[0035] In the third stage of online query response, when a user submits a query (e.g., "early applications of neural networks") through user terminal 150, the enhanced retrieval module 123 is activated. First, in step S205, the user query is received. Then, in step S206, a preliminary retrieval is performed to obtain initial slices. The enhanced retrieval module 123 uses the same sentence transformation model as in the previous steps to convert the user query text into a query vector, and calculates the cosine similarity between it and all knowledge slice vectors stored in the knowledge base 130, thereby selecting the batch of slices with the highest similarity (e.g., returning the 3 most relevant slices, or returning all slices with a similarity greater than 0.8), forming the initial slice set.

[0036] Step S207 is an extended retrieval based on the semantic association graph. For example... Figure 3 As shown in the sequence diagram, after the enhanced retrieval module 123 obtains the initial slices (signalings 2 and 3) from the knowledge base 130, it will use the semantic relationship graph (RelationshipGraph) constructed in the second stage for knowledge expansion. The system sets a preset second semantic similarity threshold (hereinafter referred to as...). delta In this embodiment, the threshold is set to 0.8. In a preferred embodiment, delta The value can be set from 0.7 to 0.9 to achieve a balance between recall and precision. The enhanced retrieval module 123 iterates through each slice in the initial slice set and queries the association degree of all other slices related to that slice in the RelationshipGraph. If the association degree of a slice pair is greater than a threshold... delta Then, the associated slices are considered as strongly related extended knowledge, and their indices are added to the extended slice set. This process corresponds to... Figure 3 Signaling 4 and 5 in the code.

[0037] In step S208, the final context information is generated by combining the fragments. At this point, the system merges and deduplicates all fragment indices from the initial fragment set and the expanded fragment set, and then rearranges them according to their order in the original document. The system then extracts the complete fragment text from the knowledge base 130 according to this order and concatenates them into a long text. This long text is the final, comprehensive, and logically coherent context information.

[0038] Accordingly, the enhanced retrieval module 123 sends the combined context information, along with the user's original query, to the large language model (Language Model Interface 140) via the large language model interface. Figure 3 (Signaling 6 in the code). After receiving such rich context, the large language model is able to generate a more comprehensive, accurate, and logically deep answer. This answer is returned to the enhanced retrieval module 123 via signaling 7, and is finally presented to the user terminal 150.

[0039] Example 2 As an optional implementation, this embodiment provides a variant to illustrate the good technical compatibility of the proposed method framework. Specifically, this embodiment demonstrates how to replace the deep learning-based similarity calculation module in Embodiment 1 with a traditional text representation algorithm that has lower computational overhead in scenarios with limited computing resources (e.g., running on edge computing devices) or extremely high processing speed requirements.

[0040] The overall system architecture of this embodiment (such as...) Figure 1 (as shown) and method flow (such as) Figure 2 As shown, this embodiment is consistent with Embodiment 1, with the core difference being the specific techniques used by the segmentation generation module 121 and the association construction module 122 when performing semantic similarity or association calculations. In this embodiment, the system uses the classic term frequency-inverse document frequency algorithm to vectorize the text.

[0041] Specifically, in performing steps S202 (iterative semantic merging) and S203 (calculating inter-slice correlation), the similarity calculation process is as follows: First, in the initial stage of document processing, the entire document corpus (i.e., all initial text units generated in step S201) is preprocessed to construct a global vocabulary. This process includes word segmentation, stop word removal, and lemmatization. Second, for each word in the vocabulary, its inverse document frequency is calculated. The formula for calculating the inverse document frequency is typically:

[0042] in, It is the total number of documents in the corpus. It contains words The system calculates the number of documents. Then, when it needs to calculate the similarity between two texts, it constructs their term frequency-inverse document frequency vectors for each text. For each word in a text... The corresponding weights are calculated as follows:

[0043] in, It is a word In the text The frequency of occurrence of each word in the vocabulary is used to determine the semantic relevance score between the two word frequency-inverse document frequency vectors. Thus, each text is represented as a sparse vector with a dimension equal to the size of the vocabulary. Finally, similar to Example 1, the system calculates the cosine similarity between these two word frequency-inverse document frequency vectors to obtain their semantic relevance score.

[0044] In the workflow, whether it is the segmentation generation module 121 performing iterative semantic merging or the association construction module 122 constructing a semantic association graph, whenever similarity comparison is involved, the above-mentioned vectorization and cosine similarity calculation are performed.

[0045] Compared to Embodiment 1, the advantage of this embodiment lies in its significantly reduced hardware resource requirements. It does not require loading large deep learning models, has minimal memory footprint, and its computation primarily involves sparse vector operations, resulting in high computation speed. This allows the technical solution of this application to be deployed on resource-constrained devices or applied to servers that need to process massive amounts of documents and have extremely high throughput requirements.

[0046] Of course, it should be noted that this approach also has certain limitations. Term frequency-inverse document frequency (TNF) is a statistical method based on the bag-of-words model, which cannot understand word order, grammatical structure, and deep semantic information. Therefore, its accuracy in judging semantic similarity is generally lower than that of methods based on deep learning models. Nevertheless, in many scenarios, especially for documents with high overlap in terminology within specialized fields, TNF can still provide sufficiently effective correlation judgments, thus successfully achieving the core inventive concept of this application.

[0047] Example 3 This embodiment proposes another variant implementation scheme, mainly addressing the efficiency issues that may be encountered when processing extremely long documents (such as entire books or lengthy technical manuals). In Embodiment 1, constructing the semantic association graph (step S204) requires calculating the association degree of N×N slice pairs. When N is very large, O(N) 2 The computational complexity of this would become extremely time-consuming. This embodiment achieves an effective balance between processing efficiency and association integrity by introducing a local semantic association graph construction strategy.

[0048] The method flow of this embodiment is basically the same as that of Embodiment 1 in the first stage (steps S201-S202) and the third stage (steps S205-S208). Its core innovation lies in the optimization of the semantic association graph construction steps (S203 and S204) in the second stage.

[0049] During steps S203 and S204, the association building module 122 no longer uses a double loop to traverse all slice pairs. Instead, it performs a process for each knowledge slice. newarr [i] only calculates the semantic correlation between it and other knowledge slices within a preset number (i.e., window size k). For example, if the window size k = 10, then for the slice with index i, the system only calculates the correlation between it and the slice with index j, where the value of j ranges from i+1 to min(i+k, N-1), and N is the total number of slices.

[0050] The specific working process is as follows: After the association construction module 122 obtains the knowledge slice list newarr, it starts a single-layer loop to traverse each slice from i=0 to N-2. Inside the loop, another nested loop traverses the subsequent k slices from j=i+1 to min(i+k, N-1). For each pair (i, j), the system uses the same deep learning model and cosine similarity method as in Example 1 to calculate its semantic association degree and stores the results.

[0051] The semantic relationship graph constructed in this way is a sparse graph (or sparse matrix), where each node has direct connections only to a few nodes whose positions are close to each other in the original text. The computational complexity of this method is O(N). 2 The efficiency is significantly reduced to approximately O(N*k), and when N is much greater than k, the efficiency improvement is extremely significant.

[0052] The semantic relationship graph constructed in this way is a sparse graph (or sparse matrix), where each node has direct connections only to a few nodes whose positions are close to each other in the original text. The computational complexity of this method is O(N). 2 The efficiency is significantly reduced to approximately O(N*k), and when N is much greater than k, the efficiency improvement is extremely significant.

[0053] In the third stage, the enhanced retrieval step S207, the enhanced retrieval module 123 still searches for associated extended slices for the initial slice in the sparse semantic association graph. This means that an initial slice can only be directly extended to the vicinity of its original position. k Strongly correlated slices (within a slice range).

[0054] It is foreseeable that this embodiment enables the technical solution of this application to be efficiently applied to ultra-large-scale single-document processing scenarios, and has significant practical value. It should be noted that this strategy may lose some connections between segments that are physically far apart in the original text but semantically strongly related. However, in most cases, the most important contextual information usually appears in adjacent or nearby paragraphs. Therefore, this local association strategy can still capture most of the valuable relationships, significantly improving efficiency while the impact on the final retrieval results is generally within an acceptable range.

[0055] Example 4 This embodiment provides another variant implementation scheme, the core idea of ​​which lies in optimizing the starting point of the entire method flow, namely step S201 "obtaining the initial text unit sequence". While the physical segmentation based on punctuation marks used in Embodiment 1 is general, it may disrupt the inherent logical structure of the document. This embodiment utilizes the structured information of the document to generate higher-quality initial text units, thereby providing a better foundation for subsequent iterative semantic merging.

[0056] The main difference between this embodiment and Embodiment 1 lies in the way the segmentation generation module 121 executes step S201. Here, the system no longer simply uses line breaks for segmentation, but instead calls a document parser specific to a particular document format. For example, if the original document to be processed is a Microsoft Word file, the system can use a specialized library, such as the open-source python-docx library, to parse the file and identify various structured elements in the document, such as headings, paragraphs, list items, and table cells. Similarly, if the document is in Markdown format, the system can use a corresponding parser to identify headings, paragraphs, code blocks, quotation blocks, and list items. If the document is in HTML or XML format, the corresponding Document Object Model parser can be used to extract the structured content defined by tags.

[0057] After parsing the document structure, the system treats each identified meaningful structured element as an "initial text unit." For example, a second-level heading and its first paragraph can be combined as an initial unit, or more finely, each paragraph and each list item can be considered an independent initial unit. These initial units form a sequence of initial text units according to their order of appearance in the document. arr .

[0058] After completing step S201 of this optimization, the subsequent steps, including S202 (iterative semantic merging), S203-S204 (semantic association graph construction) and S205-S208 (enhanced retrieval), are the same as in Example 1, except that the object of operation has changed from the original "sentence sequence" to the current "structured unit sequence".

[0059] The working process of this embodiment can be summarized as follows: The segmentation generation module 121 first analyzes and utilizes the logical structure given when the document is written to deconstruct the document into a sequence composed of logical blocks such as titles, paragraphs, and lists, and then hands this high-quality sequence over to the subsequent iterative semantic merging algorithm for processing.

[0060] It is foreseeable that this approach will yield significant technical benefits. Since the initial text units themselves possess strong semantic cohesion, the subsequent iterative semantic merging process (step S202) will be more accurate and efficient, avoiding the erroneous segmentation of strongly related content in the initial stage. The resulting knowledge slices will be more complete and clear at the macro-structural level, directly improving the quality of the knowledge units in the knowledge base 130. This, in turn, will enable the enhanced retrieval stage to match and expand user queries with more logically coherent and comprehensive contexts, further enhancing the overall system performance.

[0061] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A knowledge-enhanced retrieval and generation method based on semantic association graphs, characterized in that, Includes the following steps: Step 1, Adaptive semantic segmentation generation: Obtain the initial text unit sequence of the original document; and based on a preset first semantic similarity threshold, perform iterative semantic merging on the initial text unit sequence to generate a set of knowledge slices; Step 2, Semantic Relationship Graph Construction: Based on the set of knowledge slices, calculate the semantic relationship between any two knowledge slices in the knowledge slice set, and construct a semantic relationship graph to represent the relationship between the knowledge slices based on the semantic relationship. Step 3, Enhanced Retrieval and Context Generation: In response to the user query, a preliminary retrieval is performed in the set of knowledge slices to obtain an initial set of slices semantically related to the user query. Based on the semantic association graph and the preset second semantic similarity threshold, extended slices with a higher correlation degree to the knowledge slices in the initial slice set are obtained from the set of knowledge slices to form an extended slice set; The initial slice set and the extended slice set are combined to generate context information for responding to the user query.

2. The method according to claim 1, characterized in that, In step one, the iterative semantic merging specifically includes: Each text unit in the initial text unit sequence is processed sequentially, and the semantic similarity between the current text unit to be processed and the knowledge slice currently being constructed is calculated. When the semantic similarity is greater than the first semantic similarity threshold, the content of the current text unit to be processed is merged into the end of the current knowledge slice; Otherwise, start a new knowledge slice using the currently pending text unit.

3. The method according to claim 1, characterized in that, The calculation of semantic similarity and the calculation of semantic relevance both include: A deep learning-based sentence transformation model is used to convert text into high-dimensional vectors; Quantification is achieved by calculating the cosine similarity between the high-dimensional vectors corresponding to the two texts.

4. The method according to claim 1, characterized in that, The calculation of semantic similarity and the calculation of semantic relevance both include: Construct a term frequency-inverse document frequency (TF-IDF) vector from the text; Quantification is achieved by calculating the cosine similarity between the TF-IDF vectors corresponding to the two texts.

5. The method according to claim 1, characterized in that, The process of obtaining the initial text unit sequence of the original document includes: The original document is physically segmented using preset punctuation marks to generate the initial text unit sequence.

6. The method according to claim 1, characterized in that, The process of obtaining the initial text unit sequence of the original document includes: The document parser identifies structured elements in the original document and uses each structured element as an initial text unit to form a sequence of initial text units.

7. The method according to claim 1, characterized in that, In step two, constructing the semantic association graph includes: For each knowledge slice, the semantic correlation degree between it and a predetermined number of other knowledge slices is calculated to construct the semantic correlation graph.

8. The method according to claim 1, characterized in that, In step three, combining the initial slice set and the expanded slice set includes: The knowledge slices in the initial slice set and the extended slice set are arranged in the order they appear in the original document to form the context information.

9. A knowledge-enhanced retrieval and generation system based on semantic association graphs, characterized in that, include: The segmentation generation module is used to obtain the initial text unit sequence of the original document and perform iterative semantic merging on the initial text unit sequence based on a preset first semantic similarity threshold to generate a set of knowledge slices. The association construction module is used to calculate the semantic association degree between any two knowledge slices in the knowledge slice set based on the set of knowledge slices, and construct a semantic association graph to represent the association relationship between the knowledge slices based on the semantic association degree. An enhanced retrieval module is used to respond to user queries by performing a preliminary retrieval in the set of knowledge slices to obtain an initial slice set, obtaining extended slices with a higher correlation to the knowledge slices in the initial slice set based on the semantic association graph and a preset second semantic similarity threshold to form an extended slice set, and combining the initial slice set and the extended slice set to generate context information for responding to the user query.

Images