Hybrid search-based enterprise knowledge base augmentation generation method and system

Abstract

The application provides a hybrid retrieval-based enterprise knowledge base enhancement generation method and system. A semantic segmentation process is performed on a knowledge document set in an enterprise knowledge base to obtain a plurality of knowledge segment units. Dense vector retrieval coding processing and sparse term retrieval index construction processing are synchronously performed on each knowledge segment unit to obtain a dense vector retrieval representation and a sparse term retrieval index item sharing the same knowledge segment unit identifier. A target query request is obtained and a corresponding dense vector query representation and sparse term query representation are synchronously generated. Similarity neighbor search and term hit matching are performed to obtain a fusion candidate knowledge segment set. Document source grouping and retrieval score redistribution processing is performed on the fusion candidate knowledge segment set. After a target knowledge segment set is screened out, an enhanced generation input sequence is constructed. An enhanced generation network is inputted to obtain an enterprise knowledge base enhancement generation result. The application greatly improves the knowledge accuracy of the enterprise knowledge base enhancement generation result.

Description

Technical Field

[0001] This application relates to the fields of enterprise knowledge management and large language model enhancement generation technology, specifically, to an enterprise knowledge base enhancement generation method and system based on hybrid retrieval. Background Technology

[0002] Intelligent question answering and content generation for enterprise knowledge bases can effectively improve internal information retrieval efficiency and lower the knowledge acquisition threshold. The accuracy and source matching of the generated results directly determine the actual value of the enterprise knowledge base. Currently, most enterprise knowledge base enhancement generation in the industry adopts a single retrieval mode to obtain candidate knowledge fragments. It can only rely on one of the two paths, dense vector semantic matching or sparse term keyword matching, to recall relevant content. The single retrieval mode is prone to semantic matching deviations or keyword omissions. A large number of knowledge fragments strongly related to the query request cannot be completely recalled. The subsequently generated content is prone to problems such as misaligned knowledge sources and omission of key information, which makes it difficult to meet the actual needs of enterprises for high-precision knowledge question answering and compliant content generation. Summary of the Invention

[0003] In view of this, the purpose of this application is to provide a method and system for enhancing the generation of enterprise knowledge bases based on hybrid retrieval.

[0004] According to a first aspect of this application, a method for enhancing the generation of enterprise knowledge bases based on hybrid retrieval is provided, the method comprising: Obtain a set of knowledge documents from the enterprise knowledge base, and perform semantic segmentation on the set of knowledge documents to obtain multiple knowledge fragment units carrying source attribution mapping relationships; For each knowledge segment unit, dense vector retrieval encoding processing and sparse term retrieval index construction processing are performed synchronously to obtain the dense vector retrieval representation and sparse term retrieval index item corresponding to each knowledge segment unit. The dense vector retrieval representation and the sparse term retrieval index item share the same knowledge segment unit identifier. Obtain the target query request, and simultaneously perform dense vector query representation generation processing and sparse term query representation generation processing on the target query request to obtain the dense vector query representation and sparse term query representation corresponding to the target query request; The first candidate knowledge fragment set is obtained by performing similarity nearest neighbor search in the dense vector retrieval space using the dense vector query representation, and the second candidate knowledge fragment set is obtained by performing term matching in the sparse inverted index structure using the sparse term query representation. The first candidate knowledge fragment set and the second candidate knowledge fragment set are then fused and deduplicated to obtain a fused candidate knowledge fragment set carrying dense vector retrieval score and sparse term retrieval score. The document source grouping and retrieval score redistribution processes are performed on the fusion candidate knowledge fragment set to obtain a redistributed retrieval score set. The target knowledge fragment set is selected based on the redistributed retrieval score set. An enhanced generation input sequence is generated based on the target knowledge fragment set and the target query request. The enhanced generation input sequence is input into a pre-constructed enhanced generation network to obtain the enhanced generation result of the enterprise knowledge base.

[0005] According to a second aspect of this application, a hybrid retrieval-based enterprise knowledge base enhancement generation system is provided. The hybrid retrieval-based enterprise knowledge base enhancement generation system includes a machine-readable storage medium and a processor. The machine-readable storage medium stores machine-executable instructions. When the processor executes the machine-executable instructions, the hybrid retrieval-based enterprise knowledge base enhancement generation system implements the aforementioned hybrid retrieval-based enterprise knowledge base enhancement generation method.

[0006] Based on any of the above aspects, the technical effect of this application is as follows: By acquiring a set of knowledge documents from an enterprise knowledge base, semantic segmentation is performed to obtain multiple knowledge fragment units carrying source attribution mapping relationships. For each knowledge fragment unit, two types of retrieval encoding processes are simultaneously completed to obtain a dense vector retrieval representation sharing the same identifier and a sparse term retrieval index. After obtaining the target query request, corresponding two types of query representations are generated simultaneously. Nearest neighbor search and term matching are performed in the corresponding retrieval space to obtain two sets of candidate knowledge fragments. Fusion and deduplication processing is performed on the two sets to obtain a fused candidate knowledge fragment set carrying two types of retrieval scores. Document source grouping and retrieval score redistribution processing are performed on this fused candidate knowledge fragment set to obtain a redistributed retrieval score set. Based on this redistributed retrieval score set, a target knowledge fragment set is selected. An enhanced generation input sequence is constructed based on the target query request and input into a pre-built enhanced generation network to obtain the final enhanced generation result of the enterprise knowledge base. This invention fully integrates the advantages of two retrieval modes, significantly improving the recall completeness of relevant knowledge fragments, reducing the probability of knowledge misalignment and information omission in generated content, and significantly improving the knowledge accuracy and source matching degree of the enhanced generation result of the enterprise knowledge base. Attached Figure Description

[0007] Figure 1 A flowchart illustrating the enterprise knowledge base enhancement generation method based on hybrid retrieval provided in an embodiment of this application is shown. Figure 2 This is a schematic diagram of the logical interaction of the enterprise knowledge base enhancement generation system based on hybrid retrieval provided in the embodiments of this application.

[0008] Figure 3 This is a schematic diagram of the enterprise intelligence interaction interface provided in the embodiments of this application.

[0009] Figure 4 This is a schematic diagram of the display interface of the generated results provided in the embodiments of this application. Detailed Implementation

[0010] Figure 1 This illustration shows a flowchart of the enterprise knowledge base enhancement generation method based on hybrid retrieval provided in this application embodiment. This embodiment uses a collection of knowledge documents in the enterprise knowledge base as a unified application scenario. The enterprise knowledge base contains 32 knowledge document units, from D001 to D032. Each knowledge document unit is an internal enterprise policy document, operating procedure, technical specification, or product description. After semantic segmentation, the knowledge document units are processed to obtain knowledge fragment units. Each knowledge fragment unit carries a source attribution mapping relationship. The source attribution mapping relationship includes a document source identifier. The document source identifier corresponds one-to-one with the document number of the knowledge document unit. The collection and indexing of the enterprise knowledge base have been authorized by the enterprise knowledge base management entity, and the collection scope is limited to the minimum necessary data fields required for enhancement generation. The knowledge document collection does not contain personal privacy information. Sensitive fields in the knowledge document collection have been anonymized before semantic segmentation.

[0011] The Dense Vector Encoding Network (DVC) employs a Transformer encoder architecture. It comprises six identical semantic transformation structure layers. Each semantic transformation structure layer contains one multi-head self-attention sublayer and one feedforward sublayer. Each sublayer is followed by a normalization operation and a residual connection. The input to the DVC is a sequence of terms representing knowledge fragment units. Each term unit in the sequence is mapped to a term embedding vector. The term embedding vector has a dimension of 768. The multi-head self-attention sublayer has 12 attention heads. The feedforward sublayer has a hidden dimension of 3072. The intermediate semantic convergence vector output from the final semantic transformation structure layer is mapped to a dense vector retrieval representation through a linear mapping layer. The dense vector retrieval representation has a dimension of 512. During the pre-training phase, the DVC uses all knowledge fragment units from the enterprise knowledge base for masked term prediction training. Training employs an adaptive moment estimation optimizer with an initial learning rate of 0.01%, a batch size of 32, and 50 training epochs.

[0012] The augmented generative network employs an encoder-decoder architecture. The encoder part of the augmented generative network shares parameters with the dense vector encoding network. The decoder part of the augmented generative network consists of six decoder layers. Each decoder layer contains one masked self-attention sublayer, one cross-attention sublayer, and one feedforward sublayer. Each sublayer is followed by a normalization operation and residual connections. During the pre-training phase, the augmented generative network is trained using all knowledge fragment units from the enterprise knowledge base for next-term prediction. During the fine-tuning phase, the augmented generative network is trained using manually labeled target query requests and target knowledge fragment pairs. The fine-tuning phase uses an adaptive moment estimation optimizer with an initial learning rate of 1 / 100,000, a batch size of 8, and 20 training epochs.

[0013] Step S110: Obtain the knowledge document set in the enterprise knowledge base, and perform semantic segmentation processing on the knowledge document set to obtain multiple knowledge fragment units carrying source attribution mapping relationships.

[0014] In this embodiment, the knowledge document set is read from the document storage area of ​​the enterprise knowledge base. Each knowledge document unit includes a document number field, a document title field, and a document body field. Semantic segmentation processing is performed on the document body field of each knowledge document unit. The semantic segmentation processing adopts a sliding window segmentation method based on semantic integrity. The sliding window length is 512 term units, and the sliding step size is 384 term units. The overlap area between adjacent sliding windows is 128 term units. The semantic segmentation processing detects the chapter title, paragraph start position, and period position at the boundary of the sliding window. When the sliding window boundary falls in the middle of a sentence, the boundary is moved to the nearest period position. The semantic segmentation processing yields multiple knowledge fragment units. Each knowledge fragment unit includes a knowledge fragment unit identifier, knowledge fragment content, and source attribution mapping relationship. The knowledge fragment unit identifier is a globally unique number. For example, after semantic segmentation processing, the document body of knowledge document D001 yields three knowledge fragment units: P001, P002, and P003. The source attribution mapping relationship carried by knowledge fragment unit P001 includes document source identifier DM001.

[0015] Step S120: Perform dense vector retrieval encoding and sparse term retrieval index construction simultaneously on each knowledge segment unit to obtain the dense vector retrieval representation and sparse term retrieval index item corresponding to each knowledge segment unit. The dense vector retrieval representation and the sparse term retrieval index item share the same knowledge segment unit identifier.

[0016] In this embodiment, two processing channels are executed simultaneously for each knowledge fragment unit. The first processing channel is dense vector retrieval encoding processing. This process inputs the knowledge fragment unit into a dense vector encoding network, which outputs a dense vector retrieval representation. The second processing channel is sparse term retrieval index construction processing. This process performs term segmentation, term morphology restoration, term frequency statistics, and inverted index construction on the knowledge fragment unit. Both processing channels share the same knowledge fragment unit identifier. This identifier serves as the key linking the outputs of the two processing channels.

[0017] Step S121: Input the knowledge fragment unit into the pre-constructed dense vector coding network, and perform layer-by-layer semantic compression processing on the knowledge fragment unit through the multi-layer semantic transformation structure in the dense vector coding network to obtain the dense vector retrieval representation corresponding to the knowledge fragment unit.

[0018] In this embodiment, the knowledge fragment unit is first segmented into a sequence of term units through term segmentation. Each term unit in the term unit sequence is mapped to one term embedding vector. The term embedding vector sequence is input into a dense vector encoding network. The first layer semantic transformation structure of the dense vector encoding network receives the term embedding vector sequence and outputs the first layer intermediate semantic convergence vector sequence. The second layer semantic transformation structure receives the first layer intermediate semantic convergence vector sequence and outputs the second layer intermediate semantic convergence vector sequence. And so on. The sixth layer semantic transformation structure outputs the sixth layer intermediate semantic convergence vector sequence. Average pooling is performed on the sixth layer intermediate semantic convergence vector sequence. The average pooling result is mapped to a dense vector retrieval representation through a linear mapping layer. The dense vector retrieval representation has a dimension of 512.

[0019] Step S122: In each layer of the semantic transformation structure of the dense vector encoding network, the semantic attention calculation structure within that layer is used to perform attention-weighted aggregation processing on the semantic unit sequence of the knowledge fragment unit, generating the intermediate semantic convergence vector corresponding to that layer of the semantic transformation structure.

[0020] In this embodiment, the semantic attention computation structure is a multi-head self-attention sublayer. The first layer of the semantic transformation structure receives a sequence of term embedding vectors. Three linear transformations are performed on each term embedding vector in the sequence to obtain a query vector, a key vector, and a value vector, respectively. The dimensions of the query vector, key vector, and value vector are all 64. Twelve attention heads independently perform attention weighted aggregation. The aggregated vector output by each attention head has a dimension of 64. The aggregated vectors from the twelve attention heads are concatenated along the feature dimensions to form a 768-dimensional vector. The concatenated result is mapped to a 768-dimensional intermediate semantic convergence vector through an output linear transformation layer. The intermediate semantic convergence vector is then normalized and connected via a residual connection before being input into the feedforward sublayer. The feedforward sublayer outputs the intermediate semantic convergence vector corresponding to the semantic transformation structure of that layer.

[0021] Step S123: The intermediate semantic convergence vector output by the semantic transformation structure of the last layer of the dense vector coding network is used as the dense vector retrieval representation corresponding to the knowledge fragment unit. The dense vector retrieval representation and the knowledge fragment unit identifier form the first mapping record.

[0022] In this embodiment, the intermediate semantic convergence vector output by the sixth-layer semantic transformation structure is processed by average pooling and a linear mapping layer to obtain a dense vector retrieval representation. The dense vector retrieval representation is a 512-dimensional vector. The knowledge fragment unit identifier and the dense vector retrieval representation form a first mapping record. The first mapping record is stored in the dense vector retrieval space. The dense vector retrieval space is a 512-dimensional vector space. The first mapping record includes a knowledge fragment unit identifier field and a dense vector retrieval representation field.

[0023] Step S124: Perform term segmentation processing on the knowledge fragment unit to obtain multiple original term units in the knowledge fragment unit, and perform term morphology restoration processing on each original term unit to obtain the restored term unit corresponding to each original term unit.

[0024] In this embodiment, the term segmentation process employs a term segmentation algorithm. This algorithm divides the knowledge fragment content of a knowledge segment unit into multiple original term units. Each original term unit is the smallest semantic unit within the knowledge fragment content. A term morphology restoration process is then performed on each original term unit. This process restores the variant words in the original term unit to their basic word forms in the dictionary. After all original term units undergo term morphology restoration, the corresponding restored term units are obtained.

[0025] Step S125: Perform term frequency statistics processing on each restored term unit to obtain the occurrence frequency parameter of each restored term unit in the knowledge fragment unit. Based on the occurrence frequency parameter, select the retained term units that constitute the sparse term retrieval index of the knowledge fragment unit. Construct an inverted index entry for each retained term unit. The inverted index entry records the knowledge fragment unit identifier and the list of occurrence positions of the retained term unit in the knowledge fragment unit.

[0026] In this embodiment, the term frequency statistics process counts the number of times each restored term unit appears within a knowledge fragment unit. The occurrence frequency parameter is the number of times the restored term unit appears within the knowledge fragment unit. The filtering rule is that restored term units with an occurrence frequency parameter greater than 1 are retained term units. Restored term units with an occurrence frequency parameter equal to 1 are not retained term units. One inverted index entry is constructed for each retained term unit. The inverted index entry includes a retained term unit field, a knowledge fragment unit identifier field, and an occurrence position list field. The occurrence position list field is a list of position indices of the retained term unit within the knowledge fragment unit.

[0027] Step S126: Summarize the inverted index entries of all retained term units into sparse term retrieval index entries corresponding to knowledge fragment units. The sparse term retrieval index entries and knowledge fragment unit identifiers form a second mapping record.

[0028] In this embodiment, the inverted index entries of all reserved term units within a knowledge fragment unit are summarized into a single sparse term retrieval index item. The sparse term retrieval index item is stored in list form. Each item in the list is an inverted index entry. The sparse term retrieval index item and the knowledge fragment unit identifier form a second mapping record. The second mapping record is stored in the sparse inverted index structure. The sparse inverted index structure is a single inverted index table. Each reserved term unit in the inverted index table corresponds to one inverted list. The inverted list records a list containing all knowledge fragment unit identifiers and their occurrence positions for that reserved term unit.

[0029] Step S127: Perform consistency binding processing on the knowledge fragment unit identifier, associate and store the knowledge fragment unit identifier in the first mapping record with the knowledge fragment unit identifier in the second mapping record, and obtain the correspondence between the dense vector retrieval representation and the sparse term retrieval index term that share the same knowledge fragment unit identifier.

[0030] In this embodiment, the consistency binding process checks whether the knowledge fragment unit identifier in the first mapping record is consistent with the knowledge fragment unit identifier in the second mapping record. If they are consistent, the first mapping record and the second mapping record are associated and stored. After association and storage, both the dense vector retrieval representation and the sparse term retrieval index can be accessed simultaneously through the knowledge fragment unit identifier. The correspondence between the dense vector retrieval representation and the sparse term retrieval index that share the same knowledge fragment unit identifier is stored in the knowledge fragment unit identifier binding area.

[0031] Step S130: Obtain the target query request, and simultaneously perform dense vector query representation generation processing and sparse term query representation generation processing on the target query request to obtain the dense vector query representation and sparse term query representation corresponding to the target query request.

[0032] In this embodiment, the target query request is a natural language query input by an enterprise user. The target query request contains the query statement text. Two processing channels are executed simultaneously on the target query request. The first processing channel is the dense vector query representation generation process. The dense vector query representation generation process inputs the target query request into a dense vector encoding network. The dense vector encoding network outputs a dense vector query representation. The second processing channel is the sparse term query representation generation process. The sparse term query representation generation process performs query term segmentation, query term morphology restoration, and query intent weighting on the target query request. The outputs of the two processing channels are the dense vector query representation and the sparse term query representation, respectively.

[0033] Step S131: Perform query intent sentence recognition processing on the target query request to obtain the query statement sentence type and core content of the query corresponding to the target query request.

[0034] In this embodiment, the query intent statement recognition process uses rule matching. Query statement types include definition queries, step queries, comparison queries, and fault queries. Definition queries use "definition," "what," and "concept" as key words. Step queries use "how," "steps," and "process" as key words. Comparison queries use "difference," "comparison," and "advantages and disadvantages" as key words. Fault queries use "fault," "error," and "abnormality" as key words. The core content of the query is the remaining content after removing the key words from the target query request. For example, if the target query request is "how to configure warehouse scheduling priority," the query statement type is a step query, and the core content is "configure warehouse scheduling priority."

[0035] Step S132: Input the core content of the query into the pre-constructed dense vector encoding network, and perform layer-by-layer semantic compression processing on the core content of the query through the multi-layer semantic transformation structure in the dense vector encoding network to obtain the dense vector query representation corresponding to the target query request.

[0036] In this embodiment, the core query content is segmented into a query term sequence after term segmentation. Each query term in the query term sequence is mapped to a term embedding vector. The term embedding vector sequence is input into a dense vector encoding network. The dense vector encoding network is the same network used in step S121. The dense vector encoding network outputs a dense vector query representation. The dimension of the dense vector query representation is 512. The dense vector query representation and the dense vector retrieval representation reside in the same dense vector retrieval space.

[0037] Step S133: In each semantic transformation structure of the dense vector coding network, the semantic attention calculation structure within that semantic transformation structure is used to perform attention-weighted aggregation processing on the query term sequence of the core content part of the query, generating the intermediate semantic convergence vector of the query corresponding to that semantic transformation structure.

[0038] In this embodiment, the query term sequence is processed layer by layer through the semantic transformation structures of layers 1 to 6 in the dense vector coding network. The semantic attention computation structure in each layer performs attention-weighted aggregation on the query term sequence. The first layer semantic transformation structure outputs the first layer query intermediate semantic convergence vector. The second layer semantic transformation structure outputs the second layer query intermediate semantic convergence vector, and so on. The sixth layer semantic transformation structure outputs the sixth layer query intermediate semantic convergence vector.

[0039] Step S134: The query intermediate semantic convergence vector output by the semantic transformation structure of the last layer of the dense vector coding network is used as the dense vector query representation corresponding to the target query request. The dense vector query representation and the dense vector retrieval representation are in the same dense vector retrieval space.

[0040] In this embodiment, the intermediate semantic convergence vector of the sixth-layer query is processed by average pooling and a linear mapping layer to obtain a dense vector query representation. The dense vector query representation is a 512-dimensional vector. The dense vector query representation and the dense vector retrieval representation reside in the same 512-dimensional dense vector retrieval space. Similarity nearest neighbor search processing can be performed in the dense vector retrieval space.

[0041] Step S135: Perform query term segmentation processing on the core content part of the query to obtain multiple original query term units in the core content part of the query, and perform term morphology restoration processing on each original query term unit to obtain the query restored term unit corresponding to each original query term unit.

[0042] In this embodiment, the query term segmentation process uses the same term segmentation algorithm as in step S124. The core query content is segmented into multiple original query term units. Term morphology restoration processing is performed on each original query term unit. The variant words in the original query term units are restored to their basic word forms. After all original query term units undergo term morphology restoration processing, the corresponding restored query term units are obtained.

[0043] Step S136: Perform query intent weight marking processing on each query restored term unit according to the query statement type to obtain the query intent weight mark corresponding to each query restored term unit. Summarize all query restored term units carrying query intent weight marks into a sparse term query representation corresponding to the target query request. Each query restored term unit in the sparse term query representation is in the same term space as the reserved term unit in the sparse term retrieval index.

[0044] In this embodiment, the query intent weighting process assigns one query intent weighting tag to each query restored term unit. The query intent weighting tag is numerical. When the query statement type is a fault query, the query intent weighting tag for fault-related query restored term units has a high weight. When the query statement type is a step query, the query intent weighting tag for operation step-related query restored term units has a high weight. All query restored term units, carrying query intent weighting tags, are aggregated into a sparse term query representation. Each query restored term unit in the sparse term query representation resides in the same term space as the reserved term units in the sparse term retrieval index.

[0045] Step S140: Utilize dense vector query representation to perform similarity nearest neighbor search processing in the dense vector retrieval space to obtain a first candidate knowledge fragment set, and utilize sparse term query representation to perform term matching processing in the sparse inverted index structure to obtain a second candidate knowledge fragment set. Perform fusion and deduplication processing on the first and second candidate knowledge fragment sets to obtain a fused candidate knowledge fragment set carrying dense vector retrieval scores and sparse term retrieval scores.

[0046] In this embodiment, the dense vector retrieval space is a 512-dimensional vector space. The dense vector retrieval space stores the dense vector retrieval representations corresponding to all knowledge fragment units. The similarity nearest neighbor search process uses cosine similarity, calculated as: sim(Qd, Pd) = (Qd·Pd) / (|Qd|×|Pd|), where Qd is the dense vector query representation, Pd is the dense vector retrieval representation, and sim(Qd, Pd) is the dense vector retrieval similarity score. The sparse inverted index structure stores the sparse term retrieval index entries corresponding to all knowledge fragment units. The term matching process searches the sparse inverted index structure for inverted index entries containing any query-restored term unit from the sparse term query representation.

[0047] Step S141: In the dense vector retrieval space, perform similarity nearest neighbor search processing on the dense vector query representation and the dense vector retrieval representation corresponding to all knowledge fragment units to obtain the dense vector retrieval similarity score corresponding to each knowledge fragment unit.

[0048] In this embodiment, for each dense vector retrieval representation in the dense vector retrieval space, the cosine similarity between the dense vector retrieval representation and the dense vector query representation is calculated using sim(Qd, Pd) = (Qd·Pd) / (|Qd|×|Pd|). The cosine similarity is used as the dense vector retrieval similarity score corresponding to the knowledge fragment unit. The value range of the dense vector retrieval similarity score is from -1 to 1.

[0049] Step S142: Sort all knowledge fragment units in descending order according to the dense vector retrieval similarity score, and select knowledge fragment units whose sorting position is within the preset number of nearest neighbors to form a first candidate knowledge fragment set. Each first candidate knowledge fragment in the first candidate knowledge fragment set carries the corresponding dense vector retrieval similarity score as the dense vector retrieval score.

[0050] In this embodiment, the preset range for the number of nearest neighbors is the top K. K is set to 50. All knowledge fragment units are arranged in descending order of their dense vector retrieval similarity scores. The top 50 knowledge fragment units constitute the first candidate knowledge fragment set. Each first candidate knowledge fragment in the first candidate knowledge fragment set carries its corresponding dense vector retrieval similarity score as its dense vector retrieval score.

[0051] Step S143: In the sparse inverted index structure, traverse the sparse inverted index structure according to each query restored term unit in the sparse term query representation, recall all inverted index entries containing any query restored term unit, and perform grouping and merging processing on all recalled inverted index entries according to knowledge fragment unit identifier to obtain the set of hit query restored term units corresponding to each recalled knowledge fragment unit.

[0052] In this embodiment, for each query-restored term unit in the sparse term query representation, the inverted list corresponding to that query-restored term unit is searched in the sparse inverted index structure. The inverted list contains all inverted index entries for that query-restored term unit. All inverted index entries are recalled. The recalled inverted index entries are grouped and merged according to the knowledge fragment unit identifier. Each recalled knowledge fragment unit corresponds to one set of matched query-restored term units. The set of matched query-restored term units records all query-restored term units matched by that knowledge fragment unit.

[0053] Step S144: Based on the set of term units restored by the hit query corresponding to each recalled knowledge fragment unit, and combined with the query intent weight tag corresponding to each term unit restored by the hit query, perform sparse term retrieval score calculation to obtain the sparse term retrieval score corresponding to each recalled knowledge fragment unit.

[0054] In this embodiment, for each recalled knowledge fragment unit, the set of query-restored term units is read. For each query-restored term unit in the set, the query intent weight tag corresponding to that query-restored term unit is read. The sparse term retrieval score is calculated as: Ssp = Σ(Wq), where Ssp is the sparse term retrieval score, Wq is the query intent weight tag corresponding to the matched query-restored term unit, and Σ is the summation symbol. The sparse term retrieval score is numerical.

[0055] Step S145: Sort all recalled knowledge fragment units in descending order according to the sparse term retrieval score, and select recalled knowledge fragment units whose sorting position is within the preset term hit range to form the second candidate knowledge fragment set.

[0056] In this embodiment, the preset term hit count range is the top K terms. K is set to 50. All recalled knowledge fragment units are arranged in descending order of sparse term retrieval scores. The top 50 recalled knowledge fragment units constitute the second candidate knowledge fragment set. Each second candidate knowledge fragment in the second candidate knowledge fragment set carries a corresponding sparse term retrieval score.

[0057] Step S146: Perform knowledge fragment unit identifier matching processing on each first candidate knowledge fragment in the first candidate knowledge fragment set and each second candidate knowledge fragment in the second candidate knowledge fragment set. Mark the knowledge fragment units in the first candidate knowledge fragment set that have the same knowledge fragment unit identifier as those in the second candidate knowledge fragment set as overlapping candidate knowledge fragments. Perform score merging processing on the overlapping candidate knowledge fragments. Update the dense vector retrieval score corresponding to the overlapping candidate knowledge fragments to the original dense vector retrieval score, while retaining the sparse term retrieval score corresponding to the overlapping candidate knowledge fragments.

[0058] In this embodiment, the knowledge fragment unit identifier matching process compares the identifier of each knowledge fragment unit in the first candidate knowledge fragment set with the identifier of each knowledge fragment unit in the second candidate knowledge fragment set. Knowledge fragment units with the same identifier are marked as overlapping candidate knowledge fragments. Overlapping candidate knowledge fragments carry both dense vector retrieval scores and sparse term retrieval scores. The score merging process keeps the dense vector retrieval score unchanged while retaining the sparse term retrieval score.

[0059] Step S147: The first candidate knowledge fragments in the first candidate knowledge fragment set that do not overlap with the second candidate knowledge fragment set, the second candidate knowledge fragments in the second candidate knowledge fragment set that do not overlap with the first candidate knowledge fragment set, and the overlapping candidate knowledge fragments after the score merging process are collectively summarized into a fused candidate knowledge fragment set.

[0060] In this embodiment, non-overlapping knowledge fragment units in the first candidate knowledge fragment set only carry dense vector retrieval scores. Non-overlapping knowledge fragment units in the second candidate knowledge fragment set only carry sparse term retrieval scores. Overlapping candidate knowledge fragments carry both dense vector retrieval scores and sparse term retrieval scores. These three types of knowledge fragment units are aggregated into a fused candidate knowledge fragment set. Each fused candidate knowledge fragment in the fused candidate knowledge fragment set carries a knowledge fragment unit identifier, a dense vector retrieval score, and a sparse term retrieval score.

[0061] Step S150: Perform document source grouping and retrieval score redistribution processing on the fusion candidate knowledge fragment set to obtain a redistributed retrieval score set. Filter out the target knowledge fragment set based on the redistributed retrieval score set. Generate an enhanced generation input sequence based on the target knowledge fragment set and the target query request. Input the enhanced generation input sequence into the pre-built enhanced generation network to obtain the enhanced generation result of the enterprise knowledge base.

[0062] In this embodiment, the document source grouping process groups the fusion candidate knowledge fragments according to the source attribution mapping relationship carried by the fusion candidate knowledge fragments. The retrieval score redistribution process recalculates the redistributed retrieval score for each fusion candidate knowledge fragment based on the source grouping information and the retrieval score. The target knowledge fragment set is filtered from the fusion candidate knowledge fragment set. The enhanced generation input sequence combines the target knowledge fragment content with the target query request. The enhanced generation network outputs the enhanced generation result of the enterprise knowledge base.

[0063] Step S151: Based on the source attribution mapping relationship carried by each fusion candidate knowledge fragment in the fusion candidate knowledge fragment set, fusion candidate knowledge fragments with the same document source identifier are grouped into the same document source group, resulting in multiple document source groups.

[0064] In this embodiment, for each fusion candidate knowledge fragment in the fusion candidate knowledge fragment set, the document source identifier in the source attribution mapping relationship is read. Fusion candidate knowledge fragments with the same document source identifier are grouped into the same document source group. For example, if the document source identifiers of fusion candidate knowledge fragments PQ001, PQ002, and PQ003 are all DQ001, then PQ001, PQ002, and PQ003 are grouped into document source group G001. After all fusion candidate knowledge fragments are grouped, multiple document source groups are obtained.

[0065] Step S152: Perform dense vector retrieval score aggregation processing on all fusion candidate knowledge fragments within each document source group to obtain the dense vector retrieval score aggregation result corresponding to that document source group. Perform sparse term retrieval score aggregation processing on all fusion candidate knowledge fragments within each document source group to obtain the sparse term retrieval score aggregation result corresponding to that document source group.

[0066] In this embodiment, the formula for calculating the dense vector retrieval score aggregation result is: Ad = Σ(Sd), where Ad is the dense vector retrieval score aggregation result, and Sd is the dense vector retrieval score of each fused candidate knowledge fragment within the document source group. The formula for calculating the sparse term retrieval score aggregation result is: Asp = Σ(Ssp), where Asp is the sparse term retrieval score aggregation result, and Ssp is the sparse term retrieval score of each fused candidate knowledge fragment within the document source group.

[0067] Step S153: Calculate the source grouping retrieval contribution parameter corresponding to each document source grouping based on the dense vector retrieval score aggregation result and the sparse term retrieval score aggregation result corresponding to each document source grouping.

[0068] In this embodiment, the formula for calculating the source grouping retrieval contribution parameter is: Cg = 0.5 × Ad + 0.5 × Asp, where Cg is the source grouping retrieval contribution parameter, Ad is the aggregated result of dense vector retrieval score, and Asp is the aggregated result of sparse term retrieval score.

[0069] Step S154: Perform source grouping and sorting position determination processing on each fusion candidate knowledge fragment within each document source group to obtain the source grouping and sorting position of each fusion candidate knowledge fragment within its respective document source group.

[0070] In this embodiment, for each document source group, all fusion candidate knowledge fragments within that group are arranged from largest to smallest according to Sd + Ssp. After arrangement, each fusion candidate knowledge fragment obtains one source grouping ranking position within the group. The source grouping ranking positions increment from 1.

[0071] Step S155: Based on the dense vector retrieval score, sparse term retrieval score, source grouping sorting position, and source grouping retrieval contribution parameter corresponding to the source grouping of the document to which each fusion candidate knowledge fragment belongs, perform retrieval score redistribution processing to obtain the redistributed retrieval score corresponding to each fusion candidate knowledge fragment. Summarize the redistributed retrieval scores corresponding to all fusion candidate knowledge fragments into a redistributed retrieval score set. Perform descending sorting processing on all fusion candidate knowledge fragments according to the redistributed retrieval score set, and filter out the fusion candidate knowledge fragments whose sorting position is within the preset target number range to form the target knowledge fragment set.

[0072] In this embodiment, the redistribution retrieval score is calculated as follows: Sr = 0.35 × Sd + 0.35 × Ssp + 0.2 × Cg - 0.1 × Rg, where Sr is the redistribution retrieval score, Sd is the dense vector retrieval score, Ssp is the sparse term retrieval score, Cg is the source grouping retrieval contribution parameter, and Rg is the source grouping ranking position. All fusion candidate knowledge fragments are arranged in descending order of Sr. The fusion candidate knowledge fragments ranked in the top M constitute the target knowledge fragment set. M is set to 20.

[0073] Step S156: Perform fragment content and source identifier concatenation processing on each target knowledge fragment in the target knowledge fragment set to obtain the enhanced fragment input unit corresponding to each target knowledge fragment. Combine all enhanced fragment input units and target query requests according to the preset input arrangement to form an enhanced generation input sequence. Input the enhanced generation input sequence into the pre-constructed enhanced generation network to obtain the enhanced generation result of the enterprise knowledge base.

[0074] In this embodiment, for each target knowledge fragment, the document source identifier in the knowledge fragment content and source attribution mapping relationship is read. The knowledge fragment content and document source identifier are concatenated along the text dimension to form an enhanced fragment input unit. The preset input arrangement is target query request first, followed by all enhanced fragment input units. The concatenation result is the enhanced generation input sequence. The enhanced generation input sequence is input into the enhanced generation network. The enhanced generation network outputs the enhanced generation result of the enterprise knowledge base. The enhanced generation result of the enterprise knowledge base is in text format.

[0075] Step S210: Perform document structure parsing processing on the knowledge document set to obtain the document structure tree corresponding to each knowledge document unit. The document structure tree contains chapter-level nodes and paragraph-level nodes.

[0076] In this embodiment, this step is performed before the semantic segmentation process in step S110. The document structure parsing process is performed on each knowledge document unit in the knowledge document set. The document structure parsing process reads the document body field of the knowledge document unit. The chapter title, paragraph start mark, and paragraph content in the document body field are identified. The chapter title corresponds to the chapter level node. The paragraph start mark corresponds to the paragraph level node. The document structure tree has the chapter level node as the parent node and the paragraph level node as the child node. Each node in the document structure tree carries a node identifier and the position range of the node in the document body. For example, the document structure tree of knowledge document D001 contains chapter level nodes C01, C02, and C03. Chapter level node C01 contains paragraph level nodes P01, P02, and P03.

[0077] Step S220: When performing semantic segmentation processing on each knowledge document unit, the semantic segmentation boundary is determined based on the chapter-level nodes and paragraph-level nodes in the document structure tree, resulting in multiple knowledge fragment units.

[0078] In this embodiment, the sliding window for semantic segmentation is forcibly aligned at the boundaries of chapter-level nodes and paragraph-level nodes in the document structure tree. The sliding window boundary preferentially falls on either the chapter-level node boundary or the paragraph-level node boundary. When the sliding window boundary falls in the middle of a paragraph-level node, the boundary is moved to the nearest paragraph-level node boundary. Semantic segmentation yields multiple knowledge fragment units. Each knowledge fragment unit corresponds to one or more consecutive paragraph-level nodes in the document structure tree.

[0079] Step S230: After obtaining multiple knowledge fragment units, perform document structure location extension processing on the source attribution mapping relationship corresponding to each knowledge fragment unit, and extend the source attribution mapping relationship into a source attribution extension mapping relationship that simultaneously includes document source identifier and document structure path identifier.

[0080] In this embodiment, the document structure path identifier is the path of the knowledge fragment unit in the document structure tree. The document structure path identifier is formed by concatenating the chapter-level node identifier and the paragraph-level node identifier sequentially along the document structure tree from the root node to the leaf node. For example, the source attribution extended mapping relationship of knowledge fragment unit P001 includes the document source identifier DM001 and the document structure path identifier D001-C01-P01.

[0081] Step S240: When performing dense vector retrieval encoding processing on each knowledge fragment unit, the document structure path identifier is input as an additional semantic tag into the pre-constructed dense vector encoding network, so that the document structure path semantic information is incorporated into the dense vector retrieval representation.

[0082] In this embodiment, the document structure path identifier is mapped to a document structure path embedding vector through one embedding layer. The document structure path embedding vector and the term embedding vector sequence of the knowledge fragment unit are concatenated along the sequence dimension. The concatenated sequence is input into a dense vector encoding network. The document structure path semantic information is incorporated into the dense vector retrieval representation output by the dense vector encoding network.

[0083] Step S250: When performing sparse term retrieval index construction processing for each knowledge fragment unit, add the document structure path term corresponding to the document structure path identifier to the sparse term retrieval index term.

[0084] In this embodiment, the document structure path identifiers D001-C01-P01 correspond to the document structure path terms D001, C01, and P01. These three document structure path terms are added as reserved term units to the sparse term retrieval index. The list of the occurrence positions of the document structure path terms within the knowledge fragment unit is recorded as the starting position of the knowledge fragment unit.

[0085] Step S260: When performing similarity nearest neighbor search processing based on dense vector query representation, obtain the query document structure preference tag corresponding to the target query request, and perform structure preference weighted nearest neighbor search processing based on the query document structure preference tag and the document structure path semantic information incorporated in the dense vector retrieval representation to obtain the first candidate knowledge fragment set. When performing term matching processing based on sparse term query representation, perform structure preference term matching processing based on the query document structure preference tag and the document structure path terms in the sparse term retrieval index terms to obtain the second candidate knowledge fragment set.

[0086] In this embodiment, the query document structure preference tag is extracted from the target query request. When the target query request contains document structure preference terms, the query document structure preference tag is a preferred document structure path term. The structure preference weighted nearest neighbor search processing, based on cosine similarity calculation, adds weight to the dimension incorporating document structure path semantic information in the dense vector retrieval representation. The structure preference term matching processing prioritizes recalling inverted index entries containing document structure path terms in the sparse inverted index structure.

[0087] Step S270: When performing fusion deduplication on the first candidate knowledge fragment set and the second candidate knowledge fragment set, the source attribution extended mapping relationship is used as an additional matching condition during the fusion deduplication process to obtain a fusion candidate knowledge fragment set carrying the document structure path identifier.

[0088] In this embodiment, the deduplication process adds document structure path identifier matching as a condition to the knowledge fragment unit identifier matching. Knowledge fragment units with the same knowledge fragment unit identifier and the same document structure path identifier are marked as overlapping candidate knowledge fragments.

[0089] Step S280: When performing document source grouping processing on the fusion candidate knowledge fragment set, grouping processing is performed simultaneously according to the document source identifier and document structure path identifier based on the source attribution extended mapping relationship, resulting in multiple document structure groups. When performing retrieval score redistribution processing on each document structure group, the structure depth parameter corresponding to the document structure path identifier is included in the redistribution processing process to obtain the redistributed retrieval score corresponding to each fusion candidate knowledge fragment.

[0090] In this embodiment, document structure grouping is performed according to the hierarchical relationship between document source identifiers and document structure path identifiers. Candidate knowledge fragments with the same document source identifier are first grouped into the same document source group. Within each document source group, further subdivisions are made based on the chapter-level node identifiers of the document structure path identifiers. The structure depth parameter is the number of path nodes in the document structure path identifier. The redistribution retrieval score is calculated using the formula: Sr = 0.35 × Sd + 0.35 × Ssp + 0.2 × Cg - 0.1 × Rg + 0.05 × Dp, where Dp is the structure depth parameter.

[0091] Step S310: Perform query intent expansion processing on the target query request to obtain the query intent expansion term set corresponding to the target query request, obtain each query intent expansion term in the query intent expansion term set, perform expansion term morphology restoration processing on each query intent expansion term to obtain the expansion restored term unit corresponding to each query intent expansion term.

[0092] In this embodiment, the query intent expansion process employs synonym expansion and context word expansion. The query intent expansion term set is read from the term association table in the enterprise knowledge base. Each term in the term association table is associated with multiple synonyms and context words. Term morphology restoration processing is performed on each query intent expansion term. After restoring the query intent expansion term to its basic morphology, the expanded and restored term unit is obtained.

[0093] Step S320: Add all extended and restored term units to the sparse term query representation so that the sparse term query representation contains both query restored term units and extended and restored term units.

[0094] In this embodiment, the expanded and restored term units are concatenated along the term dimension with the query restored term units. The concatenation result is the expanded sparse term query representation. Each expanded and restored term unit in the expanded sparse term query representation carries an expanded query intent weight label.

[0095] Step S330: In the sparse inverted index structure, traverse the sparse inverted index structure according to the extended restored term unit in the sparse term query representation, and recall all extended inverted index entries containing any extended restored term unit.

[0096] In this embodiment, for each extended and restored term unit in the sparse term query representation, the inverted list corresponding to that extended and restored term unit is searched in the sparse inverted index structure. The inverted list contains all inverted index entries for that extended and restored term unit. All extended and restored inverted index entries are then retrieved.

[0097] Step S340: Perform grouping and merging processing on all recalled extended inverted index entries by knowledge fragment unit identifier to obtain the extended hit term unit set corresponding to each extended recalled knowledge fragment unit. Based on the extended hit term unit set corresponding to each extended recalled knowledge fragment unit, and combined with the extended query intent weight tag corresponding to each extended restored term unit, perform extended sparse term retrieval score calculation processing to obtain the extended sparse term retrieval score corresponding to each extended recalled knowledge fragment unit.

[0098] In this embodiment, the formula for calculating the extended sparse term retrieval score is: Sex = Σ(Wex), where Sex is the extended sparse term retrieval score, Wex is the extended query intent weight label corresponding to the extended restored term unit, and Σ is the summation symbol.

[0099] Step S350: Perform knowledge fragment unit identifier matching and merging processing on the extended recall knowledge fragment unit and the second candidate knowledge fragment set, and add the knowledge fragment units in the extended recall knowledge fragment unit that do not appear in the second candidate knowledge fragment set to the second candidate knowledge fragment set.

[0100] In this embodiment, the knowledge fragment unit identifier matching process compares the identifier of the extended recall knowledge fragment unit with the identifier of each knowledge fragment unit in the second candidate knowledge fragment set. The extended recall knowledge fragment unit corresponding to the identifier that does not appear is added to the second candidate knowledge fragment set.

[0101] Step S360: Perform sparse term retrieval score fusion processing on the supplemented second candidate knowledge fragment set, and integrate the extended sparse term retrieval score into the corresponding sparse term retrieval score to obtain the extended second candidate knowledge fragment set.

[0102] In this embodiment, the sparse term retrieval score fusion processing involves a weighted sum of the extended sparse term retrieval score and the original sparse term retrieval score. The fusion formula is: Ssp = 0.7 × Ssp_original + 0.3 × Sex, where Ssp_original is the original sparse term retrieval score and Sex is the extended sparse term retrieval score.

[0103] Step S370: Perform fusion and deduplication processing on the first candidate knowledge fragment set and the expanded second candidate knowledge fragment set to obtain a fused candidate knowledge fragment set carrying the expanded sparse term retrieval score. When performing document source grouping processing and retrieval score redistribution processing on the fused candidate knowledge fragment set, the expanded sparse term retrieval score is included in the redistribution process to obtain a redistributed retrieval score set.

[0104] In this embodiment, the deduplication process is performed in the same way as steps S146 and S147. The formula for calculating the redistributed retrieval score is: Sr = 0.35 × Sd + 0.35 × Ssp + 0.2 × Cg - 0.1 × Rg, where Ssp is the sparse term retrieval score after fusion.

[0105] Step S410: Obtain the knowledge document update instruction corresponding to the enterprise knowledge base. The knowledge document update instruction includes the knowledge document unit to be updated and the update operation type.

[0106] In this embodiment, the knowledge document update instruction is read from the enterprise knowledge base management interface. The update operation type is either a add operation, a delete operation, or a modify operation. The knowledge document unit to be updated is the knowledge document that requires the update operation.

[0107] Step S420: When the update operation type is an add operation, perform semantic segmentation processing on the knowledge document unit to be updated to obtain the new knowledge fragment unit, and establish a new source attribution mapping relationship for each new knowledge fragment unit. Perform dense vector retrieval encoding processing and sparse term retrieval index construction processing on each new knowledge fragment unit simultaneously to obtain the new dense vector retrieval representation and new sparse term retrieval index item corresponding to each new knowledge fragment unit. Insert the new dense vector retrieval representation corresponding to each new knowledge fragment unit into the dense vector retrieval space, and insert the new sparse term retrieval index item corresponding to each new knowledge fragment unit into the sparse inverted index structure.

[0108] In this embodiment, the semantic segmentation processing for the new operation is performed in the same way as the semantic segmentation processing in step S110. When a new dense vector retrieval representation is inserted into the dense vector retrieval space, a new first mapping record is created in the dense vector retrieval space. When a new sparse term retrieval index term is inserted into the sparse inverted index structure, the corresponding inverted list is updated in the sparse inverted index structure.

[0109] Step S430: When the update operation type is a deletion operation, based on the document source identifier corresponding to the knowledge document unit to be updated, locate and remove all dense vector retrieval representations belonging to the knowledge document unit to be updated in the dense vector retrieval space, and based on the document source identifier corresponding to the knowledge document unit to be updated, locate and remove all sparse term retrieval index terms belonging to the knowledge document unit to be updated in the sparse inverted index structure.

[0110] In this embodiment, the deletion operation searches for and deletes all first mapping records in the dense vector retrieval space that match the document source identifier. The deletion operation also searches for and removes all inverted index entries in the sparse inverted index structure that match the document source identifier.

[0111] Step S440: When the update operation type is a modification operation, based on the document source identifier corresponding to the knowledge document unit to be updated, remove all original dense vector retrieval representations belonging to the knowledge document unit to be updated in the dense vector retrieval space. Based on the document source identifier corresponding to the knowledge document unit to be updated, remove all original sparse term retrieval index terms belonging to the knowledge document unit to be updated in the sparse inverted index structure. Re-perform semantic segmentation processing on the modified knowledge document unit to be updated to obtain modified knowledge fragment units, and establish a modification source attribution mapping relationship for each modified knowledge fragment unit.

[0112] In this embodiment, the modification operation first performs a deletion operation and then a addition operation. The document source identifier in the source attribution mapping relationship is matched with the document number of the knowledge document unit to be updated.

[0113] Step S450: For each modified knowledge fragment unit, perform dense vector retrieval encoding processing and sparse term retrieval index construction processing simultaneously to obtain the modified dense vector retrieval representation and the modified sparse term retrieval index item. Insert the modified dense vector retrieval representation into the dense vector retrieval space and insert the modified sparse term retrieval index item into the sparse inverted index structure.

[0114] In this embodiment, the insertion method for modifying the dense vector retrieval representation and modifying the sparse term retrieval index is the same as the addition operation.

[0115] Step S510: Obtain the fragment access frequency record of each knowledge fragment unit in the knowledge document set corresponding to the enterprise knowledge base. The fragment access frequency record includes the cumulative number of times the knowledge fragment unit has been recalled and used in the historical retrieval process.

[0116] In this embodiment, the fragment access frequency record is read from the access log of the enterprise knowledge base. The cumulative count is incremented by 1 each time a knowledge fragment unit is recalled and adopted by the enhanced generative network during historical retrieval. The fragment access frequency record is stored according to the knowledge fragment unit identifier.

[0117] Step S520: Perform segment access popularity level classification processing on each knowledge segment unit. Determine the segment access popularity level corresponding to each knowledge segment unit based on the cumulative number of times in the segment access frequency record. In the dense vector retrieval space, perform popularity weighting processing on the dense vector retrieval representation corresponding to each knowledge segment unit. Integrate the popularity weighting parameter corresponding to the segment access popularity level into the dense vector retrieval representation to obtain the popularity weighted dense vector retrieval representation.

[0118] In this embodiment, the access popularity of fragments is divided into three levels: high popularity, medium popularity, and low popularity. High popularity is defined as a fragment whose cumulative access count is greater than 1.5 times the median cumulative access count of all knowledge fragment units. Low popularity is defined as a fragment whose cumulative access count is less than 0.5 times the median. All others are classified as medium popularity. The popularity weighting parameter is numerical. The popularity weighting process multiplies the dense vector retrieval representation dimension by the popularity weighting parameter. High popularity corresponds to a popularity weighting parameter of 1.2. Medium popularity corresponds to a popularity weighting parameter of 1.0. Low popularity corresponds to a popularity weighting parameter of 0.8.

[0119] Step S530: In the sparse inverted index structure, perform popularity index marking processing on the sparse term retrieval index item corresponding to each knowledge fragment unit, and record the fragment access popularity level in the sparse term retrieval index item.

[0120] In this embodiment, a popularity level field is added to the sparse term retrieval index of each knowledge fragment unit. The popularity level field can be categorized as high popularity, medium popularity, or low popularity.

[0121] Step S540: Obtain the target query request, and simultaneously perform dense vector query representation generation processing and sparse term query representation generation processing on the target query request to obtain the dense vector query representation and sparse term query representation corresponding to the target query request. In the dense vector retrieval space, perform similarity nearest neighbor search processing based on the dense vector query representation and the popularity-weighted dense vector retrieval representation to obtain the first candidate knowledge fragment set carrying the fragment access popularity level.

[0122] In this embodiment, the similarity nearest neighbor search processing is performed in the same way as step S141, except that the dense vector retrieval representation is replaced by the heat-weighted dense vector retrieval representation.

[0123] Step S550: In the sparse inverted index structure, term matching processing is performed based on the sparse term query representation. During the term matching process, the fragment access popularity level recorded in the sparse term retrieval index is used as an additional sorting criterion to obtain a second candidate knowledge fragment set carrying the fragment access popularity level. The first candidate knowledge fragment set and the second candidate knowledge fragment set are then fused and deduplicated to obtain a fused candidate knowledge fragment set carrying the fragment access popularity level.

[0124] In this embodiment, when the sparse term retrieval scores are the same, the knowledge segment unit with the higher access popularity level is ranked first in the term matching process.

[0125] Step S560: When performing document source grouping and retrieval score redistribution on the fusion candidate knowledge fragment set, the fragment access popularity level is included in the redistribution process to obtain a redistributed retrieval score set. The target knowledge fragment set is selected based on the redistributed retrieval score set. An enhanced generation input sequence is generated based on the target knowledge fragment set and the target query request. The enhanced generation input sequence is input into the pre-built enhanced generation network to obtain the enhanced generation result of the enterprise knowledge base.

[0126] In this embodiment, the formula for calculating the redistribution retrieval score is: Sr = 0.35 × Sd + 0.35 × Ssp + 0.2 × Cg - 0.1 × Rg + 0.1 × H, where H is the popularity score corresponding to the fragment access popularity level. High popularity corresponds to a popularity score of 3. Medium popularity corresponds to a popularity score of 2. Low popularity corresponds to a popularity score of 1. The subsequent target knowledge fragment set filtering and enhanced generation network processing are performed in the same manner as steps S155 and S156.

[0127] For example, step S610: Obtain the content summary of each knowledge segment unit in the enterprise knowledge base. The content summary is generated during the semantic segmentation process and is used to summarize the semantic topic of each knowledge segment unit.

[0128] In this embodiment, the fragment content summary is generated synchronously during the semantic segmentation process in step S110. The fragment content summary is a semantic topic summary text of the knowledge fragment unit. The length of the fragment content summary does not exceed 64 term units.

[0129] Step S620: Perform dense vector retrieval encoding processing on the content summary of each knowledge segment unit to obtain the dense vector retrieval representation of the summary corresponding to each knowledge segment unit. Perform association storage processing on the dense vector retrieval representation of the summary corresponding to each knowledge segment unit and the dense vector retrieval representation of the knowledge segment unit to obtain the fragment content summary association vector pair corresponding to each knowledge segment unit.

[0130] In this embodiment, the summary dense vector retrieval encoding process inputs the fragment content summary into a dense vector encoding network. The dense vector encoding network outputs a summary dense vector retrieval representation. The summary dense vector retrieval representation has a dimension of 512. The fragment content summary association vector pair contains both the summary dense vector retrieval representation and the dense vector retrieval representation.

[0131] Step S630: Obtain the target query request, perform dense vector query representation generation processing on the target query request to obtain the dense vector query representation corresponding to the target query request, and perform summary similarity nearest neighbor search processing in the dense vector retrieval space based on the dense vector query representation and the dense vector retrieval representation of all summaries to obtain a set of candidate knowledge fragments for the summary.

[0132] In this embodiment, the formula for calculating the nearest neighbor search processing of the summary similarity is: sim(Qd, As) = (Qd·As) / (|Qd|×|As|), where Qd is the dense vector query representation and As is the summary dense vector retrieval representation.

[0133] Step S640: In the dense vector retrieval space, perform full-text similarity nearest neighbor search processing based on the dense vector query representation and all dense vector retrieval representations to obtain a set of full-text candidate knowledge fragments.

[0134] In this embodiment, the calculation formula for full-text similarity nearest neighbor search is: sim(Qd, Pd) = (Qd·Pd) / (|Qd|×|Pd|), where Pd is the representation of dense vector retrieval.

[0135] Step S650: Perform candidate knowledge fragment fusion and deduplication processing on the summary candidate knowledge fragment set and the full-text candidate knowledge fragment set to obtain a summary-full-text fusion candidate knowledge fragment set carrying the summary retrieval score and the full-text retrieval score. Based on the fragment content summary association vector pair corresponding to each fusion candidate knowledge fragment in the summary-full-text fusion candidate knowledge fragment set, perform summary-full-text score balancing processing on the summary retrieval score and the full-text retrieval score to obtain a balanced candidate knowledge fragment set.

[0136] In this embodiment, the formula for balancing the abstract and full-text scores is: Sb = 0.6 × Ssu + 0.4 × Sfu, where Sb is the balanced retrieval score, Ssu is the abstract retrieval score, and Sfu is the full-text retrieval score.

[0137] Step S660: Perform document source grouping and retrieval score redistribution processing on the balanced candidate knowledge fragment set to obtain a redistributed retrieval score set. Filter out the target knowledge fragment set based on the redistributed retrieval score set. Generate an enhanced generation input sequence based on the target knowledge fragment set and the target query request. Input the enhanced generation input sequence into the pre-built enhanced generation network to obtain the enhanced generation result of the enterprise knowledge base.

[0138] In this embodiment, the formula for calculating the redistributed retrieval score is: Sr = 0.35 × Sb + 0.35 × Ssp + 0.2 × Cg - 0.1 × Rg, where Sb is the retrieval score after balancing. Subsequent processing is performed in the same manner as steps S155 and S156.

[0139] Step S710: Perform semantic tomography recognition processing on the dense vector retrieval representations corresponding to all knowledge fragment units in the dense vector retrieval space. Use the abrupt change position of the adjacent distance between dense vector retrieval representations as the semantic tomography boundary to divide the dense vector retrieval space into multiple dense vector semantic tomography regions. The dense vector retrieval representations in each dense vector semantic tomography region share the same semantic tomography identifier.

[0140] In this embodiment, the adjacency distance is the Euclidean distance between any two dense vector retrieval representations in the dense vector retrieval space. Hierarchical clustering is performed on all dense vector retrieval representations. The hierarchical position where the merging distance between adjacent clusters changes abruptly during clustering is used as the semantic fault boundary. The semantic fault boundary divides the dense vector retrieval space into multiple dense vector semantic fault regions. Dense vector retrieval representations within each dense vector semantic fault region share one semantic fault identifier. The semantic fault identifier is numbered from F01 to F0N. For example, dense vector semantic fault region F01 contains dense vector retrieval representations corresponding to knowledge fragment units P001 to P020.

[0141] Step S720: Perform internal structure analysis on each dense vector semantic fault region. Within each dense vector semantic fault region, using knowledge fragment unit identifiers as clues, extract the document source identifier to which each dense vector retrieval representation belongs based on the source attribution mapping relationship corresponding to the knowledge fragment unit identifier, thereby obtaining the document source distribution structure within each dense vector semantic fault region.

[0142] In this embodiment, for each dense vector semantic fault region, the knowledge fragment unit identifier corresponding to all dense vectors within that region is retrieved. For each knowledge fragment unit identifier, the document source identifier in the source attribution mapping relationship is read. The number of times each document source identifier appears within the dense vector semantic fault region is counted. The statistical results constitute the document source distribution structure within the fault. The document source distribution structure within the fault is a mapping from document source identifiers to their frequency of occurrence.

[0143] Step S730: In the sparse inverted index structure, perform term breakdown analysis on the sparse term retrieval index terms corresponding to all knowledge fragment units. Taking the reserved term units contained in each sparse term retrieval index term as clues, mark the reserved term units that simultaneously cross multiple dense vector semantic breakdown regions and whose frequency of occurrence in all dense vector semantic breakdown regions exceeds the preset frequency threshold as term breakdown penetration units.

[0144] In this embodiment, for each retained term unit, the frequency of its occurrence is counted across all dense vector semantic segment regions. The frequency of occurrence is the number of times the retained term unit appears in the sparse term retrieval index of all knowledge fragment units within the corresponding dense vector semantic segment region. The preset frequency threshold is the average frequency of all retained term units across all dense vector semantic segment regions. When a retained term unit appears simultaneously in at least two dense vector semantic segment regions, and its frequency in each dense vector semantic segment region exceeds the preset frequency threshold, the retained term unit is marked as a term segment penetration unit.

[0145] Step S740: Based on the number of dense vector semantic fault regions traversed by each term fault penetration unit, construct a fault penetration path record corresponding to each term fault penetration unit. The fault penetration path record contains a sequence of dense vector semantic fault region identifiers that the term fault penetration unit penetrates sequentially.

[0146] In this embodiment, for each term tomographic penetration unit, all dense vector semantic tomographic region identifiers that the term tomographic penetration unit has appeared are read. These identifiers are arranged in numerical order. The arrangement result is the tomographic penetration path record. For example, if term tomographic penetration unit W01 sequentially penetrates dense vector semantic tomographic regions F01, F03, and F05, then the tomographic penetration path record is F01-F03-F05.

[0147] Step S750: Perform penetration direction parsing processing on the fault penetration path record corresponding to each term fault penetration unit, and determine the fault penetration direction mark corresponding to the term fault penetration unit according to the arrangement order of the region identifiers in the dense vector semantic fault region identifier sequence.

[0148] In this embodiment, when the dense vector semantic tomographic region identifier sequence is arranged in ascending order by number, the fault penetration direction is marked as forward penetration. When the dense vector semantic tomographic region identifier sequence is arranged in descending order by number, the fault penetration direction is marked as reverse penetration. When the dense vector semantic tomographic region identifier sequence has both ascending and descending order by number, the fault penetration direction is marked as mixed penetration.

[0149] Step S760: Obtain the target query request, and simultaneously perform dense vector query representation generation processing and sparse term query representation generation processing on the target query request to obtain the dense vector query representation and sparse term query representation corresponding to the target query request.

[0150] In this embodiment, this step is executed in the same way as step S130.

[0151] Step S770: In the dense vector retrieval space, perform fault region localization processing based on the dense vector query representation and each dense vector semantic fault region to determine the target dense vector semantic fault region in which the dense vector query representation falls, and perform similarity nearest neighbor search processing within the target dense vector semantic fault region to obtain the first candidate knowledge fragment set.

[0152] In this embodiment, the fault region localization process calculates the Euclidean distance between the dense vector query representation and the center point of each dense vector semantic fault region. The dense vector semantic fault region with the smallest Euclidean distance is the target dense vector semantic fault region. The center point of the dense vector semantic fault region is the mean vector of all dense vector retrieval representations within that region. The similarity nearest neighbor search process is performed within the target dense vector semantic fault region, and the calculation formula is the same as in step S141.

[0153] Step S780: In the sparse inverted index structure, perform term matching processing based on the sparse term query representation. During the term matching process, prioritize recalling knowledge fragment units that contain term breakdown penetration units in the sparse term retrieval index to obtain the second candidate knowledge fragment set.

[0154] In this embodiment, when traversing the sparse inverted index structure, the term match processing checks whether the sparse term retrieval index of the knowledge fragment unit to which the inverted index entry belongs contains a term breakdown unit during the term matching process. Knowledge fragment units containing term breakdown units are prioritized over knowledge fragment units that do not contain term breakdown units during the sorting process.

[0155] Step S790: Perform fusion and deduplication processing on the first candidate knowledge fragment set and the second candidate knowledge fragment set to obtain a fusion candidate knowledge fragment set carrying fault region identifiers and fault penetration direction markers. Perform document source grouping processing and retrieval score redistribution processing on the fusion candidate knowledge fragment set to obtain a redistributed retrieval score set. Filter out the target knowledge fragment set based on the redistributed retrieval score set. Generate an enhanced generation input sequence based on the target knowledge fragment set and the target query request. Input the enhanced generation input sequence into the pre-built enhanced generation network to obtain the enhanced generation result of the enterprise knowledge base.

[0156] In this embodiment, the deduplication process is executed in the same manner as steps S146 and S147. The formula for calculating the redistribution retrieval score is: Sr = 0.35 × Sd + 0.35 × Ssp + 0.2 × Cg - 0.1 × Rg + 0.05 × Ft, where Ft is the direction score corresponding to the fault penetration direction marker. A forward penetration corresponds to a direction score of 3. A reverse penetration corresponds to a direction score of 2. A mixed penetration corresponds to a direction score of 1. Subsequent processing is executed in the same manner as steps S155 and S156.

[0157] Step S810: Perform document internal semantic evolution link construction processing on each knowledge document unit in the knowledge document set. Taking the sequential arrangement of multiple knowledge fragment units obtained from semantic segmentation within the knowledge document unit as clues, connect the dense vector retrieval representations corresponding to adjacent knowledge fragment units in sequence to obtain the document internal semantic evolution link corresponding to each knowledge document unit.

[0158] In this embodiment, for each knowledge document unit, all knowledge fragment units within that knowledge document unit are read. These knowledge fragment units are then sorted according to their sequential order within the knowledge document unit. After sorting, the dense vector retrieval representations corresponding to adjacent knowledge fragment units are connected sequentially. The connection result is a semantic evolution link within the document. This semantic evolution link within the document is an ordered sequence of dense vector retrieval representations.

[0159] Step S820: Perform evolution direction extraction processing on the semantic evolution link within the document corresponding to each knowledge document unit. Compare the semantic offset direction of the dense vector retrieval representation corresponding to adjacent knowledge fragment units in the order of their arrangement along the semantic evolution link within the document to obtain the document evolution direction tag set corresponding to each knowledge document unit. Perform binding and storage processing with the document source identifier corresponding to the knowledge document unit to obtain the document evolution mapping relationship between the document source identifier and the document evolution direction tag set.

[0160] In this embodiment, the semantic offset direction is the main direction of the difference vector between the dense vector retrieval representations corresponding to adjacent knowledge fragment units. The main direction of the difference vector is the direction corresponding to the dimension with the largest absolute value in the difference vector. Multiple semantic offset directions are obtained by sequentially comparing adjacent knowledge fragment units. All semantic offset directions constitute a set of document internal evolution direction tags. The set of document internal evolution direction tags is bound and stored with the document source identifier.

[0161] Step S830: In the dense vector retrieval space, perform cross-document semantic traction analysis processing based on the dense vector retrieval representations corresponding to all knowledge fragment units. Compare the direction overlap between the document internal evolution direction marker set corresponding to one document source identifier and the document internal evolution direction marker set corresponding to another document source identifier to obtain a cross-document semantic traction relationship set. Construct cross-document semantic traction chains based on the cross-document semantic traction relationship set. Each cross-document semantic traction chain contains multiple document source identifiers and semantic traction direction markers between adjacent document source identifiers.

[0162] In this embodiment, the direction overlap ratio is used to compare and count the proportion of identical semantic offset directions in the evolution direction marker sets within two documents to the total number of semantic offset directions. Two document source identifiers with a direction overlap ratio greater than a preset overlap ratio threshold have a cross-document semantic traction relationship. The preset overlap ratio threshold is the median of the direction overlap ratios of all document source identifier pairs. Document source identifiers with a cross-document semantic traction relationship are sequentially connected to form a cross-document semantic traction chain. The semantic traction direction marker is the semantic offset direction with the highest direction overlap ratio between adjacent document source identifiers.

[0163] Step S840: Perform backbone extraction processing on each cross-document semantic traction chain, and filter the backbone document source identifier set according to the number of knowledge fragment units corresponding to each document source identifier in the cross-document semantic traction chain.

[0164] In this embodiment, for each document source identifier in the cross-document semantic traction chain, the number of knowledge fragment units corresponding to that document source identifier is counted. Document source identifiers whose number of knowledge fragment units is greater than the median number of knowledge fragment units corresponding to all document source identifiers are selected into the main document source identifier set of the traction chain.

[0165] Step S850: Obtain the target query request, perform dense vector query representation generation processing on the target query request to obtain the dense vector query representation corresponding to the target query request, perform traction backbone nearest neighbor search processing based on the dense vector query representation and the dense vector retrieval representation corresponding to the traction chain backbone document source identifier set to obtain the first candidate knowledge fragment set, and perform sparse term query representation generation processing on the target query request to obtain the sparse term query representation corresponding to the target query request, perform term matching processing based on the sparse term query representation in the sparse inverted index structure, and prioritize recalling knowledge fragment units belonging to the traction chain backbone document source identifier set during the term matching process to obtain the second candidate knowledge fragment set.

[0166] In this embodiment, the traction backbone nearest neighbor search processing is performed only in the dense vector retrieval representation corresponding to the traction chain backbone document source identifier set. The term matching processing prioritizes knowledge fragment units belonging to the traction chain backbone document source identifier set during sorting.

[0167] Step S860: Perform fusion and deduplication processing on the first candidate knowledge fragment set and the second candidate knowledge fragment set to obtain a fusion candidate knowledge fragment set carrying cross-document semantic traction chain identifiers. Perform document source grouping processing and retrieval score redistribution processing on the fusion candidate knowledge fragment set to obtain a redistributed retrieval score set. Filter out the target knowledge fragment set based on the redistributed retrieval score set. Generate an enhanced generation input sequence based on the target knowledge fragment set and the target query request. Input the enhanced generation input sequence into the pre-built enhanced generation network to obtain the enhanced generation result of the enterprise knowledge base.

[0168] In this embodiment, the deduplication process is performed in the same manner as steps S146 and S147. The formula for calculating the redistribution retrieval score is: Sr = 0.35 × Sd + 0.35 × Ssp + 0.2 × Cg - 0.1 × Rg + 0.05 × Tc, where Tc is the chain score corresponding to the cross-document semantic traction chain identifier. The more document source identifiers included in the cross-document semantic traction chain, the higher the chain score. Subsequent processing is performed in the same manner as steps S155 and S156.

[0169] Step S910: In the dense vector retrieval space, perform semantic reverse mapping processing on the dense vector retrieval representation corresponding to each knowledge fragment unit, and reverse map each dimension of the semantic component in each dense vector retrieval representation to the specific term unit position in the original term unit sequence corresponding to the knowledge fragment unit, so as to obtain the source record of the dense vector semantic component corresponding to each knowledge fragment unit.

[0170] In this embodiment, the semantic back-mapping process employs attention weight back-tracking. Each semantic component in the dense vector retrieval representation corresponds to a weighted aggregation result of the output of one attention head in the semantic transformation structure of the last layer of the dense vector coding network. The attention weight corresponding to each term unit in this weighted aggregation result constitutes the basis for back-mapping. Each semantic component is back-mapped to the term unit position with the largest attention weight. The dense vector semantic component tracing record is a mapping from the semantic component dimension to the term unit position.

[0171] Step S920: Perform source heat distribution construction processing on the source traceability records of dense vector semantic components corresponding to each knowledge fragment unit. Based on the cumulative number of times the position of each original term unit is back-mapped by the dense vector semantic components, construct the source heat distribution corresponding to each knowledge fragment unit.

[0172] In this embodiment, for each term unit position within a knowledge segment unit, the cumulative number of times that term unit position is back-mapped in the source tracing record of the dense vector semantic component is counted. The cumulative number constitutes the source tracing heat value of that term unit position. The source tracing heat values ​​of all term unit positions constitute the source tracing heat distribution. The source tracing heat distribution is a mapping from term unit positions to source tracing heat values.

[0173] Step S930: In the sparse inverted index structure, perform index term tracing association processing on the sparse term retrieval index term corresponding to each knowledge fragment unit, and associate each reserved term unit in the sparse term retrieval index term with the heat position corresponding to the reserved term unit in the tracing heat distribution to obtain the dual-channel tracing association structure corresponding to each knowledge fragment unit.

[0174] In this embodiment, for each reserved term unit, the source heat value of the corresponding term unit position is found in the source heat distribution. Reserved term units are associated with source heat values. The dual-channel source association structure includes a list of reserved term units and the source heat value associated with each reserved term unit.

[0175] Step S940: Perform source tracing tomography processing on the dual-channel source tracing association structure corresponding to each knowledge fragment unit. In the dual-channel source tracing association structure, scan the source heat distribution and the misalignment offset between the location of the retained term unit and the location of the source tracing heat distribution along the word arrangement direction in the knowledge fragment unit to obtain the source misalignment offset record corresponding to each knowledge fragment unit.

[0176] In this embodiment, the misalignment offset is the difference in term unit position between the thermal peak position and the position where the retained term unit appears in the source thermal distribution. Multiple misalignment offset values ​​are obtained after scanning position by position. All misalignment offset values ​​constitute the source misalignment offset record.

[0177] Step S950: Perform misalignment pattern clustering processing on the source misalignment offset records corresponding to all knowledge fragment units, and group knowledge fragment units with similar source misalignment offset features into the same misalignment pattern group to obtain multiple misalignment pattern groups and the misalignment pattern representative features corresponding to each misalignment pattern group.

[0178] In this embodiment, K-means clustering is used for misalignment pattern clustering. The clustering features are the mean and standard deviation of the misalignment offset values ​​in the source misalignment offset records. The clustering results in multiple misalignment pattern groups. The cluster center of each misalignment pattern group serves as the representative feature of the misalignment pattern.

[0179] Step S960: Obtain the target query request, and simultaneously perform dense vector query representation generation processing and sparse term query representation generation processing on the target query request to obtain the dense vector query representation and sparse term query representation corresponding to the target query request.

[0180] In this embodiment, this step is executed in the same way as step S130.

[0181] Step S970: In the dense vector retrieval space, perform similarity nearest neighbor search processing based on the dense vector query representation. During the similarity nearest neighbor search processing, compare the degree of misalignment pattern matching between the dense vector query representation and each misalignment pattern representative feature to obtain a first candidate knowledge fragment set carrying misalignment pattern matching tags. In the sparse inverted index structure, perform term hit matching processing based on the sparse term query representation. During the term hit matching processing, use the dual-channel source tracing association structure to perform source tracing heat verification on the hit retained term units to obtain a second candidate knowledge fragment set carrying source tracing heat verification tags.

[0182] In this embodiment, the mismatch pattern matching degree is the cosine similarity between the dense vector query representation and the mismatch pattern representative features. Knowledge fragment units with a mismatch pattern matching degree greater than a preset matching threshold carry a mismatch pattern matching tag. Source heat verification checks whether the source heat value associated with the matched term unit is greater than a preset heat value. If it is greater than the preset heat value, it carries a source heat verification tag.

[0183] Step S980: Perform fusion deduplication processing on the first candidate knowledge fragment set and the second candidate knowledge fragment set. Use the misaligned pattern matching tag and the source heat verification tag as additional fusion criteria for fusion deduplication processing to obtain a fusion candidate knowledge fragment set. Perform document source grouping processing and retrieval score redistribution processing on the fusion candidate knowledge fragment set to obtain a redistributed retrieval score set. Filter out the target knowledge fragment set based on the redistributed retrieval score set. Generate an enhanced generation input sequence based on the target knowledge fragment set and the target query request. Input the enhanced generation input sequence into the pre-built enhanced generation network to obtain the enhanced generation result of the enterprise knowledge base.

[0184] In this embodiment, the deduplication process adds misalignment pattern matching and source tracing thermal verification matching to the knowledge fragment unit identifier matching. The formula for calculating the redistribution retrieval score is: Sr = 0.35 × Sd + 0.35 × Ssp + 0.2 × Cg - 0.1 × Rg + 0.05 × Mp + 0.05 × Hp, where Mp is the matching score corresponding to the misalignment pattern matching mark, and Hp is the verification score corresponding to the source tracing thermal verification mark. Subsequent processing is performed in the same manner as steps S155 and S156.

[0185] Step S1010: Perform retrieval channel source differentiation identification processing on each fusion candidate knowledge fragment in the fusion candidate knowledge fragment set. Based on the difference between the dense vector retrieval score and the sparse term retrieval score carried by each fusion candidate knowledge fragment, divide the fusion candidate knowledge fragments into three channel source differentiation types: dense channel-dominated fusion candidate knowledge fragments, sparse channel-dominated fusion candidate knowledge fragments, and dual-channel balanced fusion candidate knowledge fragments.

[0186] In this embodiment, the formula for calculating the score difference magnitude is: Df = |Sd - Ssp| / (Sd + Ssp + ε), where Df is the score difference magnitude, Sd is the dense vector retrieval score, Ssp is the sparse term retrieval score, and ε is the smoothing term with a value of 0.001. When Df is greater than 0.6 and Sd is greater than Ssp, the fusion candidate knowledge fragment is a dense channel-dominated fusion candidate knowledge fragment. When Df is greater than 0.6 and Ssp is greater than Sd, the fusion candidate knowledge fragment is a sparse channel-dominated fusion candidate knowledge fragment. When Df is less than or equal to 0.6, the fusion candidate knowledge fragment is a dual-channel balanced fusion candidate knowledge fragment.

[0187] Step S1020: Perform channel type internal sorting processing on the fusion candidate knowledge fragments corresponding to each channel source differentiation type. Within the dense channel-dominated fusion candidate knowledge fragment set, sort them in descending order according to the dense vector retrieval score. Within the sparse channel-dominated fusion candidate knowledge fragment set, sort them in descending order according to the sparse term retrieval score. Within the dual-channel balanced fusion candidate knowledge fragment set, sort them in descending order according to the sum of the dense vector retrieval score and the sparse term retrieval score.

[0188] In this embodiment, the dense channel-dominated fusion candidate knowledge fragment set is arranged in descending order of Sd. The sparse channel-dominated fusion candidate knowledge fragment set is arranged in descending order of Ssp. The dual-channel balanced fusion candidate knowledge fragment set is arranged in descending order of Sd + Ssp. After arrangement, each fusion candidate knowledge fragment obtains one internal ranking position within its respective channel type.

[0189] Step S1030: Perform channel type interleaving weaving processing on the fusion candidate knowledge fragments corresponding to the three channel source differentiation types. In a fixed rotation order, alternately select fusion candidate knowledge fragments from the dense channel-dominated fusion candidate knowledge fragment set, the sparse channel-dominated fusion candidate knowledge fragment set, and the dual-channel balanced fusion candidate knowledge fragment set to obtain the channel type interleaving weaving sequence.

[0190] In this embodiment, the fixed rotation order is dense channel-dominated, sparse channel-dominated, and dual-channel balanced. In the first round, the first-ranked fusion candidate knowledge fragment is selected from the dense channel-dominated fusion candidate knowledge fragment set. Then, the first-ranked fusion candidate knowledge fragment is selected from the sparse channel-dominated fusion candidate knowledge fragment set. Finally, the first-ranked fusion candidate knowledge fragment is selected from the dual-channel balanced fusion candidate knowledge fragment set. In the second round, the second-ranked fusion candidate knowledge fragment is selected, and so on. The selection results form an interleaved sequence of channel types. After the selection of fusion candidate knowledge fragments for a certain channel type is completed, that channel type is skipped and the rotation continues.

[0191] Step S1040: Perform channel type coherence injection processing on the channel type interlaced weaving sequence, and integrate the channel source differentiation type label corresponding to each fusion candidate knowledge fragment into the transition position between adjacent fusion candidate knowledge fragments in the channel type interlaced weaving sequence to obtain an interlaced weaving sequence carrying channel type coherence labels.

[0192] In this embodiment, the channel source differentiation type is labeled as dense channel-dominated, sparse channel-dominated, or dual-channel balanced. A channel type coherence marker is inserted between adjacent fusion candidate knowledge fragments in the channel type interleaving sequence. The channel type coherence marker is a marker pair composed of the channel source differentiation type markers of two adjacent fusion candidate knowledge fragments. For example, the channel type coherence marker between a dense channel-dominated fusion candidate knowledge fragment and a sparse channel-dominated fusion candidate knowledge fragment is dense-sparse.

[0193] Step S1050: Obtain the query channel bias features corresponding to the target query request, perform query channel bias identification processing on the target query request, and determine the query channel bias type corresponding to the target query request based on the distribution density of query restoration term units and query intent weight labels contained in the target query request.

[0194] In this embodiment, the query restoration term unit distribution density is the number of query restoration term units in the target query request divided by the text length of the target query request. The mean of the query intent weight label is the average of the query intent weight labels of all query restoration term units. When the query restoration term unit distribution density is greater than a preset distribution density threshold and the mean of the query intent weight labels is greater than a preset weight mean threshold, the query channel bias type is sparse channel bias. When the query restoration term unit distribution density is less than or equal to the preset distribution density threshold, the query channel bias type is dense channel bias. In other cases, the query channel bias type is dual-channel balanced bias. The preset distribution density threshold is the median of the query restoration term unit distribution density of all historical target query requests. The preset weight mean threshold is the average of the mean of the query intent weight labels of all historical target query requests.

[0195] Step S1060: Perform biased rearrangement on the interlaced weaving sequence carrying the channel type coherence marker according to the query channel bias type, and move the sorting position of the fusion candidate knowledge fragments corresponding to the channel source differentiation type that matches the query channel bias type to the front in the interlaced weaving sequence to obtain the biased rearranged fusion candidate knowledge fragment sequence.

[0196] In this embodiment, when the query channel bias type is sparse channel bias, the sorting position of sparse channel dominant fusion candidate knowledge fragments in the interlaced weaving sequence is moved forward as a whole. When the query channel bias type is dense channel bias, the sorting position of dense channel dominant fusion candidate knowledge fragments in the interlaced weaving sequence is moved forward as a whole. When the query channel bias type is dual-channel balanced bias, the sorting position of dual-channel balanced fusion candidate knowledge fragments in the interlaced weaving sequence is moved forward as a whole. After the overall forward movement, the remaining fusion candidate knowledge fragments maintain their original relative order and are moved backward.

[0197] Step S1070: Perform document source grouping processing on each fusion candidate knowledge fragment in the biased rearrangement sequence to obtain multiple document source groups. The fusion candidate knowledge fragments in each document source group share the same document source identifier.

[0198] In this embodiment, the document source grouping process is executed in the same way as step S151. Each fusion candidate knowledge fragment in the biased rearranged fusion candidate knowledge fragment sequence is grouped into its corresponding document source group according to the document source identifier in the source attribution mapping relationship.

[0199] Step S1080: Perform retrieval score redistribution processing on each document source group. Use the channel source differentiation type label and query channel bias type as additional ranking weight sources in the redistribution process to obtain the redistributed retrieval score corresponding to each fusion candidate knowledge fragment. Filter out the target knowledge fragment set based on the redistributed retrieval scores corresponding to all fusion candidate knowledge fragments. Perform fragment content and source identifier concatenation processing on each target knowledge fragment in the target knowledge fragment set to obtain the enhanced fragment input unit. Combine all enhanced fragment input units with the target query request to form an enhanced generation input sequence. Input the enhanced generation input sequence into the pre-built enhanced generation network to obtain the enterprise knowledge base enhanced generation result.

[0200] In this embodiment, the formula for calculating the redistribution retrieval score is: Sr = 0.3 × Sd + 0.3 × Ssp + 0.15 × Cg - 0.1 × Rg + 0.1 × Ct + 0.05 × Qb, where Ct is the type score corresponding to the channel source differentiation type marker, and Qb is the bias score corresponding to the query channel bias type. When the dense channel-dominant fusion candidate knowledge fragment has a dense channel bias, Ct = 3. When the sparse channel-dominant fusion candidate knowledge fragment has a sparse channel bias, Ct = 3. When the dual-channel balanced fusion candidate knowledge fragment has a dual-channel balanced bias, Ct = 3. For other combinations, Ct = 1. Qb is 2 when the query channel bias type matches the channel source differentiation type, and 1 when they do not. All fusion candidate knowledge fragments are arranged in descending order of Sr. The fusion candidate knowledge fragments ranked in the top M constitute the target knowledge fragment set. M is 20. The subsequent processing is performed in the same manner as step S156.

[0201] Step S1110: In the sparse inverted index structure, perform index term sparse density sensing processing on the sparse term retrieval index term corresponding to each knowledge fragment unit, and determine the index term sparse density sensing result corresponding to each knowledge fragment unit based on the number of retained term units contained in each sparse term retrieval index term and the distribution span of all retained term units within the knowledge fragment unit.

[0202] In this embodiment, the number of retained term units is the total number of retained term units contained in the sparse term retrieval index. The distribution span is the difference between the last and first positions in the list of occurrence positions of all retained term units within the knowledge fragment unit. The formula for calculating the sparse density perception result of the index item is: Ds = Nw / (Lw + ε), where Ds is the sparse density perception result of the index item, Nw is the number of retained term units, Lw is the distribution span, and ε is a smoothing term with a value of 0.001. A larger Ds indicates a denser distribution of retained term units within the knowledge fragment unit.

[0203] Step S1120: Perform sparse density layering processing on the sparse density perception result of the index item corresponding to each knowledge fragment unit, and divide all knowledge fragment units into three sparse density layers: sparse density dense layer, sparse density intermediate layer, and sparse density dispersed layer. Knowledge fragment units within each sparse density layer share the sparse density perception result of similar index items.

[0204] In this embodiment, the sparse density layering is performed by arranging the index item sparse density perception results from largest to smallest. If the index item sparse density perception result is greater than 1.5 times the median of the sparse density perception results of all knowledge fragment units, it is assigned to the dense sparse density layer. If it is less than 0.5 times the median, it is assigned to the dispersed sparse density layer. The rest are assigned to the intermediate sparse density layer.

[0205] Step S1130: In the dense vector retrieval space, perform dense vector distribution density perception processing on the dense vector retrieval representation corresponding to each knowledge fragment unit. Determine the dense vector distribution density perception result corresponding to each knowledge fragment unit based on the spatial distance distribution between each dense vector retrieval representation and its nearest neighbor dense vector retrieval representation in the dense vector retrieval space. Perform dense vector density layering processing on the dense vector distribution density perception result corresponding to each knowledge fragment unit, and divide all knowledge fragment units into three dense vector density layers: dense vector compact layer, dense vector regular layer, and dense vector loose layer.

[0206] In this embodiment, the dense vector distribution density perception result is the average of the Euclidean distances between the dense vector retrieval representation and the 10 most recent dense vector retrieval representations. If the dense vector distribution density perception result is less than 0.5 times the median of the dense vector distribution density perception results for all knowledge fragment units, it is assigned to the dense vector compact layer. If it is greater than 1.5 times the median, it is assigned to the dense vector sparse layer. The rest are assigned to the dense vector regular layer.

[0207] Step S1140: Perform density level cross-combination processing on the sparse density level and dense vector density level corresponding to each knowledge fragment unit to obtain the dual-channel density cross type corresponding to each knowledge fragment unit. The dual-channel density cross type is composed of the sparse density level and the dense vector density level.

[0208] In this embodiment, the dual-channel density cross type is a combination of sparse density layers and dense vector density layers. The combination method is sparse density layers first, followed by dense vector density layers. For example, a sparse density layer combined with a dense vector compact layer is a dense-compact dual-channel density cross type. There are a total of 9 dual-channel density cross types.

[0209] Step S1150: Obtain the target query request, and simultaneously perform dense vector query representation generation processing and sparse term query representation generation processing on the target query request to obtain the dense vector query representation and sparse term query representation corresponding to the target query request.

[0210] In this embodiment, this step is executed in the same way as step S130.

[0211] Step S1160: In the dense vector retrieval space, perform similarity nearest neighbor search processing based on the dense vector query representation. During the similarity nearest neighbor search processing, perform density adaptive nearest neighbor expansion based on the dual-channel density cross type corresponding to each knowledge fragment unit to obtain the first candidate knowledge fragment set. In the sparse inverted index structure, perform term matching processing based on the sparse term query representation. During the term matching processing, perform density adaptive term expansion based on the dual-channel density cross type corresponding to each knowledge fragment unit to obtain the second candidate knowledge fragment set.

[0212] In this embodiment, after obtaining the top K knowledge fragment units through similarity nearest neighbor search processing, for each knowledge fragment unit, if its dual-channel density cross type is dense-compact, then the knowledge fragment units corresponding to the 5 nearest dense vector retrieval representations of that knowledge fragment unit in the dense vector retrieval space are added to the first candidate knowledge fragment set. If it is scattered-sparse, then the 10 nearest knowledge fragment units are added. If it is other combinations, then the 7 nearest knowledge fragment units are added. After obtaining the hit knowledge fragment units through term matching processing, for each hit knowledge fragment unit, if its dual-channel density cross type is dense-compact, then all retained term units in the sparse term retrieval index of that knowledge fragment unit are used as extended terms, and the inverted index entries containing the extended terms are recalled. If it is scattered-sparse, then no extension is performed.

[0213] Step S1170: Perform fusion and deduplication processing on the first candidate knowledge fragment set and the second candidate knowledge fragment set to obtain a fusion candidate knowledge fragment set carrying a dual-channel density cross type. Perform document source grouping processing and retrieval score redistribution processing on the fusion candidate knowledge fragment set to obtain a redistributed retrieval score set. Filter out the target knowledge fragment set based on the redistributed retrieval score set. Generate an enhanced generation input sequence based on the target knowledge fragment set and the target query request. Input the enhanced generation input sequence into the pre-built enhanced generation network to obtain the enhanced generation result of the enterprise knowledge base.

[0214] In this embodiment, the deduplication process is performed in the same manner as steps S146 and S147. The formula for calculating the redistribution retrieval score is: Sr = 0.35 × Sd + 0.35 × Ssp + 0.2 × Cg - 0.1 × Rg + 0.05 × Dc, where Dc is the density score corresponding to the dual-channel density cross type. The density score for the dense-compact dual-channel density cross type is 3. The density score for the dispersed-loose dual-channel density cross type is 1. The density score for the remaining dual-channel density cross types is 2. Subsequent processing is performed in the same manner as steps S155 and S156.

[0215] like Figure 2As shown, the original knowledge documents in the enterprise knowledge base undergo data flow processes such as semantic segmentation, dual-path candidate fusion, and document update and index synchronization, ultimately presenting the enhanced knowledge base generation results on the interface. Taking an employee of a large enterprise looking up internal expense reimbursement policies in daily work as an example, when the employee enters a query request on the knowledge base search interface, the query request triggers two search paths simultaneously. One path uses dense vector semantic matching technology to find semantically similar knowledge fragments in the vector space; the other path uses sparse term inverted index technology to accurately match document fragments hit by keywords. The candidate knowledge fragments obtained from the two searches are merged and deduplicated, and then grouped according to document source and redistributed according to search scores to select the most matching document fragment as the basis for generation. The final enhanced result is marked with the corresponding citation source on the interface, and the knowledge base document is updated and the index is synchronized simultaneously, ensuring that the answer seen by the employee is both accurate and authoritative.

[0216] like Figure 3 As shown, the enterprise knowledge base presents a concrete interactive interface on the user terminal. This interface includes functional areas such as inputting enterprise knowledge questions, recent searches, popular questions, my favorites, browsing the knowledge base, document uploads, search history, and preference settings. The bottom navigation bar provides access to the homepage, knowledge base, question history, and my profile. Taking a query about internal regulations within this large enterprise as an example, after an employee enters a question about the business trip reimbursement process into the input box on their mobile terminal, the query request is sent to the enterprise knowledge base backend. The backend's hybrid retrieval module simultaneously performs dense vector semantic matching and sparse term inverted index retrieval. The matched knowledge fragments are merged using a dual-path candidate fusion mechanism, and the most relevant paragraphs are selected according to document source grouping and retrieval score redistribution rules. Then, this paragraph, along with the user's original question, is concatenated into an enhanced generation input sequence, which is fed into a pre-built enhanced generation network. Finally, a detailed answer to the question is generated on the interface, which also displays the specific document fragments cited, ensuring that the employee clearly understands the source and basis of the answer.

[0217] like Figure 4As shown, the enhanced generation results are presented in a specific interactive display on the interface. The top of the interface displays the detailed answer content generated by AI, while the bottom lists the document names of the cited fragments, their relevant versions, and specification types. The bottom of the interface indicates that the result is generated based on a hybrid retrieval of the enterprise knowledge base and includes an interactive button to generate an enhanced answer. Taking an employee's query about the business trip reimbursement process through the system as an example, after the employee submits a related question, the knowledge base system uses a dual-path retrieval mechanism that integrates dense vector semantic matching and sparse term inverted index to locate relevant paragraphs in authoritative documents such as the company's financial system V2.0 and administrative service guides. Then, based on document source grouping and retrieval score redistribution, the most matching knowledge fragment is selected and input into the enhanced generation network, thereby generating a detailed answer that includes steps such as submitting reimbursement applications and vouchers, department head review and confirmation, finance department verification of invoices, and completion of payment and reimbursement. In the results display area, the answer is clearly linked to the company's financial system V2.0 and administrative service guidelines. While viewing the answer, employees can directly trace the source of the reference to confirm the compliance of the process. They can also obtain more complete and accurate answers by clicking the "Generate Enhanced Answer" button, which effectively improves employees' trust in the accuracy of the query results.

[0218] Furthermore, an enterprise knowledge base enhancement generation system based on hybrid retrieval provided in this application embodiment includes a processor and a memory. The processor and memory are connected, for example, via a bus. Optionally, the enterprise knowledge base enhancement generation system based on hybrid retrieval may further include a transceiver, which can be used for data interaction between this enterprise knowledge base enhancement generation system and other enterprise knowledge base enhancement generation systems based on hybrid retrieval, such as sending and / or receiving data. It should be noted that in actual scheduling, the transceiver is not limited to one, and the structure of this enterprise knowledge base enhancement generation system based on hybrid retrieval does not constitute a limitation on the embodiments of this application.

[0219] The memory stores program code for executing the embodiments of this application, and its execution is controlled by a processor. The processor executes the program code stored in the memory to implement the steps shown in the foregoing method embodiments.

[0220] The above description is only an optional implementation method for some implementation scenarios of this application. It should be noted that for those skilled in the art, other similar implementation methods based on the technical concept of this application, without departing from the technical concept of this application, also fall within the protection scope of the embodiments of this application.

Claims

1. A hybrid search based enterprise knowledge base augmentation generation method, characterized in that, The method includes: Obtain a set of knowledge documents from the enterprise knowledge base, and perform semantic segmentation on the set of knowledge documents to obtain multiple knowledge fragment units carrying source attribution mapping relationships; For each knowledge segment unit, dense vector retrieval encoding processing and sparse term retrieval index construction processing are performed synchronously to obtain the dense vector retrieval representation and sparse term retrieval index item corresponding to each knowledge segment unit. The dense vector retrieval representation and the sparse term retrieval index item share the same knowledge segment unit identifier. Obtain the target query request, and simultaneously perform dense vector query representation generation processing and sparse term query representation generation processing on the target query request to obtain the dense vector query representation and sparse term query representation corresponding to the target query request; The first candidate knowledge fragment set is obtained by performing similarity nearest neighbor search in the dense vector retrieval space using the dense vector query representation, and the second candidate knowledge fragment set is obtained by performing term matching in the sparse inverted index structure using the sparse term query representation. The first candidate knowledge fragment set and the second candidate knowledge fragment set are then fused and deduplicated to obtain a fused candidate knowledge fragment set carrying dense vector retrieval score and sparse term retrieval score. The document source grouping and retrieval score redistribution processes are performed on the fusion candidate knowledge fragment set to obtain a redistributed retrieval score set. The target knowledge fragment set is selected based on the redistributed retrieval score set. An enhanced generation input sequence is generated based on the target knowledge fragment set and the target query request. The enhanced generation input sequence is input into a pre-constructed enhanced generation network to obtain the enhanced generation result of the enterprise knowledge base.

2. The hybrid search based enterprise knowledge base augmentation generation method of claim 1, wherein, The process of synchronously performing dense vector retrieval encoding and sparse term retrieval index construction on each knowledge segment unit yields a dense vector retrieval representation and a sparse term retrieval index for each knowledge segment unit, including: The knowledge fragment unit is input into a pre-constructed dense vector coding network, and the knowledge fragment unit is subjected to layer-by-layer semantic compression processing through the multi-layer semantic transformation structure in the dense vector coding network to obtain the dense vector retrieval representation corresponding to the knowledge fragment unit. In each layer of the dense vector coding network, the semantic unit sequence of the knowledge fragment unit is subjected to attention weighted aggregation processing by the semantic attention computing structure within that layer of the semantic transformation structure, thereby generating the intermediate semantic convergence vector corresponding to that layer of the semantic transformation structure. The intermediate semantic convergence vector output by the last layer semantic transformation structure of the dense vector coding network is used as the dense vector retrieval representation corresponding to the knowledge fragment unit, and the dense vector retrieval representation and the knowledge fragment unit identifier form a first mapping record; The knowledge fragment unit is subjected to term segmentation processing to obtain multiple original term units in the knowledge fragment unit, and term morphology restoration processing is performed on each original term unit to obtain the restored term unit corresponding to each original term unit; For each restored term unit, term frequency statistics processing is performed to obtain the occurrence frequency parameter of each restored term unit in the knowledge fragment unit. Based on the occurrence frequency parameter, the retained term units that constitute the sparse term retrieval index of the knowledge fragment unit are selected. An inverted index entry is constructed for each retained term unit. The inverted index entry records the knowledge fragment unit identifier and the list of occurrence positions of the retained term unit in the knowledge fragment unit. The inverted index entries of all retained term units are summarized into sparse term retrieval index items corresponding to the knowledge fragment unit, and the sparse term retrieval index items and the knowledge fragment unit identifier form a second mapping record; A consistency binding process is performed on the knowledge fragment unit identifier, and the knowledge fragment unit identifier in the first mapping record is associated with the knowledge fragment unit identifier in the second mapping record for storage, so as to obtain the correspondence between the dense vector retrieval representation and the sparse term retrieval index term that share the same knowledge fragment unit identifier.

3. The hybrid search based enterprise knowledge base augmentation generation method of claim 1, wherein, The process of obtaining the target query request involves simultaneously performing dense vector query representation generation and sparse term query representation generation on the target query request to obtain the dense vector query representation and sparse term query representation corresponding to the target query request, including: Perform query intent sentence recognition processing on the target query request to obtain the query statement sentence type and core content of the query corresponding to the target query request; The core content of the query is input into a pre-constructed dense vector coding network. The core content of the query is then subjected to layer-by-layer semantic compression through the multi-layer semantic transformation structure in the dense vector coding network to obtain the dense vector query representation corresponding to the target query request. In each semantic transformation structure of the dense vector coding network, the semantic attention calculation structure within that semantic transformation structure is used to perform attention weighted aggregation processing on the query term sequence of the core content part of the query, generating the query intermediate semantic convergence vector corresponding to that semantic transformation structure. The query intermediate semantic convergence vector output by the last layer semantic transformation structure of the dense vector coding network is used as the dense vector query representation corresponding to the target query request. The dense vector query representation and the dense vector retrieval representation are in the same dense vector retrieval space. The core content of the query is segmented into multiple original query term units. Each original query term unit is then restored to its original form to obtain the restored query term unit. According to the query statement type, each query restored term unit is processed to perform query intent weight marking, and the query intent weight mark corresponding to each query restored term unit is obtained. All query restored term units carrying query intent weight marks are summarized into the sparse term query representation corresponding to the target query request. Each query restored term unit in the sparse term query representation is in the same term space as the reserved term unit in the sparse term retrieval index.

4. The enterprise knowledge base enhancement generation method based on hybrid retrieval according to claim 1, characterized in that, The first candidate knowledge fragment set is obtained by performing similarity nearest neighbor search processing in the dense vector retrieval space using the dense vector query representation, and the second candidate knowledge fragment set is obtained by performing term matching processing in the sparse inverted index structure using the sparse term query representation. The first and second candidate knowledge fragment sets are then fused and deduplicated to obtain a fused candidate knowledge fragment set carrying both dense vector retrieval scores and sparse term retrieval scores, including: In the dense vector retrieval space, a similarity nearest neighbor search is performed on the dense vector query representation and the dense vector retrieval representation corresponding to all knowledge fragment units to obtain the dense vector retrieval similarity score corresponding to each knowledge fragment unit. All knowledge fragment units are sorted in descending order based on the dense vector retrieval similarity score, and knowledge fragment units whose sorting position is within a preset number of nearest neighbors are selected to form a first candidate knowledge fragment set. Each first candidate knowledge fragment in the first candidate knowledge fragment set carries a corresponding dense vector retrieval similarity score as a dense vector retrieval score. In the sparse inverted index structure, the sparse inverted index structure is traversed according to each query recovery term unit in the sparse term query representation, and all inverted index entries containing any query recovery term unit are recalled. All recalled inverted index entries are grouped and merged according to knowledge fragment unit identifiers to obtain the set of hit query recovery term units corresponding to each recalled knowledge fragment unit. Based on the set of term units restored by the hit query corresponding to each recalled knowledge fragment unit, and combined with the query intent weight tag corresponding to each restored term unit, the sparse term retrieval score is calculated to obtain the sparse term retrieval score corresponding to each recalled knowledge fragment unit. Based on the sparse term retrieval score, all recalled knowledge fragment units are sorted in descending order, and recalled knowledge fragment units whose sorting position is within the preset term hit range are selected to form a second candidate knowledge fragment set. For each first candidate knowledge fragment in the first candidate knowledge fragment set and each second candidate knowledge fragment in the second candidate knowledge fragment set, a knowledge fragment unit identifier matching process is performed. Knowledge fragment units in the first candidate knowledge fragment set that have the same knowledge fragment unit identifier as those in the second candidate knowledge fragment set are marked as overlapping candidate knowledge fragments. A score merging process is performed on the overlapping candidate knowledge fragments. The dense vector retrieval score corresponding to the overlapping candidate knowledge fragments is updated to the original dense vector retrieval score, while the sparse term retrieval score corresponding to the overlapping candidate knowledge fragments is retained. The first candidate knowledge fragments in the first candidate knowledge fragment set that do not overlap with the second candidate knowledge fragment set, the second candidate knowledge fragments in the second candidate knowledge fragment set that do not overlap with the first candidate knowledge fragment set, and the overlapping candidate knowledge fragments after the score merging process are collectively summarized into a fused candidate knowledge fragment set.

5. The enterprise knowledge base enhancement generation method based on hybrid retrieval according to claim 1, characterized in that, The document source grouping and retrieval score redistribution processes are performed on the fusion candidate knowledge fragment set to obtain a redistributed retrieval score set. A target knowledge fragment set is then selected based on the redistributed retrieval score set. An enhanced generation input sequence is generated based on the target knowledge fragment set and the target query request. This enhanced generation input sequence is then input into a pre-constructed enhanced generation network to obtain the enterprise knowledge base enhanced generation result, including: Based on the source attribution mapping relationship carried by each fusion candidate knowledge fragment in the fusion candidate knowledge fragment set, fusion candidate knowledge fragments with the same document source identifier are grouped into the same document source group, resulting in multiple document source groups; Dense vector retrieval score aggregation is performed on all fusion candidate knowledge fragments within each document source group to obtain the dense vector retrieval score aggregation result corresponding to that document source group. Sparse term retrieval score aggregation is performed on all fusion candidate knowledge fragments within each document source group to obtain the sparse term retrieval score aggregation result corresponding to that document source group. Based on the aggregated results of dense vector retrieval scores and sparse term retrieval scores corresponding to each document source group, calculate the source grouping retrieval contribution parameter corresponding to the document source group; For each fusion candidate knowledge fragment within each document source group, perform source grouping and sorting position determination processing to obtain the source grouping and sorting position of each fusion candidate knowledge fragment within its respective document source group; Based on the dense vector retrieval score, sparse term retrieval score, source grouping sorting position, and source grouping retrieval contribution parameter corresponding to the source grouping of the document to which it belongs, a retrieval score redistribution process is performed to obtain the redistributed retrieval score corresponding to each fusion candidate knowledge fragment. The redistributed retrieval scores corresponding to all fusion candidate knowledge fragments are summarized into a redistributed retrieval score set. Based on the redistributed retrieval score set, all fusion candidate knowledge fragments are sorted in descending order, and fusion candidate knowledge fragments whose sorting positions are within a preset target number range are selected to form a target knowledge fragment set. For each target knowledge fragment in the target knowledge fragment set, the fragment content and source identifier are concatenated to obtain the enhanced fragment input unit corresponding to each target knowledge fragment. All enhanced fragment input units and the target query request are combined with the target query request according to a preset input arrangement to form an enhanced generation input sequence. The enhanced generation input sequence is input into a pre-constructed enhanced generation network to obtain the enhanced generation result of the enterprise knowledge base.

6. The enterprise knowledge base enhancement generation method based on hybrid retrieval according to claim 1, characterized in that, The method further includes: The knowledge document set is subjected to document structure parsing processing to obtain the document structure tree corresponding to each knowledge document unit. The document structure tree contains chapter-level nodes and paragraph-level nodes. When performing semantic segmentation processing on each knowledge document unit, the semantic segmentation boundary is determined according to the chapter-level nodes and paragraph-level nodes in the document structure tree, resulting in multiple knowledge fragment units; After obtaining multiple knowledge fragment units, document structure location extension processing is performed on the source attribution mapping relationship corresponding to each knowledge fragment unit, and the source attribution mapping relationship is extended into a source attribution extended mapping relationship that simultaneously includes document source identifier and document structure path identifier; When performing dense vector retrieval encoding processing on each knowledge fragment unit, the document structure path identifier is input as an additional semantic tag into the pre-constructed dense vector encoding network, so that the document structure path semantic information is incorporated into the dense vector retrieval representation; When performing sparse term retrieval index construction processing on each knowledge fragment unit, the document structure path term corresponding to the document structure path identifier is added to the sparse term retrieval index term; When performing similarity nearest neighbor search processing based on the dense vector query representation, the query document structure preference tag corresponding to the target query request is obtained. Based on the query document structure preference tag and the document structure path semantic information incorporated in the dense vector retrieval representation, structure preference weighted nearest neighbor search processing is performed to obtain a first candidate knowledge fragment set. When performing term matching processing based on the sparse term query representation, structure preference term matching processing is performed based on the query document structure preference tag and the document structure path terms in the sparse term retrieval index to obtain a second candidate knowledge fragment set. When performing fusion deduplication on the first candidate knowledge fragment set and the second candidate knowledge fragment set, the source attribution extended mapping relationship is used as an additional matching condition during the fusion deduplication process to obtain a fusion candidate knowledge fragment set carrying the document structure path identifier; When performing document source grouping processing on the set of fused candidate knowledge fragments, grouping processing is performed simultaneously according to the document source identifier and document structure path identifier based on the source attribution extended mapping relationship, resulting in multiple document structure groups. When performing retrieval score redistribution processing on each document structure group, the structure depth parameter corresponding to the document structure path identifier is included in the redistribution process to obtain the redistributed retrieval score corresponding to each fused candidate knowledge fragment.

7. The enterprise knowledge base enhancement generation method based on hybrid retrieval according to claim 1, characterized in that, The method further includes: Perform query intent expansion processing on the target query request to obtain a set of query intent expansion terms corresponding to the target query request. Obtain each query intent expansion term in the set of query intent expansion terms. Perform expansion term morphology restoration processing on each query intent expansion term to obtain the expansion restored term unit corresponding to each query intent expansion term. All extended and restored term units are added to the sparse term query representation, so that the sparse term query representation contains both query restored term units and extended and restored term units. In the sparse inverted index structure, the sparse inverted index structure is traversed according to the extended restored term unit in the sparse term query representation, and all extended inverted index entries containing any extended restored term unit are recalled. All recalled extended inverted index entries are grouped and merged by knowledge fragment unit identifier to obtain the extended hit term unit set corresponding to each extended recalled knowledge fragment unit. Based on the extended hit term unit set corresponding to each extended recalled knowledge fragment unit, and combined with the extended query intent weight tag corresponding to each extended restored term unit, the extended sparse term retrieval score calculation is performed to obtain the extended sparse term retrieval score corresponding to each extended recalled knowledge fragment unit. The knowledge fragment unit identifier matching and merging process is performed between the extended recall knowledge fragment unit and the second candidate knowledge fragment set, and the knowledge fragment units in the extended recall knowledge fragment unit that do not appear in the second candidate knowledge fragment set are added to the second candidate knowledge fragment set. The sparse term retrieval score fusion processing is performed on the supplemented second candidate knowledge fragment set to integrate the extended sparse term retrieval score into the corresponding sparse term retrieval score, thus obtaining the extended second candidate knowledge fragment set. The first candidate knowledge fragment set and the expanded second candidate knowledge fragment set are subjected to fusion and deduplication processing to obtain a fusion candidate knowledge fragment set carrying the expanded sparse term retrieval score. When the fusion candidate knowledge fragment set is subjected to document source grouping processing and retrieval score redistribution processing, the expanded sparse term retrieval score is included in the redistribution process to obtain a redistributed retrieval score set.

8. The enterprise knowledge base enhancement generation method based on hybrid retrieval according to claim 1, characterized in that, The method further includes: Obtain the knowledge document update instruction corresponding to the enterprise knowledge base, wherein the knowledge document update instruction includes the knowledge document unit to be updated and the update operation type; When the update operation type is an add operation, semantic segmentation processing is performed on the knowledge document unit to be updated to obtain the new knowledge fragment unit, and a new source attribution mapping relationship is established for each new knowledge fragment unit. Dense vector retrieval encoding processing and sparse term retrieval index construction processing are performed simultaneously on each new knowledge fragment unit to obtain the new dense vector retrieval representation and new sparse term retrieval index item corresponding to each new knowledge fragment unit. The new dense vector retrieval representation corresponding to each new knowledge fragment unit is inserted into the dense vector retrieval space, and the new sparse term retrieval index item corresponding to each new knowledge fragment unit is inserted into the sparse inverted index structure. When the update operation type is a deletion operation, according to the document source identifier corresponding to the knowledge document unit to be updated, all dense vector retrieval representations belonging to the knowledge document unit to be updated are located and removed in the dense vector retrieval space, and according to the document source identifier corresponding to the knowledge document unit to be updated, all sparse term retrieval index terms belonging to the knowledge document unit to be updated are located and removed in the sparse inverted index structure. When the update operation type is a modification operation, according to the document source identifier corresponding to the knowledge document unit to be updated, all original dense vector retrieval representations belonging to the knowledge document unit to be updated are removed from the dense vector retrieval space. According to the document source identifier corresponding to the knowledge document unit to be updated, all original sparse term retrieval index terms belonging to the knowledge document unit to be updated are removed from the sparse inverted index structure. Semantic segmentation processing is then performed on the modified knowledge document unit to be updated to obtain modified knowledge fragment units. A modification source attribution mapping relationship is established for each modified knowledge fragment unit. For each modified knowledge fragment unit, dense vector retrieval encoding processing and sparse term retrieval index construction processing are performed simultaneously to obtain the modified dense vector retrieval representation and the modified sparse term retrieval index. The modified dense vector retrieval representation is inserted into the dense vector retrieval space, and the modified sparse term retrieval index is inserted into the sparse inverted index structure.

9. The enterprise knowledge base enhancement generation method based on hybrid retrieval according to claim 1, characterized in that, The method further includes: Obtain the fragment access frequency record of each knowledge fragment unit in the knowledge document set corresponding to the enterprise knowledge base. The fragment access frequency record includes the cumulative number of times the knowledge fragment unit has been recalled and used in the historical retrieval process. For each knowledge fragment unit, a fragment access popularity level classification process is performed. The fragment access popularity level corresponding to each knowledge fragment unit is determined according to the cumulative number in the fragment access frequency record. In the dense vector retrieval space, a popularity weighting process is performed on the dense vector retrieval representation corresponding to each knowledge fragment unit. The popularity weighting parameter corresponding to the fragment access popularity level is integrated into the dense vector retrieval representation to obtain the popularity weighted dense vector retrieval representation. In the sparse inverted index structure, a popularity index marking process is performed on the sparse term retrieval index item corresponding to each knowledge fragment unit, and the access popularity level of the fragment is recorded in the sparse term retrieval index item; Obtain the target query request, and simultaneously perform dense vector query representation generation processing and sparse term query representation generation processing on the target query request to obtain the dense vector query representation and sparse term query representation corresponding to the target query request. In the dense vector retrieval space, perform similarity nearest neighbor search processing based on the dense vector query representation and the popularity-weighted dense vector retrieval representation to obtain the first candidate knowledge fragment set carrying the fragment access popularity level. In the sparse inverted index structure, term matching processing is performed based on the sparse term query representation. During the term matching process, the fragment access popularity level recorded in the sparse term retrieval index is used as an additional sorting criterion to obtain a second candidate knowledge fragment set carrying the fragment access popularity level. The first candidate knowledge fragment set and the second candidate knowledge fragment set are then fused and deduplicated to obtain a fused candidate knowledge fragment set carrying the fragment access popularity level. When performing document source grouping and retrieval score redistribution on the fusion candidate knowledge fragment set, the access popularity level of the fragment is included in the redistribution process to obtain a redistributed retrieval score set. The target knowledge fragment set is selected based on the redistributed retrieval score set. An enhanced generation input sequence is generated based on the target knowledge fragment set and the target query request. The enhanced generation input sequence is input into a pre-built enhanced generation network to obtain the enhanced generation result of the enterprise knowledge base.

10. A hybrid retrieval-based enterprise knowledge base enhancement generation system, characterized in that, The method includes a processor and a computer-readable storage medium storing machine-executable instructions that, when executed by the processor, implement the enterprise knowledge base enhancement generation method based on hybrid retrieval as described in any one of claims 1-9.

Images