Training, retrieval enhancement methods, devices, electronic equipment and storage media for context fusion models

By training a multi-objective context fusion model on text block sequences, and utilizing the semantic features of the preceding text and local context features, the accuracy of text block retrieval is improved and the computational complexity is reduced, thus solving the problems of high computational cost and poor representation accuracy in existing technologies.

CN123115347APending Publication Date: 2026-11-17SUZHOU YIZHU INTELLIGENT TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611415873.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-09-11
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

In existing technologies, segmenting long documents into multiple text blocks results in high computational complexity for the model and poor accuracy of contextual representation of feature vectors, which affects the accuracy of text block retrieval.

Method used

By acquiring the initial semantic features of the sample text block sequence, the preceding semantic features and local context semantic features are determined, and a context fusion model is used for multi-objective training, including preceding semantic loss, local context loss and self-semantic preservation loss, to improve the accuracy of context representation while reducing computational overhead.

Benefits of technology

It significantly improves the accuracy of context representation, reduces model computational overhead, enhances robustness to discontinuous and jumpy local contexts, and solves the problems of high computational complexity and poor representation accuracy in existing technologies.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123115347A_ABST
    Figure CN123115347A_ABST
Patent Text Reader

Abstract

This disclosure proposes a training and retrieval enhancement method, apparatus, electronic device, and storage medium for a context fusion model, comprising: inputting the initial semantic features of multiple sample text blocks in a sample text block sequence into the context fusion model to be trained; performing context fusion on the initial semantic features of the current sample text block and the initial semantic features of multiple sample text blocks to obtain the enhanced semantic features of the current sample text block; and training the context fusion model to be trained based on the contextual semantic loss between the enhanced semantic features and the contextual semantic features of the current sample text block, the local contextual loss between the enhanced semantic features and at least one local contextual semantic feature of the current sample text block, and the self-semantic preservation loss between the enhanced semantic features and the initial semantic features of the current sample text block to obtain the context fusion model, which can balance the computational cost of the model and the accuracy of semantic representation.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a method, apparatus, electronic device, and storage medium for training and retrieval enhancement of a context fusion model. Background Technology

[0002] In natural language processing applications such as information retrieval, question answering systems, and document understanding, it is often necessary to convert text into feature vectors (called embeddings) in order to perform similarity calculations and semantic matching.

[0003] In existing technologies, long documents are divided into multiple text chunks. In order to capture the context information of each text chunk in the original text, the context text is usually concatenated with the current text chunk and then input into the embedding model / encoding model to obtain the feature vector of the current text chunk. Since the text length and the model computational complexity have a square relationship, the model computational overhead is high, and the accuracy of the context representation of the generated feature vector is poor, which affects the accuracy of text chunk retrieval. Summary of the Invention

[0004] This disclosure provides a method, apparatus, electronic device, and storage medium for training and retrieval enhancement of a context fusion model, which can significantly improve the accuracy of context representation in the context fusion model while effectively reducing the computational overhead of the model.

[0005] According to one aspect of this disclosure, a method for training a context fusion model is provided, comprising: acquiring a sequence of sample text blocks and initial semantic features of each of a plurality of sample text blocks in the sequence of sample text blocks; for a current sample text block among the plurality of sample text blocks, determining a contextual semantic feature of the current sample text block based on the current sample text block and at least one sample text block preceding the current sample text block; determining at least one local contextual semantic feature of the current sample text block based on the current sample text block and at least one other sample text block selected from the sequence of sample text blocks; and inputting the initial semantic features of each of the plurality of sample text blocks into the context fusion model to be trained. The initial semantic features of the current sample text block are fused with the initial semantic features of each of the plurality of sample text blocks to obtain the enhanced semantic features of the current sample text block; the preceding semantic loss is determined based on the difference between the enhanced semantic features and the preceding semantic features, the local context loss is determined based on the difference between the enhanced semantic features and at least one local context semantic feature, and the self-semantic preservation loss is determined based on the difference between the enhanced semantic features and the initial semantic features of the current sample text block; the context fusion model to be trained is trained based on the preceding semantic loss, the local context loss, and the self-semantic preservation loss to obtain the context fusion model.

[0006] According to one aspect of this disclosure, a retrieval enhancement method is provided, comprising: obtaining initial semantic features of each of multiple text blocks in a text block sequence of a document to be entered into a database; inputting the initial semantic features of each of the multiple text blocks into a context fusion model, performing context fusion of the initial semantic features of the current text block with the initial semantic features of the multiple text blocks to obtain enhanced semantic features of the current text block; and adding the enhanced semantic features of each of the multiple text blocks in the text block sequence to a retrieval knowledge base; wherein the context fusion model is trained based on the training method of the context fusion model described above.

[0007] According to one aspect of this disclosure, a training apparatus for a context fusion model is provided, comprising: a sample acquisition module, configured to acquire a sequence of sample text blocks and initial semantic features of each of a plurality of sample text blocks in the sequence of sample text blocks; a context semantic determination module, configured to determine, for a current sample text block among the plurality of sample text blocks, the context semantic features of the current sample text block based on the current sample text block and at least one sample text block preceding the current sample text block; a local context semantic determination module, configured to determine at least one local context semantic feature of the current sample text block based on the current sample text block and at least one other sample text block selected from the sequence of sample text blocks; and a context fusion module, configured to fuse the initial semantic features of each of the plurality of sample text blocks. The system inputs a context fusion model to be trained, performs context fusion on the initial semantic features of the current sample text block and the initial semantic features of the multiple sample text blocks to obtain the enhanced semantic features of the current sample text block; a loss determination module is used to determine the context semantic loss based on the difference between the enhanced semantic features and the context semantic features, determine the local context loss based on the difference between the enhanced semantic features and at least one local context semantic feature, and determine the self-semantic preservation loss based on the difference between the enhanced semantic features and the initial semantic features of the current sample text block; a model training module is used to train the context fusion model to be trained based on the context semantic loss, the local context loss and the self-semantic preservation loss to obtain the context fusion model.

[0008] According to one aspect of this disclosure, a retrieval enhancement apparatus is provided, comprising: a feature acquisition module, configured to acquire initial semantic features of each of multiple text blocks in a text block sequence of a document to be entered into a database; a semantic enhancement module, configured to input the initial semantic features of each of the multiple text blocks into a context fusion model, and perform context fusion of the initial semantic features of the current text block with the initial semantic features of the multiple text blocks to obtain enhanced semantic features of the current text block; and a document entry module, configured to add the enhanced semantic features of each of the multiple text blocks in the text block sequence to a retrieval knowledge base; wherein the context fusion model is trained based on the training method of the context fusion model described above.

[0009] According to one aspect of this disclosure, an electronic device is proposed, the electronic device including a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for implementing connection communication between the processor and the memory, wherein the program is executed by the processor to implement the training method of the context fusion model as described above or the retrieval enhancement method as described above.

[0010] According to one aspect of this disclosure, a computer-readable storage medium is proposed that stores one or more programs, which can be executed by one or more processors to implement the training method of the context fusion model as described above or the retrieval enhancement method as described above.

[0011] This disclosure proposes a training and retrieval enhancement method, apparatus, electronic device, and storage medium for a context fusion model. The method includes: acquiring a sequence of sample text blocks and initial semantic features of multiple sample text blocks within the sequence; for the current sample text block among the multiple sample text blocks, determining the contextual semantic features of the current sample text block based on the current sample text block and at least one sample text block preceding the current sample text block; determining at least one local contextual semantic feature of the current sample text block based on the current sample text block and at least one other sample text block selected from the sequence of sample text blocks; and inputting the initial semantic features of the multiple sample text blocks into the training and retrieval enhancement method. A context fusion model is trained by fusing the initial semantic features of the current sample text block with the initial semantic features of multiple sample text blocks to obtain enhanced semantic features of the current sample text block. The contextual semantic loss is determined based on the difference between the enhanced semantic features and the preceding context semantic features; the local context loss is determined based on the difference between the enhanced semantic features and at least one local context semantic feature; and the self-semantic preservation loss is determined based on the difference between the enhanced semantic features and the initial semantic features of the current sample text block. The context fusion model is then trained based on the preceding context semantic loss, local context loss, and self-semantic preservation loss to obtain the context fusion model. This disclosure performs multi-objective training on the context fusion model based on the preceding context semantic loss, local context loss, and self-semantic preservation loss of each sample text block in the sample text block sequence, driving the model to learn to adaptively fuse contextual information and improve the accuracy of context representation. Furthermore, compared to performing context fusion after concatenating the original text, performing context fusion based on the initial semantic features of multiple text blocks can significantly reduce the model's computational overhead.

[0012] Furthermore, by acquiring the text content corresponding to the current sample text block and at least some sample text blocks preceding the current sample text block, the text content is summarized to obtain the context summary text, and semantic features are extracted from the context summary text to obtain the context semantic features as a global supervision signal, thereby training the global context alignment capability of the context fusion model.

[0013] Furthermore, at least one local context text is constructed, wherein, for each local context text, if a sample text block exists before the current sample text block, at least one first sample text block is selected from the sample text blocks before the current sample text block; if a sample text block exists after the current sample text block, at least one second sample text block is selected from the sample text blocks after the current sample text block. The current sample text block and the selected sample text blocks are combined according to their respective arrangement order in the sample text block sequence to obtain the corresponding local context text, and semantic features are extracted from at least one local context text to obtain at least one local context semantic feature as a local supervision signal, which can enhance the robustness of the model to discontinuous and jumpy local contexts.

[0014] Furthermore, if there are no sample text blocks before the current sample text block, at least one second sample text block can be selected from the sample text blocks following the current sample text block, and the selected at least one second sample text block can be combined with the current sample text block in the order of arrangement to obtain the corresponding local context text. If there are no sample text blocks after the current sample text block, at least one first sample text block can be selected from the sample text blocks preceding the current sample text block, and the selected at least one first sample text block can be combined with the current sample text block in the order of arrangement to obtain the corresponding local context text. This ensures that sample text blocks at the beginning and end of the sequence can also obtain local context supervision, improving the completeness of training coverage.

[0015] Furthermore, the context fusion model to be trained includes a context feature aggregation network layer and a feature fusion network layer. Through the context feature aggregation network layer, the semantic relevance weights of each of the multiple sample text blocks are determined based on the semantic relevance between the initial semantic features of the current sample text block and the initial semantic features of each of the multiple sample text blocks. Based on the semantic relevance weights, the initial semantic features of each of the multiple sample text blocks are aggregated to obtain context aggregation features that can reflect the semantic role of the current sample text block in the entire document. Through the feature fusion network layer, the initial semantic features and context aggregation features of the current sample text block are fused to obtain enhanced semantic features, thus solving semantic ambiguity problems such as referential resolution and omission recovery.

[0016] Furthermore, based on the initial semantic features of the current sample text block and its corresponding context aggregation features, a learnable gating function is used to dynamically calculate the fusion weights that indicate how much original semantics should be retained and how much contextual information should be incorporated into the final output. Finally, the initial semantic features and context aggregation features are weighted and summed according to these weights to generate the enhanced semantic features of the current sample text block. This process ensures that the model can adaptively adjust the information fusion strategy for text blocks with different characteristics (such as independent terms or context-dependent pronouns).

[0017] Furthermore, based on the initial semantic features of the current sample text block and its corresponding context aggregation features, and based on the multi-dimensional gating weights corresponding to the initial semantic features of the current sample text block, multi-dimensional feature selection is performed on the context aggregation features to obtain multiple context selection features. The multiple context selection features and the current sample text block mapping features corresponding to the initial semantic features are then fused to obtain enhanced semantic features. Through multi-dimensional feature fusion, the comprehensiveness of the model's context feature fusion is improved.

[0018] Other features and advantages of this disclosure will be set forth in the following description and will be apparent in part from the description or may be learned by practicing the disclosure. The objectives and other advantages of this disclosure may be realized and obtained by means of the structures particularly pointed out in the description, claims and drawings. Attached Figure Description

[0019] The accompanying drawings are provided to further understand the technical solutions of this disclosure and constitute a part of the specification. They are used together with the embodiments of this disclosure to explain the technical solutions of this disclosure and do not constitute a limitation on the technical solutions of this disclosure.

[0020] Figure 1 This is an architecture diagram of the training method for the context fusion model applied in the embodiments of this disclosure;

[0021] Figure 2 This is a flowchart of a training method for a context fusion model according to an embodiment of the present disclosure;

[0022] Figure 3 This is a flowchart of a training method for a context fusion model according to another embodiment of the present disclosure;

[0023] Figure 4 This is a flowchart of a training method for a context fusion model according to another embodiment of the present disclosure;

[0024] Figure 5a yes Figure 4 A sub-flowchart of step S403;

[0025] Figure 5bThis is a schematic diagram of the structure of a feature fusion network layer proposed in one embodiment of this disclosure;

[0026] Figure 6a yes Figure 4 Another sub-flowchart of step S403;

[0027] Figure 6b This is a schematic diagram of the structure of a feature fusion network layer proposed in another embodiment of this disclosure;

[0028] Figure 7 This is a flowchart of a training method for a context fusion model according to another embodiment of the present disclosure;

[0029] Figure 8a This is a flowchart of a retrieval enhancement method according to an embodiment of this disclosure;

[0030] Figure 8b This is a flowchart of a retrieval enhancement method according to another embodiment of this disclosure;

[0031] Figure 9a This is a schematic diagram of a context fusion model proposed in one embodiment of this disclosure;

[0032] Figure 9b This is a schematic diagram of a context fusion model proposed in another embodiment of this disclosure;

[0033] Figure 10 This is a schematic diagram of the structure of a training device for a context fusion model proposed in one embodiment of this disclosure;

[0034] Figure 11 This is a schematic diagram of the structure of a retrieval enhancement device proposed in one embodiment of the present disclosure;

[0035] Figure 12 This is a schematic diagram of the structure of an electronic device proposed in one embodiment of the present disclosure. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of this disclosure clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the scope of this disclosure.

[0037] Before providing a further detailed description of the embodiments of this disclosure, the terms and concepts used in these embodiments are explained, and they are subject to the following interpretations:

[0038] Token: A token is the smallest semantic unit used in large-scale model language processing. It represents the basic input unit formed after the original input text has been segmented. A token can be a character, a word, or a phrase, and it is the basic unit for constructing sequences, generating vector representations, and performing attention calculations.

[0039] Query Vector: A query vector is a vector representation generated from the current lexical unit, used in self-attention mechanisms to proactively request attention from other lexical units. This vector is obtained through the model's embedding layer and linear transformations, representing the semantic direction of the current lexical unit.

[0040] Key vector: A key vector is a vector representation generated for each word, used for matching and accessed by query vectors of other words. In the attention mechanism, the degree of attention the current word pays to other words is determined by calculating the similarity between the query vector and each key vector.

[0041] Value Vector: The value vector is a semantic representation of each lexical unit, used for weighted extraction in the attention mechanism. The attention mechanism weights and aggregates the value vectors based on the attention scores calculated from the query vector and key vector to generate the final contextual representation.

[0042] Attention Score: The attention score is a measure of the similarity between the query vector and the key vector of the target term, typically calculated using a dot product. This score reflects the degree of attention one term pays to another; a higher score indicates a stronger association. After normalization, the attention score is used to extract the corresponding value vector using weighted averages.

[0043] In existing technologies, long documents are divided into multiple text chunks. In order to capture the context information of each text chunk in the original text, the context text is usually concatenated with the current text chunk and then input into the embedding model / encoding model to obtain the feature vector of the current text chunk. Since the text length and the model computational complexity have a square relationship, the model computational overhead is high, and the accuracy of the context representation of the generated feature vector is poor, which affects the accuracy of text chunk retrieval.

[0044] Based on this, this disclosure proposes a training and retrieval enhancement method, device, electronic device, and storage medium for a context fusion model. Based on the contextual semantic loss, local context loss, and self-semantic preservation loss of each sample text block in the sample text block sequence, the context fusion model to be trained is subjected to multi-objective training, driving the model to learn to adaptively fuse contextual information and improve the accuracy of context representation. In addition, compared with context fusion after splicing the original text, context fusion based on the initial semantic features of multiple text blocks can significantly reduce the computational cost of the model.

[0045] System architecture description applied in the embodiments of this disclosure

[0046] Figure 1 This is a system architecture diagram of the training method of the context fusion model in this embodiment of the present disclosure, which includes: object terminal 110, Internet 120, gateway 130, and server 140.

[0047] The object terminal 110 is a device used by the object to input a sequence of sample text blocks. It can take various forms, including desktop computers, laptops, PDAs (personal digital assistants), mobile phones, in-vehicle terminals, home theater terminals, and dedicated terminals. Furthermore, it can be a single device or a collection of multiple devices. For example, multiple devices can be connected via a local area network, sharing a single display device to work collaboratively, forming a single terminal. The object terminal 110 can also communicate with the Internet 120 via wired or wireless means to exchange data.

[0048] Gateway 130, also known as an internetwork connector or protocol converter, is a computer system or device that enables network interconnection at the transport layer and acts as a translator. It bridges the gap between two systems using different communication protocols, data formats, languages, or even completely different architectures. Gateway 130 also provides filtering and security functions. Messages sent from target terminal 110 to server 140 are forwarded to the corresponding server 140 via gateway 130. Messages sent from server 140 to target terminal 110 are also forwarded to the corresponding target terminal 110 via gateway 130.

[0049] Server 140 refers to a computer system that provides services to target terminal 110. Compared to target terminal 110, server 140 has higher requirements in terms of stability, security, and performance. Server 140 can be a single high-performance computer in a network platform, a cluster of multiple high-performance computers, a portion of a single high-performance computer (e.g., a virtual machine), or a combination of portions of multiple high-performance computers (e.g., virtual machines). Server 140 can also communicate with the Internet 120 via wired or wireless means to exchange data.

[0050] Overall Implementation of the Training Method for the Context Fusion Model in the Embodiments of this Disclosure

[0051] This disclosure proposes a training method for a context fusion model, applied to a training device for a context fusion model, with reference to... Figure 2 The training method for the context fusion model includes:

[0052] Step S210: Obtain the sample text block sequence and the initial semantic features of each sample text block in the sample text block sequence;

[0053] Step S220: For the current sample text block among multiple sample text blocks, determine the contextual semantic features of the current sample text block based on the current sample text block and at least one sample text block preceding the current sample text block.

[0054] Step S230: Determine at least one local contextual semantic feature of the current sample text block based on the current sample text block and at least one other sample text block selected from the sample text block sequence.

[0055] Step S240: Input the initial semantic features of each of the multiple sample text blocks into the context fusion model to be trained, and perform context fusion on the initial semantic features of the current sample text block and the initial semantic features of each of the multiple sample text blocks to obtain the enhanced semantic features of the current sample text block.

[0056] Step S250: Determine the contextual semantic loss based on the difference between the enhanced semantic features and the contextual semantic features, determine the local contextual loss based on the difference between the enhanced semantic features and at least one local contextual semantic feature, and determine the semantic preservation loss based on the difference between the enhanced semantic features and the initial semantic features of the current sample text block.

[0057] Step S260: Based on the semantic loss, local context loss and self-semantic preservation loss, train the context fusion model to be trained to obtain the context fusion model.

[0058] In step S210, the sequence of sample text blocks can be multiple consecutive text blocks obtained by semantic segmentation of the sample document. Specifically, the initial semantic features of each sample text block are used to characterize the semantic features within each sample text block itself, that is, the semantic features that have not undergone fusion processing.

[0059] Specifically, in one embodiment, such as Figure 3 As shown, step S210 may include the following steps:

[0060] In step S301, the sample document is segmented to obtain a sequence of sample text blocks arranged in order of their position in the sample document;

[0061] In step S302, semantic features are extracted from multiple sample text blocks using a semantic feature extraction model to obtain the initial semantic features of each sample text block.

[0062] In one specific embodiment, the initial semantic features of each sample text block can be obtained by encoding each sample text block using a pre-trained general vectorization model, and the initial semantic features can be represented as feature vectors. Illustratively, the general vectorization model can be any existing artificial intelligence model with text encoding capabilities; this disclosure does not impose any particular limitation on it. For example, the general vectorization model can adopt a BGE (Bidirectional Guided Embedding) model.

[0063] In step S220, the current sample text block can be the sample text block that is currently being fused with context among multiple sample text blocks in the sample text block sequence. The contextual semantic features of the current sample text block are used to characterize the global contextual features of the current sample text block at its position in the sample text block sequence.

[0064] Specifically, see Figure 3 In one embodiment, step S220 may include the following steps:

[0065] In step S303, the text content corresponding to the current sample text block and at least some sample text blocks preceding the current sample text block is obtained;

[0066] In step S304, the text content is summarized to obtain the above summary text;

[0067] In step S305, semantic features are extracted from the above summary text to obtain the semantic features of the above text.

[0068] Specifically, at least some sample text blocks preceding the current sample text block can be all sample text blocks preceding the current sample text block.

[0069] Specifically, the preceding summary text is used to summarize the text content of the current sample text block and at least some sample text blocks preceding the current sample text block. The preceding summary text can be obtained by a text summarization model after summarizing the text content. Illustratively, the text summarization model can be any artificial intelligence model with text summarization capabilities in the existing technology, such as the Large Model Model (LLM).

[0070] It should be noted that when there are no sample text blocks before the current sample text block, such as when the current sample text block is the first sample text block in the sequence of sample text blocks, the contextual semantic features can be determined solely based on the text content corresponding to the current sample text block. For example, the text content of the current sample text block itself can be summarized and semantic features extracted, and the resulting semantic features can be determined as the contextual semantic features of the current sample text block. Alternatively, the determination of the contextual semantic loss for the current sample text block can be omitted, that is, the context fusion model can be trained solely based on the local context loss and its own semantic preservation loss.

[0071] Specifically, the semantic features of the preceding text can be obtained by extracting semantic features from the preceding text using a pre-trained general vectorization model, and the semantic features of the preceding text can be represented as feature vectors.

[0072] By summarizing at least a portion of the sample text blocks preceding the current sample text block and the current sample text block, a context summary text is obtained. Semantic features are then extracted from the context summary text to obtain context semantic features as a global supervision signal, thereby training the global context alignment capability of the context fusion model.

[0073] In step S230, each local contextual semantic feature of the current sample text block is used to characterize the semantic features of the current sample text block under a local context window in the sample text block sequence.

[0074] Specifically, see Figure 3 In one embodiment, step S230 may include the following steps:

[0075] In step S306, at least one local context text is constructed, wherein, for each local context text, at least one first sample text block is selected from the sample text blocks preceding the current sample text block, and at least one second sample text block is selected from the sample text blocks following the current sample text block; the at least one first sample text block, the current sample text block, and the at least one second sample text block are combined according to their respective order of arrangement in the sample text block sequence to obtain the corresponding local context text;

[0076] In step S307, semantic features are extracted from at least one local context text to obtain at least one local context semantic feature that corresponds one-to-one with at least one local context text.

[0077] Specifically, at least one of the first sample text block and at least one second sample text block includes a sample text block that is not adjacent to the current sample text block.

[0078] It should be noted that the current sample text block may be located at the beginning or end of the sample text block sequence. In this case, there may be no sample text blocks available for selection before or after the current sample text block. In response to this situation, the construction method of the local context text can be adaptively adjusted.

[0079] Specifically, when there are no sample text blocks before the current sample text block, for example, when the current sample text block is the first sample text block in the sample text block sequence, for each local context text, at least one second sample text block can be selected from the sample text blocks located after the current sample text block, and the at least one second sample text block can be combined with the current sample text block according to their respective order in the sample text block sequence to obtain the corresponding local context text; at this time, the local context text does not include the first sample text block.

[0080] Similarly, when there are no sample text blocks after the current sample text block, for example, when the current sample text block is the last sample text block in the sample text block sequence, for each local context text, at least one first sample text block can be selected from the sample text blocks located before the current sample text block, and the at least one first sample text block can be combined with the current sample text block according to their respective order in the sample text block sequence to obtain the corresponding local context text; in this case, the local context text does not include the second sample text block.

[0081] Therefore, sample text blocks located at the beginning or end of the sample text block sequence can also construct local context text and obtain local context supervision signals, avoiding the situation where the first and last sample text blocks cannot participate in training due to the lack of selectable sample text blocks on one side. This ensures the training coverage of sample text blocks at different positions in the sample document, which is conducive to the context fusion model learning the context representation of sample text blocks at each position in a balanced manner.

[0082] Optionally, if there are no sample text blocks before or after the current sample text block, it is not necessary to construct local context text for the current sample text block, and the determination of local context loss for the current sample text block is omitted accordingly. That is, the context fusion model is trained only based on the semantic loss of the preceding text and its own semantic preservation loss.

[0083] Optionally, if there are no other sample text blocks in the sample text block sequence besides the current sample text block, for example, if the sample text block sequence includes only one sample text block, the current sample text block itself can be determined as the local context text; or, the sample document that includes only one sample text block can be excluded from the sample set used for training.

[0084] It should be noted that the semantic features above and the local contextual semantic features are only used to provide supervision signals during the training phase. During the retrieval enhancement phase, only the initial semantic features of each text block in the text block sequence need to be input into the trained contextual fusion model to obtain their respective enhanced semantic features. Whether the current text block is at the beginning or end of the text block sequence does not affect the inference of the contextual fusion model.

[0085] Specifically, the local context semantic features corresponding to each local context text can be obtained by extracting semantic features from the corresponding local context text using a pre-trained general vectorization model, and the representation of the local context semantic features can be a feature vector.

[0086] Optionally, at least one local context text includes multiple candidate local context texts; at least one of the first sample text block and the second sample text block included in different candidate local context texts is different; accordingly, semantic features are extracted from at least one local context text to obtain at least one local context semantic feature corresponding to at least one local context text, including: semantic features are extracted from multiple candidate local context texts to obtain multiple candidate local context semantic features corresponding to multiple candidate local context texts.

[0087] By constructing at least one local context text and extracting semantic features from each local context text, the local context semantic features can be obtained as local supervision signals, which can enhance the robustness of the model to discontinuous and jumpy local contexts.

[0088] In step S240, the enhanced semantic features of the current sample text block can be used to characterize the context fusion features between the initial semantic features of the current sample text block and the initial semantic features of multiple sample text blocks.

[0089] Specifically, see Figure 4 In one embodiment, the context fusion model to be trained includes: a context feature aggregation network layer and a feature fusion network layer, and step S240 may include the following steps:

[0090] In step S401, the context feature aggregation network layer determines the semantic relevance weights of each of the multiple sample text blocks based on the semantic relevance between the initial semantic features of the current sample text block and the initial semantic features of each of the multiple sample text blocks.

[0091] In step S402, the initial semantic features of multiple sample text blocks are aggregated according to the semantic relevance weights to obtain the context aggregation features of the current sample text block.

[0092] In step S403, the initial semantic features and context aggregation features of the current sample text block are fused through the feature fusion network layer to obtain enhanced semantic features.

[0093] In step S401, the context aggregation feature can be a semantic feature obtained by aggregating the initial semantic features of multiple sample text blocks based on the semantic relevance between multiple sample text blocks and the current sample text block. The context aggregation feature can be represented as a feature vector. The semantic relevance between each sample text block and the current sample text block is used to characterize the degree of semantic association between the initial semantic features of each sample text block and the initial semantic features of the current sample text block.

[0094] Specifically, in one embodiment, step S401 may include the following steps:

[0095] Perform a first linear transformation on the initial semantic features of the current sample text block to obtain the query vector;

[0096] The initial semantic features of multiple sample text blocks are subjected to a second linear transformation and a third linear transformation respectively to obtain multiple key vectors and multiple value vectors.

[0097] Based on the similarity between the query vector and multiple key vectors, determine the semantic relevance weights of each of the multiple sample text blocks;

[0098] Step S402 may include the following steps:

[0099] Multiple value vectors are weighted and aggregated based on semantic relevance weights to obtain context-aggregated features.

[0100] In one embodiment, the context feature aggregation network layer can be an attention fusion layer. Accordingly, the semantic relevance between each sample text block and the current sample text block can be an attention score. Further, attention mapping is performed on the initial semantic features of multiple sample text blocks to obtain the query vector corresponding to the current sample text block, the key vector corresponding to each of the multiple sample text blocks, and the value vector corresponding to each of the multiple sample text blocks. The query vector corresponding to the current sample text block and the key vector corresponding to each of the multiple sample text blocks are respectively multiplied by a dot product to obtain the attention score of each of the multiple sample text blocks. The attention score is normalized by the Softmax function to obtain the attention weight of each of the multiple sample text blocks. Then, the value vectors corresponding to the multiple sample text blocks are weighted and summed based on the attention weights of the multiple sample text blocks to obtain the context aggregation feature.

[0101] For example, let q be the initial semantic features of the current sample text block. i∈Rd, the initial semantic features of multiple sample text blocks in the sample text block sequence are X=[x0,x1,...,x n ]∈Rn×d, the embedding dimension d is usually set to 768 or 1024.

[0102] Through a learnable weight matrix W Q The initial semantic features q of the current sample text block i Mapping to the query vector space, we get the query vector Query=q i W Q Through a learnable weight matrix W K The initial semantic features of multiple sample text blocks are mapped to the key vector space to obtain the key vector Key=XW. K Through a learnable weight matrix W V The initial semantic features of each of the multiple sample text blocks are mapped to the value vector space to obtain the value vector Value=XW. V All linear layer weights are initialized using standard Xavier uniform initialization.

[0103] Calculate the dot product similarity between the query and all keys: Score = QK T / sqrt(d k ), where d k The dimension of the Key is defined; attention weights A∈R1×n are obtained through the Softmax normalization function: A=Softmax(Score), and the attention weights are used to perform a weighted summation of all Value vectors to generate a contextual aggregation feature v that incorporates the global context. fusion =A×V.

[0104] By using a self-attention mechanism, the association weight between the initial semantic features of the current sample text block and the initial semantic features of all sample text blocks in the sample document is calculated. Based on this, the vector information of all sample text blocks is weighted and summarized, thereby generating a context vector for the current sample text block that reflects its semantic role in the entire document.

[0105] Specifically, in an optional embodiment, such as Figure 5a As shown, step S403 may include the following steps:

[0106] In step S501, a first gating weight is determined based on the initial semantic features of the current sample text block, and feature selection is performed on the initial semantic features of the current sample text block based on the first gating weight to obtain the first selected feature.

[0107] Specifically, the first selected feature can be used to characterize the retained features in the initial semantic features of the current sample text block. The first selected feature can be the retained feature obtained after selecting features from the initial semantic features of the current sample text block based on the first gating weight. The first selected feature is represented as a feature vector.

[0108] For example, the first selection feature is chosen. local =g local ⊙q i (⊙ indicates element-wise multiplication), g local =Sigmoid(q i ).

[0109] In step S502, a second gating weight is determined based on the context aggregation features, and feature selection is performed on the context aggregation features based on the second gating weight to obtain the second selected features.

[0110] Specifically, the second selection feature can be used to characterize the features introduced into the current sample text block from the context aggregation features. The second selection feature can be the introduced feature obtained after selecting the context aggregation features based on the second gating weight corresponding to the context aggregation features. The second selection feature is represented as a feature vector.

[0111] For example, the second selection feature choose glocal =g glocal ⊙q i (⊙ indicates element-wise multiplication), g glocal =Sigmoid(v fusion ).

[0112] In step S503, the first selected feature and the second selected feature are fused to obtain an intermediate fused feature.

[0113] Specifically, the intermediate fusion feature can be a fusion of the first and second selected features, and the intermediate fusion feature is represented as a feature vector. For example, the intermediate fusion feature is obtained by adding the first and second selected features.

[0114] In step S504, the intermediate fusion features are subjected to a nonlinear transformation to obtain enhanced semantic features.

[0115] Specifically, enhanced semantic features can be obtained by performing a nonlinear transformation on intermediate fused features, and the enhanced semantic features are represented as feature vectors.

[0116] Specifically, see Figure 5bThe feature fusion network layer may include: a first gating layer, a second gating layer, a first fusion layer, and a first nonlinear transformation layer. Further, the first gating layer determines a first gating weight based on the initial semantic features of the current sample text block, and performs feature selection on the initial semantic features of the current sample text block according to the first gating weight to obtain a first selected feature; the second gating layer determines a second gating weight based on the context aggregation features, and performs feature selection on the context aggregation features according to the second gating weight to obtain a second selected feature; the first fusion layer fuses the first selected feature and the second selected feature to obtain an intermediate fused feature; the first nonlinear transformation layer performs a nonlinear transformation on the intermediate fused feature to obtain enhanced semantic features. For example, the first and second gating layers may be Sigmoid gating layers, and the first nonlinear transformation layer may be an FFN (feedforward network) layer. FFN typically consists of two fully connected layers, with the intermediate dimension typically increased by a factor of 4 (i.e., dff = 4d), and the activation function used is GELU or ReLU.

[0117] The feature fusion network layer provided in this embodiment dynamically calculates the fusion weights (i.e., the gate value) based on the initial semantic features of the current sample text block and its corresponding context aggregation features through a learnable gating function. This gate value is automatically learned by the model and is used to indicate how much original semantics should be retained and how much contextual information should be incorporated in the final output. Finally, the initial semantic features and context aggregation features of the current sample text block are weighted and summed according to this weight to generate the enhanced semantic features of the current sample text block. This process ensures that the model can adaptively adjust the information fusion strategy for text blocks with different characteristics (such as independent terms or context-dependent pronouns).

[0118] Specifically, in another alternative embodiment, such as Figure 6a As shown, step S403 further includes the following steps:

[0119] In step S601, the context aggregation features are subjected to multi-dimensional feature mapping to obtain multiple context sub-features.

[0120] Specifically, multiple context selection features can be mapping features obtained by performing multi-dimensional feature mapping on context aggregation features, and the context selection features are represented as feature vectors.

[0121] In step S602, multiple gate weights are determined based on the initial semantic features of the current sample text block, and the multiple gate weights correspond one-to-one with multiple context sub-features.

[0122] Specifically, the multiple gating weights can be multiple feature selection weights obtained after feature activation of the initial semantic features of the current sample text block. For example, the feature activation function can be the sigmoid function.

[0123] In step S603, feature selection is performed on the corresponding context sub-features according to multiple gating weights to obtain multiple context selection features.

[0124] Specifically, multiple context selection features can be introduced features obtained by performing multi-dimensional feature selection on context aggregation features, and the context selection features are represented as feature vectors.

[0125] In step S604, feature mapping is performed on the initial semantic features of the current sample text block to obtain the mapped features of the current sample text block.

[0126] Specifically, the current sample text block mapping feature can be obtained by mapping the initial semantic features of the current sample text block, and the current sample text block mapping feature is represented as a feature vector.

[0127] In step S605, multiple context selection features and the current sample text block mapping features are fused to obtain enhanced semantic features.

[0128] Specifically, enhanced semantic features can be fused features of multiple context selection features and current sample text block mapping features, with the intermediate fused features represented as feature vectors.

[0129] Specifically, see Figure 6b The feature fusion network layer may include: N first feature mapping layers, N gate weight layers corresponding one-to-one with the N first feature mapping layers, N feature selection layers corresponding one-to-one with the N first feature mapping layers, a second feature mapping layer, and a second fusion layer. Further, each first feature mapping layer is used to perform linear feature mapping on the context aggregation features to obtain a context sub-feature; each gate weight layer is used to perform feature activation on the initial semantic features of the current sample text block to obtain a gate weight; each feature selection layer is used to perform feature selection on the corresponding context sub-feature according to the corresponding gate weight to obtain a context selection feature; the second feature mapping layer is used to perform feature mapping on the initial semantic features of the current sample text block to obtain the current sample text block mapping feature; the second fusion layer is used to fuse the N context selection features output by the N feature selection layers and the current sample text block mapping feature to obtain enhanced semantic features. For example, the N first feature mapping layers can be N linear mapping layers, the N gated weight layers can be N sigmoid gated layers, the second feature mapping layer can be a linear mapping layer, the second fusion layer can be a nonlinear transformation layer (e.g., an FFN layer), and N can be set to 3.

[0130] The feature fusion network layer provided in this embodiment performs multi-dimensional feature selection on the context aggregation features based on the multi-dimensional gating weights corresponding to the initial semantic features of the current sample text block, resulting in multiple context selection features. These multiple context selection features and the current sample text block mapping features corresponding to the initial semantic features are then fused to obtain enhanced semantic features. Through multi-dimensional feature fusion, the comprehensiveness of the model's context feature fusion is improved.

[0131] Specifically, to train the context fusion model to be trained, a multi-task training objective needs to be constructed, including: a self-information preservation task that makes the enhanced semantic features of the current sample text block as close as possible to its initial semantic features; a global context alignment task that aligns the enhanced semantic features with the contextual semantic features representing the historical contextual information of the text block; and a local context adaptation task that makes the enhanced semantic features robust to non-continuous local contextual text. By optimizing this multi-task objective, the model is trained to simultaneously master the above three capabilities. In one embodiment, step S250 includes the following steps:

[0132] In step S2501 (not shown), the semantic loss of the preceding text is determined based on the difference between the enhanced semantic features and the semantic features of the preceding text;

[0133] In step S2502 (not shown), the local context loss is determined based on the difference between the enhanced semantic features and at least one local context semantic feature;

[0134] In step S2503 (not shown), the semantic preservation loss is determined based on the difference between the enhanced semantic features and the initial semantic features of the current sample text block.

[0135] In step S2501, the contextual semantic loss is used to characterize the difference between the enhanced semantic features and the contextual semantic features. In a specific embodiment, the contextual semantic loss between the enhanced semantic features and the contextual semantic features can be determined based on a preset loss function. Optionally, the preset loss function is a cosine distance loss function.

[0136] For example, the current sample text block is the i-th text block in the sample text block sequence, i.e., chunk_i. The enhanced semantic features of the current sample text block are represented as y_i, the context semantic features of the current sample text block are represented as Yi_a, and the context semantic loss L_global is represented as: L_global = 1 - cosine_similarity(y_i, Yi_a).

[0137] In step S2502, the local context loss is used to characterize the difference between the enhanced semantic features and the local context semantic features. In a specific embodiment, the local context loss between the enhanced semantic features and the local context semantic features can be determined based on a preset loss function. Optionally, the preset loss function is a cosine distance loss function.

[0138] For example, the current sample text block is the i-th text block in the sample text block sequence, i.e., chunk_i. The enhanced semantic features of the current sample text block are represented as y_i, the local context semantic features of the current sample text block are represented as Yi_b, and the context semantic loss L_global is represented as: L_global = 1 - cosine_similarity(y_i, Yi_b).

[0139] Specifically, in one embodiment, at least one local context text includes multiple candidate local context texts, and at least one of the first sample text block and the second sample text block included in the different candidate local context texts is different. At least one local context semantic feature includes: multiple candidate local context semantic features that correspond one-to-one with the multiple candidate local context texts. Step S2502 includes the following steps:

[0140] Multiple candidate local context losses are determined based on the differences between the enhanced semantic features and the semantic features of multiple candidate local contexts.

[0141] The local context loss is obtained by averaging or weighting multiple candidate local context losses.

[0142] Specifically, each candidate context text in the plurality of candidate context texts can consist of a first sample text block, a current sample text block, and a second sample text block. The first sample text block is the sample text block preceding the current sample text block in the sample text block sequence, and the second sample text block is the sample text block following the current sample text block in the sample text block sequence. Any two candidate context texts in the plurality of candidate context texts differ in at least one of the following: either the first sample text block or the second sample text block. Optionally, the number of candidate context texts can be set to three.

[0143] Specifically, the candidate local context loss of each candidate context text is used to characterize the difference between the candidate local context semantic features and the enhanced semantic features of each candidate context text. The local context loss can be the loss information obtained by fusing the candidate local context losses of multiple candidate context texts. Optionally, the fusion process here can be the mean processing.

[0144] By determining the candidate local context loss for each candidate context text based on the candidate local context semantic features and enhanced semantic features of each candidate context text in multiple candidate context texts, and fusing the candidate local context losses of multiple candidate context texts to obtain the local context loss, the local context robustness of the context fusion model is further enhanced.

[0145] In step S2503, the self-semantic preservation loss is used to characterize the difference between the enhanced semantic features and the initial semantic features of the current sample text block. In a specific embodiment, the self-semantic preservation loss between the enhanced semantic features and the initial semantic features of the current sample text block can be determined based on a preset loss function. Optionally, the preset loss function is the cosine distance loss function.

[0146] For example, the current sample text block is the i-th text block in the sample text block sequence, i.e., chunk_i. The enhanced semantic features of the current sample text block are represented as y_i, the initial semantic features of the current sample text block are represented as Yi_c, and the self-semantic preservation loss L_self is represented as: L_self = 1 - cosine_similarity(y_i, Yi_c).

[0147] Specifically, in one embodiment, such as Figure 7 As shown, step S260 includes the following steps:

[0148] In step S701, corresponding loss weights are set for the above semantic loss, local context loss, and self-semantic preservation loss, respectively;

[0149] In step S702, the semantic loss of the preceding text, the local context loss, and the semantic preservation loss of the text itself are weighted and summed according to the loss weights to obtain the target loss;

[0150] In step S703, the model parameters of the context fusion model to be trained are updated according to the target loss.

[0151] Specifically, in step S702, the target loss can be the loss information obtained by fusing the preceding semantic loss, the local context loss, and the self-semantic preservation loss. In a specific embodiment, the target loss information can be obtained by weighted summation of the preceding semantic loss, the local context loss, and the self-semantic preservation loss.

[0152] For example, the target loss L_total = α × L_self + β × L_global + γ × L_local, where the self-loss weight α (corresponding to the self-semantic preservation loss), the global loss weight β (corresponding to the contextual semantic loss), and the local loss weight γ (corresponding to the local context loss) are adjustable hyperparameters used to balance the importance of different loss terms. Optionally, the global loss weight is greater than the self-loss weight or the local loss weight to emphasize context fusion as the primary learning objective. For example, α = 0.2, β = 0.6, and γ = 0.2 can be set.

[0153] Specifically, in one embodiment, such as Figure 7 As shown, step S260 further includes the following steps:

[0154] In S704, based on the updated context fusion model to be trained, the context fusion training iteration operation including steps S240-S250 and S701-S703 is repeated until the context fusion training convergence condition is met; the context fusion model to be trained obtained under the condition of meeting the context fusion training convergence condition is used as the context fusion model.

[0155] In an optional embodiment, the aforementioned convergence condition for context fusion training can be that the number of training iterations reaches a preset number of training iterations. Optionally, the convergence condition for context fusion training can also be that the target loss information is less than a specified threshold. In the embodiments of this specification, the preset number of training iterations and the specified threshold can be preset in conjunction with the training speed and accuracy of the network in practical applications.

[0156] The training method of the context fusion model proposed in this disclosure is based on the contextual semantic loss, local context loss and self-semantic preservation loss of each sample text block in the sample text block sequence. The training method performs multi-objective training on the context fusion model to be trained, driving the model to learn to adaptively fuse contextual information and improve the accuracy of context representation. In addition, compared with context fusion after splicing the original text, context fusion based on the initial semantic features of multiple text blocks can significantly reduce the computational cost of the model.

[0157] Furthermore, by acquiring the text content corresponding to the current sample text block and at least some sample text blocks preceding the current sample text block, the text content is summarized to obtain the context summary text, and semantic features are extracted from the context summary text to obtain the context semantic features as a global supervision signal, thereby training the global context alignment capability of the context fusion model.

[0158] Furthermore, at least one local context text is constructed, wherein for each local context text, at least one first sample text block is selected from the sample text blocks preceding the current sample text block, and at least one second sample text block is selected from the sample text blocks following the current sample text block; the at least one first sample text block, the current sample text block, and the at least one second sample text block are combined according to their respective order in the sample text block sequence to obtain the corresponding local context text, and semantic features are extracted from each of the at least one local context text to obtain at least one local context semantic feature as a local supervision signal, which can enhance the robustness of the model to discontinuous and jumpy local contexts.

[0159] Furthermore, the context fusion model to be trained includes a context feature aggregation network layer and a feature fusion network layer. Through the context feature aggregation network layer, the semantic relevance weights of each of the multiple sample text blocks are determined based on the semantic relevance between the initial semantic features of the current sample text block and the initial semantic features of each of the multiple sample text blocks. Based on the semantic relevance weights, the initial semantic features of each of the multiple sample text blocks are aggregated to obtain context aggregation features that can reflect the semantic role of the current sample text block in the entire document. Through the feature fusion network layer, the initial semantic features and context aggregation features of the current sample text block are fused to obtain enhanced semantic features, thus solving semantic ambiguity problems such as referential resolution and omission recovery.

[0160] Furthermore, based on the initial semantic features of the current sample text block and its corresponding context aggregation features, a learnable gating function is used to dynamically calculate the fusion weights that indicate how much original semantics should be retained and how much contextual information should be incorporated into the final output. Finally, the initial semantic features and context aggregation features are weighted and summed according to these weights to generate the enhanced semantic features of the current sample text block. This process ensures that the model can adaptively adjust the information fusion strategy for text blocks with different characteristics (such as independent terms or context-dependent pronouns).

[0161] Furthermore, based on the initial semantic features of the current sample text block and its corresponding context aggregation features, and based on the multi-dimensional gating weights corresponding to the initial semantic features of the current sample text block, multi-dimensional feature selection is performed on the context aggregation features to obtain multiple context selection features. The multiple context selection features and the current sample text block mapping features corresponding to the initial semantic features are then fused to obtain enhanced semantic features. Through multi-dimensional feature fusion, the comprehensiveness of the model's context feature fusion is improved.

[0162] This disclosure provides a retrieval enhancement method, applied to a retrieval enhancement device, with reference to... Figure 8a The retrieval enhancement method includes:

[0163] Step S810: Obtain the initial semantic features of each of the multiple text blocks in the text block sequence of the document to be added to the database;

[0164] Step S820: Input the initial semantic features of each of the multiple text blocks into the context fusion model, and perform context fusion on the initial semantic features of the current text block and the initial semantic features of the multiple text blocks to obtain the enhanced semantic features of the current text block.

[0165] Step S830: Add the enhanced semantic features of each of the multiple text blocks in the text block sequence to the retrieval knowledge base;

[0166] The context fusion model is trained using the training method described above.

[0167] In step S810, the document to be added to the database can be the original document to be added to the retrieval knowledge base, and the text block sequence can be multiple consecutive text blocks obtained by semantic segmentation of the document to be added. Specifically, the initial semantic features of each text block are used to characterize the semantic features within each text block itself, that is, the semantic features that have not undergone fusion processing.

[0168] In one specific embodiment, the initial semantic features of each text block can be obtained by encoding each text block using a pre-trained general vectorization model, and the initial semantic features can be represented as feature vectors. Illustratively, the general vectorization model can be any existing artificial intelligence model with text encoding capabilities; this disclosure does not impose any particular limitation on it. For example, the general vectorization model can adopt a BGE (Bidirectional Guided Embedding) model.

[0169] In step S820, the context fusion model is obtained by training the context fusion model to be trained based on the contextual semantic loss of the current sample text block, the local context loss of the current sample text block, and the self-semantic preservation loss of the current sample text block in the sample text block sequence. The contextual semantic loss is the loss between the enhanced semantic features of the current sample text block and the contextual semantic features of the current sample text block; the local context loss is the loss between the enhanced semantic features and the local contextual semantic features of the current sample text block; the self-semantic preservation loss is the loss between the enhanced semantic features and the initial semantic features of the current sample text block; the enhanced semantic features are obtained by inputting the initial semantic features of multiple sample text blocks in the sample text block sequence into the context fusion model to be trained, and then performing context fusion on the initial semantic features of the current sample text block and the initial semantic features of multiple sample text blocks.

[0170] Specifically, in one embodiment, the context fusion model includes: a context feature aggregation network layer and a feature fusion network layer, such as... Figure 8b As shown, step S820 includes the following steps:

[0171] In step S8201, the context feature aggregation network layer determines the semantic relevance weights of each of the multiple text blocks based on the semantic relevance between the initial semantic features of the current text block and the initial semantic features of each of the multiple text blocks.

[0172] In step S8202, the initial semantic features of multiple text blocks are aggregated according to the semantic relevance weights to obtain the context aggregation features of the current text block.

[0173] In step S8203, the initial semantic features and context aggregation features of the current text block are fused through the feature fusion network layer to obtain the enhanced semantic features of the current text block.

[0174] In an optional embodiment, the feature fusion layer includes: a first gating layer, a second gating layer, a first fusion layer, and a first nonlinear transformation layer. Further, the initial semantic features of the current text block are input into the first gating layer to determine a first gating weight, and feature selection is performed on the initial semantic features of the current text block according to the first gating weight to obtain a first selected feature; the context aggregation features are input into the second gating layer to determine a second gating weight, and feature selection is performed on the context aggregation features according to the second gating weight to obtain a second selected feature; the first selected feature and the second selected feature are input into the first fusion layer for feature fusion processing to obtain an intermediate fused feature; the intermediate fused feature is input into the first nonlinear transformation layer for nonlinear transformation to obtain the enhanced semantic features of the current text block.

[0175] For example, such as Figure 9a In the context fusion model shown, the context feature aggregation layer can be a self-attention fusion layer, the first gating layer and the second gating layer can be Sigmoid gating layers respectively, and the first nonlinear transformation layer can be an FFN layer.

[0176] In another optional embodiment, the feature fusion layer includes: N first feature mapping layers, N gate weight layers corresponding one-to-one with the N first feature mapping layers, N feature selection layers corresponding one-to-one with the N first feature mapping layers, a second feature mapping layer, and a second fusion layer. Further, the context aggregation features are input into the N first feature mapping layers for N-dimensional linear feature mapping to obtain N context sub-features; the initial semantic features of the current sample text block are input into the N gate weight layers for feature activation to obtain N gate weights; the N gate weights and N context sub-features are input into the corresponding feature selection layers, and feature selection is performed on the corresponding context sub-features according to the corresponding gate weights to obtain a context selection feature; the initial semantic features of the current sample text block are input into the second feature mapping layer for feature mapping to obtain the current sample text block mapping feature; the N context selection features output by the N feature selection layers and the current sample text block mapping feature are input into the second fusion layer for fusion to obtain enhanced semantic features.

[0177] For example, such as Figure 9b In the context fusion model shown, the context feature aggregation layer can be a self-attention fusion layer, N can be 3, each gate weight layer includes: 1 liner layer and 1 sigmoid gate layer, each first feature mapping layer is 1 liner layer, each second feature mapping layer is 1 liner layer, and the second fusion layer can be an FFN layer.

[0178] In step S830, the retrieval knowledge base can provide information assistance for downstream retrieval tasks.

[0179] The retrieval enhancement method proposed in this disclosure obtains the initial semantic features of multiple text blocks in the text block sequence of the document to be entered into the database. The initial semantic features of multiple text blocks are then input into a context fusion model trained using the aforementioned context fusion model training method. The initial semantic features of the current text block are then fused with the initial semantic features of multiple text blocks to obtain the enhanced semantic features of the current text block. Finally, the enhanced semantic features of multiple text blocks in the text block sequence are added to the retrieval knowledge base, generating an enhanced embedding vector for each text block in the document that retains its own semantics while incorporating the context information of the document in which it is located. This significantly improves the accuracy of downstream retrieval tasks.

[0180] Description of apparatus and devices according to embodiments of this disclosure

[0181] See Figure 10This disclosure also provides a training device 1000 for a context fusion model, including a sample acquisition module 1010, a context semantic determination module 1020, a local context semantic determination module 1030, a context fusion module 1040, a loss determination module 1050, and a model training module 1060.

[0182] The sample acquisition module 1010 is used to acquire the sample text block sequence and the initial semantic features of each of the multiple sample text blocks in the sample text block sequence;

[0183] The context semantic determination module 1020 is used to determine the context semantic features of the current sample text block based on the current sample text block and at least one sample text block preceding the current sample text block among multiple sample text blocks.

[0184] The local context semantic determination module 1030 is used to determine at least one local context semantic feature of the current sample text block based on the current sample text block and at least one other sample text block selected from the sequence of sample text blocks.

[0185] The context fusion module 1040 is used to input the initial semantic features of multiple sample text blocks into the context fusion model to be trained, and to perform context fusion on the initial semantic features of the current sample text block and the initial semantic features of multiple sample text blocks to obtain the enhanced semantic features of the current sample text block.

[0186] The loss determination module 1050 is used to determine the contextual semantic loss based on the difference between the enhanced semantic features and the contextual semantic features, to determine the local contextual loss based on the difference between the enhanced semantic features and at least one local contextual semantic feature, and to determine its own semantic preservation loss based on the difference between the enhanced semantic features and the initial semantic features of the current sample text block.

[0187] The model training module 1060 is used to train the context fusion model to be trained based on the semantic loss of the preceding text, the local context loss, and the semantic preservation loss of the model itself, so as to obtain the context fusion model.

[0188] The training apparatus 1000 for the context fusion model disclosed herein is used to execute the training method of the context fusion model as described in the above embodiments. Its specific processing procedure is the same as that of the training method of the context fusion model in the above embodiments, and will not be repeated here.

[0189] See Figure 11 This disclosure also provides a retrieval enhancement device 1100, including a feature acquisition module 1111, a semantic enhancement module 1120, and a document storage module 1130, wherein...

[0190] The feature acquisition module 1111 is used to acquire the initial semantic features of each of the multiple text blocks in the text block sequence of the document to be entered into the database;

[0191] The semantic enhancement module 1120 is used to input the initial semantic features of multiple text blocks into the context fusion model, and to perform context fusion of the initial semantic features of the current text block with the initial semantic features of multiple text blocks to obtain the enhanced semantic features of the current text block.

[0192] The document entry module 1130 is used to add the enhanced semantic features of each of the multiple text blocks in the text block sequence to the retrieval knowledge base;

[0193] The context fusion model is trained based on the context fusion model training method described above.

[0194] The search enhancement device 1100 disclosed herein is used to execute the search enhancement method as described in the above embodiments. Its specific processing procedure is the same as that of the search enhancement method described in the above embodiments, and will not be repeated here.

[0195] This disclosure also provides an electronic device 1210, including:

[0196] At least one processor, and,

[0197] A memory that is communicatively connected to at least one processor; wherein,

[0198] The memory stores instructions that are executed by at least one processor to cause the at least one processor to perform the method as described in any of the above embodiments of this application when executing the instructions.

[0199] The following is combined Figure 12 The hardware structure of the electronic device is described in detail. The electronic device includes: a processor 1210, a memory 1220, an input / output interface 1230, a communication interface 1240, and a bus 1250.

[0200] The processor 1210 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this disclosure.

[0201] The memory 1220 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 1220 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1220 and is called and executed by the processor 1210 to execute the training method of the context fusion model of the embodiments of this disclosure.

[0202] The input / output interface 1230 is used to implement information input and output.

[0203] The communication interface 1240 is used to enable communication and interaction between this device and other devices. Communication can be achieved via wired means (e.g., USB, Ethernet cable) or wireless means (e.g., mobile network, Wi-Fi, Bluetooth).

[0204] Bus 1250 transmits information between various components of the device (e.g., processor 1210, memory 1220, input / output interface 1230, and communication interface 1240);

[0205] The processor 1210, memory 1220, input / output interface 1230 and communication interface 1240 are connected to each other within the device via bus 1250.

[0206] This application also provides a computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement the training method of the context fusion model in the above embodiments, which will not be described in detail here.

[0207] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in this disclosure and the foregoing drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “including,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatuses.

[0208] It should be understood that in this disclosure, "at least one item" means one or more, and "more than one" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0209] It should be understood that in the description of the embodiments of this disclosure, "multiple" means two or more, "greater than", "less than", "exceeding" etc. are understood to exclude the number itself, and "above", "below", "within" etc. are understood to include the number itself.

[0210] In the several embodiments provided in this disclosure, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0211] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0212] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0213] It should also be understood that the various implementation methods provided in this disclosure can be combined arbitrarily to achieve different technical effects.

[0214] The above is a detailed description of the embodiments of this disclosure. However, this disclosure is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this disclosure. All such equivalent modifications or substitutions are included within the scope defined by the claims of this disclosure.

Claims

1. A training method for a context fusion model, characterized in that, The method comprises: obtaining a sample text block sequence and initial semantic features of each of a plurality of sample text blocks in the sample text block sequence; determining, for a current sample text block in the plurality of sample text blocks, a preceding semantic feature of the current sample text block according to the current sample text block and at least one sample text block located before the current sample text block; determining at least one local context semantic feature of the current sample text block according to the current sample text block and at least one other sample text block selected from the sample text block sequence; inputting the initial semantic features of each of the plurality of sample text blocks into a context fusion model to be trained, and performing context fusion on the initial semantic feature of the current sample text block and the initial semantic features of each of the plurality of sample text blocks to obtain an enhanced semantic feature of the current sample text block; determining a preceding semantic loss according to a difference between the enhanced semantic feature and the preceding semantic feature, determining a local context loss according to a difference between the enhanced semantic feature and the at least one local context semantic feature, and determining a self semantic retention loss according to a difference between the enhanced semantic feature and the initial semantic feature of the current sample text block; performing model training on the context fusion model to be trained according to the preceding semantic loss, the local context loss, and the self semantic retention loss to obtain a context fusion model. 2.The method of claim 1, wherein, Obtaining a sample text block sequence and initial semantic features of each of a plurality of sample text blocks in the sample text block sequence comprises: segmenting a sample document to obtain the sample text block sequence arranged in a position order in the sample document; respectively extracting semantic features of the plurality of sample text blocks by a semantic feature extraction model to obtain the initial semantic features of each of the plurality of sample text blocks. 3.The method of claim 1, wherein, Determining the preceding semantic feature according to the current sample text block and at least one sample text block located before the current sample text block comprises: obtaining text content corresponding to the current sample text block and at least part of the sample text blocks located before the current sample text block; performing abstract processing on the text content to obtain a preceding abstract text; extracting semantic features of the preceding abstract text to obtain the preceding semantic feature. 4.The method of claim 1, wherein, Determining at least one local context semantic feature of the current sample text block according to the current sample text block and at least one other sample text block selected from the sample text block sequence comprises: Construct at least one local context text, wherein, for each local context text, if a sample text block exists before the current sample text block, at least one first sample text block is selected from the sample text blocks preceding the current sample text block; if a sample text block exists after the current sample text block, at least one second sample text block is selected from the sample text blocks following the current sample text block; combine the current sample text block and the selected sample text blocks according to their respective order in the sample text block sequence to obtain the corresponding local context text; Semantic features are extracted from each of the at least one local context text to obtain at least one local context semantic feature that corresponds one-to-one with the at least one local context text.

5. The method of claim 4, wherein, At least one of the at least first sample text block and the at least second sample text block includes a sample text block that is not adjacent to the current sample text block.

6. The method of claim 4, wherein, The at least one local context text includes multiple candidate local context texts; at least one of the first sample text block and the second sample text block included in different candidate local context texts is different; Semantic features are extracted from each of the at least one local context text to obtain at least one local context semantic feature corresponding to each of the at least one local context text, including: Semantic features are extracted from the multiple candidate local context texts to obtain multiple candidate local context semantic features that correspond one-to-one with the multiple candidate local context texts.

7. The method of claim 6, wherein, Determining the local context loss based on the difference between the enhanced semantic features and the at least one local context semantic feature includes: Based on the differences between the enhanced semantic features and the multiple candidate local context semantic features, multiple candidate local context losses are determined. The local context loss is obtained by averaging or weighting the multiple candidate local context losses. 8.The method of claim 1, wherein, Based on the aforementioned semantic loss, the local context loss, and the self-semantic preservation loss, the context fusion model to be trained is trained to obtain a context fusion model, including: Set corresponding loss weights for the above semantic loss, the local context loss, and the self-semantic preservation loss, respectively; The target loss is obtained by weighting and summing the above semantic loss, the local context loss, and the self-semantic preservation loss according to the loss weights. The model parameters of the context fusion model to be trained are updated based on the target loss. 9.The method of claim 1, wherein, The context fusion model to be trained includes a context feature aggregation network layer and a feature fusion network layer; Contextual fusion is performed on the initial semantic features of the current sample text block and the initial semantic features of each of the plurality of sample text blocks, including: The context feature aggregation network layer determines the semantic relevance weights of each of the multiple sample text blocks based on the semantic relevance between the initial semantic features of the current sample text block and the initial semantic features of each of the multiple sample text blocks. Based on the semantic relevance weights, feature aggregation is performed on the initial semantic features of each of the multiple sample text blocks to obtain the context aggregation features of the current sample text block; The enhanced semantic features are obtained by fusing the initial semantic features and the context aggregation features of the current sample text block through the feature fusion network layer.

10. The method of claim 9, wherein, Based on the semantic relevance between the initial semantic features of the current sample text block and the initial semantic features of each of the plurality of sample text blocks, the semantic relevance weights corresponding to each of the plurality of sample text blocks are determined, including: A first linear transformation is performed on the initial semantic features of the current sample text block to obtain the query vector; The initial semantic features of the multiple sample text blocks are subjected to a second linear transformation and a third linear transformation respectively to obtain multiple key vectors and multiple value vectors; Based on the similarity between the query vector and the multiple key vectors, the semantic relevance weights corresponding to each of the multiple sample text blocks are determined; Based on the semantic relevance weights, feature aggregation is performed on the initial semantic features of each of the multiple sample text blocks to obtain the context aggregation features of the current sample text block, including: The multiple value vectors are weighted and aggregated according to the semantic relevance weights to obtain the context aggregation feature.

11. The method of claim 9, wherein, The feature fusion network layer performs feature fusion on the initial semantic features and the context aggregation features of the current sample text block, including: A first gating weight is determined based on the initial semantic features of the current sample text block, and feature selection is performed on the initial semantic features of the current sample text block based on the first gating weight to obtain the first selected feature; A second gating weight is determined based on the context aggregation features, and feature selection is performed on the context aggregation features based on the second gating weight to obtain a second selected feature; The first selection feature and the second selection feature are fused to obtain an intermediate fused feature; The intermediate fusion features are subjected to a nonlinear transformation to obtain the enhanced semantic features.

12. The method of claim 9, wherein, The feature fusion network layer performs feature fusion on the initial semantic features and the context aggregation features of the current sample text block, including: The context aggregation features are subjected to multi-dimensional feature mapping to obtain multiple context sub-features; Multiple gate weights are determined based on the initial semantic features of the current sample text block, and the multiple gate weights correspond one-to-one with the multiple context sub-features; Each context sub-feature is selected based on the multiple gating weights to obtain multiple context selection features; The initial semantic features of the current sample text block are mapped to obtain the mapped features of the current sample text block; The enhanced semantic features are obtained by fusing the multiple context selection features and the current sample text block mapping features.

13. A method of retrieving augmentation, characterized by, The method includes: Obtain the initial semantic features of each text block in the text block sequence of the document to be imported into the database; The initial semantic features of each of the multiple text blocks are input into the context fusion model, and the initial semantic features of the current text block are fused with the initial semantic features of the multiple text blocks to obtain the enhanced semantic features of the current text block. The enhanced semantic features of each of the multiple text blocks in the text block sequence are added to the retrieval knowledge base; The context fusion model is trained using the training method of the context fusion model according to any one of claims 1 to 12.

14. A training device for a context fusion model, characterized in that, The device includes: The sample acquisition module is used to acquire a sequence of sample text blocks and the initial semantic features of each of the multiple sample text blocks in the sequence of sample text blocks. The context semantic determination module is used to determine the context semantic features of the current sample text block based on the current sample text block and at least one sample text block preceding the current sample text block among the plurality of sample text blocks. The local context semantic determination module is used to determine at least one local context semantic feature of the current sample text block based on the current sample text block and at least one other sample text block selected from the sequence of sample text blocks; The context fusion module is used to input the initial semantic features of each of the multiple sample text blocks into the context fusion model to be trained, and to perform context fusion on the initial semantic features of the current sample text block and the initial semantic features of the multiple sample text blocks to obtain the enhanced semantic features of the current sample text block. The loss determination module is used to determine the contextual semantic loss based on the difference between the enhanced semantic features and the contextual semantic features, to determine the local contextual loss based on the difference between the enhanced semantic features and the at least one local contextual semantic features, and to determine its own semantic preservation loss based on the difference between the enhanced semantic features and the initial semantic features of the current sample text block. The model training module is used to train the context fusion model to be trained based on the above semantic loss, the local context loss and the self-semantic preservation loss, so as to obtain the context fusion model.

15. A retrieval augmentation device, comprising: The device includes: The feature acquisition module is used to acquire the initial semantic features of each of the multiple text blocks in the text block sequence of the document to be imported into the database; The semantic enhancement module is used to input the initial semantic features of each of the multiple text blocks into the context fusion model, and perform context fusion on the initial semantic features of the current text block and the initial semantic features of the multiple text blocks to obtain the enhanced semantic features of the current text block. The document entry module is used to add the enhanced semantic features of each of the multiple text blocks in the text block sequence to the retrieval knowledge base; The context fusion model is trained using the training method of the context fusion model according to any one of claims 1 to 12.

16. An electronic device, comprising: The electronic device includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for establishing communication between the processor and the memory. The program, when executed by the processor, implements the training method of the context fusion model as described in any one of claims 1 to 12 or the retrieval enhancement method as described in claim 13.

17. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs, which can be executed by one or more processors to implement the training method of the context fusion model as described in any one of claims 1 to 12 or the retrieval enhancement method as described in claim 13.