A vertical domain adaptation method and system based on entity knowledge distribution alignment

Abstract

This invention provides a method and system for vertical domain adaptation based on entity knowledge distribution alignment, relating to the fields of natural language processing and artificial intelligence. First, by constructing entity knowledge representations from multi-perspective features of the same entity knowledge, this invention achieves a comprehensive and systematic characterization of the identity, structure, and semantic information of entity knowledge. Second, through a cross-perspective knowledge distribution alignment mechanism based on contrastive learning, it ensures semantic consistency of knowledge from different sources before fusion. Finally, by using a compression module based on learnable query tokens and precisely injecting the compressed knowledge tokens into the self-attention calculation of a specified layer in the encoder of a pre-trained language model, it achieves low-intrusive deep knowledge injection, enabling the pre-trained language model to adaptively coordinate internal and external knowledge without modifying the main parameters, significantly reducing the cost of vertical domain adaptation.

Description

Technical Field

[0001] This invention relates to the fields of natural language processing and artificial intelligence, and in particular to a vertical domain adaptation method, system, and method for storing knowledge-enhanced data based on entity knowledge distribution alignment. Background Technology

[0002] Pre-trained Language Models (PLMs) and Large Language Models (LLMs) perform well on general natural language understanding tasks, but they often perform poorly in vertical domain applications (such as news recommendation, medical question answering, and financial analysis) due to a lack of domain-specific entity knowledge.

[0003] Existing technical solutions mainly fall into the following categories: Entity knowledge representation learning methods based on knowledge graphs (such as R-GCN): These methods can learn the relational representation of entity knowledge in graph structures, but their representation space has a significant distributional difference from the internal representation space of pre-trained language models, making direct fusion difficult.

[0004] Knowledge fusion methods in the pre-training stage [such as ERNIE (Enhanced Language Representation with Informative Entities) and KnowBERT (Knowledge Enhanced Contextual Word Representations)]: These methods incorporate knowledge in the model pre-training stage and have significant effects, but they require large-scale re-training, which is costly and difficult to adapt to rapidly updated domain knowledge.

[0005] Retrieval-Augmented Generation (RAG) method: This method dynamically retrieves relevant knowledge during inference and concatenates it into the input as text, making it simple to implement. However, its knowledge injection remains at the input level, which is a shallow injection and is easily affected by the quality of retrieval. Furthermore, long texts can dilute key entity knowledge information, making it difficult to achieve deep semantic alignment.

[0006] Lightweight knowledge adaptation methods (such as adapters): These methods achieve domain adaptation by adding a few parameters, making them less invasive. However, they typically only consider knowledge from a single source and lack a unified modeling and alignment mechanism for knowledge from multiple heterogeneous entities, leading to unstable knowledge fusion.

[0007] In summary, the core problem with existing technologies is that there are semantic misalignments and distribution differences among multi-source heterogeneous entity knowledge, and there is a lack of an effective solution to achieve deep, stable, and low-intrusion knowledge injection while freezing the main parameters of the pre-trained language model. Summary of the Invention

[0008] In view of this, the core technical problem to be solved by the present invention is: how to provide a deep, stable, and low-intrusion method for vertical domain knowledge adaptation and injection, so as to significantly improve the performance of pre-trained language models in vertical domain tasks.

[0009] On one hand, this invention provides a vertical domain adaptation method based on entity knowledge distribution alignment, including: S1, entity knowledge extraction: performing entity knowledge recognition on the input vertical domain text to obtain a set of entity knowledge contained in the text; S2, multi-source knowledge construction: constructing multi-view features for each entity knowledge in the entity knowledge set; S3, knowledge distribution alignment: optimizing the distribution alignment of the multi-view features of the same entity knowledge through a contrastive learning loss function to obtain an entity knowledge representation with higher semantic consistency; S4, knowledge compression: after deep modeling the entity knowledge representation through an encoder, obtaining an enhanced entity knowledge representation, compressing the enhanced entity knowledge representation using a compression module based on learnable query tokens to generate a fixed number of compressed knowledge tokens; S5, knowledge injection: injecting the compressed knowledge tokens into the self-attention calculation of a specified layer of the pre-trained language model encoder to enhance the representational capability of the pre-trained language model, thereby achieving low-intrusion vertical domain knowledge enhancement.

[0010] Optionally, in step S2, the multi-perspective features include entity identification features, entity relationship features, and entity description features.

[0011] Optionally, in step S2, entity identification features are obtained from the embedding space to carry identity information; entity relationship features are extracted based on knowledge graph triples to carry structural information; entity description features are generated using a large language model and encoded by a pre-trained language encoder to carry semantic information.

[0012] Optionally, step S3 specifically includes: using multi-perspective features of the same entity knowledge as positive samples and multi-perspective features of other entity knowledge as negative samples, and optimizing the semantic consistency between entity knowledge representations from different sources by contrastive learning loss function.

[0013] Optionally, in step S4, generating a fixed number of compressed knowledge tokens using a compression module based on learnable query tokens specifically includes the following steps: S41, initializing a fixed number of learnable query tokens and processing the query tokens through self-attention; S42, performing cross-attention with the query tokens as query requests and entity knowledge augmentation representations as keys and values, selectively extracting key entity knowledge information; S43, generating the final compressed knowledge tokens from the selectively extracted entity knowledge information through a feedforward network.

[0014] Optionally, in step S5, injecting the compressed knowledge token into the self-attention calculation of a specified layer of the pre-trained language model encoder means concatenating the compressed knowledge token into the key and value of the self-attention mechanism of the specified layer of the pre-trained language model encoder, while the query remains unchanged.

[0015] Optionally, in step S5, the specified layer is the middle and high layers of the pre-trained language model encoder.

[0016] Optionally, the following steps are also included: S6, Model Training: Jointly optimize the downstream task loss function and the contrastive learning loss function to train the parameters of the vertical adapter, while keeping the main parameters of the pre-trained language model frozen.

[0017] On the other hand, this invention provides a vertical domain adaptation method and a vertical domain adaptation system based on entity knowledge distribution alignment, used to implement the vertical domain adaptation method based on entity knowledge distribution alignment as described above. The vertical domain adaptation system comprises: a multi-source entity knowledge construction module, used to construct multi-view features for each entity knowledge set in the input text; a cross-view knowledge alignment module, used to perform distribution alignment of the multi-view features based on contrastive learning to obtain entity knowledge representations; an entity knowledge compression module, used to perform deep modeling and compression of the entity knowledge representations, outputting a fixed number of compressed knowledge tokens; and an encoding layer knowledge injection module, used to inject the compressed knowledge tokens into the self-attention mechanism of a specified layer of the pre-trained language model encoder.

[0018] On the other hand, the present invention also provides a method for storing knowledge-enhanced data, comprising: performing the above-described vertical domain adaptation method based on entity knowledge distribution alignment to generate compressed knowledge tokens; and storing the compressed knowledge tokens or a pre-trained language model enhanced by compressed knowledge tokens into a storage medium.

[0019] Implementing this invention has the following beneficial effects: The top-level beneficial effect of this invention is that it realizes a deep, stable, and low-intrusive fusion of external entity knowledge and the internal representation space of the pre-trained language model, thereby significantly improving the performance of the pre-trained language model on vertical domain tasks. Specifically, this invention constructs entity knowledge representations from multiple perspectives of the same entity knowledge, achieving a comprehensive and systematic characterization of the identity, structure, and semantic information of entity knowledge, providing a rich knowledge foundation for fusion. Secondly, through a cross-perspective knowledge distribution alignment mechanism based on contrastive learning, it effectively alleviates the semantic misalignment and distribution differences between multi-source heterogeneous knowledge, ensuring semantic consistency of knowledge from different sources before fusion, resulting in enhanced entity knowledge representations. Next, through a compression module based on learnable query tokens, the fused enhanced entity knowledge representations are condensed into a small number of high-information-density compressed knowledge tokens, retaining key information while controlling redundancy caused by injection, achieving a balance between knowledge capacity and computational efficiency. Finally, by precisely injecting compressed knowledge tokens into the self-attention calculation of a specified layer in the encoder of a pre-trained language model, low-intrusive deep knowledge injection is achieved, enabling the pre-trained language model to adaptively coordinate internal and external knowledge without modifying the main parameters, greatly reducing the cost of vertical domain adaptation. With the synergistic effect of the above-mentioned technical features, it is ultimately possible to overcome the semantic misalignment and distribution differences between multi-source heterogeneous entity knowledge without modifying the main parameters of the pre-trained language model, and significantly improve the performance of the pre-trained language model in vertical domain tasks. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating a vertical domain adaptation method based on entity knowledge distribution alignment in one embodiment. Figure 2 This is an overall flowchart of a vertical domain adapter based on entity knowledge distribution alignment in one embodiment. Detailed Implementation

[0021] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. The step numbers in the following embodiments are only for ease of explanation and do not limit the order of the steps. The execution order of each step in the embodiments can be adapted according to the understanding of those skilled in the art.

[0022] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the embodiments of this application. The singular forms “a,” “the,” and “the” used in the embodiments of this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0023] In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims. In the description of this application, it should be understood that the terms "first," "second," "third," etc., are used only to distinguish similar objects and are not necessarily used to describe a specific order or sequence, nor should they be construed as indicating or implying relative importance. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0024] Furthermore, in the description of this application, unless otherwise stated, "multiple" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0025] Example 1 In this embodiment, as Figure 1 The vertical domain adaptation method shown includes: S1. Entity knowledge extraction: Entity knowledge recognition is performed on the input vertical domain text to obtain the set of entity knowledge contained in the text; S2. Multi-source knowledge construction: For each entity knowledge in the entity knowledge set, construct multi-perspective features respectively; S3, Cross-perspective knowledge distribution alignment: Multi-perspective features of the same entity knowledge are optimized for distribution alignment through a contrastive learning loss function to obtain entity knowledge representations with higher semantic consistency; S4. Knowledge Compression: After deep modeling the entity knowledge representation through an encoder, an enhanced entity knowledge representation is obtained. The enhanced entity knowledge representation is then compressed using a compression module based on learnable query tokens (lexical units) to generate a fixed number of compressed knowledge tokens. S5. Knowledge Injection: Inject compressed knowledge tokens into the self-attention computation of a specified layer of the pre-trained language model encoder to enhance the representational power of the pre-trained language model, thereby achieving low-intrusion vertical domain knowledge enhancement.

[0026] In this embodiment, step S1 is as follows: For each text in a vertical domain (such as a news article), entity knowledge extraction is first performed to identify the entity knowledge sets in the text title and abstract. Entity knowledge types cover people, places, organizations, events, and domain themes. Entity knowledge extraction can be accomplished using existing mature entity knowledge recognition tools or pre-trained language models.

[0027] In this embodiment, the multi-view features in step S2 include entity ID features, entity relation features, and entity description features.

[0028] Specifically, entity identification features are obtained from the embedding space to carry identity information; entity relationship features are extracted based on knowledge graph triples to carry structural information; and entity description features are generated using a large language model and encoded through a pre-trained language encoder to carry semantic information.

[0029] In this embodiment, the entity identification features are constructed as follows: Knowledge for each entity Query the unique identifier of entity knowledge in an external knowledge base. And a random embedding vector is initialized as the latent space feature representation of entity knowledge, thereby obtaining entity identification features. : Equation (1); Entity identification features This indicates that during subsequent training, optimization is performed through gradient updates to gradually learn discriminative representations capable of distinguishing different entity knowledge identities. The initial dimension of the entity identifier features is consistent with the input dimension of the subsequent Transformer encoder.

[0030] In this embodiment, the external knowledge base selected is Wikipedia.

[0031] In this embodiment, an embedding vector is randomly initialized using Xavier (Xavier Initialization) as a latent space feature representation of entity knowledge.

[0032] In this embodiment, the entity relationship features are constructed as follows: First, construct the knowledge graph relationship structure between entity knowledge, assuming the set of all entities is denoted as . Assign a unique index to each entity. (Note:) and This refers to the index of the entity in the index table. These represent the head and tail entities corresponding to the subscript, respectively. For entities in a knowledge graph... and Through triples Describe the relationship structure between entities and extract entity relationship features from it. , express and The relationship types between entities. This representation carries structural information about entities, characterizing their neighborhood structure and multi-hop dependencies within the knowledge network, and obtaining entity relationship features. :

[0033] Equation (2); Entity relation features encode the structural location and neighborhood association information of an entity in the knowledge network. For entities that do not exist in the knowledge graph, the semantic similarity between their text description and known entities is calculated. The average of the relation features of the top K entities with the highest similarity is selected as the imputation value, and a similarity threshold is set. If the highest similarity is lower than the threshold, zero vectors are used for imputation.

[0034] In this embodiment, the entity description features are constructed as follows: Utilizing large language models for knowledge of each entity Generate context-aware background descriptions : Equation (3); The generated prompt is designed as follows: "Please provide the entity [ A concise background description is generated, including its basic attributes, category, and relevant contextual information. The generated description text is then encoded using a pre-trained language model encoder to obtain entity description features. :

[0035] Equation (4); In this embodiment, the pre-trained language model encoder is BERT (Bidirectional Encoder Representations from Transformers). The hidden state corresponding to the [CLS] token of the BERT encoder is used as the final representation of the entity description features.

[0036] In this embodiment, the generated text from the large language model is described as an offline preprocessing step and does not participate in the gradient calculation for subsequent pre-trained language model training.

[0037] In this embodiment, the large language model selected is GPT-4o.

[0038] It should be noted that entity identification features, entity relationship features, and entity description features are constructed independently of each other, each adopting the representation method most suitable for its information characteristics. This is the fundamental reason why cross-perspective knowledge alignment is needed in the future.

[0039] For new entities not present in the training set (Out-of-Vocabulary, OOV), use an independent, learnable... Features serve as inputs to new entities in a pre-trained language model.

[0040] In this embodiment, step S3 specifically includes: using the multi-perspective features of the same entity knowledge as positive samples and the multi-perspective features of other entity knowledge as negative samples, and optimizing the semantic consistency between entity knowledge representations from different sources by using a contrastive learning loss function.

[0041] In this embodiment, since the multi-view features come from different pre-trained language models and training targets, there are significant distribution differences between their representation spaces. This invention achieves knowledge alignment through a multi-view contrastive learning mechanism, which specifically includes the following steps: constructing positive and negative samples, contrastive learning optimization, and feature fusion.

[0042] In this embodiment, the construction of positive and negative samples is specifically as follows: Positive sample construction: For each entity's entity knowledge features in the entity set, select other entity knowledge features of the same entity as positive samples. .

[0043] Negative sample construction: Randomly select three entity knowledge features from other entities in the same batch to form negative samples. .

[0044] To ensure the quality and diversity of negative samples, each positive sample corresponds to multiple negative samples, and the selection of negative samples is completed within the same training batch.

[0045] In this embodiment, the contrastive learning optimization specifically involves: Through the following loss function ( Optimize feature representation by aggregating positive sample features and extrapolating negative sample features: Equation (5); in, Using cosine similarity, temperature parameters To control the sharpness of the distribution, a small-scale grid search is performed on the validation set. ∈{0.01, 0.05, 0.07, 0.1, 1.0}), and found that when The model performance was optimal when the coefficient of performance was 0.07, so 0.07 was used in all subsequent experiments.

[0046] By minimizing Pre-trained language models can enhance the consistency of multi-view feature representations of entity knowledge within the same text and improve the discriminativeness of entity knowledge in cross-text scenarios, thus providing a semantically aligned representation basis for subsequent feature fusion.

[0047] In this embodiment, feature fusion specifically refers to: After alignment, the feature vectors from the three perspectives (i.e., entity identifier features, entity relation features, and entity description features) are concatenated to obtain the entity knowledge representation. : Equation (6); This stitching operation preserves the complete information of the features from each viewpoint, and the subsequent Transformer encoder and compression module are responsible for further interaction and fusion of cross-viewpoint features.

[0048] In this embodiment, step S4, after deep modeling the entity knowledge representation through an encoder, yields an enhanced entity knowledge representation, specifically including: Representing entity knowledge Input a Transformer encoder block. The encoder employs a standard multi-head self-attention mechanism and a feedforward network structure, with two layers and the hidden dimension consistent with the pre-trained language model. Through the self-attention mechanism, the encoder can capture high-level semantic relationships between different entities and between features of the same entity from different perspectives, generating an entity knowledge-enhanced representation Z:

[0049] Equation (7).

[0050] In this embodiment, step S4, which uses a compression module based on learnable query tokens to generate a fixed number of compressed knowledge tokens, specifically includes the following steps: S41. Initialize a fixed number of learnable query tokens and process the query tokens through self-attention; S42. Perform cross-attention using the query token as the query request (Query) and the entity knowledge augmentation representation as the key (Key) and value (Value) to selectively extract key entity knowledge information; S43. Generate the final compressed knowledge token by selectively extracting entity knowledge information through a feedforward network.

[0051] In this embodiment, the compression module is a knowledge compression module based on the Qformer (Query Transformer) structure, which compresses the entity knowledge augmented representation into a fixed number of N compressed knowledge tokens.

[0052] In this embodiment, step S41 specifically includes: Initialize using Xavier Each query token is a learnable query token, with dimensions matching the output dimension of the Transformer encoder. The query token undergoes self-attention layer processing, enabling it to possess preliminary internal interactive capabilities before compression.

[0053] Equation (8).

[0054] In this embodiment, step S42 specifically includes: The query token after self-attention processing ( As a query, entity knowledge augmentation representation Using these as Key and Value, a cross-attention operation is performed, enabling the selective extraction of the most critical information from rich entity knowledge and its aggregation into a fixed number of query tokens, resulting in compressed knowledge tokens. The specific formula is as follows: Equation (9); The cross-attention mechanism employs multi-head attention (8 heads), enabling different attention heads to focus on different aspects of entity knowledge.

[0055] In this embodiment, step S43 specifically includes: The output of the cross-attention operation undergoes nonlinear transformation and feature refinement via a feedforward network to obtain the final compressed knowledge token. : Equation (10).

[0056] In this embodiment, the feedforward network adopts a two-layer fully connected structure, with the middle layer using the GELU activation function and configured with residual connections and layer normalization.

[0057] In this embodiment, the optimal compression quantity was determined through systematic ablation experiments. Tests were conducted separately. Five settings were used, with the MINER (Multi-Interest Matching Network for News Recommendation) model's experimental results on the MIND-small dataset as an example: When N=1, the AUC was 63.15; when N=2, it was 63.87; when N=4, it reached its optimum at 64.58; when N=8, it decreased to 64.12; and when N=15, it further decreased to 63.53. The experimental results showed an inverted U-shaped trend of first increasing and then decreasing. The results indicate that when N=4, the pre-trained language model achieves the best balance between information preservation and computational efficiency, with all evaluation metrics reaching their peak. Too few tokens (N=1 or 2) lead to excessive information compression, irreversibly losing some entity semantics; too many tokens (N=8 or 15) introduce redundant representations, occupying too much attention resources during subsequent encoder injection, diluting the pre-trained language model's focus on the original text semantics. N=4 achieves the best balance between information preservation and representation compactness; therefore, this invention recommends setting N=4.

[0058] In this embodiment, step S5, injecting the compressed knowledge token into the self-attention calculation of a specified layer of the pre-trained language model encoder, means concatenating the compressed knowledge token into the key and value of the self-attention mechanism of the specified layer of the pre-trained language model encoder, while the query remains unchanged.

[0059] Specifically, compress knowledge tokens As an external knowledge enhancement signal, it is concatenated into the Key and Value of the self-attention mechanism of the specified layer in the pre-trained language model encoder. The original encoder attention layer is calculated as follows:

[0060] Equation (11); The computation method after introducing knowledge enhancement is as follows: Equation (12); in This is the hidden state representation of the original text input after a pre-encoding layer. The core design of this injection method is to only extend the length of the Key and Value without changing the source of the Query, allowing the pre-trained language model to adaptively determine the degree of attention it pays to external entity knowledge through attention weights. When external knowledge has significant supplementary value to the current text semantics, the attention weights naturally allocate more attention to the compressed knowledge token; otherwise, they mainly rely on the original text information. Throughout the process, the main parameters of the pre-trained language model remain frozen, and only the parameters of the EDA (Entity-aware DomainAdapter) are updated.

[0061] In this embodiment, in step S5, the designated layer is the middle and high layers of the pre-trained language model encoder.

[0062] In this embodiment, the location of the specified layer is determined through the following experiments: Based on the MINER model, we systematically compared the effects of connecting EDA to different layers of the BERT 12-layer Transformer (layers 0 to 11) on the MIND-small (Microsoft News Dataset - small) dataset, and also tested two additional multi-layer joint augmentation strategies (Layer [0-5] and Layer [6-11]).

[0063] The experimental results revealed the following key patterns: (a) The optimal injection layer is layer 8: This layer achieved the best results across all metrics, with an AUC of 64.58 and an MRR of 35.43. Layer 8 is located in the mid-to-high level of the encoder. At this point, the MINER model has completed sufficient contextual semantic encoding, and the text representation possesses relatively rich high-level semantic information, while still retaining space for further fusion and refinement at higher levels. Introducing compressed knowledge tokens at this position enables deep interaction between external knowledge and textual semantics at the most expressive stage.

[0064] (b) Shallow layers perform best, deep layers perform worst: Layer 1, as the second-best position (AUC 64.31), can play a "knowledge-guiding" role by introducing entity knowledge in the early stages of encoding, enabling external knowledge to gradually collaborate with text features in subsequent multi-layer propagation. Layers 10 and 11 perform close to the baseline level (AUC 62.47 and 62.31, respectively). When external knowledge is introduced at the top layer of the encoder, the MINER model is already close to the output stage and lacks sufficient subsequent layers to fully absorb and integrate the newly added entity information.

[0065] (c) Single-layer injection outperforms multi-layer joint injection: Inserting EDA into the first half of the layers (Layer [0-5], AUC 63.35) or the second half of the layers (Layer [6-11], AUC 62.68) simultaneously yields significantly lower results than inserting it into the 8th layer alone (64.58). Simultaneous multi-layer injection introduces excessive external signal interference, negatively impacting gradient propagation and feature learning between layers due to redundant information. This finding validates the conclusion that "precise local augmentation" is superior to "extensive multi-layer augmentation."

[0066] In summary, the overall distribution exhibits an inverted U-shaped trend, with optimal performance at mid-to-high levels, superior performance at shallower levels, and worst performance at deeper levels. Based on this, this invention recommends selecting layer 8 as the knowledge injection location in BERT-based applications. For encoders with other architectures, the optimal injection layer needs to be determined through similar ablation experiments.

[0067] In this embodiment, the vertical domain adaptation method based on entity knowledge distribution alignment further includes the following steps: S6. Model Training: Jointly optimize the downstream task loss function and the contrastive learning loss function to train the parameters of the vertical adapter, while keeping the main parameters of the pre-trained language model frozen.

[0068] In this embodiment, the downstream task loss is calculated as follows: Taking recommendation tasks as an example, the final representation of an item is obtained through an item encoder enhanced with entity knowledge. The final representation of the user is obtained through the user encoder. Calculate the cosine similarity between the user and the candidate items as the matching score:

[0069] Equation (13); In the experimental setup, each positive sample was paired with four negative samples, and the cross-entropy loss function was used. Optimize sample sorting: Equation (14); in The actual labels for the sample are (1 for clicks, 0 for no clicks). Predict probabilities for the model.

[0070] In this embodiment, the contrastive learning loss function is: that is, the loss function in step S3. Used to enhance the semantic consistency of multi-perspective entity representations.

[0071] The final loss function is obtained by a weighted combination of the downstream task loss function and the contrastive learning loss function: Equation (15); in To compare the weighting coefficients of the loss, a small-scale grid search was performed on the validation set, and it was found that when... The pre-trained language model exhibits optimal and stable performance around 0.05, therefore subsequent experiments were conducted using this model. Based on a value of 0.05, the pre-trained language model is optimized not only in supervised ranking tasks, but also the consistency of its multi-view features is simultaneously enhanced through joint optimization.

[0072] In this embodiment, the training configuration of the vertical adapter is as follows: The AdamW optimizer was used, with bf16 mixed precision training (AMP) enabled for acceleration. Training consisted of multiple epochs, with evaluation performed on the validation set after each epoch. The checkpoint with the highest NDCG (Normalized Discounted Cumulative Gain) @10 value was selected. An early stopping strategy was adopted: training was terminated if the validation set metric did not improve after three consecutive iterations. The main parameters of the pre-trained language model encoder were kept frozen; only the parameters of the EDA adapter modules (including entity ID embedding, Transformer encoder, and Qformer compression module) and cross-view knowledge alignment parameters were updated.

[0073] Example 2 This embodiment, based on Embodiment 1, evaluates the performance of the enhanced pre-trained language model from multiple dimensions.

[0074] In this embodiment, a multi-dimensional evaluation index system is used to comprehensively evaluate the performance of the enhanced pre-trained language model. The specific evaluation indexes include: (1) AUC (Area Under Curve): measures the probability that a pre-trained language model will rank positive samples before negative samples. It reflects the overall ranking ability of the pre-trained language model and is not sensitive to the sample distribution.

[0075] (2) MRR (Mean Reciprocal Rank): Evaluates the average of the reciprocals of the first appearance of the correct recommendation in the ranking list, focusing on the accuracy of the first recommendation.

[0076] (3) NDCG@5 and NDCG@10 (Normalized Discounted Cumulative Gain): Position-sensitive ranking evaluation metrics. The higher the position of a highly relevant item, the greater its contribution. They are used to evaluate the ranking quality of the Top-5 and Top-10 recommendation lists, respectively.

[0077] Different base models were selected to pre-train language models for implementation and validation. This invention has been implemented and validated on various architectural models to demonstrate the universality of the method: (1) LSTUR (Long- and Short-Term User Representations) model: a dual-channel architecture that combines user ID embedding to capture long-term preferences with GRU to model short-term interests. After introducing EDA, the AUC on the MIND-small dataset increased from 62.07 to 65.11, and the NDCG@10 increased from 37.69 to 40.63.

[0078] (2) NRMS (Neural News Recommendation Approach with Multi-head Self-Attention) model: It is an end-to-end architecture based entirely on multi-head self-attention. After introducing EDA, the AUC increased from 62.46 to 65.62, achieving the highest AUC and NDCG@5 scores, indicating that the attention mechanism has significant advantages in aggregating entity information.

[0079] (3) MINER model: a multi-interest modeling architecture that extracts multiple interest vectors from user history. After introducing EDA, the MRR increased from 32.55 to 35.43, proving that multi-interest modeling combined with entity knowledge can more accurately locate the user's primary interests.

[0080] (4) IISAN (Intra- and Inter-modal Side Adapted Network for Multimodal Representation): An efficient adaptation model for multimodal sequence recommendation, proposing a decoupled parameter efficient fine-tuning paradigm. After introducing EDA, the MRR increased from 32.75 to 35.65, achieving the highest MRR score.

[0081] The same enhancement effect was also verified on the EB-NeRD (Ekstra Bladet News Recommendation Dataset). All the basic models achieved stable improvement in various indicators after introducing EDA, which proves the generalization ability of the vertical domain adaptation method based on entity knowledge distribution alignment of the present invention across datasets.

[0082] Different encoders were selected for implementation verification: To verify the applicability of the vertical domain adaptation method based on entity knowledge distribution alignment in this invention to a stronger encoder, the item encoder was replaced from BERT with LLaMA-1 for experimentation.

[0083] Experimental results show that: (1) When LLaMA-1 is used as the encoder, the baseline performance of all pre-trained language models is significantly improved (e.g., the baseline AUC of IISAN increases from 61.35 to 65.02); (2) Even on the basis of the stronger encoder LLaMA-1, the EDA module can still bring stable and significant performance gains (e.g., the AUC of IISAN+EDA is further increased to 67.68, an increase of 2.66 percentage points).

[0084] These results demonstrate a clear complementarity between structured external entity knowledge and the implicit knowledge learned by the pre-trained language model. Even with significant enhancements in the capabilities of the pre-trained language model itself, explicit entity knowledge can still serve as an effective vertical domain enhancement signal. Furthermore, under the LLaMA-1 encoder, the advantages of multi-interest modeling architectures (IISAN, MINER) over single-interest architectures (NRMS, LSTUR) are further amplified. IISAN+EDA achieves the best performance across all metrics, indicating that a stronger encoder can provide a richer feature base for multi-interest modeling.

[0085] Specific implementation of each ablation experiment Validation of the independent contributions of three types of entity knowledge: entity identifier features, entity relation features, and entity description features: We introduced entity identifier features, entity relationship features, and entity description features separately, and compared them with the complete EDA scheme. Experimental results show that entity identifier features have the largest independent contribution (LSTUR+ID AUC reaches 64.81), followed by entity relationship features (LSTUR+Relation AUC is 63.27), while entity description features contribute relatively less (LSTUR+Description AUC is 62.94). However, the effect of fully integrating the three types of entity knowledge (LSTUR+EDA AUC is 65.11) is better than any single knowledge source, demonstrating the complementarity of multi-perspective features and the necessity of joint modeling.

[0086] The reason why entity description features contribute relatively little independently is that the BERT encoder has already learned a relatively rich contextual semantics from the text, and the information gain brought by explicit entity description text is relatively limited. However, when used in conjunction with entity identifier features and entity relation features, entity description features can provide semantic interpretation at the natural language level, making up for the shortcomings of entity identifier features and entity relation features in terms of semantic interpretability.

[0087] Validation of the contribution of cross-perspective knowledge distribution alignment: Cross-perspective knowledge distribution alignment includes both knowledge alignment steps and EDA schemes without them. Experimental results show that the knowledge distribution alignment step brings stable positive gains to all pre-trained language models. Taking the IISAN model as an example, AUC improves from 64.85 to 65.12, and MRR improves from 35.65 to 35.92. Although the absolute increase is relatively modest, the consistent improvement across all models and all metrics demonstrates the effectiveness and robustness of cross-perspective knowledge alignment in mitigating the differences in heterogeneous knowledge distributions.

[0088] Sensitivity analysis of the number N of compressed knowledge tokens: Based on the MINER model on the MIND-small dataset, five settings of N∈{1, 2, 4, 8, 15} were tested. The experimental results show an inverted U-shaped trend of first increasing and then decreasing: the AUC is 63.15 when N=1 (due to excessive information loss), reaches the optimum of 64.58 when N=4, and drops to 63.53 when N=15 (due to redundant noise interference). These results provide clear guidance for the selection of hyperparameters in practical deployments.

[0089] Systematic analysis of specified layer locations for knowledge injection: Based on the MINER model, the injection effect of each of the 12 layers of BERT and two multi-layer joint injection strategies were tested. Experiments showed that layer 8 was the optimal injection position (AUC 64.58), layer 1 was also good (AUC 64.31), and layer 11 performed the worst (AUC 62.31). Multi-layer joint injection (AUC 63.35 for Layer [0-5] and AUC 62.68 for Layer [6-11]) was inferior to the optimal single-layer injection. These results reveal an inverted U-shaped pattern in knowledge injection positions, providing a rule of thumb for selecting the optimal injection position under different encoder architectures.

[0090] Example 3 Based on the above embodiments, this embodiment provides a system for implementing the above-described vertical domain adaptation method based on entity knowledge distribution alignment.

[0091] In this embodiment, as Figure 2 As shown, the vertical domain adaptation system based on entity knowledge distribution alignment includes: The multi-source entity knowledge construction module is used to construct multi-perspective features for each entity knowledge set in the input text (vertical text, such as news).

[0092] The cross-perspective knowledge alignment module is used to perform distribution alignment of multi-perspective features based on contrastive learning to obtain entity knowledge representations. The entity knowledge compression module is used to perform deep modeling and compression of entity knowledge representations and output a fixed number of compressed knowledge tokens. The knowledge injection module for the encoding layer is used to inject compressed knowledge tokens into the self-attention mechanism of a specified layer of the pre-trained language model encoder.

[0093] In this embodiment, the multi-view features include entity identification features, entity relationship features, and entity description features. Entity identification features are obtained from the embedding space to carry identity information; entity relationship features are extracted based on knowledge graph triples to carry structural information; entity description features are generated using a large language model and encoded through a pre-trained language encoder to carry semantic information.

[0094] In this embodiment, the cross-perspective knowledge alignment module uses the multi-perspective features of the same entity knowledge as positive samples and the multi-perspective features of other entity knowledge as negative samples. It optimizes the semantic consistency between entity knowledge representations from different sources by using a contrastive learning loss function.

[0095] In this embodiment, the entity knowledge compression module initializes a fixed number of learnable query tokens and processes the query tokens through self-attention; it performs cross-attention using the query tokens as query requests and the entity knowledge augmentation representation as keys and values, selectively extracting key entity knowledge information; and it generates the final compressed knowledge token from the selectively extracted entity knowledge information through a feedforward network.

[0096] In this embodiment, the knowledge injection module of the encoding layer is used to concatenate the compressed knowledge token into the key and value of the self-attention mechanism of the specified layer of the pre-trained language model encoder, while the query remains unchanged.

[0097] In this embodiment, the vertical adaptation system based on entity knowledge distribution alignment also includes a joint training module. The joint training module is used to jointly optimize the downstream task loss function and the contrastive learning loss function to train the parameters of the vertical adapter, while keeping the main parameters of the pre-trained language model frozen.

[0098] Example 4 This embodiment provides a method for storing knowledge-enhanced data, based on the above embodiments.

[0099] In this embodiment, the method for storing knowledge-enhanced data includes: The vertical domain adaptation method based on entity knowledge distribution alignment is implemented for the transfer of ownership on the mountain to generate compressed knowledge tokens; And store compressed knowledge tokens or pre-trained language models enhanced with compressed knowledge tokens to the storage medium.

[0100] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A vertical domain adaptation method based on entity knowledge distribution alignment, characterized in that, include: S1. Entity knowledge extraction: Entity knowledge recognition is performed on the input vertical domain text to obtain the set of entity knowledge contained in the text; S2. Multi-source knowledge construction: For each entity knowledge in the entity knowledge set, construct multi-view features respectively; S3, Cross-perspective knowledge distribution alignment: Multi-perspective features of the same entity knowledge are optimized for distribution alignment through a contrastive learning loss function to obtain entity knowledge representations with higher semantic consistency; S4. Knowledge Compression: After deep modeling the entity knowledge representation through an encoder, an enhanced entity knowledge representation is obtained. The enhanced entity knowledge representation is then compressed using a compression module based on learnable query tokens to generate a fixed number of compressed knowledge tokens. S5. Knowledge Injection: The compressed knowledge token is injected into the self-attention calculation of a specified layer of the pre-trained language model encoder to enhance the representation ability of the pre-trained language model, thereby achieving low-intrusion vertical domain knowledge enhancement. 2.The method of claim 1, wherein, In step S2, the multi-view features include entity identification features, entity relationship features, and entity description features. 3.The method of claim 2, wherein, In step S2, the entity identification feature is obtained from the embedding space to carry identity information; The entity relationship features are extracted based on knowledge graph triples to carry structural information; The entity description features are generated using a large language model and encoded using a pre-trained language encoder to carry semantic information.

4. The method of claim 1, wherein, Step S3 specifically includes: using multi-perspective features of the same entity knowledge as positive samples and multi-perspective features of other entity knowledge as negative samples, and optimizing the semantic consistency between entity knowledge representations from different sources by comparing and learning the loss function.

5. The method of claim 1, wherein, In step S4, generating a fixed number of compressed knowledge tokens using a compression module based on learnable query tokens specifically includes the following steps: S41. Initialize a fixed number of learnable query tokens and process the query tokens through self-attention; S42. Perform cross-attention using the query token as the query request and the entity knowledge augmentation representation as the key and value, and selectively extract key entity knowledge information; S43. The selectively extracted entity knowledge information is used to generate the final compressed knowledge token through a feedforward network.

6. The method of claim 1, wherein, In step S5, injecting the compressed knowledge token into the self-attention calculation of a specified layer of the pre-trained language model encoder means: The compressed knowledge token is concatenated to the key and value in the self-attention mechanism of the specified layer of the pre-trained language model encoder, while the query remains unchanged.

7. The method of claim 1, wherein, In step S5, the designated layer is the middle and high layers of the pre-trained language model encoder. 8.The method of vertical domain adaptation based on entity knowledge distribution alignment according to claim 1, wherein, It also includes the following steps: S6. Model Training: Jointly optimize the downstream task loss function and the contrastive learning loss function to train the parameters of the vertical adapter, while keeping the main parameters of the pre-trained language model frozen.

9. An entity knowledge distribution alignment based vertical adaptation system for implementing the entity knowledge distribution alignment based vertical adaptation method according to any one of claims 1 to 8, characterized in that, The vertical domain adaptation system includes: The multi-source entity knowledge construction module is used to construct multi-view features for each entity knowledge set in the input text. The cross-perspective knowledge alignment module is used to perform distribution alignment of the multi-perspective features based on contrastive learning to obtain entity knowledge representations. The entity knowledge compression module is used to perform deep modeling and compression on the entity knowledge representation and output a fixed number of compressed knowledge tokens. The encoding layer knowledge injection module is used to inject the compressed knowledge token into the self-attention mechanism of a specified layer of the pre-trained language model encoder.

10. A method of storing knowledge augmentation data, characterized by, include: The vertical domain adaptation method based on entity knowledge distribution alignment as described in any one of claims 1 to 8 is used to generate a compressed knowledge token; And store the compressed knowledge token or the pre-trained language model enhanced with the compressed knowledge token to the storage medium.

Images