Domain adaptation method and device for text embedding models

Abstract

The application provides a domain adaptation method of a text embedding model, comprising: generating a query sentence from a data set by a large language model, the query sentence focusing on a specific domain event in the data set; for the query sentence, constructing a ranking list of retrieval results by using a keyword-based retrieval method, the ranking list comprising the query sentence, samples of the ranked retrieval results, and retrieval scores corresponding to the samples; and training the text embedding model by using the ranking list and a list-level loss function based on the ranking list to obtain an adjusted text embedding model.

Description

Technical Field

[0001] This invention generally relates to the fields of natural language processing and machine learning. More specifically, this invention relates to methods and apparatus for adapting general-purpose text embedding models to private datasets using keyword-based retrieval methods. Background Technology

[0002] Text embedding is the cornerstone of various artificial intelligence (AI) applications, playing a central role, especially in information retrieval and retrieval augmentation (RAG) systems. With the widespread adoption of AI, companies like OpenAI and Cohere have launched general-purpose text embedding APIs, enabling organizations to quickly integrate AI into their RAG systems. However, while these general-purpose text embedding models perform well in general benchmarks, applying them to private datasets (such as domain-specific or enterprise-specific private data, which often contains a large amount of technical jargon and industry terminology) often presents numerous challenges.

[0003] For example, a pharmaceutical company plans to build a RAG system based on its large internal dataset. Employees might use the system to query information about internal product codes (e.g., product code: PHX-121). However, because the general-purpose model was not trained on this private dataset, it may fail to correctly understand or retrieve relevant documents containing such specific terms, resulting in suboptimal system output.

[0004] Current RAG systems commonly attempt to address this problem by combining traditional keyword-based retrieval with embedding-based retrieval. One mainstream hybrid approach is Reverse Rank Fusion (RRF), which reorders retrieval results based on a mathematical formula without requiring fine-tuning of the underlying embedding model. While simple and effective, RRF is essentially a heuristic method, and its effectiveness may be limited by the lack of fine-tuning for private datasets. Therefore, how to fine-tune a general embedding model to better align it with private datasets has become a pressing issue.

[0005] One of the core challenges in adapting embedding models to domain-specific datasets lies in the lack of available fine-tuning signals. General-purpose embedding models are typically trained on well-organized, large question-answering (QA) datasets using contrastive learning; however, private datasets are often unlabeled, free text data, which presents unique challenges for such adaptation efforts. This situation underscores the importance of developing effective methods to generate supervisory signals to adapt general-purpose embedding models to private, unlabeled datasets.

[0006] Therefore, there is an urgent need for a method that can enable general text embedding models to adapt to private unlabeled datasets by generating effective supervision signals without relying on manual annotation. Summary of the Invention

[0007] On one hand, the present invention provides a domain adaptation method for a text embedding model, comprising: Query statements are generated from a dataset using a large language model, and these queries focus on domain-specific events within the dataset. For the query statement, a keyword-based retrieval method is used to construct a sorted list of retrieval results. The sorted list includes the query statement, a sample of sorted retrieval results, and a retrieval score corresponding to the sample. The text embedding model is trained using the sorted list and a list-level loss function based on the sorted list to obtain an adjusted text embedding model.

[0008] In embodiments of the present invention, generating query statements from a dataset using a large language model includes: The first major language model is used to extract events and their related parameters from the dataset. The dataset, along with the extracted events and their related parameters, is provided to the second language model. By prompting the second language model, a query statement is generated for each event.

[0009] In an embodiment of the present invention, for the query statement, constructing a sorted list of search results using a keyword-based retrieval method includes: The dataset is divided into multiple data blocks. Calculate the retrieval score for each data block. Based on the retrieval scores of the data blocks, a subset of data blocks are selected from the plurality of data blocks. The selected data block is divided into multiple intervals. The sorted list is obtained by sampling each of the plurality of intervals.

[0010] In an embodiment of the present invention, selecting a subset of data blocks from the plurality of data blocks includes selecting the top-scoring data blocks from the plurality of data blocks. Data blocks.

[0011] In an embodiment of the present invention, the list-level loss function is: in, The query statement, For the collection of generated query statements, The number of the plurality of intervals, This is a sample obtained by sampling the interval. The retrieval score corresponding to the sample, The embedding vector and sample corresponding to the query statement The similarity between the corresponding embedding vectors and They are and The distribution obtained by normalization using the softmax function.

[0012] In an embodiment of the present invention, in, It is the temperature scaling factor.

[0013] In an embodiment of the present invention, a fine-to-coarse partitioning strategy is adopted to divide the selected data blocks into multiple intervals, such that the partitioning granularity of the interval with higher ranking is higher than that of the interval with lower ranking.

[0014] In an embodiment of the present invention, the keyword-based retrieval method includes BM25, and the retrieval score is the BM25 score between the query statement and the data block.

[0015] In an embodiment of the present invention, the dataset is a collection of multiple domain-specific documents.

[0016] On the other hand, the present invention also provides an electronic device, comprising: at least one processor; and a memory communicatively connected to said at least one processor. The memory stores one or more computer programs executable by said at least one processor, said one or more computer programs being executed by said at least one processor to enable said at least one processor to perform the method according to the present invention.

[0017] On the other hand, the present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method according to the present invention.

[0018] The method of this invention has been evaluated on multiple domains and datasets, using two general text embedding models of different scales. Compared with the basic text embedding model, the text embedding model generated by the method of this invention consistently achieves a significant improvement in retrieval accuracy. In addition, experiments show that its performance is superior to or equal to that of two commonly used techniques in current RAG systems: (1) fine-tuning technique based on intra-batch negative sample contrastive learning; and (2) RRF hybrid method.

[0019] To gain a deeper understanding of the internal operating mechanism of the method of this invention, the alignment and uniformity characteristics of the adapted text embedding model were studied. The results show that BMEmbed improves the uniformity of the text embedding model while maintaining good alignment, thereby enhancing retrieval performance.

[0020] In summary, this invention provides a simple and efficient method for adapting general text embedding models to private datasets. With the increasing adoption of RAG systems across industries, the proposed method is considered a practical solution for improving domain-specificity, thereby enabling more accurate and context-aware retrieval results in real-world applications. Attached Figure Description

[0021] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof. The above and other features and advantages will become more apparent to those skilled in the art from the detailed description of exemplary embodiments with reference to the accompanying drawings, in which: Figure 1 This diagram illustrates how a text embedding model can be adapted to a specific private domain. Figure 2 An overall architecture diagram of the domain adaptation method for text embedding models according to the present invention is shown; Figure 3 The charts show statistical information and implementation details of the dataset; Figure 4 The charts show the retrieval performance of different methods on three datasets, and indicate the best results for each text embedding model on each dataset. Figure 5 Showing different Retrieval performance, measured by MAP@10, under the value and sampling strategy; Figure 6 Showing different Alignment and uniformity results under the value and sampling strategy; Figure 7 Showing different At this value, retrieval performance is measured in MAP@10. Figure 8 Showing different Alignment and uniformity results under the given values; Figure 9 The charts present the alignment and uniformity data for each text embedding model, where lower alignment (↓) and higher uniformity (↑) are better, and the best results for each text embedding model on each dataset are marked. Figure 10 An example comparing the original query with its masked version is shown; Figure 11 The chart presents the results of a comparative experiment involving query masks between the two models; Figure 12 The prompt words used for event retrieval are shown; Figure 13 The suggested words generated for the query are shown; Figure 14 The process of generating a real query is shown, including the input corpus, extracted events, and generated query; Figure 15 The charts summarize the results of the ablation experiments; Figure 16 The prompts used for masked keyword extraction are shown. Figure 17 This is a block diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0022] To enable those skilled in the art to better understand the technical solutions of the present invention, exemplary embodiments of the present invention are described below in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0023] Where there is no conflict, the various embodiments of the present invention and the features thereof may be combined with each other.

[0024] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having the meaning consistent with their meaning in the context of the relevant art and the invention, and will not be interpreted as having an idealized or overly formal meaning unless expressly so defined herein.

[0025] This invention proposes a method to adapt a general text embedding model to a private dataset. This method utilizes a keyword-based retrieval function to generate a supervision signal from the ranking of keyword retrieval results.

[0026] Figure 1 A schematic diagram illustrating the adaptation of a text embedding model to a specific private domain is shown. The method according to the present invention comprises the following three main parts: (1) Domain query generation: prompting the large language model to generate synthetic queries based on specific domain events extracted from a private corpus (or private dataset) (i.e., the generated queries are automatically synthesized). (2) Relevant sampling: Use keyword-based retrieval methods to retrieve relevant paragraphs and samples at the word level from different intervals of the sorted list to obtain effective signals; (3) List-level fine-tuning: On the sorted list, the text embedding model is optimized using a list-level loss function to make full use of the sorting supervision information.

[0027] Unlike traditional in-batch negative sample contrastive learning, the method according to the present invention uses the ranking results obtained by the keyword-based retrieval method to guide the fine-tuning process of the text embedding model.

[0028] In this article, "private corpus / dataset" refers to a collection of data that has a specific affiliation (such as a company, organization, or specific field), is not publicly disclosed, and usually contains domain-specific terminology or industry jargon.

[0029] Text embedding refers to the numerical representation of text to capture its semantic information and transform text of varying lengths into fixed-dimensional vectors. In the past, fine-tuning models like BERT and T5 to adapt to downstream embedding tasks was the mainstream approach. However, with the development of large language models, this landscape is changing. Current research has shifted towards building general embedding models based on large language models, including Qwen, LLM2Vec, and NV-Embed. These large language model-based embedding models have demonstrated superior performance on large-scale text datasets, such as the MTEB large-scale text embedding benchmark.

[0030] Current embedding models are primarily trained through contrastive learning, with the widely adopted objective function being the InfoNCE loss function, which aims to distinguish semantically related text pairs from semantically unrelated ones. While contrastive learning is highly effective, its performance largely depends on the selection of high-quality positive and negative samples. Constructing relevant and irrelevant samples from private corpora is a challenging task when adapting embedding models to specific domains. This invention proposes a keyword-based retrieval method to construct lexical-level relevant samples, addressing the sample selection challenge in an unsupervised manner.

[0031] In embodiments of the present invention, the keyword-based retrieval method includes the Best Match 25 (BM25) algorithm. The following description will use the BM25 algorithm as an example to illustrate the technical solution of the present invention. It is understood that mentioning or using BM25 should not be construed as limiting the scope of protection of the present invention solely to BM25. Based on the teachings of this invention, other keyword-based retrieval methods can also replace BM25. In other words, the use of BM25 represents only one implementation of the technical solution disclosed herein; in some implementations, other keyword-based retrieval methods may also be used instead of BM25.

[0032] BM25 is a retrieval method based on Term Frequency-Inverse Document Frequency (TF-IDF). This method ranks documents by considering the uniqueness and importance of terms relevant to a given query. For query q, the BM25 score of document d is defined as follows: in, Representation of terms In the document Word frequency in; A normalized value representing the document length; Representation of terms Inverse document frequency in a corpus; and The hyperparameters are used to control the influence of word frequency and document length on the results, respectively. It has been confirmed that using BM25 as a weak supervision signal for training small models is effective.

[0033] Despite significant progress in dense retrieval, BM25 remains a robust retrieval algorithm. Its rule-based keyword matching mechanism exhibits strong generalization capabilities, maintaining competitive performance even in scenarios where keyword matching is more critical than semantic matching. Therefore, existing research has employed hybrid methods (such as RRF) to fuse and reorder the results of dense retrieval models (based on embeddings) and sparse retrieval models (based on BM25). However, RRF relies on heuristic rules to rank these hybrid results; in contrast, this invention aims to fine-tune a general text embedding model to a specific dataset, achieving true adaptation, rather than simply fusing the results of different retrieval methods.

[0034] The following section introduces a domain adaptation method for the general text embedding model according to the present invention, aiming to adapt the general text embedding model to private datasets composed of unlabeled data. The method comprises three steps, and the overall process is as follows: Figure 2 As shown.

[0035] During the domain query generation stage, a large language model (such as GPT-4) is prompted to generate synthetic queries. The synthetic queries are directed to specific domain events with predefined domain event types in the private corpus (i.e., factual units that can be identified and extracted from the corpus as <event type, participating entities, time and attributes>, such as profit warning, merger and acquisition announcement or claim refusal), while targeting non-general concept topics.

[0036] In embodiments of the present invention, all events and their related parameters are extracted from a private corpus using a large language model; at the same time, the original context on which the event extraction is based is also generated, which will serve as the basis for the query used by the baseline method in subsequent experiments.

[0037] Subsequently, the private corpus and extracted events are provided to the large language model, which then prompts the model to automatically generate queries for each event. . Figure 12 and Figure 13 An example of large language model prompt words used in the domain query generation phase is shown.

[0038] During the relevant sampling phase, the keyword retrieval method BM25 was used to construct a sorted list of retrieval results.

[0039] The private corpus is divided into multiple text chunks, and the calculation is performed for each query. BM25 score between each text block, select the top score The text block (denoted as) ), where each text block Each of these corresponds to its BM25 score This is related. Understandably, each query can correspond to multiple extracted events. Within the corpus, multiple matching query results can be generated for a single event.

[0040] Further set Divided into Let the intervals be denoted as . This method can sample positive and negative samples from different intervals, thereby expanding the sampling space across different relevance levels and effectively reducing noise in BM25 pseudo-labels. The partitioning can employ a uniform partitioning strategy or a fine-to-coarse partitioning strategy. The uniform partitioning strategy divides the BM25 score range into equally sized segments, ensuring a consistent distribution of samples across all intervals. In contrast, the fine-to-coarse partitioning strategy prioritizes finer segmentation of high-relevance scores, thus achieving more refined sampling of the top-ranking positive sample classes. For example, when... When using a fine-to-coarse strategy, the top 20 ranked list can be divided into the following intervals according to the ranking: [0,2), [2,6), [6,12), [12,20), while the uniform strategy divides it into the following intervals: [0,5), [5,10), [10,15), [15,20).

[0041] For each interval Randomly select a sample and its search score To form a sorted list .

[0042] In the list-level fine-tuning stage, the basic text embedding model is fine-tuned using a list-level loss function.

[0043] Since the BM25 retrieval results generate a ranked list, assuming this list contains valuable ranking information, a list-level training objective can better utilize this ranking information compared to the commonly used intra-batch negative sample contrast learning objective (which typically ignores ranking information). Therefore, this invention employs a list-level training objective to fully utilize the ranking information obtained from the BM25 retrieval.

[0044] Given and basic text embedding models First, obtain the query. With sample ( The embeddings of ) are denoted as follows: and Then, the cosine similarity between the two is calculated, i.e. The list-level loss function is calculated as follows: in, , and They are and The distribution obtained after normalization using the softmax function. (In the target score list) Introducing a temperature scaling factor : here, Used to control the steepness of the target distribution: The smaller the value, the more concentrated the target distribution; The larger the value, the smoother the target distribution.

[0045] In this invention, the basic embedding model can adopt the following two general embedding models: gte-Qwen2-1.5B-instruct (a small but powerful model) and e5-mistral-7B-instruct (a large model based on Mistral-7B). Both models have shown strong competitiveness on the MTEB leaderboard.

[0046] The fine-tuned embedding model based on the method of the present invention (hereinafter referred to as BMEmbed) is compared with the following benchmark methods: 1) BM25 (parameters set to k1=1.2, b=0.75); 2) Basic Embedded Model (Base); 3) An embedding model (CL) fine-tuned using the contrastive target InfoNCE loss function, where evidence generated by the large language model is used as positive samples, combined with in-batch negative samples; 4) Reverse Rank Fusion (RRF), a hybrid retrieval method that merges ranking results from multiple sources into a unified ranking: in It's a document. It is a set of sorters (retrieval machines). It is a document In the sorter The sorting in It is a constant set to 40. Here, the ranking results of BM25 will be combined with the ranking results of the underlying embedding model. 5) Combination scheme of BM25 with the model finely tuned by the method (BMEmbed) according to the present invention (RRF+BMEmbed).

[0047] In the experiments (main experiments) provided by this invention, three publicly available retrieval datasets were selected as evaluation benchmarks. However, these datasets were all released after the base embedding model, meaning that the base embedding model is unlikely to have used these datasets as training data. Therefore, although these datasets are publicly available, they can effectively simulate "private" datasets in this experimental setup, while ensuring the fairness of the comparison and the reproducibility of the results.

[0048] Specifically, the three datasets are: Multihop-RAG (a multi-hop question-answering (QA) dataset from the financial news field), Finance-RAG (a long-text QA dataset based on financial reports, released as part of the ACM-ICAIF'24 FinanceRAG competition), and LegalBench-RAG (a challenging long-text legal QA dataset). Each dataset contains questions, corresponding relevant evidence, and the original corpus, where the evidence is used as labels to evaluate retrieval performance. See [link to detailed statistics]. Figure 3 The chart.

[0049] In the domain query generation stage, GPT-4o was used for precise event extraction, and GPT-4o-mini was used for query synthesis to minimize costs. Based on the corpus size, 5972, 1009, and 685 queries were generated for Multihop-RAG, Finance-RAG, and LegalBench-RAG, respectively. Figure 3 As shown. Figure 14This demonstrates a real-world query generation process, showcasing the input document, extracted events, and the final generated query. During the relevance sampling phase, the text block size for Multihop-RAG is set to 256, while the other two long text datasets use a block size of 1024. A "fine-to-coarse" partitioning strategy is used by default. The parameter 'i' for Multihop-RAG is set to 9, while the other two datasets use 'i' set to 6; the parameters for Multihop-RAG and Finance-RAG... The value is 1000, for LegalBench-RAG. The value is 4000. The impact of different 𝑚 values ​​and partitioning strategies is further discussed below. See the experimental results at the value. Figure 15 During the list-level fine-tuning phase, the batch size for contrastive learning (CL) is fixed at 16, and the batch size of BMEmbed in this invention is set to the same value as... Values ​​are equal; temperature scaling factor The rank was set to a moderate value between 1.0 and 5.0, and further adjusted based on the dataset and model type, which is discussed in detail below. The model was fine-tuned using the low-rank adaptation (LoRA) technique with a rank of 16 and 1000 fine-tuning steps. Training the Qwen model on four 3090 GPUs took approximately 1.5 hours, while training the e5-mistral model on eight H800 GPUs took approximately 1 hour.

[0050] The adjusted embedding model can respond to new user queries and output a sorted list of relevant text fragments retrieved from the private dataset.

[0051] Figure 4 Experimental results of the present invention's BMEmbed and all baseline models on two basic embedding models and three datasets are presented, with specific observations as follows: 1) The original embedding model performs poorly in specific domains. In most cases, even large-sized base embedding models outperform BM25 on the Multihop-RAG and Finance-RAG datasets. This finding suggests that further domain adaptation is necessary when applying general embedding models to specific domains. 2) Contrastive learning (CL) fails to consistently deliver performance improvements in embedding model adaptation. Surprisingly, this invention finds that applying contrastive learning training to the base embedding model does not always improve performance. This is presumably because noise in the positive sample evidence generated by large language models may interfere with the model optimization process. This result indicates that contrastive learning is sensitive to the quality of positive and negative samples, and this method does not always achieve the desired performance improvement in embedding model adaptation tasks. 3) The BMEmbed model provided by this invention achieves continuous and stable performance improvement by leveraging the supervision signal provided by BM25. The method of this invention enhances the performance of the base model on all embedding models and datasets, especially excelling in the Hit@4 metric. Compared with the RRF method, which fuses BM25 ranking information with dense retrieval results of the embedding model, the BMEmbed model provided by this invention achieves significant improvement. This demonstrates that the method of this invention can deeply analyze the ranking confidence signal of BM25, thereby achieving better domain adaptation of the embedding model. 4) Furthermore, the method of this invention can be combined with other hybrid retrieval methods to further improve performance. This is verified in the comparative experiments of RRF+BMEmbed and RRF alone: ​​in most cases, RRF+BMEmbed showed a significant performance gain, except in the LegalBench-RAG dataset—in which the BM25 baseline model itself performed poorly, so RRF+BMEmbed failed to achieve further performance improvement.

[0052] To further explore the underlying mechanism by which the method of this invention improves performance, the interaction between sampler selection and temperature scaling factor on retrieval performance was studied through ablation experiments. In addition, core characteristics reflecting embedding quality, namely alignment and uniformity, were introduced to provide a deeper theoretical explanation.

[0053] Ablation experiments were conducted using the Multihop-RAG dataset and the Qwen2-1.5B model by default.

[0054] Specifically, on the Multihop-RAG dataset, three sets of experiments were conducted using the Tongyi Thousand Questions model, controlling different variables according to the process to explore four key factors: Choice: Explored The cases when taking values ​​of 200, 500, and 1000. Choice: Examined The case where the value is between 6 and 10. Sampling strategies: Two approaches were compared: fine-to-coarse and uniform. The first partition was fixed at 0 to 3 to ensure information-rich positive samples, while the remaining partitions were divided according to the chosen strategy. Specifically, when using a fine-to-coarse strategy, for a given... and The length of each subsequent interval is twice that of the previous interval. This can be expressed by the formula: hyperparameters For convenience, its reciprocal is used ( The values ​​are set to 0.1, 0.2, 0.5, 0.7 and 1.0.

[0055] The experiment was set up as follows: 1. Fix the temperature scaling factor to 1. Fixed at 1000, testing different Values ​​and sampling strategies. 2. Fix the temperature scaling factor to 1. The value was fixed at 10, and a fine-to-coarse strategy was adopted to explore different... The impact of the value. 3. Fixed at 500 The value was fixed at 10, and a fine-to-coarse strategy was adopted. The effect of changing the temperature scaling factor was then examined.

[0056] Ablation test results (e.g.) Figure 15 As shown in the figure, fine-tuned embedding models with lower alignment but higher uniformity tend to achieve better results in retrieval tasks. There is a strong correlation between retrieval performance and these two attributes: specifically, embedding models with better alignment tend to produce better retrieval results; furthermore, when alignment is similar, models with higher uniformity perform better. These findings suggest that the proposed strategy can be used to adjust alignment and uniformity to optimize retrieval performance.

[0057] Ablation experiments empirically demonstrate a strong correlation between embedding characteristics and retrieval performance, implying that the performance improvement of the embedding model adjusted according to the method of this invention stems from the optimization of embedding characteristics. The core observations and conclusions will be discussed in detail below.

[0058] High-quality embeddings should cluster similar data points more closely while retaining enough useful information to distinguish different data points, ultimately achieving lower alignment and higher uniformity. This paper uses alignment and uniformity as evaluation metrics for embedding quality.

[0059] Alignment measures the degree of positional matching of similar data points in the embedding space. It is quantified by calculating the average Euclidean distance between the embedding vectors of all positive sample pairs. Uniformity reflects the distribution of data points in the embedding space. It is quantified using a Gaussian kernel (this method captures the pairwise similarity between all data points in the distribution). The representations of both are as follows: in Represents positive sample pairs in the dataset. It is the data distribution of all data points. It is an embedding model that maps input data points to corresponding embedding vectors in a high-dimensional space. In this experiment, The reference question is associated with its corresponding block of evidence, which is randomly sampled from each document and forms the basis for calculating uniformity. gather.

[0060] Because fine-tuning further corrects the model's alignment, making comparisons between different models difficult, a scaling factor is introduced to address this issue. A model with high alignment doesn't necessarily perform worse than a model with low alignment in retrieval tasks. If a high-alignment model can also ensure that negative samples are more dispersed relative to positive samples, it can still achieve excellent retrieval performance. Based on this, the distance between the query and its nearest neighbor embedding vector in the database is defined as the alignment scaling factor. In subsequent experiments, a normalized version of the alignment is used, represented as follows: in Refers to the embedding of questions in the database The closest embedding. Furthermore, since the original uniformity was negative, the absolute value of the uniformity was used for analysis in this experiment. This facilitates comparative analysis, where a larger absolute value indicates a more uniform distribution of the embedding model.

[0061] To investigate the impact of different partitioning methods on the BM25-based correlation sampling process in this invention, the roles of various partitioning factors, including the number of partitions and the partitioning strategy, were explored. Specifically, this invention sets... The value range is 6 to 10, and two schemes are adopted: uniform sampling strategy and "fine to coarse" sampling strategy, among which the temperature scaling factor Set to 1, The value is set to 1000. Figure 5 Showing different The relationship between the retrieval metric MAP@10 and performance of the fine-tuned model under the value and sampling strategy; Figure 6 The uniformity and alignment of the fine-tuned model shown in the previous figure were compared. Experiments revealed that, compared to the uniform sampling strategy, the "fine-to-coarse" sampling strategy achieves better retrieval performance and alignment. In contrast, the uniform sampling strategy, due to its overly uniform embedding distribution, suffers from alignment loss, resulting in poor retrieval performance. Furthermore, under the "fine-to-coarse" sampling strategy, when... Increasing the value from 6 to 7 significantly improved MAP@10 performance, indicating that moderately expanding the sampling range can capture more relevant samples; however, further increasing the value... This value can cause performance fluctuations, and the overall performance gradually declines. These findings indicate that precise calibration is necessary. Values ​​are used to optimize retrieval performance.

[0062] This invention explores the temperature scaling factor The effect of taking different values. For ease of experimental analysis, this invention uses its reciprocal. , The values ​​were set to 0.1, 0.2, 0.5, 0.7, and 1.0 respectively; the experimental parameters were set to... , The sampling strategy adopts a "fine-to-coarse" approach.

[0063] Figure 7 Showing different At this value, the trend of the MAP@10 index and performance of the fine-tuned model, and the corresponding uniformity and alignment results are as follows: Figure 8 As shown. Analysis indicates that the smaller the temperature scaling factor (i.e., ... The larger the temperature scaling factor, the better the uniformity of the embedding distribution, leading to better retrieval performance; in contrast, as the temperature scaling factor increases (i.e., ... The uniformity gradually decreases, even falling below the uniformity level of the base model. This phenomenon is due to the fact that a large temperature scaling factor smooths the label distribution, reduces the discriminative power between training samples, and leads to excessive clustering of embedding vectors. This excessive clustering impairs the performance of downstream tasks that require clear differentiation of embedding vectors—as observed in this experiment, retrieval performance drops significantly in this case.

[0064] Ablation experiments and related analyses show that the "fine-to-coarse" strategy, combined with a small temperature scaling factor, is an effective way to fully utilize BM25. This conclusion is supported by both theoretical basis and empirical evidence. Since the main experiment described above is based on this strategy, this invention... Figure 9 The paper discloses uniformity and alignment data for the corresponding fine-tuned embedding model for further analysis.

[0065] The fine-tuned embedding model BMEmbed, compared to the base model, significantly improves uniformity while maintaining relatively low alignment, thus achieving better retrieval results. In contrast, the contrastive learning (CL) model using in-batch negative samples, while achieving a significant improvement in uniformity, failed to effectively maintain or improve the alignment of the base model. This imbalance leads to unstable retrieval performance, and in some cases, even performance degradation. Specifically, this invention clarifies the ideal optimization direction for embedding characteristics (such as...). Figure 6 (As shown by the arrow in the image), the fine-tuned embedding model BMEmbed achieved this theoretical optimization goal on both the Multihop-RAG and Finance-RAG datasets, fully demonstrating its technical potential to balance uniformity and alignment optimization.

[0066] To explore how BM25-driven signals promote domain adaptation of the embedding model, this invention conducted keyword masking experiments on query statements. Specifically, this invention compared the retrieval performance of the base model Qwen2-1.5B and its fine-tuned variant (BMEmbed) under two conditions: 1) original query statements; 2) keyword-masked query statements. The experimental procedure consisted of three steps: First, a large language model was used to extract domain-specific keywords from each query statement. The suggested words used are described in [link to relevant documentation]. Figure 16 The second step is to mask the identified keywords (see example). Figure 10 The process involves preserving the syntactic structure of the statements, ultimately resulting in 2255 query statement pairs. The third step is to evaluate the performance degradation of the two models when processing masked query statements and original query statements.

[0067] like Figure 11 As shown, the retrieval performance of BMEmbed and the base model under keyword masking conditions was compared. Experimental results show that after masking the keywords, the performance advantage of BMEmbed is completely eliminated, and the accuracy of the two models drops to similar levels. This result indicates that the performance improvement of BMEmbed's fine-tuning is mainly due to the model's increased attention to domain-specific keywords, consistent with the core assumption of this invention. Furthermore, the above findings also demonstrate that the list-level fine-tuning strategy of this invention, through BM25-guided pattern learning, can effectively enable BMEmbed to internalize private domain features.

[0068] As AI becomes increasingly prevalent in practical applications (especially RAG systems), adapting general-purpose models to domain-specific data remains a key challenge. This invention proposes a method for adapting text embedding models to domain-specific datasets (e.g., enterprise-specific data). Since private datasets typically contain specialized terminology and domain-specific expressions, this invention uses keyword-based retrieval as a supervisory signal to fine-tune the general embedding model. Experimental results show that BMEmbed effectively improves retrieval performance, producing more accurate query results on private datasets. As AI continues to empower various industries, the method proposed in this invention will further promote the implementation and adaptation of AI in specific domain applications, achieving more efficient and context-sensitive accurate retrieval.

[0069] Figure 17 This is a block diagram of an electronic device provided in an embodiment of the present invention.

[0070] Reference Figure 17This invention provides an electronic device comprising: at least one processor 1701; at least one memory 1702; and one or more I / O interfaces 1703 connected between the processor 1701 and the memory 1702; wherein the memory 1702 stores one or more computer programs executable by the at least one processor 1701, and the one or more computer programs are executed by the at least one processor 1701 to enable the at least one processor 1701 to perform the above-described method for self-calibrating an omnidirectional radiation field.

[0071] The modules in the aforementioned electronic device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of the computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0072] This invention also provides a computer-readable storage medium storing a computer program thereon, wherein the computer program, when executed by a processor / processor core, implements the aforementioned domain adaptation method for the embedded model. The computer-readable storage medium may be volatile or non-volatile.

[0073] This invention also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code. When the computer-readable code is run in a processor of an electronic device, the processor in the electronic device executes the domain adaptation method of the above-described embedding model.

[0074] Each module in the above system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0075] Those skilled in the art will understand that all or some of the steps, systems, and apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software can be distributed on a computer-readable storage medium, which may include computer storage media (or non-transitory media) and communication media (or transient media).

[0076] As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable program instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technologies, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is known to those skilled in the art that communication media typically contain computer-readable program instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0077] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0078] The computer program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions. This electronic circuitry can execute the computer-readable program instructions to implement various aspects of the invention.

[0079] The computer program product described herein can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0080] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0081] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0082] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0083] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0084] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for illustrative purposes only and should be construed as such, and is not intended to be limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in conjunction with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in conjunction with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of the invention as set forth in the appended claims.

Claims

1. A domain adaptation method for a text embedding model, comprising: Query statements are generated from a dataset using a large language model, and these queries focus on domain-specific events within the dataset. For the query statement, a keyword-based retrieval method is used to construct a sorted list of retrieval results. The sorted list includes the query statement, a sample of sorted retrieval results, and a retrieval score corresponding to the sample. The text embedding model is trained using the sorted list and a list-level loss function based on the sorted list to obtain an adjusted text embedding model.

2. The method according to claim 1, wherein, Generating query statements from a dataset using a large language model includes: The first major language model is used to extract events and their related parameters from the dataset. The dataset, along with the extracted events and their related parameters, is provided to the second language model. By prompting the second language model, a query statement is generated for each event.

3. The method according to claim 1, wherein, For the query statement, a sorted list of search results is constructed using a keyword-based retrieval method, including: The dataset is divided into multiple data blocks. Calculate the retrieval score for each data block. Based on the retrieval scores of the data blocks, a subset of data blocks are selected from the plurality of data blocks. The selected data block is divided into multiple intervals. The sorted list is obtained by sampling each of the plurality of intervals.

4. The method according to claim 3, wherein, Selecting a subset of data blocks from the plurality of data blocks includes selecting the data blocks with the highest retrieval scores. Data blocks.

5. The method according to claim 4, wherein, The list-level loss function is: in, The query statement, For the collection of generated query statements, The number of the plurality of intervals, This is a sample obtained by sampling the interval. The retrieval score corresponding to the sample, The embedding vector and sample corresponding to the query statement The similarity between the corresponding embedding vectors and They are and The distribution obtained by normalization using the softmax function.

6. The method according to claim 4, wherein, in, It is the temperature scaling factor.

7. The method according to claim 3, wherein, A fine-to-coarse partitioning strategy is adopted to divide the selected data blocks into multiple intervals, so that the partitioning granularity of the interval with higher ranking is higher than that of the interval with lower ranking.

8. The method according to any one of claims 1-7, wherein, The keyword-based retrieval method includes BM25, and the retrieval score is the BM25 score between the query statement and the data block.

9. The method according to any one of claims 1-7, wherein, The dataset is a collection of documents from multiple specific domains.

10. An electronic device, comprising: At least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores one or more computer programs that can be executed by the at least one processor to enable the at least one processor to perform the method as described in any one of claims 1 to 9.

11. A computer-readable storage medium having a computer program stored thereon, wherein, The computer program, when executed by a processor, implements the method as described in any one of claims 1 to 9.

Images