A fine-tuning method of a pre-trained text representation model

Abstract

The application relates to the technical field of natural language processing, and provides a fine-tuning method of a pre-trained text representation model, which obtains a final text representation model through multiple rounds of fine-tuning, in each round of iteration, based on the similarity of a query and candidate documents, candidate documents ranked in an interval are intercepted from high to low to form a difficult negative sample candidate set of the query, and a plurality of documents are randomly sampled from the difficult negative sample candidate set to form a difficult negative sample set of the query; finally, based on the query and the difficult negative sample set, training samples for this round of training are constructed, and based on the training samples, this round of fine-tuning is completed. Through truncation sampling and random sampling, the obtained difficult negative samples have high confusion and low false negative sample risk; through multi-stage iteration, difficult-to-distinguish negative samples are found again in each stage, so that the model continuously faces new high-confusion samples, and the problem of gradient weakening is alleviated. The application is suitable for text representation in tasks such as question answering, case matching, document recommendation and business auditing.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and more specifically to a method for fine-tuning a pre-trained text representation model. Background Technology

[0002] In tasks such as semantic retrieval, similar text matching, question answering recall, and professional knowledge base retrieval, text representation models are typically used to map queries and candidate texts to a unified vector space and determine the semantic relevance between texts through vector similarity. The quality of text representation directly affects the recall accuracy and ranking performance of the retrieval system. If the text representation model cannot accurately express the true semantic relationship between the query and candidate texts, even if the retrieval system adopts an efficient vector index structure, it will be difficult to obtain stable and reliable retrieval results.

[0003] Existing text representation models, typically pre-trained or fine-tuned based on large-scale general corpora, can handle general semantic similarity matching tasks, but still have significant shortcomings in professional scenarios. Taking financial texts as an example, financial announcements, research reports, financial statements, regulations, and business Q&As contain a large number of proper nouns, abbreviations, numerical constraints, time conditions, subject relationships, and causal relationships. Some texts are highly similar in literal expression, but their true semantics, applicable objects, or conclusions are completely different; other texts have significant literal differences, but are actually related in business semantics. Existing text representation models tend to capture literal similarity more easily, but struggle to reliably distinguish the aforementioned fine-grained differences, causing retrieval systems to recall noisy or semantically confusing documents to the top of the search results.

[0004] In existing technologies, to improve the domain adaptability of text representation models, contrastive learning is typically used for fine-tuning. This involves constructing positive and negative samples for the query, allowing the model to learn the relative distance relationships between the query and the positive and negative samples. To enhance the training signal, existing methods usually select high-ranking documents that are not labeled as positive samples from the current model's retrieval results as hard negative samples. A high ranking means that the current model considers these documents to be very close to the query in the vector space; not being labeled as positive samples excludes the true correct answer. Therefore, hard negative samples refer to those that do not match the positive samples but appear superficially very similar to them.

[0005] However, in specialized domain data, annotations are often incomplete. A query may have multiple truly relevant documents, but the training data only annotates a portion of them. If the top-ranking unannotated document is directly selected as a negative sample, this document is likely a missed truly relevant document. Forcing such documents as negative samples causes the model to push away semantic representations that should be similar, thereby disrupting the text vector space structure and reducing the model's recall ability for truly relevant text.

[0006] Secondly, existing methods often mine difficult negative samples all at once before fine-tuning and reuse them in subsequent training. After several rounds of training, the originally difficult negative samples may become easy to distinguish, the loss function may approach saturation, the training gradient may weaken, and the model may struggle to learn deeper semantic differences, easily getting stuck in a local optimum. However, if the fine-tuning process continues to focus on increasingly difficult negative samples, the model in the final stage may become more biased towards distinguishing highly confusing samples in a specific training set, weakening the general semantic matching ability already possessed by the initial model, leading to catastrophic forgetting or a decline in generalization ability. Summary of the Invention

[0007] The technical problem to be solved by this invention is to propose a fine-tuning method for a pre-trained text representation model that can continuously provide effective hard negative sample training signals while reducing the probability of introducing false negative samples.

[0008] The technical solution adopted by the present invention to solve the above-mentioned technical problems is as follows:

[0009] A method for fine-tuning a pre-trained text representation model includes the following steps:

[0010] S1. For the pre-trained text representation model, construct a query set, a positive sample set, and a candidate document set for fine-tuning it; the query set is a set of queries, the positive sample set is a set of labeled positive sample documents, and the candidate document set is a set of unlabeled candidate documents in the target domain.

[0011] S2. Construct query samples for this round of training; each query sample includes a query and at least one positive sample document that matches the query; using the current text representation model, encode the query of each query sample and each candidate document contained in the candidate document set to obtain their corresponding vector representation.

[0012] S3. For each query sample, process it separately according to the following steps:

[0013] S31. Calculate the similarity between the vector representation of each candidate document in the candidate document set and the vector representation of the query in the query sample, and sort the candidate documents from high to low similarity.

[0014] S32. From the candidate documents sorted in step S31, extract those ranked... The candidate documents within the specified interval constitute the candidate set of hard-negative samples for this query sample. As a preset lower bound, the As a preset upper limit for truncation, the ;

[0015] S33. Randomly sample from the candidate set of hard-to-bear samples of the query sample. The document constitutes the hard-burden sample set for this query sample. ;

[0016] S34. Construct training samples corresponding to the query sample, wherein the query of the training sample is the query of the corresponding query sample, the positive samples of the training sample are the positive sample documents of the corresponding query sample, and the negative samples of the training sample include the candidate documents contained in the hard negative sample set of the corresponding query sample.

[0017] S4. Based on the training samples corresponding to each query sample obtained in step S3 for this round of training, the parameters of the current text representation model are fine-tuned using the contrastive learning algorithm.

[0018] S5. Determine whether the preset number of fine-tuning iterations has been reached; if not, use the text representation model fine-tuned in step S4 as the new current text representation model and return to step S2; if it has been reached, obtain the final text representation model.

[0019] Specifically, in step S4, the contrast loss is calculated according to the following formula:

[0020]

[0021] in, Indicates the first The encoding function of the text representation model obtained from rounds of iteration. Represents the similarity function. Represents an exponential function with the natural constant as its base; For temperature coefficient, For query, As a positive sample, For negative samples, Indicates query The negative sample set, wherein the negative samples contained in the negative sample set include documents contained in the hard negative sample set.

[0022] Furthermore, in step S5, if the preset number of fine-tuning iterations is reached, the parameters of the text representation model obtained in each round of fine-tuning iterations are fused to obtain the parameters of the final text representation model.

[0023] Furthermore, the final model parameters are obtained by fusing the parameter increments of the text representation model obtained in each iteration relative to the pre-trained text representation model in step S1, as follows:

[0024] S51. Calculate the parameters of the text representation model obtained in each iteration according to the following formula, relative to the initial text representation model parameters. parameter increment :

[0025]

[0026] in, For the first The parameters of the text representation model obtained from rounds of iteration;

[0027] S52. Calculate the parameter increment of the text representation model obtained in each iteration using the following formula. Sparsification is performed to obtain the sparsified parameter increments. :

[0028]

[0029] in, The preset discard probability, According to the probability of discard The generated Bernoulli random mask, This indicates element-wise multiplication;

[0030] S53. Using the following formula, fuse the sparsified parameter increments of the text representation model obtained from each iteration. To obtain the final model parameters :

[0031]

[0032] in, This is the preset number of fine-tuning iterations. For the first The preset weights of the text representation model obtained from rounds of iteration.

[0033] Optimally, in step S52, the discard probability ,and The For the preset number of fine-tuning iterations, the This is the preset discard rate.

[0034] Optimally, in step S53, the first step is calculated according to the following formula. Preset weights of the text representation model obtained from rounds of iteration :

[0035]

[0036] in, Indicates the iteration round, This is the preset number of fine-tuning iterations.

[0037] Optimally, in step S32, the lower bound is truncated. The above and truncation upper bounds In step S33, the number of documents sampled. .

[0038] Optimal, in step S2, the batch size of query samples used for this round of training is 32-64, and in step S33, the number of documents sampled... .

[0039] Optimal, the number of fine-tuning iterations .

[0040] Optimally, in step S33, random sampling is performed from the candidate set of hard negative samples of the query sample. Each document constitutes a hard negative sample set corresponding to the query sample; the random sampling is uniform random sampling or similarity segmented random sampling.

[0041] The beneficial effects of this invention are:

[0042] First, this invention constructs a candidate set of difficult-to-negative samples through truncated sampling, avoiding the direct use of top-ranking unlabeled documents as negative samples. This reduces the probability of missed positive samples being misused as negative samples and alleviates the damage of false negative sample noise to the text vector space. Then, random sampling is performed from the candidate set of difficult-to-negative samples for the query sample. These documents constitute the hard negative sample set for the query sample, so that the final hard negative sample has both high confusion and low risk of false negative samples.

[0043] Secondly, this invention uses multi-stage iterative fine-tuning to rediscover negative samples that the current model struggles to distinguish at each stage, allowing the model to continuously face new highly confusing samples and alleviating the problem that statically difficult negative samples become too easy to distinguish in the later stages of training, causing the loss to tend to saturate and the gradient to weaken.

[0044] Therefore, the fine-tuning method of this invention can improve the text representation model's ability to distinguish between texts that are literally similar but semantically different, and improve the matching ability between semantically related texts that have different literal expressions, thereby improving the recall accuracy and ranking quality of professional corpus retrieval. This invention can also reduce the probability of highly similar but low-relevance texts appearing at the top of search results, providing more reliable textual evidence for subsequent tasks such as knowledge base question answering, similar case matching, document recommendation, and business review. Attached Figure Description

[0045] Figure 1 This is a flowchart illustrating an embodiment of the present invention;

[0046] Figure 2 The NDCG@20 performance of each model on each subset dataset is shown. Detailed Implementation

[0047] This invention aims to provide a fine-tuning method for a pre-trained text representation model. This method uses a pre-trained text representation model as a base model and performs multi-stage iterative fine-tuning. In each stage, truncated sampling is used to mine hard negative samples, resulting in the final text representation model. Specifically, it includes the following steps:

[0048] S1. For the pre-trained text representation model, construct a query set, a positive sample set, and a candidate document set for fine-tuning it; the query set is a set of queries, the positive sample set is a set of labeled positive sample documents, and the candidate document set is a set of unlabeled candidate documents in the target domain.

[0049] S2. Construct query samples for this round of training; each query sample includes a query and at least one positive sample document that matches the query; using the current text representation model, encode the query of each query sample and each candidate document contained in the candidate document set to obtain their corresponding vector representation.

[0050] S3. For each query sample, process it separately according to the following steps:

[0051] S31. Calculate the similarity between the vector representation of each candidate document in the candidate document set and the vector representation of the query in the query sample, and sort the candidate documents from high to low similarity.

[0052] S32. From the candidate documents sorted in step S31, extract those ranked... The candidate documents within the specified interval constitute the candidate set of hard-negative samples for this query sample. As a preset lower bound, the As a preset upper limit for truncation, the ;

[0053] S33. Randomly sample from the candidate set of hard-to-bear samples of the query sample. The document constitutes the hard-burden sample set for this query sample. ;

[0054] S34. Construct training samples corresponding to the query sample, wherein the query of the training sample is the query of the corresponding query sample, the positive samples of the training sample are the positive sample documents of the corresponding query sample, and the negative samples of the training sample include the candidate documents contained in the hard negative sample set of the corresponding query sample.

[0055] S4. Based on the training samples corresponding to each query sample obtained in step S3 for this round of training, the parameters of the current text representation model are fine-tuned using the contrastive learning algorithm.

[0056] S5. Determine whether the preset number of fine-tuning iterations has been reached; if not, use the text representation model fine-tuned in step S4 as the new current text representation model and return to step S2; if it has been reached, obtain the final text representation model.

[0057] In professional domain data, annotations are often incomplete. A query may have multiple truly relevant documents, but the training data only annotates a portion of them. The top-ranking unannotated documents may be missed positive samples, and directly using them as negative samples would introduce false negative noise. Therefore, this invention addresses this issue by using a preset lower bound for truncation. and truncation of the upper boundary Sampling is performed using truncated intervals to avoid high-risk samples, selecting difficult-to-pass samples from relatively reliable intervals that are still difficult to distinguish. Simultaneously, random sampling is performed from the candidate set of difficult-to-pass samples for the query sample. These documents constitute the hard negative sample set for the query sample, so that the final hard negative sample has both high confusion and low risk of false negative samples.

[0058] Based on this, the present invention uses multi-stage iterative fine-tuning. In each stage, the model obtained from the previous stage of fine-tuning is used to rediscover difficult-to-distinguish negative samples, so that the model continuously faces new highly confused samples, which alleviates the problem that static difficult negative samples become too easy in the later stages of training and the loss tends to saturate and the gradient weakens.

[0059] Therefore, the fine-tuning method of this invention can continuously provide effective hard-negative sample training signals while reducing the probability of introducing false negative samples. It should be noted that the core of this invention lies in multi-stage iterative fine-tuning and truncated sampling of hard-negative samples; therefore, it can be used for fine-tuning any existing text representation model. That is, the pre-trained text representation model can employ a BERT-type dual-tower encoding model, an E5-type text embedding model, a BGE-type text embedding model, or a Qwen-type text embedding model.

[0060] The lower bound of the above cutoff interval The purpose of this is to define high-risk areas for false negatives, excluding unlabeled documents that rank too high in the search results from the negative sample candidate set. That is, documents ranked 1st to 2nd in the search results... Unlabeled documents with a very high probability of missing positive samples can be directly used as negative samples, causing the model to push away the semantic representations that should be brought closer together, severely disrupting the text vector space structure. However, if Setting the value too high will exclude a large number of documents that are only confusing in some dimensions, resulting in an overly simplistic candidate set of hard-to-bear samples. This weakens the training signal, preventing the model from fully learning fine-grained semantic distinctions. The core consideration is that the sparser the corpus annotation, the higher the risk of missing positive samples. The larger the value, the lower the value can be when the corpus annotation is relatively complete or the coverage of labeled positive samples is high. This is to retain more difficult-to-bear samples with effective discriminative value. Preferably, the lower bound of the truncation is... In particular, for scenarios with extremely sparse annotations, the preferred method can be selected. For scenarios with high annotation coverage, the preferred method can be selected. .

[0061] The upper bound of the above cutoff interval This is used to define the tail position of the candidate set of hard-to-bear samples, and... Together they form the negative sample screening window. If If the setting is too small, the documents in the candidate set will have a low degree of semantic confusion with the query, failing to provide an effective contrastive learning gradient and resulting in insufficient training signals; if... If the threshold is set too large, the candidate set will contain a large number of easily separable documents that are largely irrelevant to the query semantics. This degenerates the hard-negative sample sampling into an approximately random negative sample sampling, weakening the core advantage of hard-negative sample mining. Therefore, The settings need to ensure that candidate documents are located near the model's decision boundary, i.e., possessing a certain degree of obfuscation but not being clearly irrelevant to the query. For typical corpus sizes, preferably, the above and truncation upper bounds are... For corpora with large scale, such as millions of document fragments, and where the model's initial retrieval has already been able to recall a large number of relevant documents to relatively high positions, the scale can be appropriately expanded. Up to [200, 500]; for scenarios with a small corpus or highly focused queries, the following can be used: .generally, and Maintain at least an order of magnitude difference between them to ensure that the candidate set has reasonable capacity and diversity.

[0062] The above-mentioned number of difficult-to-handle samples , representing the number of hard-to-bear samples contained in each training sample, due to the truncation lower bound. and truncation of the upper boundary The size of the determined candidate set of hard-to-bear samples should be greater than or equal to the number of samples. Therefore, the following is required:

[0063]

[0064] A larger value provides more difficult-to-discriminate samples, enhancing the model's discriminative ability, but it may also introduce more moderately confusing samples, shifting the training focus away from the most difficult-to-discriminate samples. Conversely, a smaller value... If the sample size is too small, it will be difficult to cover the limited confusion patterns, making it difficult for the model to steadily improve its discriminative ability. Ideally, the number of document samples should be... For cases where the query set is small or the labels are extremely sparse, a smaller [size / method] can be used. For example, 2~4, to reduce the risk of residual false negative samples that may be introduced by a single query; for scenarios with a sufficient number of queries and high domain differentiation difficulty, a larger value can be used. For example, 6~10, to fully expose the model's shortcomings in the confusion region.

[0065] Furthermore, intra-batch negative samples typically provide additional negative samples, that is, difficult negative samples from other training samples in the same batch are sampled as ordinary negative samples for this training sample. This also needs to be coordinated with the batch size. Experiments have shown that in step S2, the batch size of query samples used for this round of training is 32-64, and in step S33, the number of documents sampled... It can effectively balance training efficiency and model discrimination ability.

[0066] Multi-stage iterative fine-tuning can directly utilize the final stage model or the optimal model from the validation set. However, early-stage models retain more general semantic matching capabilities, while later-stage models possess stronger domain-specific fine-grained discrimination capabilities. Therefore, this invention further proposes that in step S5, if a preset number of fine-tuning iterations is reached, the parameters of the text representation model obtained in each round of fine-tuning iterations are fused to obtain the parameters of the final text representation model. By fusing multi-stage models, the capabilities and characteristics of different training stages are taken into account, reducing the risk of overfitting and catastrophic forgetting caused by using only the final checkpoint.

[0067] It should be noted that existing technologies employ multi-stage training methods, such as those used for classification model training. The primary purpose of multi-stage training is to improve the model's robustness and generalization ability under different data distributions or task scenarios. Merging multiple models is also mainly to integrate the capabilities gained from different tasks into a single model. Furthermore, existing technologies that perform multi-stage training and fusion typically emphasize the complementarity between models generated from different tasks, training data, hyperparameter configurations, or random initializations, rather than focusing on the fusion of checkpoint parameters across multiple stages within the same training process. Especially in iterative processes with progressively advancing stages, subsequent stage models are often considered as further optimizations of earlier stage models. It is more natural to choose the final stage model or the optimal model on the validation set as the final model, rather than reintroducing earlier stage models into the final parameters. Correspondingly, existing methods do not address the scenario of reconstructing hard-to-bear samples based on the text representation capabilities of each stage.

[0068] Directly fusing parameters across multiple stages may dull the newly learned capabilities. Therefore, this invention goes a step further by fusing parameters based on the parameter increments of each stage model relative to the initial model. The parameter increments represent the direction of change introduced in that stage, i.e., what new capabilities have been learned, thus balancing the general semantic capabilities of the early and mid-stages with the domain-specific fine-grained distinguishing capabilities of the later stages.

[0069] The above fine-tuning iteration count The number of stages determines how many iterations the model undergoes in the "dynamic update-fine-tuning of hard-to-bear samples" process. Too few stages, such as only one, are equivalent to only performing a single static hard-to-bear sample mining and single-stage fine-tuning, failing to fully reflect the continuous training signal gain brought by dynamic updates and making it difficult to leverage the fine-grained discrimination advantage of later stages. Too many stages may result in highly overlapping parameter increments in later stages, diminishing fusion benefits, and a linear increase in training time and computational cost. Therefore, the optimal number of fine-tuning iterations is... In cases where the semantic differences in the target domain text are rich and the labeled data is large, a more sophisticated approach can be adopted. or This allows for further stratification to capture confusion samples of varying difficulty; for scenarios with limited resources or relatively simple corpora, a more sophisticated approach can be adopted. They can still benefit from dynamic updates and incremental fusion.

[0070] Secondly, the parameter increments are fused, and sparsity processing can also be introduced. Specifically, the parameter increments of the text representation model obtained in each iteration relative to the pre-trained text representation model in step S1 are fused to obtain the final model parameters:

[0071] S51. Calculate the parameters of the text representation model obtained in each iteration according to the following formula, relative to the initial text representation model parameters. parameter increment :

[0072]

[0073] in, For the first The parameters of the text representation model obtained from rounds of iteration;

[0074] S52. Calculate the parameter increment of the text representation model obtained in each iteration using the following formula. Sparsification is performed to obtain the sparsified parameter increments. :

[0075]

[0076] in, The preset discard probability, According to the probability of discard The generated Bernoulli random mask, This indicates element-wise multiplication;

[0077] S53. Using the following formula, fuse the sparsified parameter increments of the text representation model obtained from each iteration. To obtain the final model parameters :

[0078]

[0079] in, This is the preset number of fine-tuning iterations. For the first The preset weights of the text representation model obtained from rounds of iteration.

[0080] Sparsity processing involves applying parameter increments independently for each stage. The probability of randomly setting the value to zero, and the retained portion is then... Scaling ensures that the increment remains constant in the desired sense, which can reduce collinearity interference between increments at different stages, reduce the risk of overfitting, and force the fusion process to rely on a more robust direction of parameter change.

[0081] parameter Unlike conventional model fusion techniques that either do not employ sparsity or only use a globally uniform dropout rate, the value of this approach lies in specifically using sparsity for the fusion of parameter increments at different stages of the same training process, thus selectively suppressing the dimensionality of noise increments at each stage. If the value is too low, such as close to 0, sparsity reduction essentially fails, and fusion may excessively inherit local overfitting patterns from certain stages; if... If the value is too high, such as exceeding 0.5, too much effective domain-specific information is discarded, weakening the model's ability to distinguish specific domains. Therefore, the preferred value is... When there is limited training data in the domain or a large number of model parameters, the performance can be appropriately increased. This is to enhance the regularization effect.

[0082] At the same time, the parameter increments in each iteration are calculated according to the discard probability. When performing sparse fusion, the same parameter dimension should be ensured across all dimensions. The probability of being "always discarded" in a round is low enough that... .in, The preset discard rate, also known as the globally uniform discard rate, is set as follows: A value of 0.01 indicates that at most 1% of the parameter dimensions are not preserved at all during the fusion process. When, the probability of dropping The upper limit is approximately 0.215, and the actual discard probability used is... The integer value can be set to 0.2.

[0083] Phase fusion weight The parameters at each stage determine their contribution to the final model. The design principle is that the early-stage model retains more general semantic matching capabilities, while the later-stage model possesses stronger domain-specific fine-grained discrimination capabilities; both play an irreplaceable role in the final retrieval performance. Therefore, the weight settings must balance generality and specialization, rather than simply averaging or using only the weights from the last stage.

[0084] As one implementation method, a linear incremental strategy can be adopted. Specifically, in step S53, the first step is calculated according to the following formula. Preset weights of the text representation model obtained from rounds of iteration :

[0085]

[0086] in, Indicates the iteration round.

[0087] A linearly increasing strategy is adopted, allowing the weights to gradually increase in later stages, reflecting the asymptotic accumulation of domain capabilities. When The weights were 1 / 6, 2 / 6, and 3 / 6 respectively. Experimentally, this strategy demonstrated superior retrieval quality compared to average weighting or retaining only the final stage. Alternatively, automatic search based on validation set performance, exponential incrementing, or allocation only between adjacent stages could be used, but it should be ensured that later stages receive greater weight while not discarding contributions from earlier stages.

[0088] In step S33, random samples are taken from the candidate set of hard negative samples of the query sample. Each document constitutes a hard negative sample set corresponding to the query sample; the random sampling is uniform random sampling or similarity segmented random sampling.

[0089] The following description, in conjunction with specific examples, provides further details.

[0090] Example:

[0091] A fine-tuning method for a pre-trained text representation model, such as Figure 1 As shown, it includes the following steps:

[0092] S1. Data Preparation

[0093] In this step, for the pre-trained text representation model, a query set, a positive sample set, and a candidate document set are constructed for fine-tuning the model. The query set is a collection of queries, the positive sample set is a collection of labeled positive sample documents, and the candidate document set is a collection of unlabeled candidate documents in the target domain.

[0094] In this embodiment, the target domain is finance. The training data includes financial business questions, corresponding labeled positive sample documents, and a candidate document set consisting of announcements, research reports, financial statements, rules and regulations, business knowledge base Q&A, and historical materials.

[0095] S2, Query Sample Construction

[0096] In this step, query samples for this round of training are constructed; each query sample includes a query and a positive sample document that matches the query; using the current text representation model, the query of each query sample and each candidate document contained in the candidate document set are encoded to obtain their corresponding vector representations.

[0097] In this embodiment, if a query has multiple positive sample documents, a positive sample document is randomly selected. The preset batch size is 64.

[0098] S3, Truncation Sampling

[0099] In this step, each query sample is processed separately according to the following steps:

[0100] S31, Re-search

[0101] In this step, the similarity between the vector representation of each candidate document in the candidate document set and the vector representation of the query in the query sample is calculated, and the candidate documents are sorted from high to low according to the similarity.

[0102] S32, Select Candidates

[0103] In this step, from the candidate documents sorted in step S31, the documents ranked in the top... The candidate documents within the specified interval constitute the candidate set of hard-negative samples for this query sample. As a preset lower bound, the As a preset upper limit for truncation, the In this embodiment, It is 20. It is 200.

[0104] S33, Sampling difficulties

[0105] In this step, random samples are taken from the candidate set of hard negative samples of the query sample. The document constitutes the hard-burden sample set for this query sample. .

[0106] In this embodiment, uniform random sampling is used. It is 5.

[0107] S34. Construct training samples

[0108] In this step, training samples are constructed corresponding to the query sample. The query of the training sample is the query of the corresponding query sample. The positive samples of the training sample are the positive sample documents of the corresponding query sample. The negative samples of the training sample include the candidate documents contained in the hard negative sample set of the corresponding query sample.

[0109] S4, Model Fine-tuning

[0110] In this step, based on the training samples corresponding to each query sample obtained in step S3 for this round of training, the parameters of the current text representation model are fine-tuned using a contrastive learning algorithm.

[0111] In this embodiment, the contrast loss is calculated according to the following formula:

[0112]

[0113] in, Indicates the first The encoding function of the text representation model obtained from rounds of iteration. Represents the similarity function. Represents an exponential function with the natural constant as its base; For temperature coefficient, For query, As a positive sample, For negative samples, Indicates query The negative sample set, wherein the negative samples contained in the negative sample set include documents contained in the hard negative sample set.

[0114] S5, parameter fusion

[0115] In this step, it is determined whether the preset number of fine-tuning iterations has been reached; if not, the text representation model fine-tuned in step S4 is used as the new current text representation model, and the process returns to step S2; if it has been reached, the final text representation model is obtained.

[0116] In this embodiment, The final model parameters are obtained by fusing the parameter increments of the text representation model obtained in each iteration relative to the pre-trained text representation model in step S1, following these steps:

[0117] S51. Calculate the parameters of the text representation model obtained in each iteration according to the following formula, relative to the initial text representation model parameters. parameter increment :

[0118]

[0119] in, For the first The parameters of the text representation model obtained from rounds of iteration;

[0120] S52. Calculate the parameter increment of the text representation model obtained in each iteration using the following formula. Sparsification is performed to obtain the sparsified parameter increments. :

[0121]

[0122] in, The preset discard probability, According to the probability of discard The generated Bernoulli random mask, This indicates element-wise multiplication;

[0123] S53. Using the following formula, fuse the sparsified parameter increments of the text representation model obtained from each iteration. To obtain the final model parameters :

[0124]

[0125] in, This is the preset number of fine-tuning iterations. For the first The preset weights of the text representation model obtained from rounds of iteration.

[0126] In step S52, a preset The probability of being discarded is 0.003. The upper limit is approximately 0.144, the probability of dropping. Round to 0.1.

[0127] In step S53, the number of steps is calculated according to the following formula. Preset weights of the text representation model obtained from rounds of iteration :

[0128]

[0129] in, Indicates the iteration round.

[0130] test:

[0131] Using the method described in the above embodiments, with the BERT-type dual-tower coding model as the base model, the following settings are made: It is 3. It is 20. The number of random samples is 200. The temperature coefficient is 5. The probability of being dropped is 0.05. The value was 0.1, and tests were conducted. Verification metrics included: Normalized Loss Cumulative Gain (NDCG@K), Inverse Ranking Mean (MRR@K), and Average Precision Mean (MAP@K).

[0132] Among them, such as Figure 2As shown, the adaptability of the model under different financial scenarios is verified, and the performance of each model on the NDCG@20 dataset is demonstrated. It can be observed that the implementation method, namely Ours in the figure, achieves the best results on four datasets except SmoothNLP, especially on the AlphaFi dataset containing complex inference chains and the DuEE-fin dataset focusing on event element extraction, where it improves performance by 5.59% and 4.85% respectively compared to the base model Qwen3-Embedding, demonstrating the advantage of the implementation method in handling high-order financial semantic tasks. Overall, the implementation method not only leads in comprehensive metrics but also shows significant improvements in financial sub-scenarios with significant distribution differences, such as news information, professional research reports, event extraction, and interactive question answering, fully validating the effectiveness of the domain knowledge enhancement method proposed in this paper.

[0133] Then, the performance of each model on the comprehensive financial dataset was compared and analyzed through experiments. The results are shown in Table 1. First, a longitudinal comparison was made between the general domain model and the domain-transfer fine-tuned model. It can be found that all models fine-tuned with financial corpora show a significant advantage in overall performance, significantly outperforming the general baseline model in NDCG, MRR, and MAP. Even SimnCSE, which performed relatively weakly among the fine-tuned models, had an NDCG@10 index 2.6% higher than Qwen3-Embedding, the best-performing general model. These results indicate that adaptive fine-tuning for the financial vertical domain is significantly necessary. General models trained on open-domain data struggle to capture the professional semantic features of the financial domain, resulting in poor retrieval performance. However, through fine-tuning with domain data, the model can effectively adapt to the financial semantic space, significantly improving retrieval results. The implementation method, namely Ours in the table, achieved a 3.9% improvement in the NDCG@10 index compared to the base model. This demonstrates that this method successfully injected professional financial knowledge into the general embedding model, effectively alleviating the understanding bias of the general model when dealing with financial terminology and complex semantics. Secondly, a horizontal comparison was made with various domain transfer fine-tuning methods. The Example Method, also known as Ours in the table, achieved the best experimental results across all performance metrics. Specifically, the Example Method achieved 86.82%, 85.14%, and 85.06% in NDCG@10, MRR@10, and MAP@10, respectively, all higher than other baseline models. Compared to the classic static hard-negative sample mining method ANCE, the Example Method achieved a 1.07% higher NDCG@10 metric; and compared to NV-Retriever, which employs a sample denoising strategy, it also achieved a 0.46% improvement. These results demonstrate that the continuous dynamic adjustment of sample difficulty in this method can more effectively enhance the text representation model's ability to capture fine-grained semantic features of financial text.

[0134] Table 1. Comparative Experiment Results

[0135]

[0136] Finally, it should be noted that the above embodiments are merely preferred embodiments and are not intended to limit the present invention. It should be pointed out that those skilled in the art can make various modifications, equivalent substitutions, and improvements without departing from the spirit and scope of the claims, and all such modifications, substitutions, and improvements should be included within the scope of protection of the present invention.

Claims

1. A method for fine-tuning a pre-trained text representation model, comprising the following steps: S1. For the pre-trained text representation model, construct a query set, a positive sample set, and a candidate document set for fine-tuning it; the query set is a set of queries, the positive sample set is a set of labeled positive sample documents, and the candidate document set is a set of unlabeled candidate documents in the target domain. S2. Construct query samples for this round of training; each query sample includes a query and at least one positive sample document that matches the query; using the current text representation model, encode the query of each query sample and each candidate document contained in the candidate document set to obtain their corresponding vector representation. S3. For each query sample, process it separately according to the following steps: S31. Calculate the similarity between the vector representation of each candidate document in the candidate document set and the vector representation of the query in the query sample, and sort the candidate documents from high to low similarity. S32, from the candidate documents sorted in step S31, extracting candidate documents ranked in the top interval, to form a hard-negative sample candidate set of the query sample, wherein the is a preset lower bound of the truncation, and the is a preset upper bound of the truncation, and the ; S33, randomly sampling from the hard negative sample candidate set of the query sample a document, to form a hard negative sample set of the query sample, wherein the ; S34. Construct training samples corresponding to the query sample, wherein the query of the training sample is the query of the corresponding query sample, the positive samples of the training sample are the positive sample documents of the corresponding query sample, and the negative samples of the training sample include the candidate documents contained in the hard negative sample set of the corresponding query sample. S4. Based on the training samples corresponding to each query sample obtained in step S3 for this round of training, the parameters of the current text representation model are fine-tuned using the contrastive learning algorithm. S5. Determine whether the preset number of fine-tuning iterations has been reached; If the desired result is not achieved, the text representation model fine-tuned in step S4 will be used as the new current text representation model, and the process will return to step S2. If this is achieved, the final text representation model is obtained.

2. The fine-tuning method for a pre-trained text representation model as described in claim 1, characterized in that: In step S4, the contrast loss is calculated according to the following formula: ； wherein, represents the i-th iterative obtained text representation model, represents a similarity function, represents an exponential function with a natural constant as base; is a temperature coefficient, is a query, is a positive sample, is a negative sample, represents a query negative sample set, the negative samples contained in the negative sample set including the documents contained in the difficult negative sample set.

3. The fine-tuning method for a pre-trained text representation model as described in claim 1, characterized in that: In step S5, if the preset number of fine-tuning iterations is reached, the parameters of the text representation model obtained in each round of fine-tuning iterations are fused to obtain the parameters of the final text representation model.

4. The fine-tuning method for a pre-trained text representation model as described in claim 3, characterized in that: The final model parameters are obtained by fusing the parameter increments of the text representation model obtained in each iteration relative to the pre-trained text representation model in step S1, following these steps: S51, calculate the parameter of the text representation model obtained in each round of iteration, relative to the initial text representation model parameter the parameter increment of the text representation model : ； in, For the first The parameters of the text representation model obtained from rounds of iteration; S52. Calculate the parameter increment of the text representation model obtained in each iteration using the following formula. Sparsification is performed to obtain the sparsified parameter increments. : ； in, The preset discard probability, According to the probability of discard The generated Bernoulli random mask, This indicates element-wise multiplication; S53. Using the following formula, fuse the sparsified parameter increments of the text representation model obtained from each iteration. To obtain the final model parameters : ； in, This is the preset number of fine-tuning iterations. For the first The preset weights of the text representation model obtained from rounds of iteration.

5. The fine-tuning method for a pre-trained text representation model as described in claim 4, characterized in that: In step S52, the discard probability ,and The For the preset number of fine-tuning iterations, the This is the preset discard rate.

6. The fine-tuning method for a pre-trained text representation model as described in claim 4, characterized in that: In step S53, the number of steps is calculated according to the following formula. Preset weights of the text representation model obtained from rounds of iteration : ； in, Indicates the iteration round, This is the preset number of fine-tuning iterations.

7. A method for fine-tuning a pre-trained text representation model as described in any one of claims 1 to 6, characterized in that: In step S32, the lower bound is truncated. The above and truncation upper bounds ; In step S33, the number of documents sampled. .

8. The fine-tuning method for a pre-trained text representation model as described in claim 7, characterized in that: In step S2, the batch size of query samples used for this round of training is 32-64. In step S33, the number of documents sampled... .

9. A method for fine-tuning a pre-trained text representation model as described in any one of claims 1 to 6, characterized in that: The number of fine-tuning iterations .

10. A method for fine-tuning a pre-trained text representation model as described in any one of claims 1 to 6, characterized in that: In step S33, random samples are taken from the candidate set of hard-negative samples of the query sample. Each document constitutes a hard negative sample set corresponding to the query sample; the random sampling is uniform random sampling or similarity segmented random sampling.

Images