Sensitive field-oriented text embedding model localization fine-tuning method and system

Abstract

This invention provides a method and system for localizing and fine-tuning a text embedding model for sensitive domains, belonging to the field of model localization technology. The method includes: obtaining an initial sensitive domain corpus; filtering and retaining high-value text paragraphs from the domain corpus; generating domain-synthesized queries to obtain multiple sets of sample domain retrieval queries, constructing multiple sets of query-positive document pairs; filtering low-quality samples, then removing false negatives and mining hard negatives to form a contrastive learning training sample set; performing grouped contrastive learning training to obtain multiple domain-adaptation low-rank parameter matrices; and completing localization adaptation for highly sensitive domains. This solves the technical problem of existing technologies that use general text embedding models to directly perform sensitive domain text retrieval, resulting in insufficient model representation of sensitive domain terminology, leading to reduced semantic matching accuracy between query text and domain documents, and failing to meet the information retrieval needs of highly specialized domains.

Description

Technical Field

[0001] This invention relates to the field of model localization technology, and in particular to a method and system for fine-tuning the localization of text embedding models for sensitive domains. Background Technology

[0002] Text embedding models map query text and document text to a unified vector space and achieve semantic matching based on vector similarity. Compared with traditional keyword retrieval methods, they can effectively identify potential semantic associations under different expressions and improve information retrieval capabilities in complex text scenarios.

[0003] However, texts from different domains often possess strong specialized characteristics. For example, sensitive domains may include military, energy, and industrial security fields. Taking the military domain as an example, its documents typically contain a large number of technical terms, organizational entities, technical descriptions, and specific semantic relationships. While general text embedding models are primarily pre-trained on general corpora and possess strong general language understanding capabilities, they lack sufficient learning of specialized concepts, domain expressions, and implicit knowledge relationships within specific domains. This results in insufficient semantic representation capabilities for specialized domain retrieval tasks. Related research indicates that text embedding models with strong general capabilities do not directly translate to performance in sensitive domain retrieval; domain adaptation remains a crucial approach to improving specialized retrieval effectiveness.

[0004] Existing technologies typically use domain data to fine-tune general text embedding models, enabling the models to learn text representation features of the target domain. One approach is to conduct supervised training by manually constructing query-document annotation data. However, sensitive domains often face challenges such as data acquisition limitations, high costs of manual annotation, and limited domain expert resources, making it difficult to continuously construct large-scale, high-quality training data.

[0005] Therefore, how to achieve efficient localization adaptation of general text embedding models using limited domain data while ensuring the security of data in sensitive domains, and improve the model's ability to represent specialized semantics in sensitive domains, has become a problem that needs to be solved in the current field of text retrieval technology. Summary of the Invention

[0006] To address the aforementioned deficiencies or improvement needs of existing technologies, this invention provides a method and system for localized fine-tuning of text embedding models for sensitive domains. This solves the technical problem that existing technologies use general text embedding models to directly perform text retrieval in sensitive domains, resulting in insufficient model representation of sensitive domain terminology. Consequently, the semantic matching accuracy between query text and domain documents is reduced, failing to meet the information retrieval needs of highly specialized domains.

[0007] The specific technical solution is as follows:

[0008] According to a first aspect of the present invention, a method for localizing and fine-tuning a text embedding model for sensitive domains is provided, the method comprising:

[0009] The initial sensitive document set, collected based on the target sensitive domain, is parsed and filtered offline to obtain an initial sensitive domain corpus. The Qwen3-4B-Instruct model is used to score the professionalism of the initial sensitive domain corpus based on semantic information density, and a corpus of high-value text paragraphs in the domain is selected and retained. Multiple sample paragraphs in the high-value text paragraph corpus are used to generate domain synthesis queries based on the Qwen3-32B model, resulting in multiple sets of sample domain retrieval queries. Based on mapping relationships, multiple sets of query-positive document pairs are constructed. A multidimensional quality assessment is used to evaluate the low-quality performance of these multiple sets of query-positive document pairs. After filtering the quality samples, false negative examples are removed and hard negative examples are mined based on the similarity of the embedding vectors to form a contrastive learning training sample set. Based on freezing the original weight parameters in the pre-constructed general text embedding model, multiple low-rank adaptation modules are introduced, and the contrastive learning training sample set is used as training data to perform grouped contrastive learning training on the multiple low-rank adaptation modules based on functional relevance clustering until the preset convergence condition is met, resulting in multiple domain-adapted low-rank parameter matrices. The multiple domain-adapted low-rank parameter matrices are merged into the general text embedding model through matrix addition to complete the high-sensitivity domain localization adaptation of the general text embedding model.

[0010] In one implementation, the Qwen3-4B-Instruct model is used to perform a professionalism score based on semantic information density on the initial sensitive domain corpus, and to filter and retain a corpus of high-value text paragraphs in the domain, including:

[0011] The Qwen3-4B-Instruct model is deployed locally using a vLLM inference engine. The first sensitive paragraph from the initial sensitive domain corpus is input into the Qwen3-4B-Instruct model, driving a professionalism scoring process guided by corpus quality scoring prompts, and outputting the first paragraph score. If the first paragraph score is not lower than a preset scoring threshold, the first sensitive paragraph is retained as the first sample paragraph. Similarly, the remaining sensitive paragraphs from the initial sensitive domain corpus are input into the Qwen3-4B-Instruct model one by one, and paragraph filtering based on professionalism scores is performed to select and retain the high-value text paragraph corpus of the domain.

[0012] In one implementation, the Qwen3-4B-Instruct model includes a cascaded embedding layer, a multi-layer Transformer decoder, a final normalization layer, and a linear projection output layer. The embedding layer is equipped with corpus quality scoring prompt templates and query scoring prompt templates. The corpus quality scoring prompt templates require the Qwen3-4B-Instruct model, acting as an expert in the target sensitive domain, to perform a three-dimensional comprehensive evaluation of the input paragraph based on terminology density, information substance depth, and domain relevance. The query scoring prompt templates require the Qwen3-4B-Instruct model to perform a three-dimensional comprehensive evaluation of the query-positive document pair based on the domain semantic relevance of the target sensitive domain, paragraph content coverage, and semantic independence from context.

[0013] In one implementation, the first sensitive paragraph from the initial sensitive domain corpus is input into the Qwen3-4B-Instruct model to drive a professionalism scoring process guided by corpus quality scoring prompt word templates, and outputs the first paragraph score, including:

[0014] In the embedding layer, the first sensitive paragraph is used as the text to be scored and concatenated to the end of the corpus quality scoring prompt word template to construct a first complete input sequence. The first complete input sequence is then segmented to obtain an input token sequence. The input token sequence is fed into the multi-layer Transformer decoder to perform autoregressive inference and generate a hidden state sequence. The final normalization layer performs layer normalization on the hidden state sequence to obtain a normalized hidden state. The linear projection output layer maps the normalized hidden state to the vocabulary space, generates an output text sequence, and then performs score value parsing and extraction on the output text sequence to output the score of the first paragraph.

[0015] In one implementation, multiple sample paragraphs from the high-value text paragraph corpus in the domain are used to generate domain synthesis queries based on the Qwen3-32B model, resulting in multiple sets of sample domain retrieval queries. Based on the mapping relationship, multiple sets of query-positive document pairs are constructed, including:

[0016] The number of queries to be generated is determined based on the text length of the first sample paragraph; the first sample paragraph is used as the anchor text for query generation and appended to the end of the synthetic query generation prompt template to construct a second complete input sequence; the second complete input sequence is fed into the Qwen3-32B model to drive autoregressive generation and output the first set of sample domain retrieval queries, wherein the synthetic query generation prompt template requires the Qwen3-32B model to generate diverse target sensitive domain retrieval queries based on the content of the input paragraph from the perspective of the commander of the target sensitive domain; the first sample paragraph is used as the first positive example document to establish a mapping relationship between the first set of sample domain retrieval queries and the first positive example document, constructing the first set of query-positive example document pairs.

[0017] In one implementation, after filtering low-quality samples from the multiple sets of query-positive document pairs using multidimensional quality assessment, false negative example removal and hard negative example mining based on embedding vector similarity are performed to form a contrastive learning training sample set, including:

[0018] Multiple query-positive document pairs from the first set of query-positive document pairs are fed into the Qwen3-4B-Instruct model to drive a multi-dimensional quality assessment guided by the query scoring prompt word template, resulting in multiple first domain semantic relevance scores, multiple first paragraph content coverage scores, and multiple first context-independent semantic scores. A preset relevance threshold, preset coverage threshold, and preset independence threshold are used to map and compare the first domain semantic relevance scores, first paragraph content coverage scores, and first context-independent semantic scores to determine the retention of the multiple query-positive document pairs, resulting in the first set of retained pairs. Similarly, the Qwen3-4B-Instruct model is applied to filter low-quality samples from the multiple sets of query-positive document pairs. Multiple sets of retained pairs are obtained; the inter-group separations of the multiple sets of retained pairs are merged to obtain K retained pairs, from which K domain retrieval queries and K positive example documents are extracted; the K retained domain retrieval queries drive the un-fine-tuned general text embedding model to perform a dense search in the domain high-value text paragraph corpus to obtain K candidate document sets; using the similarity scores of the K positive example documents and the K candidate document sets as anchors, false negative examples are removed by combining a preset similarity ratio threshold to obtain K filtering candidate sets; a preset number of documents are randomly sampled from the K filtering candidate sets to construct K hard negative example document sets; the K retained domain retrieval queries, the K positive example documents, and the K hard negative example document sets are combined one by one to construct K contrastive learning training samples, which are then aggregated to form the contrastive learning training sample set.

[0019] In one implementation, based on freezing the original weight parameters in a pre-built general text embedding model, multiple low-rank adaptation modules are introduced, and the contrastive learning training sample set is used as training data. Grouped contrastive learning training based on functional association clustering is performed on the multiple low-rank adaptation modules until a preset convergence condition is met, resulting in multiple domain-adaptation low-rank parameter matrices, including:

[0020] After freezing the intra-layer weight parameter matrices of multiple target layers in the general text embedding model, the multiple low-rank adaptation modules are initialized by mapping. Each low-rank adaptation module contains a low-rank decomposition matrix A and a low-rank decomposition matrix B, forming a bypass injection path parallel to the corresponding intra-layer weight parameter matrix. Based on the layer depth of the multiple target layers, the multiple low-rank adaptation modules are divided into multiple groups of low-rank adaptation modules. According to the target layer semantic level of the multiple groups of low-rank adaptation modules, the contrastive learning training sample set is divided into multiple training sample subsets. In the general text embedding model, the multiple training sample subsets are used as grouped training data to perform polling and freezing low-rank adaptation training of the multiple groups of low-rank adaptation modules until a preset convergence condition is met, thus obtaining the multiple domain-adapted low-rank parameter matrices.

[0021] In one implementation, it further includes:

[0022] Using a first subset of training samples as grouped training data, during the low-rank adaptation training of the first group of low-rank adaptation modules, only the first group of low-rank adaptation modules is activated to participate in parameter updates, while the remaining low-rank adaptation modules that are not activated remain temporarily frozen. The multiple groups of low-rank adaptation modules are traversed alternately in a preset round order to perform parameter updates under polling freeze until a preset convergence condition is met, resulting in multiple groups of domain-adapted low-rank parameter matrices. These multiple groups of domain-adapted low-rank parameter matrices are then split into groups to obtain multiple domain-adapted low-rank parameter matrices. The product of the low-rank decomposition matrix A and the low-rank decomposition matrix B in the multiple domain-adapted low-rank parameter matrices is then added and merged with the corresponding multiple in-layer weight parameter matrices to complete the high-sensitivity domain-localization adaptation of the general text embedding model.

[0023] In one implementation, the original sensitive document set collected based on the target sensitive domain is parsed and filtered locally offline to obtain an initial sensitive domain corpus, including:

[0024] Based on the knowledge system of the target sensitive domain, the domain tag set is expanded; the domain tag set is randomly grouped into multiple domain tag constraints by a preset number of tags; the multiple domain tag constraints are used to collect targeted documents to obtain the original sensitive document set; after formatting and parsing the original sensitive document set, format noise is filtered to produce the initial sensitive domain corpus.

[0025] According to a second aspect of the present invention, a localization fine-tuning system for text embedding models in sensitive domains is provided, the system comprising:

[0026] The parsing and filtering module performs local offline parsing and filtering on the original sensitive document set collected based on the target sensitive domain to obtain an initial sensitive domain corpus; the professionalism scoring module uses the Qwen3-4B-Instruct model to perform professionalism scoring on the initial sensitive domain corpus based on semantic information density, and filters and retains a corpus of high-value text paragraphs in the domain; the domain synthesis query module generates domain synthesis queries based on the Qwen3-32B model from multiple sample paragraphs in the high-value text paragraph corpus, obtaining multiple sets of sample domain retrieval queries, and constructs multiple sets of query-positive example document pairs based on mapping relationships; the hard negative example mining module uses multidimensional quality assessment to perform multidimensional quality assessment on the hard negative example document pairs. After filtering low-quality samples from group query-positive document pairs, false negatives are removed and hard negatives are mined based on embedding vector similarity to form a contrastive learning training sample set. The learning training module, based on the frozen original weight parameters in the pre-built general text embedding model, introduces multiple low-rank adaptation modules and uses the contrastive learning training sample set as training data to perform grouped contrastive learning training on the multiple low-rank adaptation modules based on functional association clustering until a preset convergence condition is met, resulting in multiple domain-adapted low-rank parameter matrices. The localization adaptation module merges the multiple domain-adapted low-rank parameter matrices into the general text embedding model through matrix addition, completing the high-sensitivity domain localization adaptation of the general text embedding model.

[0027] Beneficial effects of the embodiments of the present invention:

[0028] By performing local offline parsing and format noise filtering on the original sensitive document set collected from the target sensitive domain, unified processing of documents from heterogeneous sources is achieved, reducing the interference of invalid format information, duplicate content, and non-semantic noise on subsequent model training. The Qwen3-4B-Instruct model is used to evaluate the semantic information density of text paragraphs in the initial sensitive domain corpus, automatically identifying high-quality samples containing high-density professional knowledge, key technical descriptions, and domain-related information from a large amount of domain text. This allows subsequent query generation and adaptation training processes to focus more on high-quality domain knowledge content. The Qwen3-32B model is used to generate domain-synthesized queries from high-value domain text paragraphs, enabling training samples to cover various expressions of real user search needs, improving query semantic richness and domain adaptability. By combining a multi-dimensional quality assessment mechanism, query-positive document pairs are screened for semantic relevance, content coverage, and query independence, effectively reducing... The system automatically generates the proportion of low-quality data in the samples. Simultaneously, it performs false negative example removal through embedding vector similarity analysis and further mines high-discrimination hard negative examples, enabling the training samples to possess both accurate positive association and effective negative constraint capabilities. This improves the text embedding model's ability to distinguish similar texts within a specific domain and enhances retrieval accuracy. By freezing the original weight parameters of the general text embedding model and introducing multiple low-rank adaptation modules, it maintains the general semantic representation capabilities of the basic model while achieving efficient parameter adjustment for sensitive domain knowledge. By fusing multiple domain-adapted low-rank parameter matrices obtained during training into the general text embedding model using matrix addition, it achieves domain capability transfer without retraining the complete model parameters. The fused model retains the original general text understanding capabilities while enhancing its ability to represent target sensitive domain terms, professional knowledge, and semantic relationships, improving the accuracy of domain text retrieval, matching, and semantic analysis tasks, while also meeting the security and controllability requirements of local deployment environments.

[0029] Of course, implementing any product or method of the present invention does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 This invention illustrates a flowchart of a method for localizing and fine-tuning a text embedding model for sensitive domains, as provided by the present invention.

[0032] Figure 2 This diagram illustrates the distribution of corpus scores in the localization fine-tuning method for text embedding models in sensitive domains provided by this invention.

[0033] Figure 3 This diagram illustrates the working principle of incremental learning of domain parameters in the localization fine-tuning method for text embedding models in sensitive domains provided by this invention.

[0034] Figure 4 A schematic diagram of the localization fine-tuning system for text embedding models in sensitive domains provided by the present invention is shown.

[0035] Figure labeling: Parsing and filtering module 10, professionalism scoring module 20, domain synthesis query module 30, hard negative example mining module 40, learning and training module 50, localization adaptation module 60. Detailed Implementation

[0036] To facilitate understanding of the present invention, a more complete description of the invention will be given below with reference to the accompanying drawings, which illustrate preferred embodiments of the invention. However, the invention can be implemented in many different forms and is not limited to the embodiments described herein; rather, these embodiments are provided so that the disclosure of the invention will be more thorough and complete.

[0037] Furthermore, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0038] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," and "counterclockwise," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0039] Unless otherwise expressly stated, throughout the specification and claims, the term "comprising" or its variations such as "including" or "comprises" shall be understood to include the stated elements or components without excluding other elements or other components.

[0040] Example 1: See Figure 1 The flowchart of the localization fine-tuning method for text embedding models in sensitive domains provided in this embodiment of the invention includes:

[0041] Y100: The original sensitive document set collected based on the target sensitive domain is parsed and filtered locally offline to obtain the initial sensitive domain corpus.

[0042] For the original sensitive document set collected in a targeted sensitive area, document parsing is first performed in a local offline environment to prevent sensitive data from being transmitted externally during processing. The original sensitive document set includes various types of text data within this area, such as technical documents, specification documents, business documents, and historical records. A document parsing tool is used to uniformly convert documents of different formats, extracting structured information such as text content, heading levels, and paragraph structure. The parsed text is then filtered according to preset data cleaning rules.

[0043] The filtering process includes removing text content lacking actual semantic information, such as duplicate text, abnormal characters, residual formatting information, and excessively short text fragments, while retaining text paragraphs with complete semantic structure. For document content with chapter relationships, paragraphs are concatenated according to context to ensure that text fragments have relatively complete domain knowledge expressions. After the above processing, text paragraphs that meet the basic quality requirements are uniformly stored to form an initial sensitive domain corpus, providing a data foundation for subsequent domain text screening and training sample construction.

[0044] Y200: The Qwen3-4B-Instruct model is used to score the professionalism of the initial sensitive domain corpus based on semantic information density, and to select and retain high-value text paragraphs in the domain.

[0045] Text paragraphs from the initial sensitive domain corpus are input into the locally deployed Qwen3-4B-Instruct model according to a preset input format. A domain text evaluation prompt template is constructed to guide the model in comprehensively analyzing text paragraphs from dimensions such as terminology content, domain knowledge completeness, contextual relevance, and information effectiveness. Semantic information density describes the degree to which a unit of text contains effective domain knowledge information. Texts containing a large amount of general descriptions, repetitive explanations, or low-relevance content have low semantic information density; texts containing explicit objects, professional concepts, technical relationships, and domain logical relationships have high semantic information density. The model outputs corresponding scores according to preset scoring rules and filters text paragraphs according to set thresholds, retaining high-value text paragraphs that meet the scoring requirements. Simultaneously, the filtered texts are uniformly numbered and structured for storage, forming a high-value domain text paragraph corpus, providing a data source for subsequent construction of query-document training samples.

[0046] Y300: Generate domain synthesis queries based on the Qwen3-32B model from multiple sample paragraphs in the high-value text paragraph corpus of the domain, obtain multiple sets of sample domain retrieval query terms, and construct multiple sets of query-positive document pairs based on the mapping relationship.

[0047] Selected high-value text paragraphs from the chosen domain are input into the locally deployed Qwen3-32B model. A pre-defined query generation template is used to generate corresponding search queries based on the core entities, technical descriptions, business relationships, and implicit needs contained in the text. These synthesized queries simulate the query expressions used by actual users in sensitive domain information retrieval. The generation process maintains a semantic connection with the original domain text, ensuring that the generated queries correspond to the key knowledge content within the text. For each high-value text paragraph, one or more queries are generated, and a mapping relationship is established between the queries and the original text paragraphs, using the original text paragraphs as positive example documents for the corresponding queries. Through this method, a query-positive example document set containing query information and corresponding target documents is formed without manual annotation, providing a foundation of samples for subsequent comparative learning training.

[0048] Methods based on character count control and other rules can only filter enumerable noise patterns. The corpus still contains low-quality content unrelated to sensitive areas but without obvious formatting anomalies, such as journal introductions, author information, and table of contents page numbers. To ensure the professionalism and information density of the training data, a locally deployed Qwen3-4B-Instruct model was used, with each paragraph scored using a vLLM inference engine. The corpus score distribution is as follows: Figure 2 As shown, this paper retains paragraphs with a score of 4 or higher, resulting in 90,926 high-quality text paragraphs, with a retention rate of 90.5%.

[0049] Y400: After filtering low-quality samples from the multiple sets of query-positive document pairs using multidimensional quality assessment, false negative examples are removed and hard negative examples are mined based on embedding vector similarity to form a contrastive learning training sample set.

[0050] The query and corresponding positive example documents are input into the quality assessment model. Using pre-defined evaluation rules, a comprehensive evaluation is conducted based on aspects such as the semantic relevance of the query to the document, the completeness of the query expression, and query independence. Specifically, relevance determines whether the query content accurately corresponds to the core information in the document; completeness of the query expression determines whether the query meets a clear information retrieval requirement; and query independence determines whether the query overly relies on context or lacks practical retrieval meaning.

[0051] After filtering low-quality samples, the text embedding model further generates embedding vectors corresponding to the query, positive example documents, and candidate negative example documents, and calculates the similarity between the vectors. False negative examples refer to document samples that, while not appearing to belong to the target document, actually contain the same or highly related semantic information. These samples are removed based on similarity to avoid erroneous negative samples affecting model learning. Simultaneously, samples with high similarity to the query but not actually matching are selected from the remaining candidate negative examples as hard negative examples, enabling the model to learn to distinguish fine-grained semantic differences. This ultimately forms the sample set used for contrastive learning training.

[0052] Y500: Based on freezing the original weight parameters in the pre-built general text embedding model, multiple low-rank adaptation modules are introduced, and the contrastive learning training sample set is used as training data. The multiple low-rank adaptation modules are subjected to grouped contrastive learning training based on functional correlation clustering until the preset convergence condition is met, resulting in multiple domain-adaptive low-rank parameter matrices.

[0053] While keeping the original parameters of the general text embedding model unchanged, multiple low-rank adaptation modules are introduced to perform efficient domain-adaptive training of the model's parameters. Specifically, multiple low-rank parameter modules are loaded into the target network layer of the general text embedding model. The domain-adjusted parameters to be learned are represented by low-rank matrix factorization, so that only the parameter matrices in the low-rank adaptation modules are updated during training, while the original weight parameters in the base model are frozen.

[0054] The low-rank adaptation module is used to adjust the model's representation capabilities with a relatively small number of trainable parameters. Considering the differences in semantic representation tasks undertaken by different network layers, the functional association features of multiple low-rank adaptation modules are analyzed. Based on factors such as network layer position, text feature expression capability, and semantic level of training samples, modules with similar adjustment effects are grouped into the same training group. Using the constructed contrastive learning training sample set, the low-rank adaptation modules in each group are trained. During training, the low-rank parameter matrix is ​​optimized by narrowing the distance between the query vector and the positive example document vector and widening the distance between the query vector and the negative example document vector. Training stops according to a preset training epoch or the convergence condition of the loss function, resulting in multiple domain-adapted low-rank parameter matrices.

[0055] Y600: The multiple domain adaptation low-rank parameter matrices are merged into the general text embedding model through matrix addition to complete the high-sensitivity domain localization adaptation of the general text embedding model.

[0056] Multiple low-rank parameter matrices obtained from training are matched according to their corresponding network layer positions and combined with the original weight parameters in the general text embedding model using matrix addition. This allows domain adaptation information to be applied to the model's feature extraction process. Specifically, the matrix addition fusion method involves adding the parameter increments learned by the low-rank adaptation module to the original model parameters, enabling the model to retain its original general semantic expression capabilities while introducing optimized representation capabilities for sensitive domain text characteristics. After fusion, the model is validated by inputting target sensitive domain text samples, obtaining text embedding results, and evaluating the adaptation results based on similarity ranking in a domain retrieval task. Once the model meets preset performance requirements, the general text embedding model is transformed into a sensitive domain localized adaptation model, enabling it to perform sensitive domain-oriented text representation and retrieval tasks in a local environment.

[0057] In one implementation, the Qwen3-4B-Instruct model is used to perform a professionalism score based on semantic information density on the initial sensitive domain corpus, and to filter and retain a corpus of high-value text paragraphs in the domain, including:

[0058] Y210: Deploy the Qwen3-4B-Instruct model locally using the vLLM inference engine; Y220: Input the first sensitive paragraph from the initial sensitive domain corpus into the Qwen3-4B-Instruct model, drive a professionalism scoring process guided by corpus quality scoring prompt word templates, and output the first paragraph score; Y230: If the first paragraph score is not lower than a preset scoring threshold, retain the first sensitive paragraph as the first sample paragraph; Y240: Similarly, input the remaining sensitive paragraphs from the initial sensitive domain corpus into the Qwen3-4B-Instruct model one by one, perform paragraph filtering based on professionalism scores, and select and retain the high-value text paragraph corpus of the domain.

[0059] The pre-acquired Qwen3-4B-Instruct model weight file is loaded onto the local server, and model initialization is completed according to the computing resource configuration, including model parameter loading, runtime environment configuration, and inference service startup. The vLLM inference engine optimizes the inference process of the large language model in the local environment, improving the model's efficiency in handling batch text input tasks through continuous batch processing, memory management, and request scheduling mechanisms. After model deployment, a text input interface is established, enabling subsequent sensitive domain corpus segments to be evaluated to be input into the Qwen3-4B-Instruct model for semantic analysis without external network transmission. This localized deployment ensures that text data during sensitive domain document processing remains within a controlled computing environment, meeting the security isolation requirements of sensitive domain data processing.

[0060] The first sensitive paragraph is populated into a pre-constructed corpus quality scoring prompt word template. This template constrains the model's evaluation process, ensuring the model analyzes text content according to a unified evaluation standard. The corpus quality scoring prompt word template limits the model's evaluation dimensions and output format, including evaluation criteria such as the degree of text domain relevance, usage of specialized terminology, completeness of knowledge expression, and effectiveness of technical information. The Qwen3-4B-Instruct model performs semantic understanding of the domain knowledge information contained in the first sensitive paragraph based on the input paragraph text and scoring rules, and outputs the corresponding first paragraph score. This first paragraph score represents the knowledge value of the text paragraph within the target sensitive domain, providing a quantitative basis for subsequent text selection.

[0061] A scoring threshold is pre-set based on the requirements for constructing the target sensitive domain corpus. When the score of the first paragraph reaches or exceeds the preset scoring threshold, it is determined that the first sensitive paragraph has high domain information value, and it is retained and marked as the first sample paragraph. The first sample paragraph represents the valid domain text sample retained after professional evaluation. Compared to directly using all the original corpus for model training, filtering by scoring threshold can reduce the impact of general descriptions, low information density text, and content with low domain relevance on the subsequent training process. Simultaneously, sensitive paragraphs that do not reach the scoring threshold are not included in the subsequent sample construction process, thereby increasing the proportion of valid knowledge content in the final domain corpus.

[0062] The remaining sensitive paragraphs are sequentially input into the Qwen3-4B-Instruct model, and a professionalism score is performed using the same corpus quality scoring prompt word template to obtain the score results for each sensitive paragraph. For each sensitive paragraph, based on the comparison between the corresponding score result and the preset scoring threshold, text paragraphs that meet the requirements are retained, and they are uniformly numbered, classified, and stored, along with their corpus information is recorded. By performing the above scoring and filtering process on all sensitive paragraphs, low-value, low-relevance, and incomplete text content is eliminated, ultimately forming a domain-high-value text paragraph corpus composed of multiple high-quality domain text paragraphs.

[0063] The high-value text paragraph corpus in the domain serves as the data foundation for subsequent domain query generation and comparative learning sample construction. Its text content has strong semantic features of the target domain, which can improve the accuracy of the association between the subsequently generated query and the training samples.

[0064] In one implementation, Y211: The Qwen3-4B-Instruct model includes a cascaded embedding layer, a multi-layer Transformer decoder, a final normalization layer, and a linear projection output layer; Y212: The embedding layer is equipped with a corpus quality scoring prompt word template and a query scoring prompt word template; Y213: The corpus quality scoring prompt word template requires the Qwen3-4B-Instruct model, acting as an expert in the target sensitive domain, to perform a three-dimensional comprehensive evaluation of the input paragraph based on terminology density, information substance depth, and domain relevance; Y214: The query scoring prompt word template requires the Qwen3-4B-Instruct model to perform a three-dimensional comprehensive evaluation of the query-positive document pair based on the domain semantic relevance of the target sensitive domain, paragraph content coverage, and semantic independence from context.

[0065] The Qwen3-4B-Instruct model adopts a generative language model structure based on the Transformer architecture, which includes a sequentially connected embedding layer, a multi-layer Transformer decoder, a final normalization layer, and a linear projection output layer. The embedding layer converts the input text sequence into its corresponding vector representation, mapping character and lexical information in the text to a continuous vector space. The multi-layer Transformer decoder extracts contextual relationships from the input text layer by layer based on a self-attention mechanism, enabling the model to learn semantic connections within the text. The final normalization layer standardizes the hidden states after multi-layer feature transformation to improve the stability of the output features. The linear projection output layer maps the hidden states to the target output space, generating the corresponding evaluation results or text output.

[0066] In the process of evaluating the quality of corpora in sensitive domains, after the text to be evaluated is input into the model, the text information is sequentially processed through the above network structure to complete semantic feature extraction and result generation, enabling the model to understand the text content in combination with contextual information and output a scoring result or analysis result that meets the evaluation requirements.

[0067] To enable the Qwen3-4B-Instruct model to perform sensitive domain text analysis according to preset evaluation criteria, corpus quality scoring prompt word templates and query scoring prompt word templates are configured at the model input end. These prompt word templates constrain the text task input to the model, guiding the model to complete text evaluation according to unified rules through preset role descriptions, evaluation dimension explanations, and output format requirements. Specifically, when screening domain corpora, the corpus quality scoring prompt word template is invoked, combining the sensitive paragraphs to be evaluated with the corresponding evaluation rules to form a complete model input; when evaluating the quality of query-positive document pairs, the query scoring prompt word template is invoked, inputting the query to be evaluated and the corresponding positive document content into the model according to a preset format.

[0068] The prompt word template does not change the basic network parameters of the Qwen3-4B-Instruct model, but constrains the model's inference process at the input text level, enabling the same model to perform corresponding evaluation tasks for different processing stages, thereby improving the consistency and repeatability of the evaluation process.

[0069] For the text segments to be screened in the initial sensitive domain corpus, the text segments are input into the Qwen3-4B-Instruct model configured with corpus quality scoring prompt word templates. The corpus quality scoring prompt word templates set the model to perform text analysis as an expert in the target sensitive domain, and require the model to make a comprehensive evaluation of the input segments based on preset evaluation dimensions.

[0070] The three-dimensional evaluation includes terminology density, information substance depth, and domain relevance. Terminology density evaluates the proportion of target-sensitive domain-specific vocabulary, expressions, and domain concepts in the text, determining whether the text possesses significant domain characteristics. Information substance depth evaluates whether the text contains specific technical descriptions, process explanations, parameter relationships, or knowledge logic, rather than merely general descriptions. Domain relevance evaluates the matching relationship between the text content and the task requirements of the target-sensitive domain. The Qwen3-4B-Instruct model performs semantic analysis on the input paragraph based on these three dimensions and outputs a corresponding comprehensive score, serving as an important basis for determining whether the text should be retained.

[0071] To reduce semantic bias or incomplete expression in synthesized queries, the query-positive document pairs are input into the Qwen3-4B-Instruct model, which is configured with query scoring prompt word templates, for quality evaluation of domain retrieval queries and corresponding positive example document pairs.

[0072] The query scoring suggestion template requires the model to comprehensively evaluate three aspects: domain semantic relevance, paragraph content coverage, and semantic independence from context. Domain semantic relevance determines whether the query accurately expresses the core information needs of the target sensitive domain; paragraph content coverage determines whether the query corresponds to the main knowledge content in the example document, avoiding queries that only relate to irrelevant local information in the document; and semantic independence from context determines whether the query still has a clear meaning without the assistance of the original document, avoiding the generation of invalid queries that depend on the original text context for understanding.

[0073] The quality of the generated query-positive document pairs is quantified using the three-dimensional evaluation method described above, and low-quality samples are filtered out based on the evaluation results, thereby improving the effectiveness and domain adaptability of subsequent comparative learning training data.

[0074] In one implementation, the first sensitive paragraph from the initial sensitive domain corpus is input into the Qwen3-4B-Instruct model, driving a professionalism scoring process guided by corpus quality scoring prompt word templates, and outputting the first paragraph score, including:

[0075] Y221: In the embedding layer, the first sensitive paragraph is used as the text to be scored and concatenated to the end of the corpus quality scoring prompt word template to construct a first complete input sequence. The first complete input sequence is then segmented to obtain an input token sequence. Y222: The input token sequence is fed into the multi-layer Transformer decoder to perform autoregressive inference and generate a hidden state sequence. Y223: The final normalization layer performs layer normalization on the hidden state sequence to obtain a normalized hidden state. Y224: The linear projection output layer maps the normalized hidden state to the vocabulary space, generates an output text sequence, and then performs score value parsing and extraction on the output text sequence to output the score of the first paragraph.

[0076] During the model input construction phase, the first sensitive paragraph is added to the end of the corpus quality scoring prompt word template according to preset text concatenation rules, forming a complete input sequence with the evaluation role, evaluation dimension, output format requirements, and text to be evaluated in the prompt word template. The corpus quality scoring prompt word template provides the model with an evaluation task description, enabling the model to analyze the input paragraph according to preset domain text evaluation standards; the first sensitive paragraph serves as the actual evaluation object, providing the domain text content to be analyzed. After concatenating the prompt word template and the first sensitive paragraph, the first complete input sequence is segmented, converting the continuous text into a sequence of input tokens that the model can recognize. Tokens are the basic semantic units used by large language models for text processing. Segmentation converts words, symbols, and special identifiers in the text into corresponding numbers, enabling the subsequent Transformer decoder to perform semantic feature extraction based on the serialized input.

[0077] In each Transformer decoder layer, a self-attention mechanism is used to calculate the relationships between input tokens. This allows the model to comprehensively consider the evaluation requirements in the corpus quality scoring prompts and the text content in the first sensitive paragraph, and to extract domain semantic features from the text layer by layer. The self-attention mechanism is used to assign feature weights based on the degree of correlation between different tokens, enabling the model to recognize technical terms, knowledge descriptions, and contextual relationships in the text.

[0078] After continuous processing by multiple Transformer decoders, the input token sequence is converted into a corresponding hidden state sequence. The hidden state sequence represents the feature representation of the input text content after the model performs deep semantic encoding, containing contextual semantic information related to the professionalism evaluation of the first sensitive paragraph, and providing a feature basis for the subsequent output scoring results.

[0079] Layer normalization is used to standardize and adjust the feature parameters in the hidden state sequence, keeping the feature values ​​of different dimensions within a stable range and reducing feature fluctuations caused by differences in input text length and word distribution during model inference. By normalizing the hidden state, the consistency of text semantic feature expression can be maintained, improving the stability of subsequent output mapping processes. Specifically, the normalized hidden state retains the domain semantic information extracted by the Transformer decoder from the first sensitive paragraph, and after scaling, it can adapt to the mapping requirements of the linear projection output layer, providing input features for generating the output text sequence corresponding to the scoring results.

[0080] The obtained normalized hidden state is input into a linear projection output layer, which maps the hidden state features to the model vocabulary space, generating a corresponding output text sequence. This output text sequence is the evaluation result text generated by the Qwen3-4B-Instruct model based on the corpus quality scoring prompt word template requirements, and its content includes score values, evaluation labels, or structured scoring information. Since the model output is in text form, after obtaining the output text sequence, score value parsing and extraction processing is performed on the output text sequence to identify and extract the corresponding numerical score information from the output content, obtaining the first paragraph score.

[0081] Specifically, based on preset scoring format rules, the scoring fields in the output text sequence are located, and the scoring values ​​representing the text's professionalism are extracted. For example, when the model outputs "Overall Score: 85" according to the preset format, the value "85" is extracted as the score for the first paragraph using the scoring parsing rules. This first paragraph score is then compared with a preset scoring threshold to determine whether the first sensitive paragraph meets the requirements for selecting high-value texts in the domain.

[0082] In one implementation, multiple sample paragraphs from the high-value text paragraph corpus of the aforementioned domain are used to generate domain synthesis queries based on the Qwen3-32B model, resulting in multiple sets of sample domain retrieval queries. Based on mapping relationships, multiple sets of query-positive document pairs are constructed, including:

[0083] Y310: Determine the number of queries to be generated based on the text length of the first sample paragraph; Y320: Use the first sample paragraph as the anchor text for query generation, and append it to the end of the synthetic query generation prompt template to construct a second complete input sequence; Y330: Feed the second complete input sequence into the Qwen3-32B model to drive autoregressive generation and output the first set of sample domain retrieval queries, wherein the synthetic query generation prompt template requires the Qwen3-32B model to generate diverse target sensitive domain retrieval queries based on the content of the input paragraph from the perspective of the commander of the target sensitive domain; Y340: Use the first sample paragraph as the first positive example document, establish a mapping relationship between the first set of sample domain retrieval queries and the first positive example document, and construct the first set of query-positive example document pairs.

[0084] The text length of the first sample paragraph is statistically analyzed. The text length can be represented by the number of characters, tokens, or semantic units. The number of queries to be generated is determined according to a preset query generation rule. For shorter first sample paragraphs, the domain knowledge information they contain is relatively concentrated; therefore, a smaller number of synthesized queries are used to avoid generating highly repetitive query content. For longer first sample paragraphs, the entities, technical relationships, and business descriptions they contain are richer, so the number of queries to be generated is increased to ensure that different knowledge dimensions are covered by the queries. By dynamically determining the number of queries generated based on the text length, the generated sample domain retrieval queries match the scale of the original text information, improving the data diversity of subsequent query-positive document pairs.

[0085] During the input construction phase, the first sample paragraph is appended to the end of the synthetic query generation prompt template, ensuring that the task description, role setting, query generation rules, and the text to be analyzed within the prompt form continuous input content. The synthetic query generation prompt template defines the model's query generation direction, enabling the model to understand key knowledge information in the input paragraph based on the characteristics of the target sensitive domain and generate corresponding queries according to actual retrieval needs. The first sample paragraph serves as anchor text for query generation, providing factual basis for the query generation process and ensuring that the generated queries maintain semantic relevance to the real-world domain text content.

[0086] By constructing the second complete input sequence in the above manner, the subsequent Qwen3-32B model can simultaneously obtain the query generation task requirements and the corresponding domain text information, providing an input basis for generating retrieval queries with domain characteristics.

[0087] The Qwen3-32B model employs a generative language model structure based on the Transformer architecture, comprising a cascaded embedding layer, a multi-layer Transformer decoder, a final normalization layer, and a linear projection output layer. The embedding layer carries synthetic query generation prompt word templates, converting the task description and anchor text in the prompt words into a unified vector representation. The multi-layer Transformer decoder extracts features from the input sequence layer by layer through a self-attention mechanism, enabling the model to comprehensively understand the query generation requirements in the prompt words and the domain semantic information in the first sample paragraph. The final normalization layer and the linear projection output layer map the extracted semantic features to the vocabulary space and gradually generate the query text through an autoregressive approach.

[0088] During the model inference process, the second complete input sequence is first processed by word segmentation, converted into the corresponding token sequence, and input into the embedding layer for vectorization representation; the multi-layer Transformer decoder performs contextual semantic analysis on the prompt word information and the content of the first sample paragraph in the input sequence based on the self-attention mechanism, and extracts knowledge features related to the target sensitive domain; after the final normalization layer processing, it is mapped to the word space by the linear projection output layer, and the query text is gradually generated through autoregression.

[0089] Specifically, the synthetic query generation prompt template requires the Qwen3-32B model to understand the content of the first sample paragraph from the perspective of personnel in the target sensitive domain, and to generate diverse target sensitive domain retrieval queries based on actual information acquisition needs. The generated queries should reflect the query expressions that domain personnel might use when performing tasks, analyzing problems, or acquiring information, while maintaining a semantic correspondence with the first sample paragraph. Finally, the model outputs a first set of sample domain retrieval queries containing multiple query phrases.

[0090] Each query in the first set of sample domain retrieval queries is taken as a query sample, and the first sample paragraph on which the query is based is taken as the target matching document. The relationship between the two is recorded. Since the query is generated from the content of the first sample paragraph, the first positive example document can serve as the semantic matching target of the corresponding query, representing the correct retrieval result of the query in the target sensitive domain.

[0091] The query-positive document pairs are used in the subsequent comparative learning training process. By enabling the model to learn the semantic relationship between the query and the corresponding document, the text embedding model's ability to represent text features of the target sensitive domain is improved. Furthermore, since the query generation process is based on anchored real-domain text, compared to completely manually constructed training samples, it can reduce the problem of samples deviating from actual domain knowledge, providing a foundation for quality control of subsequent training samples.

[0092] In one implementation, after filtering low-quality samples from the multiple sets of query-positive document pairs using multidimensional quality assessment, false negative example removal and hard negative example mining based on embedding vector similarity are performed to form a contrastive learning training sample set, including:

[0093] Y410: Feed multiple query-positive document pairs from the first set of query-positive document pairs into the Qwen3-4B-Instruct model to drive a multi-dimensional quality assessment guided by the query scoring prompt word template, resulting in multiple first domain semantic relevance scores, multiple first paragraph content coverage scores, and multiple first context-independent semantic independence scores; Y420: Use a preset relevance threshold, preset coverage threshold, and preset independence threshold to map and compare the first domain semantic relevance scores, first paragraph content coverage scores, and first context-independent semantic independence scores, and make a retention judgment on the multiple query-positive document pairs to obtain the first set of retained pairs; Y430: Similarly, apply the Qwen3-4B-Instruct model to filter low-quality samples from the multiple sets of query-positive document pairs to obtain multiple sets of Y440: Merge the inter-group separations of the multiple retained pairs to obtain K retained pairs, and then extract K retained domain retrieval queries and K retained positive example documents from them; Y450: Use the K retained domain retrieval queries to drive the un-fine-tuned general text embedding model to perform a dense search in the domain high-value text paragraph corpus to obtain K candidate document sets; Y460: Use the similarity scores of the K retained positive example documents and the K candidate document sets as anchors, and combine them with a preset similarity ratio threshold to remove false negative examples, to obtain K filtering candidate sets; Y470: Randomly sample a preset number of documents from the K filtering candidate sets to construct a set of K hard negative example documents; Y480: Combine the K retained domain retrieval queries, the K retained positive example documents, and the K hard negative example document sets one by one to construct K contrastive learning training samples, and aggregate them to form the contrastive learning training sample set.

[0094] Each query and its corresponding example document are input into the model, allowing the model to perform a comprehensive analysis based on the semantic relationship between the query content and the document content. The query scoring suggestion template is used to define the model's evaluation rules, requiring the model to score the query-example document pair from three dimensions: domain semantic relevance, paragraph content coverage, and semantic independence from context. Specifically, the domain semantic relevance score evaluates whether the query accurately reflects the information needs in the target sensitive domain; the paragraph content coverage score evaluates whether the information needs expressed by the query correspond to the main content of the example document; and the semantic independence from context score evaluates whether the query possesses a complete and clear retrieval meaning without relying on the original paragraph.

[0095] Through the above evaluation process, multiple scores for semantic relevance in the first domain, content coverage in the first paragraph, and semantic independence from context were obtained for the first set of query-positive document pairs, providing a quantitative basis for subsequent low-quality sample filtering.

[0096] For any query-positive document pair, if its first domain semantic relevance score is not lower than the preset relevance threshold, it indicates that the query is strongly related to the target sensitive domain knowledge; if its first paragraph content coverage score is not lower than the preset coverage threshold, it indicates that the query can effectively point to the main knowledge content in the positive document; if its first context-independent semantic independence score is not lower than the preset independence threshold, it indicates that the query still has clear retrieval significance under independent input conditions.

[0097] When all three evaluation dimensions meet their corresponding threshold requirements, the query-positive document pair is considered a valid sample and retained, forming the first set of retained pairs. For any query-positive document pair with a score below the corresponding threshold in any dimension, it is considered a low-quality sample and will not be included in the subsequent training sample construction process.

[0098] Each query-positive document pair is sequentially input into the Qwen3-4B-Instruct model, and a three-dimensional quality assessment is performed using query scoring prompt word templates. This yields scores for domain semantic relevance, paragraph content coverage, and semantic independence from context for each query-positive document pair. Based on preset relevance thresholds, coverage thresholds, and independence thresholds, a retention decision is made for each query-positive document pair.

[0099] By traversing and processing the entire set of query-positive document pairs, low-quality samples that are unclear in their expression, insufficiently related to the documents, or unable to independently express the retrieval requirements are filtered out. Only valid query-positive document pairs that meet the multidimensional evaluation criteria are retained, resulting in multiple sets of retained pairs.

[0100] Multiple retention pairs are aggregated according to a preset data organization method, and the number of valid samples after merging is counted. When the number of valid samples after merging reaches K, it is defined as K retention pairs. From the K retention pairs, the corresponding retention domain search queries and retention positive example documents are extracted, resulting in K retention domain search queries and K retention positive example documents. The retention domain search queries serve as query samples in subsequent contrastive learning, and the retention positive example documents serve as positive matching samples corresponding to the queries.

[0101] The general text embedding model is used to vectorize K domain-preserving search queries and each text segment in a corpus of high-value domain text segments, resulting in query embedding vectors and document embedding vectors. The text embedding vectors convert text semantic information into a computable vector form, enabling semantic matching between different texts based on distance relationships in the vector space. The similarity between the query embedding vector corresponding to each domain-preserving search query and the document embedding vectors in the corpus is calculated, and a predetermined number of candidate documents are selected according to the similarity ranking, forming a candidate document set corresponding to each query. Since the search process is based on the pre-fine-tuned general text embedding model, the candidate document set reflects the model's initial semantic retrieval capability before domain adaptation, providing a source of candidate samples for subsequent false negative example identification and hard negative example construction.

[0102] For any reserved domain retrieval query, the similarity score between the corresponding reserved positive document and each candidate document is obtained. Using the similarity score of the positive document as a baseline, and combined with a preset similarity ratio threshold, a candidate document is determined to be a false negative. A false negative is a document that is considered a negative sample during sample construction but actually has a high semantic relevance to the query or positive document. If the similarity ratio between the candidate document and the positive document exceeds the preset threshold, the document is considered to potentially contain domain knowledge content similar to the positive document and is removed from the candidate document set.

[0103] This paper uses contrastive loss as the training objective. Its core idea is to learn discriminative embedding representations by maximizing the similarity between the query and positive documents while minimizing the similarity with negative documents. For the query... , positive example document and the set of negative examples The loss function is defined as:

[0104]

[0105] in For cosine similarity, This is a temperature hyperparameter. Negative examples consist of two parts: explicit hard negative examples and batch negative examples during training. Together, they provide diverse negative example signals to learn more refined semantic discrimination boundaries.

[0106] By using the above-mentioned false negative example filtering method, the interference of erroneous negative samples on the subsequent comparative learning process is reduced, and the model is prevented from incorrectly learning the exclusion relationship between similar semantic documents and queries during training, ultimately resulting in K filtering candidate sets.

[0107] For each domain-preserving retrieval query, based on the similarity ranking between the remaining candidate documents in the filtered candidate set and the query, several candidate documents are randomly sampled or selected according to preset rules as hard negative example documents. Hard negative example documents are negative sample documents that have a high degree of semantic similarity to the query but are not the target document corresponding to the query. Compared to ordinary random negative examples, hard negative examples have semantic features closer to positive examples, increasing the difficulty for the model to distinguish similar semantic content and enabling the text embedding model to learn more fine-grained semantic boundaries during training. By constructing a corresponding hard negative example document set for each query, training data containing highly discriminative negative samples is formed, providing a sample foundation for subsequent comparative learning and optimization.

[0108] For each retained domain retrieval query, it is matched with its corresponding retained positive example documents to form a positive semantic association. Simultaneously, the set of hard negative example documents corresponding to the query is combined with the query to form a negative semantic sample relationship. Thus, each training sample includes a query, a positive example document, and multiple hard negative example documents. The contrastive learning training samples are used to guide the text embedding model in learning the semantic distance relationship between queries and documents, enhancing the vector proximity between queries and relevant documents and increasing the vector discriminability between queries and irrelevant documents during training. After aggregating K corresponding training samples, a complete contrastive learning training sample set is formed, which is used for the subsequent domain-specific training of the low-rank adaptation module.

[0109] In one implementation, based on freezing the original weight parameters in a pre-built general text embedding model, multiple low-rank adaptation modules are introduced. The contrastive learning training sample set is used as training data to perform grouped contrastive learning training on the multiple low-rank adaptation modules based on functional association clustering until a preset convergence condition is met, resulting in multiple domain-adaptation low-rank parameter matrices, including:

[0110] Y510: After freezing the intra-layer weight parameter matrices of multiple target layers in the general text embedding model, the multiple low-rank adaptation modules are initialized by mapping. Each low-rank adaptation module contains a low-rank decomposition matrix A and a low-rank decomposition matrix B, forming a bypass injection path parallel to the corresponding intra-layer weight parameter matrix. Y520: Based on the layer depth of the multiple target layers, the multiple low-rank adaptation modules are divided into multiple groups of low-rank adaptation modules. Y530: Based on the semantic level of the target layer of the multiple groups of low-rank adaptation modules, the contrastive learning training sample set is divided into multiple training sample subsets. Y540: In the general text embedding model, the multiple training sample subsets are used as grouped training data to perform polling and freezing low-rank adaptation training of the multiple groups of low-rank adaptation modules until a preset convergence condition is met, thus obtaining the multiple domain-adaptation low-rank parameter matrices.

[0111] Since the contrastive learning training sample set is constructed from domain-synthesized data, its data size is limited compared to a general pre-training dataset. Directly performing full parameter updates on the general text embedding model could easily lead to overfitting to the limited domain samples, while also increasing the number of parameter updates and computational resource consumption during training. Therefore, this scheme employs a low-rank adaptation method for incremental learning of domain parameters in the general text embedding model. Its working principle is as follows: Figure 3 As shown. Specifically, let the original weight parameter matrix of the target layer to be adapted in the general text embedding model be: Its update process can generally be described in the following form:

[0112]

[0113]

[0114] in, It is the original parameter matrix. The parameter matrix to be updated. These are the merged parameters. and It is a parameter matrix obtained through low-rank decomposition. During fine-tuning, the original weights of the model remain frozen, and only the low-rank decomposition matrix injected through the side path is updated to adapt to the target domain. During forward propagation, for the input... The original intermediate state is:

[0115]

[0116] After introducing the low-rank adaptation module, the computation state is updated as follows:

[0117]

[0118] After training, parameters can be merged using simple matrix addition without increasing computational overhead during the inference phase. Because it only requires updating a small number of parameters, it enables effective knowledge transfer on small to medium-sized, high-quality datasets, making it particularly suitable for scenarios where data is scarce in vertical domains.

[0119] Specifically, to address the domain adaptation requirements of general text embedding models, multiple target layers for text feature extraction are first selected in the model. Then, the in-layer weight parameter matrices of these target layers are frozen to ensure that the basic model parameters remain unchanged during subsequent training. The in-layer weight parameter matrices include trainable parameters in the Transformer network used for feature mapping, attention calculation, and semantic transformation. Freezing avoids large-scale overall updates to model parameters, reduces computational resource consumption during domain adaptation, and preserves the original semantic representation capabilities of the general text embedding model.

[0120] After freezing the target layer weights, multiple low-rank adaptation modules are mapped and initialized at the corresponding positions of each target layer. Each low-rank adaptation module includes a low-rank decomposition matrix A and a low-rank decomposition matrix B, which represent the incremental adjustment parameters for the original weight parameter matrix through matrix multiplication. The low-rank adaptation modules are connected in parallel with the weight parameter matrices in the corresponding layers to form a bypass injection path, so that the input features are superimposed with the domain feature information adjusted by the low-rank adaptation modules while being calculated by the original network.

[0121] The low-rank adaptation module does not directly modify the weights of the original model. Instead, it adjusts the model's capabilities by learning the changes in the low-dimensional parameter space, thereby providing a parameter adaptation basis for subsequent sensitive domain text representation while maintaining general semantic capabilities.

[0122] Based on the position of the target layer in the Transformer structure, multiple target layers are sorted according to their hierarchical relationship from shallow to deep, and the corresponding low-rank adaptation modules are divided into multiple groups according to a preset hierarchical division rule. For example, the low-rank adaptation modules corresponding to shallow target layers closer to the input side are divided into a low-level semantic adaptation group, and the low-rank adaptation modules corresponding to deep target layers closer to the output side are divided into a high-level semantic adaptation group.

[0123] In this system, network layers of different depths play different roles in representing textual features. Shallow networks are typically used to extract lexical, grammatical, and local structural information, while deep networks are used to form more abstract semantic relationships. Therefore, by grouping the low-rank adaptation modules according to the target layer depth, different module groups can perform targeted parameter adjustments for semantic representation features at different layers, improving the effectiveness of parameter updates during domain adaptation.

[0124] This study analyzes the semantic role of different low-rank adaptation module groups in the text representation process, corresponding to the target layer, and determines the sample features to focus on during adaptation training based on their corresponding semantic levels. For example, low-rank adaptation module groups related to shallow semantic representation are more suitable for learning domain terms, keyword expressions, and local semantic relationships; while low-rank adaptation module groups related to deep semantic representation are more suitable for learning domain knowledge relationships, contextual associations, and overall semantic matching relationships.

[0125] Based on the semantic features contained in the query terms, positive example documents, and hard negative example documents in the comparative learning training samples, the training samples are classified and matched, and samples with corresponding semantic features are assigned to the corresponding training sample subsets. This partitioning method allows different low-rank adaptation module groups to optimize their parameters using training data that matches their semantic level, enhancing the model's ability to learn knowledge features at different levels in sensitive domains.

[0126] During a training session, a set of low-rank adaptation modules is selected as the current training module, and a subset of training samples corresponding to its semantic level is loaded for comparative learning training. Simultaneously, the parameters of other low-rank adaptation modules are frozen, keeping their current parameters unchanged. The current training module adjusts the parameters of the corresponding low-rank decomposition matrices A and B based on the semantic relationship between the query and positive example documents, as well as the semantic difference between the query and hard negative example documents. This enables the text embedding vectors generated by the model to more accurately distinguish between sensitive domain-related and irrelevant text.

[0127] After training a set of low-rank adaptation modules, the process switches to the next set and repeats. Through round-robin optimization among multiple sets of modules, the low-rank adaptation parameters at each level gradually converge. Preset convergence conditions may include the training loss function reaching a set range, model performance metrics meeting requirements, or reaching a preset number of training rounds. Once these convergence conditions are met, low-rank adaptation training stops, and the low-rank decomposition matrix A and low-rank decomposition matrix B from each low-rank adaptation module are extracted to form multiple domain-specific low-rank parameter matrices. This provides a parameter basis for subsequent parameter fusion to achieve sensitive domain-specific localization adaptation.

[0128] One implementation also includes:

[0129] Y550: Using the first training sample subset as group training data, during the low-rank adaptation training of the first group of low-rank adaptation modules, only the first group of low-rank adaptation modules is activated to participate in parameter updates, while the remaining low-rank adaptation modules that are not activated remain temporarily frozen; Y560: The multiple groups of low-rank adaptation modules are traversed alternately in a preset round order to perform parameter updates under polling freeze until the preset convergence condition is met, resulting in multiple groups of domain adaptation low-rank parameter matrices; Y570: The multiple groups of domain adaptation low-rank parameter matrices are split into groups to obtain multiple domain adaptation low-rank parameter matrices; Y580: The product of the low-rank decomposition matrix A and the low-rank decomposition matrix B in the multiple domain adaptation low-rank parameter matrices is combined with the corresponding multiple in-layer weight parameter matrices by matrix addition to complete the high-sensitivity domain localization adaptation of the general text embedding model.

[0130] The query terms, positive example documents, and hard negative example documents from the first training sample subset are input into a general text embedding model. Corresponding vector representations are obtained through text embedding calculation, and the training loss is calculated based on the contrastive learning objective. During this training process, only the first group of low-rank adaptation modules is activated to participate in gradient calculation and parameter updates, allowing the low-rank decomposition matrices A and B in the first group of low-rank adaptation modules to be adjusted according to the training loss. Simultaneously, the remaining low-rank adaptation modules not participating in the current round of training are temporarily frozen, keeping their parameters unchanged. This temporary freezing is used to avoid parameter interference caused by simultaneous updates between low-rank adaptation modules at different semantic levels, enabling the current training module to learn domain features for samples at the corresponding semantic level. In this way, the first group of low-rank adaptation modules prioritizes learning the domain semantic features contained in the first training sample subset, providing initialization parameters for subsequent round-robin training of multiple low-rank adaptation modules.

[0131] In each training round, the current target low-rank adaptation module group is selected, its corresponding training sample subset is used as input data, and the low-rank adaptation modules in this group are activated to participate in backpropagation calculation. Simultaneously, other low-rank adaptation module groups not participating in the current training round are frozen, keeping their low-rank decomposition matrices A and B unchanged. After training the current module group is completed, the process switches to the next low-rank adaptation module group according to a preset order, and repeats the above training process.

[0132] The polling-freeze training method allows different low-rank adaptation module groups to optimize parameters for different levels of semantic features, while reducing feature shift issues caused by simultaneous updates from multiple modules. As the training rounds increase, each low-rank adaptation module group gradually learns the terminology, semantic relationships, and domain knowledge structure in the target-sensitive domain.

[0133] When the training process meets the preset convergence conditions, the polling and training is stopped and frozen. The preset convergence conditions may include the training loss reaching a set range, the verification index reaching a preset requirement, or the completion of a preset number of training iterations, and finally obtaining multiple sets of domain-adaptive low-rank parameter matrices.

[0134] Based on the grouping relationship of low-rank adaptation modules during training, the parameter results corresponding to each group of low-rank adaptation modules are identified and separated, and the group-level low-rank parameter matrix containing parameters of multiple modules is split into multiple individual low-rank parameter matrices. Each domain-adaptive low-rank parameter matrix corresponds to a target layer or a low-rank adaptation module in the general text embedding model, representing the parameter adjustment information learned by that target layer for the target-sensitive domain.

[0135] By splitting the process, the domain adaptation parameters corresponding to different target layers are stored independently, which facilitates the accurate mapping of each low-rank parameter matrix to the corresponding weight parameter matrix according to the model network structure, thereby achieving precise fusion of domain adaptation information.

[0136] For any target layer, the low-rank decomposition matrix A and low-rank decomposition matrix B in the corresponding low-rank adaptation module are multiplied to obtain the domain parameter increment matrix for that target layer. This domain parameter increment matrix is ​​then combined with the corresponding layer's weight parameter matrix through matrix addition, resulting in the original weight parameters being infused with target-sensitive domain feature adjustment information. The product of low-rank decomposition matrix A and low-rank matrix B represents the parameter changes learned during low-rank adaptation training, which is used to modify the model's feature representation capability without retraining the complete model parameters. After matrix addition fusion, the general text embedding model retains its original general semantic understanding capability while also possessing the ability to extract features from target-sensitive domain text.

[0137] After completing the fusion of all target layer parameters, a text embedding model with high sensitivity domain localization adaptation is obtained. This model can perform target sensitive domain text representation, semantic matching, and information retrieval tasks in a local computing environment.

[0138] In one implementation, the original sensitive document set collected based on the target sensitive domain is parsed and filtered offline locally to obtain an initial sensitive domain corpus, including:

[0139] Y110: Expand the domain tag set based on the knowledge system of the target sensitive domain; Y120: Randomly group the domain tag set into a preset number of tags to construct multiple domain tag constraints; Y130: Use the multiple domain tag constraints to collect targeted documents to obtain the original sensitive document set; Y140: After formatting and parsing the original sensitive document set, perform format noise filtering to produce the initial sensitive domain corpus.

[0140] A basic knowledge system corresponding to the target sensitive domain is acquired. This knowledge system includes a domain classification structure, core concept nodes, professional entities, and the relationships between entities. Key knowledge nodes in this knowledge system are semantically expanded, generating extended tags by combining domain synonyms, hierarchical concepts, related business descriptions, and professional expressions. The expanded domain tag set is used to represent multi-level knowledge features in the target sensitive domain, enabling subsequent document collection to cover domain content at different semantic granularities. The domain tag set includes basic domain tags, extended semantic tags, and related feature tags, used to improve the coverage and domain relevance of sensitive domain document resources.

[0141] According to a preset single-group tag capacity, multiple domain tags are randomly selected from the domain tag set to form a domain tag group, and the tag selection process is repeated until all domain tags are grouped, resulting in multiple domain tag constraints. Each domain tag constraint is used to limit the semantic retrieval scope in a single document acquisition task, enabling different tag combinations to cover different knowledge directions within the target sensitive domain. By randomly combining domain tags, the acquisition bias caused by fixed tag combinations can be reduced, resulting in a higher breadth of domain coverage for the subsequently obtained original sensitive document set.

[0142] Each domain tag constraint is used as a document retrieval condition. Text data related to the corresponding tag constraint is obtained from preset document sources through keyword matching, semantic retrieval, or domain knowledge association retrieval. These document sources may include publicly available technical materials, industry standard documents, professional databases, and domain-specific business materials. During document collection, independent searches are performed for each domain tag constraint, and the document results obtained under different tag constraints are aggregated to form an initial set of sensitive documents. This targeted collection method, driven by multiple sets of domain tag constraints, ensures that the collected results cover different knowledge branches within the target sensitive domain, while reducing the proportion of non-domain-related documents.

[0143] The system parses the file structure, character encoding, paragraph hierarchy, and text identifier information of the original sensitive documents to extract valid text content. It then segments the text into paragraphs and unifies the format according to preset text organization rules. The parsed text undergoes format noise filtering, including removing invalid characters, duplicate content, webpage structure information, abnormal symbols, and non-semantic tags. Furthermore, it performs validity checks on the text content based on preset quality rules, filtering out document fragments with missing content, semantic anomalies, or insufficient domain relevance, retaining only sensitive domain text data that meets quality requirements.

[0144] Through the above processing, the original sensitive documents with heterogeneous sources and inconsistent formats are transformed into structured, high-quality text resources, forming an initial sensitive domain corpus, which provides a data foundation for subsequent high-value text paragraph screening, query generation, and domain adaptation training of text embedding models.

[0145] Example 2: Based on the same inventive concept as the text embedding model localization fine-tuning method for sensitive domains in the foregoing examples, this invention provides a text embedding model localization fine-tuning system for sensitive domains. See [link to example]. Figure 4 The system includes:

[0146] The parsing and filtering module 10 is used to perform local offline parsing and filtering on the original sensitive document set collected based on the target sensitive domain, to obtain an initial sensitive domain corpus; the professionalism scoring module 20 is used to perform professionalism scoring on the initial sensitive domain corpus based on semantic information density using the Qwen3-4B-Instruct model, and to filter and retain the corpus of high-value text paragraphs in the domain; the domain synthesis query module 30 is used to generate domain synthesis queries based on the Qwen3-32B model from multiple sample paragraphs in the corpus of high-value text paragraphs in the domain, to obtain multiple sets of sample domain retrieval queries, and to construct multiple sets of query-positive document pairs based on mapping relationships; the hard negative example mining module 40 is used to perform multidimensional quality assessment on the... After filtering low-quality samples from multiple sets of query-positive document pairs, false negative example removal and hard negative example mining based on embedding vector similarity are performed to form a contrastive learning training sample set. The learning training module 50 is used to introduce multiple low-rank adaptation modules based on the frozen original weight parameters in the pre-constructed general text embedding model, and use the contrastive learning training sample set as training data to perform grouped contrastive learning training on the multiple low-rank adaptation modules based on functional relevance clustering until the preset convergence condition is met, resulting in multiple domain-adapted low-rank parameter matrices. The localization adaptation module 60 is used to merge the multiple domain-adapted low-rank parameter matrices into the general text embedding model through matrix addition, completing the high-sensitivity domain localization adaptation of the general text embedding model.

[0147] In one implementation, the professionalism scoring module 20 is used for:

[0148] The Qwen3-4B-Instruct model is deployed locally using a vLLM inference engine. The first sensitive paragraph from the initial sensitive domain corpus is input into the Qwen3-4B-Instruct model, driving a professionalism scoring process guided by corpus quality scoring prompts, and outputting the first paragraph score. If the first paragraph score is not lower than a preset scoring threshold, the first sensitive paragraph is retained as the first sample paragraph. Similarly, the remaining sensitive paragraphs from the initial sensitive domain corpus are input into the Qwen3-4B-Instruct model one by one, and paragraph filtering based on professionalism scores is performed to select and retain the high-value text paragraph corpus of the domain.

[0149] In one implementation, the Qwen3-4B-Instruct model includes a cascaded embedding layer, a multi-layer Transformer decoder, a final normalization layer, and a linear projection output layer. The embedding layer is equipped with corpus quality scoring prompt templates and query scoring prompt templates. The corpus quality scoring prompt templates require the Qwen3-4B-Instruct model, acting as an expert in the target sensitive domain, to perform a three-dimensional comprehensive evaluation of the input paragraph based on terminology density, information substance depth, and domain relevance. The query scoring prompt templates require the Qwen3-4B-Instruct model to perform a three-dimensional comprehensive evaluation of the query-positive document pair based on the domain semantic relevance of the target sensitive domain, paragraph content coverage, and semantic independence from context.

[0150] In one implementation, the professionalism scoring module 20 is used for:

[0151] In the embedding layer, the first sensitive paragraph is used as the text to be scored and concatenated to the end of the corpus quality scoring prompt word template to construct a first complete input sequence. The first complete input sequence is then segmented to obtain an input token sequence. The input token sequence is fed into the multi-layer Transformer decoder to perform autoregressive inference and generate a hidden state sequence. The final normalization layer performs layer normalization on the hidden state sequence to obtain a normalized hidden state. The linear projection output layer maps the normalized hidden state to the vocabulary space, generates an output text sequence, and then performs score value parsing and extraction on the output text sequence to output the score of the first paragraph.

[0152] In one implementation, the domain synthesis query module 30 is used for:

[0153] The number of queries to be generated is determined based on the text length of the first sample paragraph; the first sample paragraph is used as the anchor text for query generation and appended to the end of the synthetic query generation prompt template to construct a second complete input sequence; the second complete input sequence is fed into the Qwen3-32B model to drive autoregressive generation and output the first set of sample domain retrieval queries, wherein the synthetic query generation prompt template requires the Qwen3-32B model to generate diverse target sensitive domain retrieval queries based on the content of the input paragraph from the perspective of the commander of the target sensitive domain; the first sample paragraph is used as the first positive example document to establish a mapping relationship between the first set of sample domain retrieval queries and the first positive example document, constructing the first set of query-positive example document pairs.

[0154] In one implementation, the hard negative example mining module 40 is used for:

[0155] Multiple query-positive document pairs from the first set of query-positive document pairs are fed into the Qwen3-4B-Instruct model to drive a multi-dimensional quality assessment guided by the query scoring prompt word template, resulting in multiple first domain semantic relevance scores, multiple first paragraph content coverage scores, and multiple first context-independent semantic scores. A preset relevance threshold, preset coverage threshold, and preset independence threshold are used to map and compare the first domain semantic relevance scores, first paragraph content coverage scores, and first context-independent semantic scores to determine the retention of the multiple query-positive document pairs, resulting in the first set of retained pairs. Similarly, the Qwen3-4B-Instruct model is applied to filter low-quality samples from the multiple sets of query-positive document pairs. Multiple sets of retained pairs are obtained; the inter-group separations of the multiple sets of retained pairs are merged to obtain K retained pairs, from which K domain retrieval queries and K positive example documents are extracted; the K retained domain retrieval queries drive the un-fine-tuned general text embedding model to perform a dense search in the domain high-value text paragraph corpus to obtain K candidate document sets; using the similarity scores of the K positive example documents and the K candidate document sets as anchors, false negative examples are removed by combining a preset similarity ratio threshold to obtain K filtering candidate sets; a preset number of documents are randomly sampled from the K filtering candidate sets to construct K hard negative example document sets; the K retained domain retrieval queries, the K positive example documents, and the K hard negative example document sets are combined one by one to construct K contrastive learning training samples, which are then aggregated to form the contrastive learning training sample set.

[0156] In one implementation, the learning and training module 50 is used for:

[0157] After freezing the intra-layer weight parameter matrices of multiple target layers in the general text embedding model, the multiple low-rank adaptation modules are initialized by mapping. Each low-rank adaptation module contains a low-rank decomposition matrix A and a low-rank decomposition matrix B, forming a bypass injection path parallel to the corresponding intra-layer weight parameter matrix. Based on the layer depth of the multiple target layers, the multiple low-rank adaptation modules are divided into multiple groups of low-rank adaptation modules. According to the target layer semantic level of the multiple groups of low-rank adaptation modules, the contrastive learning training sample set is divided into multiple training sample subsets. In the general text embedding model, the multiple training sample subsets are used as grouped training data to perform polling and freezing low-rank adaptation training of the multiple groups of low-rank adaptation modules until a preset convergence condition is met, thus obtaining the multiple domain-adapted low-rank parameter matrices.

[0158] In one implementation, the learning and training module 50 is used for:

[0159] Using a first subset of training samples as grouped training data, during the low-rank adaptation training of the first group of low-rank adaptation modules, only the first group of low-rank adaptation modules is activated to participate in parameter updates, while the remaining low-rank adaptation modules that are not activated remain temporarily frozen. The multiple groups of low-rank adaptation modules are traversed alternately in a preset round order to perform parameter updates under polling freeze until a preset convergence condition is met, resulting in multiple groups of domain-adapted low-rank parameter matrices. These multiple groups of domain-adapted low-rank parameter matrices are then split into groups to obtain multiple domain-adapted low-rank parameter matrices. The product of the low-rank decomposition matrix A and the low-rank decomposition matrix B in the multiple domain-adapted low-rank parameter matrices is then added and merged with the corresponding multiple in-layer weight parameter matrices to complete the high-sensitivity domain-localization adaptation of the general text embedding model.

[0160] In one implementation, the parsing and filtering module 10 is used to:

[0161] Based on the knowledge system of the target sensitive domain, the domain tag set is expanded; the domain tag set is randomly grouped into multiple domain tag constraints by a preset number of tags; the multiple domain tag constraints are used to collect targeted documents to obtain the original sensitive document set; after formatting and parsing the original sensitive document set, format noise is filtered to produce the initial sensitive domain corpus.

[0162] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

[0163] The foregoing description of specific exemplary embodiments of the invention is for illustrative and explanatory purposes. These descriptions are not intended to limit the invention to the precise forms disclosed, and it will be apparent that many changes and variations can be made in accordance with the foregoing teachings. The exemplary embodiments were chosen and described in order to explain the specific principles of the invention and its practical application, thereby enabling those skilled in the art to implement and utilize various different exemplary embodiments of the invention, as well as various different choices and variations. The scope of the invention is intended to be defined by the claims and their equivalents.

Claims

1. A method for localizing and fine-tuning text embedding models for sensitive domains, characterized in that, The method includes: The original sensitive document set collected based on the target sensitive domain is parsed and filtered locally offline to obtain the initial sensitive domain corpus; The Qwen3-4B-Instruct model was used to score the professionalism of the initial sensitive domain corpus based on semantic information density, and the corpus of high-value text paragraphs in the domain was selected and retained. Multiple sample paragraphs in the high-value text paragraph corpus of the aforementioned domain are used to generate domain synthesis queries based on the Qwen3-32B model, resulting in multiple sets of sample domain retrieval queries. Based on the mapping relationship, multiple sets of query-positive document pairs are constructed. After filtering low-quality samples from the multiple sets of query-positive document pairs using multidimensional quality assessment, false negative examples are removed and hard negative examples are mined based on embedding vector similarity to form a contrastive learning training sample set. Based on freezing the original weight parameters in the pre-built general text embedding model, multiple low-rank adaptation modules are introduced, and the contrastive learning training sample set is used as training data. The multiple low-rank adaptation modules are subjected to grouped contrastive learning training based on functional association degree clustering until the preset convergence condition is met, and multiple domain adaptation low-rank parameter matrices are obtained. The multiple domain-adapted low-rank parameter matrices are merged into the general text embedding model through matrix addition, thereby completing the high-sensitivity domain localization adaptation of the general text embedding model.

2. The method for localizing and fine-tuning text embedding models for sensitive domains as described in claim 1, characterized in that, The Qwen3-4B-Instruct model was used to perform a professionalism score based on semantic information density on the initial sensitive domain corpus, and a corpus of high-value text paragraphs in the domain was selected and retained, including: The Qwen3-4B-Instruct model is deployed locally using the vLLM inference engine; The first sensitive paragraph in the initial sensitive domain corpus is input into the Qwen3-4B-Instruct model to drive a professionalism scoring process guided by the corpus quality scoring prompt word template, and output the score of the first paragraph. If the score of the first paragraph is not lower than the preset score threshold, then the first sensitive paragraph is retained as the first sample paragraph. Similarly, the remaining sensitive paragraphs in the initial sensitive domain corpus are input one by one into the Qwen3-4B-Instruct model, and paragraph filtering based on professional score is performed to select and retain the high-value text paragraph corpus of the domain.

3. The method for localizing and fine-tuning text embedding models for sensitive domains as described in claim 2, characterized in that, The Qwen3-4B-Instruct model includes a cascaded embedding layer, a multi-layer Transformer decoder, a final normalization layer, and a linear projection output layer. The embedded layer includes a corpus quality scoring prompt word template and a query scoring prompt word template. The corpus quality scoring prompt word template requires the Qwen3-4B-Instruct model to perform a three-dimensional comprehensive evaluation of the input paragraph based on terminology density, information substance depth, and domain relevance, in the capacity of an expert in the target sensitive domain. The query scoring prompt word template requires the Qwen3-4B-Instruct model to perform a three-dimensional comprehensive evaluation of the query-positive document pair based on the domain semantic relevance of the target sensitive area, the paragraph content coverage, and the semantic independence from context.

4. The method for localizing and fine-tuning text embedding models for sensitive domains as described in claim 3, characterized in that, The first sensitive paragraph from the initial sensitive domain corpus is input into the Qwen3-4B-Instruct model, driving a professionalism scoring process guided by corpus quality scoring prompt word templates, and outputting the first paragraph score, including: In the embedding layer, the first sensitive paragraph is used as the text to be scored and is concatenated to the end of the corpus quality scoring prompt word template to construct a first complete input sequence. The first complete input sequence is then segmented to obtain an input token sequence. The input token sequence is fed into the multi-layer Transformer decoder to perform autoregressive inference and generate a hidden state sequence. The final normalization layer performs layer normalization processing on the hidden state sequence to obtain the normalized hidden state; The linear projection output layer maps the normalized hidden state to the vocabulary space, generates an output text sequence, and then performs score value parsing and extraction on the output text sequence to output the score of the first paragraph.

5. The method for localizing and fine-tuning text embedding models for sensitive domains as described in claim 3, characterized in that, Multiple sample paragraphs from the high-value text paragraph corpus in the aforementioned domain are used to generate domain-synthesized queries based on the Qwen3-32B model, resulting in multiple sets of sample domain retrieval queries. Based on the mapping relationship, multiple sets of query-positive document pairs are constructed, including: The number of query entries to be generated is determined based on the text length of the first sample paragraph; The first sample paragraph is used as the anchor text for query generation and is appended to the end of the synthetic query generation prompt word template to construct the second complete input sequence; The second complete input sequence is fed into the Qwen3-32B model to drive autoregressive generation and output the first set of sample domain search query terms. The synthetic query generation prompt word template requires the Qwen3-32B model to generate diverse target sensitive domain search queries based on the input paragraph content from the perspective of the commander of the target sensitive domain. Using the first sample paragraph as the first positive example document, establish a mapping relationship between the first set of sample domain retrieval queries and the first positive example document, and construct the first set of query-positive example document pairs.

6. The method for localizing and fine-tuning text embedding models for sensitive domains as described in claim 5, characterized in that, After filtering low-quality samples from the multiple query-positive document pairs using multidimensional quality assessment, false negative example removal and hard negative example mining based on embedding vector similarity are performed to form a contrastive learning training sample set, including: Multiple query-positive document pairs from the first group of query-positive document pairs are fed into the Qwen3-4B-Instruct model to drive a multi-dimensional quality assessment guided by the query scoring prompt word template, resulting in multiple first domain semantic relevance scores, multiple first paragraph content coverage scores, and multiple first context-independent semantic scores. Using a preset relevance threshold, a preset coverage threshold, and a preset independence threshold, the semantic relevance score of the first domain, the content coverage score of the first paragraph, and the semantic independence score of the first paragraph are compared and compared to determine the retention of the multiple query-positive document pairs, resulting in the first set of retention pairs. Similarly, the Qwen3-4B-Instruct model is applied to filter low-quality samples from the multiple sets of query-positive document pairs to obtain multiple sets of retained pairs; After merging the inter-group separations of the multiple sets of reserved pairs to obtain K reserved pairs, K reserved domain retrieval queries and K reserved positive example documents are extracted from them; The general text embedding model before fine-tuning is driven by the K domain-preserving retrieval queries to perform a dense search in the domain-high-value text paragraph corpus to obtain a set of K candidate documents. Using the similarity scores of the K retained positive documents and the K candidate document sets as anchor points, and combining them with a preset similarity ratio threshold, false negative examples are removed to obtain K filtering candidate sets; A set of K hard negative example documents is constructed by randomly sampling a preset number of documents from the K filter candidate sets; The K retained domain retrieval queries, K retained positive example documents, and K hard negative example document sets are combined one by one to construct K contrastive learning training samples, which are then aggregated to form the contrastive learning training sample set.

7. The method for localizing and fine-tuning a text embedding model for sensitive domains as described in claim 6, characterized in that, Based on freezing the original weight parameters in the pre-built general text embedding model, multiple low-rank adaptation modules are introduced, and the contrastive learning training sample set is used as training data. Grouped contrastive learning training based on functional association clustering is performed on these multiple low-rank adaptation modules until a preset convergence condition is met, resulting in multiple domain-adaptation low-rank parameter matrices, including: After freezing the weight parameter matrices of multiple target layers in the general text embedding model, the multiple low-rank adaptation modules are initialized by mapping. Each low-rank adaptation module contains a low-rank decomposition matrix A and a low-rank decomposition matrix B, which form a bypass injection path parallel to the corresponding weight parameter matrix. Based on the hierarchical depth of the multiple target layers, the multiple low-rank adaptation modules are divided into multiple groups of low-rank adaptation modules; Based on the target layer semantic hierarchy of the multiple low-rank adaptation modules, the contrastive learning training sample set is divided into multiple training sample subsets; In the general text embedding model, the multiple training sample subsets are used as grouped training data to perform polling and freezing low-rank adaptation training of the multiple groups of low-rank adaptation modules until the preset convergence condition is met, thereby obtaining the multiple domain adaptation low-rank parameter matrices.

8. The method for localizing and fine-tuning a text embedding model for sensitive domains as described in claim 7, characterized in that, Also includes: Using the first training sample subset as group training data, during the low-rank adaptation training of the first group of low-rank adaptation modules, only the first group of low-rank adaptation modules are activated to participate in parameter updates, while the other low-rank adaptation modules that are not activated remain temporarily frozen. The multiple sets of low-rank adaptation modules are traversed alternately in a preset round order to update parameters under polling and freezing until the preset convergence condition is met, resulting in multiple sets of domain adaptation low-rank parameter matrices. The multiple sets of domain adaptation low-rank parameter matrices are grouped to obtain the multiple sets of domain adaptation low-rank parameter matrices; The product of the low-rank decomposition matrix A and the low-rank decomposition matrix B in the multiple domain adaptation low-rank parameter matrices is combined with the corresponding multiple in-layer weight parameter matrices by matrix addition to complete the high-sensitivity domain localization adaptation of the general text embedding model.

9. The method for localizing and fine-tuning a text embedding model for sensitive domains as described in claim 1, characterized in that, The original sensitive document set collected based on the target sensitive domain is parsed and filtered offline locally to obtain an initial sensitive domain corpus, including: Based on the knowledge system of the target sensitive areas, expand the domain tag set; The domain tag set is randomly grouped into groups with a preset number of tags to construct multiple domain tag constraints; The original sensitive document set is obtained by using the multiple domain tag constraints for targeted document collection. After formatting and parsing the original sensitive document set, format noise filtering is performed to produce the initial sensitive domain corpus.

10. A localized fine-tuning system for text embedding models in sensitive domains, characterized in that, For implementing the method steps of any one of claims 1 to 9, including: The parsing and filtering module is used to perform local offline parsing and filtering on the original sensitive document set collected based on the target sensitive domain, so as to obtain the initial sensitive domain corpus; The professionalism scoring module is used to perform a professionalism score on the initial sensitive domain corpus based on semantic information density using the Qwen3-4B-Instruct model, and to filter and retain the corpus of high-value text paragraphs in the domain. The domain synthesis query module is used to generate domain synthesis queries based on the Qwen3-32B model from multiple sample paragraphs in the high-value text paragraph corpus of the domain, obtain multiple sets of sample domain retrieval query terms, and construct multiple sets of query-positive document pairs based on the mapping relationship; The hard negative example mining module is used to filter low-quality samples from the multiple sets of query-positive document pairs using multidimensional quality assessment, and then perform false negative example removal and hard negative example mining based on embedding vector similarity to form a contrastive learning training sample set. The learning and training module is used to introduce multiple low-rank adaptation modules on the basis of freezing the original weight parameters in the pre-built general text embedding model, and to use the contrastive learning and training sample set as training data to perform grouped contrastive learning and training on the multiple low-rank adaptation modules based on functional correlation clustering until the preset convergence condition is met, so as to obtain multiple domain adaptation low-rank parameter matrices. The localization adaptation module is used to merge the multiple domain adaptation low-rank parameter matrices into the general text embedding model through matrix addition, thereby completing the high-sensitivity domain localization adaptation of the general text embedding model.

Images