Embedding data synthesis method and device fusing retrieval and large model distillation, and medium

Abstract

This application provides a method, apparatus, and medium for embedding data synthesis that integrates retrieval and large-model distillation. The method includes: preprocessing unstructured documents from a vertical domain, dividing the unstructured documents into multi-granularity text blocks with hierarchical relationships; forming a context based on the combination of multi-granularity text blocks, injecting perturbation information corresponding to prior knowledge of the vertical domain into the context, calling a generative model to generate a retrieval query based on the context, and determining the target text block corresponding to the retrieval query as the initial positive sample; filtering false negative sample text blocks according to the relationships between text blocks to form a positive sample set and a negative sample set; constructing contrastive learning training samples, and using the contrastive learning training samples to train a semantic representation model to generate embedding vectors for retrieval tasks. This application can improve the efficiency and realism of retrieval task construction, enhance the completeness of positive sample coverage, and improve the stability and retrieval accuracy of contrastive learning training.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an embedded data synthesis method, apparatus and medium that integrates retrieval and large model distillation. Background Technology

[0002] In the field of large-scale model and intelligent agent applications, especially in the field of embedded data synthesis technology for vertical industries such as finance, healthcare, and industrial manufacturing, as various industries accelerate digital transformation, a large number of unstructured documents such as research reports, medical records, technical manuals, and construction documents are accumulating. These documents need to be transformed into machine-processable semantic representations through data embedding to support applications such as enhanced retrieval generation and question-answering decision-making.

[0003] In existing technologies, documents are typically segmented, retrieval queries are constructed, and a semantic representation model is trained using contrastive learning to obtain embedding vectors for retrieval. However, on the one hand, existing methods lack an efficient toolchain for constructing retrieval tasks when faced with massive amounts of unstructured documents. The generated queries often rely on general templates or simple prompts, making it difficult to reflect the professional terminology, jargon, and real-world usage scenarios of vertical domains. This results in the synthesized embedding data being disconnected from actual retrieval needs.

[0004] On the other hand, in the process of constructing contrastive learning samples, existing technologies mostly rely on fixed-length text blocks and select positive and negative samples through simple similarity or random strategies. This approach easily leads to the fragmentation of cross-text block and cross-chapter related information, resulting in incomplete positive sample recall; negative samples are mostly easily distinguishable random negative examples, lacking systematic mining of difficult negative samples that are semantically similar but do not match business requirements. At the same time, due to the lack of effective filtering of false negative samples, samples that are semantically related but labeled as negative examples are easily introduced into the contrastive learning loss, affecting the model's convergence stability and retrieval accuracy, making it even more difficult to obtain high-quality embedding data in data-scarce vertical domains. Summary of the Invention

[0005] In view of this, embodiments of this application provide an embedded data synthesis method, apparatus and medium that integrates retrieval and large model distillation to solve the problems of low efficiency in constructing retrieval tasks and failure to fit business needs, incomplete positive sample recall, difficult negative sample mining and insufficient filtering of false negative samples in the prior art.

[0006] A first aspect of this application provides an embedded data synthesis method that integrates retrieval and large model distillation, comprising: preprocessing unstructured documents in a vertical domain, dividing the unstructured documents into multi-granularity text blocks with hierarchical relationships, and constructing a knowledge base based on the multi-granularity text blocks and their relationships; forming a context based on the combination of multi-granularity text blocks in the knowledge base, injecting perturbation information corresponding to prior knowledge in the vertical domain into the context, calling a generative model to generate a retrieval query based on the context, and determining the target text block corresponding to the retrieval query as an initial positive sample; inputting the retrieval query into a semantic retrieval system, and in the knowledge base... A candidate text block set is retrieved from the knowledge base and ranked according to relevance. Based on the ranking results and the inference model, positive sample text blocks for supplementation and difficult negative sample text blocks that are semantically similar to the retrieval query but are not positive samples are mined from the candidate text block set. False negative sample text blocks are filtered according to the correlation between text blocks to form a corresponding positive sample set and negative sample set for each retrieval query. Contrastive learning training samples are constructed based on the positive sample set and negative sample set, and a semantic representation model is trained using the contrastive learning training samples to generate embedding vectors for the retrieval task.

[0007] A second aspect of this application provides an embedded data synthesis apparatus that integrates retrieval and large model distillation, comprising: a preprocessing module for preprocessing unstructured documents in a vertical domain, dividing the unstructured documents into multi-granularity text blocks with hierarchical relationships, and constructing a knowledge base based on the multi-granularity text blocks and their relationships; a generation module for forming a context based on the combination of multi-granularity text blocks in the knowledge base, injecting perturbation information corresponding to prior knowledge in the vertical domain into the context, calling a generation model to generate a retrieval query based on the context, and determining the target text block corresponding to the retrieval query as an initial positive sample; and a retrieval module for inputting the retrieval query into a semantic retrieval system. The system retrieves a set of candidate text blocks from the knowledge base and sorts them according to relevance. The mining module, based on the sorting results and the inference model, mines positive sample text blocks for supplementation and difficult negative sample text blocks that are semantically similar to the retrieval query but are not positive samples from the candidate text block set. It also filters false negative sample text blocks based on the correlation between text blocks to form a corresponding positive sample set and negative sample set for each retrieval query. The training module is used to construct contrastive learning training samples based on the positive sample set and the negative sample set, and uses the contrastive learning training samples to train the semantic representation model to generate embedding vectors for the retrieval task.

[0008] A third aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the steps of the above-described method.

[0009] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method.

[0010] The above-described technical solutions adopted in the embodiments of this application can achieve the following beneficial effects:

[0011] By preprocessing unstructured documents in a vertical domain, the documents are divided into multi-granularity text blocks with hierarchical relationships, and a knowledge base is constructed based on these multi-granularity text blocks and their relationships. A context is formed by combining these multi-granularity text blocks from the knowledge base, and perturbation information corresponding to prior knowledge of the vertical domain is injected into the context. A generative model is then invoked to generate a retrieval query based on the context, and the target text block corresponding to the retrieval query is identified as the initial positive sample. The retrieval query is input into a semantic retrieval system, which retrieves a set of candidate text blocks from the knowledge base and ranks them according to relevance. Based on the ranking results and the inference model, supplementary positive sample text blocks and difficult negative sample text blocks that are semantically similar to the retrieval query but not positive samples are extracted from the candidate text block set. False negative sample text blocks are filtered based on the relationships between text blocks to form corresponding positive and negative sample sets for each retrieval query. Contrastive learning training samples are constructed based on the positive and negative sample sets, and a semantic representation model is trained using these samples to generate embedding vectors for retrieval tasks. This application can improve the efficiency and authenticity of retrieval task construction, enhance the completeness of positive sample coverage, and improve the stability and retrieval accuracy of contrastive learning training. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a flowchart illustrating the embedded data synthesis method for fusion retrieval and large model distillation provided in this application embodiment;

[0014] Figure 2 This is a schematic diagram of the embedded data synthesis device for fusion retrieval and large model distillation provided in the embodiments of this application;

[0015] Figure 3 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0016] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0017] This application mainly relates to vertical domain embedded data synthesis technology in the field of large model and agent application. This technology is the core foundation supporting scenarios such as vertical domain retrieval augmentation generation (RAG), and the quality of the embedded data generated directly determines the retrieval accuracy and business performance of the representation model.

[0018] With the acceleration of digital transformation in vertical sectors such as finance, healthcare, and industrial manufacturing, the industry has accumulated a massive amount of unstructured documents, such as research reports, medical records, and technical manuals. These documents contain extremely high business value and need to be transformed into machine-understandable semantic representations through data embedding to support the implementation of large-scale model applications such as subsequent retrieval and analysis.

[0019] Currently, embedded data synthesis in vertical domains faces two major pain points: First, massive amounts of unstructured text are difficult to efficiently transform into retrieval tasks, and the generated queries are of low quality. The industry's accumulated massive amounts of unstructured, proprietary knowledge lacks an efficient transformation path—constructing a query is essentially constructing a retrieval task, but existing technologies lack suitable toolchains, making it impossible to efficiently generate retrieval tasks based on massive amounts of data; simultaneously, the generated queries suffer from multiple quality issues: the generated queries are severely out of touch with real business scenarios, resulting in low quality and making the embedded data unable to match actual retrieval needs, leading to significant performance discrepancies after model deployment.

[0020] Secondly, contrastive learning suffers from low-quality sample mining and lacks efficient tools. Current techniques often fail to capture complete recall of positive samples due to overly fine chunk segmentation or their scattered distribution across different chapters, failing to cover cross-chunk and cross-chapter correlations. Negative samples, on the other hand, are mostly randomly selected, lacking effective mining of "difficult" negative samples. This results in poor training performance for contrastive learning, unstable model convergence, and difficulty in balancing recall and precision. Especially in scenarios where data is scarce in vertical domains, the lack of high-quality samples further exacerbates the difficulty of embedding data synthesis.

[0021] Therefore, the main technical problem addressed in this application focuses on the core pain points of embedded data synthesis in vertical domains, as follows:

[0022] 1. Efficient and realistic retrieval task generation: Existing methods cannot efficiently convert massive amounts of data into queries, and the generated queries lack vertical domain characteristics, fail to adapt to professional terminology, jargon, and disturbance scenarios in actual business, resulting in low-quality generated queries that cannot cover multi-granular and multi-expression retrieval needs, leading to low matching degree between embedded data and real business scenarios.

[0023] 2. Incomplete positive sample recall and false sample issues in contrastive learning: Traditional methods cannot effectively mine positive and negative samples in contrastive learning, and the chunk segmentation method causes the cross-chunk and cross-chapter association information to be fragmented, making it impossible to effectively mine all positive samples and filter false samples. Single-granularity positive sample mining cannot cover the complete semantics corresponding to the query, resulting in missing positive sample information and affecting the semantic representation integrity of the embedded data.

[0024] 3. Insufficient mining of difficult negative samples and excessive false negative samples: Existing sample mining methods are unable to accurately mine "difficult" negative samples and lack an effective false negative sample filtering mechanism. This problem is fatally destructive to contrastive learning training based on infoNCE loss.

[0025] In view of the problems existing in the prior art, this application proposes an embedded data synthesis method that integrates retrieval and large model distillation. By using RAG as a data synthesis tool, it can efficiently process unstructured private "deposited" knowledge in the industry, deeply combine the inherent knowledge and reasoning ability of large models, and significantly improve the quality of embedded data through multi-granular and diversified query generation and precise sample mining strategies, thus adapting to the real business needs of vertical fields.

[0026] The core of this application's technical solution is to integrate retrieval mechanisms with large-model distillation techniques to construct a method and system for embedding data synthesis adapted to vertical domains, addressing the core pain point of high-quality embedding data synthesis in vertical domains. The core steps of this application's technical solution include:

[0027] 1. Knowledge Base Construction: Preprocess a large number of unstructured documents in the vertical domain, build a knowledge base through incremental persistence, and process the documents into a multi-granular structure when they are added to the database, so as to support the subsequent multi-granularity query generation and sample mining.

[0028] 2. Multi-granular and diversified query generation: Using 3-5 sliding windows to traverse chunks in the knowledge base as the context of the large model, important knowledge is selected from these chunks to generate retrieval tasks through the reasoning capabilities of the large model. Low-quality queries are avoided by comparing them with each other. Various noise templates (including spelling perturbation, synonym substitution of professional terms, multi-hop reasoning, etc.) are used to inject perturbations. Multiple large models such as moonshot, GPT-4o, Qwen, and DeepSeek are randomly called to generate queries. At the same time, the chunks on which the generated queries are based are recorded as positive examples to ensure the authenticity, diversity and domain adaptability of the queries.

[0029] 3. Multi-granularity retrieval and rearrangement: The generated query is input into the retrieval system, which dynamically returns relevant chunks (including fragments of different lengths) of different granularities and chapters, realizing the initial acquisition of multi-length cross-chapter samples; the retrieval results are rearranged by randomly switching between multiple rearrangement models, and the relevance judgment ability of different models is used to improve the diversity and accuracy of the results.

[0030] 4. Sample mining integrating large model distillation: Select the 30 chunks after the rearranged Top-k as context, and mine positive samples from them to enhance recall; select the chunks within the Top-k as context, identify and mine "difficult" negative samples through large model inference, and filter false positive samples, thus distilling the knowledge of the large model into the sample mining process.

[0031] 5. Long-Context Positive Example Merging and Embedded Data Generation: Utilizing the long-context reasoning capabilities of large models, relevant positive samples across chunks are dynamically concatenated into complete samples, covering complete semantics; enhancing sample integrity and diversity.

[0032] The technical solution of this application will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0033] Figure 1 This is a flowchart illustrating the embedded data synthesis method for fusion retrieval and large model distillation provided in this application. For example... Figure 1 As shown, this method for fusion retrieval and large model distillation of embedded data may specifically include:

[0034] S101 preprocesses unstructured documents in the vertical domain, divides the unstructured documents into multi-granularity text blocks with hierarchical relationships, and builds a knowledge base based on the multi-granularity text blocks and their relationships.

[0035] S102, based on the combination of multi-granularity text blocks in the knowledge base to form a context, inject the perturbation information corresponding to the prior knowledge of the vertical domain into the context, call the generative model to generate a retrieval query based on the context, and determine the target text block corresponding to the retrieval query as the initial positive sample;

[0036] S103, input the retrieval query into the semantic retrieval system, retrieve the candidate text block set from the knowledge base, and sort the candidate text block set according to relevance;

[0037] S104. Based on the ranking results and inference model, positive sample text blocks for supplementation and difficult negative sample text blocks that are semantically similar to the retrieval query but do not belong to positive samples are mined from the candidate text block set. False negative sample text blocks are filtered according to the correlation between text blocks to form a corresponding positive sample set and negative sample set for each retrieval query.

[0038] S105: Contrastive learning training samples are constructed based on positive and negative sample sets. The semantic representation model is trained using the contrastive learning training samples to generate embedding vectors for retrieval tasks.

[0039] In some embodiments, unstructured documents are divided into multi-granularity text blocks with hierarchical relationships, and a knowledge base is constructed based on the multi-granularity text blocks and their relationships, including:

[0040] Unstructured documents are cleaned of formatting and their content extracted to obtain continuous text to be processed;

[0041] Based on the table of contents title and natural paragraph boundaries, continuous text is divided into parent text blocks as the first granularity.

[0042] Based on preset length and structure rules, the parent text block is divided to obtain at least one child text block of different granularities, and a subordinate relationship is established between the parent text block and the corresponding child text block.

[0043] Based on the subordinate relationships and the order information of each text block in the original document, generate association information that represents the hierarchy and order of multi-granular text blocks;

[0044] Multi-granular text blocks and related information are stored in a knowledge base.

[0045] Specifically, the system first receives raw document files from fields such as finance, healthcare, industrial manufacturing, or construction, including financial research reports, medical records, equipment technical manuals, and construction organization designs. Files from different sources are then uniformly input into the preprocessing module. The preprocessing module cleans up formatting issues in the raw files, such as layout tags, headers and footers, watermarks, and redundant blank lines. Combining optical character recognition (OCR) results with encoding conversion, it extracts text within images and various formatted texts into a single continuous text segment. Simultaneously, it retains table of contents information, heading level markers, and page number information, providing foundational data for subsequent structure-based segmentation.

[0046] After extracting continuous text, the segmentation module performs a first-level structural division based on the document's existing table of contents and paragraph boundaries. For example, it parses out first-level headings, second-level headings, or chapter titles, dividing the continuous text segments corresponding to each chapter into parent text blocks. Each parent text block typically corresponds to a chapter in a research report, a functional chapter in a technical manual, or a sub-item description in a construction document, ensuring that each parent text block has a relatively complete semantic scope. Building on this, the segmentation module further performs fine-grained segmentation of the parent text blocks according to preset length and structure rules. For example, based on the number of characters, sentences, or paragraphs, it divides the parent text block into one or more child text blocks, ensuring that the length of each child text block falls within a preset range and avoiding segmentation within sentences, formulas, or technical terms as much as possible.

[0047] To facilitate multi-granularity scheduling and cross-block merging during subsequent sample mining, the association construction module synchronously records the hierarchical relationships between parent and child text blocks, as well as the sequential relationships between child text blocks, during the block segmentation process. For example, a unique parent block identifier is assigned to each parent text block, and a unique child block identifier is assigned to each child text block generated within it. Internally, a one-to-many mapping relationship between parent block identifiers and corresponding child block identifiers is maintained. Simultaneously, based on the start and end positions of each text block in the original document, the index continuity relationship between adjacent child text blocks is recorded, and sequence information such as chapter number, page number, and heading level is marked to indicate the arrangement order of child text blocks within the same parent text block and the chronological order of different parent text blocks within the entire text.

[0048] After generating the parent-child hierarchical and sequential relationships, the association construction module organizes the aforementioned multi-granularity text blocks and related information into structured data entries. Each entry includes at least the following fields: document identifier, parent block identifier, child block identifier, block type marker, text content, start and end positions in the original document, parent-child dependency marker, and adjacent block sequence marker. Metadata such as source business type, document date, and language category can also be added if necessary. The knowledge base storage module writes these structured data entries into the knowledge base. The knowledge base can adopt a storage structure that supports a hybrid management of support vector indexes and relational indexes. This supports semantic retrieval of text blocks at various granularities and allows for rapid recovery of chapter structure and cross-block context within the knowledge base based on parent-child relationships and index continuity.

[0049] In some specific examples, taking financial research reports as an example, the system receives a research report document containing three chapters: macroeconomic analysis, industry analysis, and company analysis. The preprocessing module first removes the cover, disclaimer page, headers, footers, and unnecessary separators, and then parses the main text into the three chapters according to the table of contents. The segmentation module treats each chapter as a parent text block, and within each parent text block, it divides it into several child text blocks, with a maximum of approximately one thousand characters per block, using sentence boundaries as the dividing points.

[0050] For example, a macroeconomic analysis chapter is divided into a parent text block P1, which generates child text blocks C11, C12, C13, etc. Similarly, equipment manuals for industrial manufacturing companies can be divided into parent text blocks based on equipment functional modules, and further refined into child text blocks based on operating procedures or fault types. The association construction module records the hierarchical relationships between P1 and C11, C12, and C13, as well as the index continuity relationships between C11 and C12, and C12 and C13, and stores these blocks and their relationship information in a knowledge base. This way, when generating subsequent retrieval queries and mining samples, individual child text blocks can be directly used as fine-grained semantic units, or the entire chapter context can be restored through parent-child relationships and index continuity relationships.

[0051] Through the above embodiments, the original unstructured vertical domain documents are transformed into multi-granularity text blocks with parent-child hierarchical and sequential relationships, and uniformly stored in the knowledge base. This allows for flexible switching and combination between different granularities and levels when generating retrieval queries, performing semantic retrieval, and mining positive and negative samples, maintaining the integrity and coherence of the text's semantic structure, and providing a structured foundation for subsequent embedded data synthesis.

[0052] In some embodiments, a context is formed based on the combination of multi-granularity text blocks in the knowledge base, and perturbation information corresponding to prior knowledge of the vertical domain is injected into the context, including:

[0053] Select multiple multi-granular text blocks with related relationships from the knowledge base according to preset window rules, and combine the multiple multi-granular text blocks into a context for generating retrieval queries;

[0054] Based on prior knowledge of the target vertical domain, the perturbation type corresponding to the prior knowledge is determined, and perturbation information is generated according to the perturbation type.

[0055] According to the preset strategy, perturbation information is injected into the context to obtain a perturbation context carrying the features of prior knowledge in the vertical domain.

[0056] Specifically, the context selection submodule first selects combinations of related text blocks as context based on multi-granularity text blocks in the knowledge base and their hierarchical and sequential relationships, according to preset window rules. The window rules can be set as sliding windows with child text blocks as the basic unit, for example, the window length is 3 to 5 child text blocks, and the window is allowed to cross adjacent child text blocks within the same parent text block, or cross the boundaries of adjacent parent text blocks when needed, so as to cover the continuous semantic range across paragraphs and sections.

[0057] In actual implementation, the context selection submodule can start from the first child text block of a parent text block, and sequentially select C1, C2, and C3 to form the first window. Then slide one child block backward to form C2, C3, and C4 to form the second window, and so on. If a chapter boundary is encountered and there is a sequential relationship between adjacent chapters, the last child block of the previous chapter and the first child block of the next chapter can be included in the window together to cover semantic segments that are cross-chapter but continuous in business.

[0058] After obtaining the basic context used to generate the retrieval query, the perturbation injection submodule determines the perturbation type based on prior knowledge of the target vertical domain. Prior knowledge includes common abbreviations and industry jargon in the financial field, commonly used abbreviations and colloquial expressions in the medical field, common names and operating codes for equipment in the industrial manufacturing field, and internal conventions for different trades in the construction field. The system pre-abstracts and organizes this prior knowledge into various perturbation types, such as spelling perturbation, synonym substitution of professional terms, multi-hop inference hints, and jargon conversion. Upon receiving a context and its corresponding domain tag, the perturbation injection submodule selects one or more suitable perturbation types based on the prior knowledge base of that domain and generates perturbation information according to the corresponding perturbation type.

[0059] For example, in the scenario of financial research reports, when the target vertical domain is the financial domain, the perturbation injection sub-module can select two types of perturbation types, namely "synonymous replacement of professional terms" and "minor spelling mistakes", from the financial prior knowledge base. For the type of "synonymous replacement of professional terms", the system extracts the corresponding relationships from the pre-constructed term mapping table. For example, "price-earnings ratio" is replaced with "PE ratio", and "return on equity" is replaced with "ROE", while keeping other key information in the context unchanged. For the type of "minor spelling mistakes", the system introduces a small number of perturbations such as letter order inversion and similar character replacement in some common words or indicator names according to the preset error templates. For example, "macroeconomics" is misspelled as "macro economy" to simulate the typing errors or non-standard expressions that may occur when real users input queries.

[0060] For the industrial manufacturing domain, according to the common jargon in equipment maintenance manuals, "concrete" can be replaced with the construction site idiom "铨", or "high-voltage switchgear" can be replaced with "high-voltage cabinet", and hints such as "the name commonly used by on-site masters" can be added to the context to make the subsequent generated retrieval queries closer to the real expressions of front-line engineers.

[0061] After generating the specific perturbation information, the perturbation injection sub-module injects the perturbation information into the context according to the preset strategy, and obtains the perturbed context carrying the prior knowledge features of the vertical domain. The preset strategy can include a perturbation position selection strategy and a perturbation intensity control strategy. The perturbation position selection strategy is used to determine which terms, sentences or paragraphs in the context are to be replaced or inserted. For example, perturbations are preferentially implemented at positions such as key business domain terms, indicator names, equipment names, and process steps, to avoid destroying the overall semantic coherence of the context.

[0062] The perturbation intensity control strategy is used to control the number and type combination of perturbations applied in a single context. For example, only 1 type of perturbation is injected in most windows, and 2 types of perturbations are injected in some windows, to introduce sufficient diversity while ensuring readability. In the implementation process, the perturbation injection sub-module can randomly select one or two text segments to be perturbed for each context, and replace or insert the aforementioned generated perturbation information at the corresponding positions to form a new perturbed context, while retaining the block index information and domain labels of the original context, which is convenient for tracing the corresponding original text block after generating the retrieval query.

[0063] For example, taking the field of construction as an example, when a construction organization design document of a certain engineering project is stored in the knowledge base, a parent text block describes "the casting process of the foundation slab concrete". After segmentation, multiple child text blocks are obtained, including content such as concrete mix ratio requirements, vibration mode, and maintenance time. The sub-module for context selection can select three consecutive child text blocks according to the sliding window rule to form the context, covering the complete process of concrete casting from batching to maintenance. The sub-module for perturbation injection can, based on the prior knowledge in the field of construction, replace "concrete" with the term "quan" commonly used at the construction site, rewrite "maintenance" as "moist maintenance" or "humidification", and introduce a tone expression such as "the on-site team leader often says so" in the sentence, making the overall expression in the perturbed context closer to the actual language usage habits of construction site technicians.

[0064] Through the above embodiments, before generating the retrieval query, the system constructs a context that can reflect cross-segment and cross-chapter semantics by using multi-granularity text blocks in the knowledge base, and injects multiple types of perturbation information based on the prior knowledge in the vertical domain, so that the context itself carries features such as domain terms, jargon, spelling mistakes, and multi-hop reasoning. Therefore, when subsequently calling the generation model to generate the retrieval query, retrieval expression forms with diverse styles and close to the real business scenario can be naturally generated. While ensuring the semantic integrity of the context, this processing method introduces the ability to model domain knowledge and real input noise, which is beneficial for the synthesized retrieval query to better cover various retrieval habits and expression methods in the vertical domain, and provides a richer and more representative input for the subsequent embedded data synthesis and contrast learning training.

[0065] In some embodiments, calling the generation model to generate a retrieval query according to the context, and determining the target text block corresponding to the retrieval query as the initial positive sample includes:

[0066] Construct a set of generation models including multiple generation models, and select a target generation model from the set of generation models according to a preset selection rule;

[0067] Input the context processed by perturbation into the target generation model, generate at least one retrieval query corresponding to the context, and obtain the association information characterizing the corresponding relationship between the retrieval query and each multi-granularity text block in the context;

[0068] According to the association information and the preset association determination rule, determine the target text block corresponding to the retrieval query from the multi-granularity text blocks included in the context, and use the target text block as the initial positive sample of the retrieval query.

[0069] Specifically, the generative model scheduling submodule first constructs a generative model set based on system deployment requirements. This set can include multiple large language models with different training corpora and stylistic features, such as a first-class model optimized for general scenarios, a second-class model enhanced on Chinese financial corpora, and a third-class model enhanced on industrial technical documents. For each model, metadata such as its model identifier, maximum supported context length, adapted vertical domain tags, and resource consumption level are registered in the model set.

[0070] The system pre-defines model selection rules, such as polling, randomization, or selection based on vertical domain priority. When a perturbation context and its domain label (e.g., finance, healthcare, industrial manufacturing, or construction) are received, the model scheduling submodule filters a subset of suitable models from the model set based on the domain label, and then selects a target model from it according to the pre-defined selection rules. For example, when the domain label is finance, models that have been fine-tuned in financial research reports are given priority; when the current model is under high load, a backup model in the same domain is switched to, thus ensuring domain adaptability while also considering system throughput.

[0071] After determining the target generation model, the generation model scheduling submodule constructs a standardized input sequence from the perturbated context and inputs it into the target generation model. The standardized input sequence includes not only the main text content resulting from the combination of the aforementioned multi-granularity text blocks, but also prompts to constrain the output format, guiding the generation model to annotate the text blocks upon which the search query is based while outputting the search query. For example, the prompts could instruct the generation model to "generate one search question that a real user might ask based on the following context, and provide the main text block number on which the question is based in the output," where the text block number corresponds to the local index assigned to each multi-granularity text block by the system when constructing the context.

[0072] Furthermore, upon receiving the perturbation context and prompts, the target generation model, combining the technical terms, jargon, and noise patterns within the context, uses its language understanding and reasoning capabilities to generate at least one retrieval query highly relevant to the context's semantics. The output structure includes annotation information, such as providing the indexes of one or more text blocks that the retrieval query primarily relies on, in the form of a list or special markers. The generation model scheduling submodule then passes this output as the original generated result to the positive sample annotation submodule.

[0073] The positive sample annotation submodule parses the search query text and the corresponding set of text block indices from the original generated results. Combining the block index mapping relationship preserved during the context construction phase, the positive sample annotation submodule can quickly locate the corresponding multi-granularity text blocks in the current context.

[0074] For example, in a financial research report scenario, the perturbation context covers three sub-text blocks in the macroeconomics section concerning interest rates, inflation, and monetary policy. The target generation model outputs the retrieval query "What impact does the current tightening of monetary policy have on stock market valuations?", and labels the text blocks that this question is primarily based on as indices C21 and C22. The positive sample labeling submodule then finds the sub-text blocks indexed C21 and C22 within the context and marks them as the target text block set corresponding to this retrieval query.

[0075] In some cases, such as when the generative model only labels one text block index or labels more than three text block indexes, the positive sample labeling submodule can filter and merge the target text blocks according to the preset association judgment rules. For example, it can prioritize keeping two text blocks that are highly related to the search query topic and have consecutive indexes, or merge text blocks with too many indexes and then register them as a single positive sample unit.

[0076] For example, the process described in this embodiment is also applicable in industrial manufacturing or construction. Taking an equipment troubleshooting manual as an example, when the disturbance context covers multiple sub-text blocks of "inverter overload fault" and, after being perturbed by jargon, includes expressions such as "inverter keeps tripping" or "motor keeps stopping," the target generation model will generate a search query that closely resembles the style of front-line engineers, such as "how to troubleshoot the cause of frequent inverter tripping," and will annotate the sub-text block indexes that are the main basis for this problem in the output. The positive sample annotation submodule parses these indexes and registers the corresponding text blocks as the initial positive samples for this problem. Subsequently, in the sample mining stage, these initial positive samples can be used to expand to other related text blocks in the knowledge base.

[0077] In some embodiments, a retrieval query is input into a semantic retrieval system, a set of candidate text blocks is retrieved from a knowledge base, and the set of candidate text blocks is sorted according to relevance, including:

[0078] The retrieval query is converted into a first semantic representation, and the second semantic representation of each multi-granularity text block is obtained from the knowledge base;

[0079] Based on the similarity between the first semantic representation and the second semantic representation, multi-granular text blocks related to the retrieval query are selected from the knowledge base as a candidate text block set;

[0080] Using at least one relevance determination model, the relevance score of each candidate text block is calculated based on the retrieval query and the semantic representation of the candidate text block. When multiple relevance determination models exist, the relevance scores are aggregated. The candidate text block set is sorted according to the aggregated relevance scores to obtain the sorted candidate text block sequence.

[0081] Specifically, during the system initialization phase, the semantic encoding submodule pre-generates a second semantic representation for each multi-granularity text block in the knowledge base. The second semantic representation can be obtained through a trained semantic representation model, which can adopt an encoder structure to encode the input parent and child text blocks separately, converting them into fixed-dimensional vector representations, and storing these vectors along with the corresponding text block identifiers and hierarchical information in a vector index structure.

[0082] For newly added documents, the system performs the same encoding process on the newly generated multi-granularity text blocks while simultaneously cleaning up the format, dividing the hierarchy, and building relationships. The second semantic representation is incrementally written into the vector index of the knowledge base. When processing online retrieval queries, the semantic encoding submodule receives the retrieval query text output by the generation model and calls the same or compatible semantic representation model to encode the retrieval query into a first semantic representation, thereby ensuring that the retrieval query and the multi-granularity text blocks are comparable within the same semantic space.

[0083] Furthermore, after obtaining the first semantic representation of the retrieval query and the second semantic representation pre-stored in the knowledge base, the candidate recall submodule calculates the similarity between the first semantic representation and each of the second semantic representations based on a preset similarity measurement rule. The similarity measurement can employ vector dot product, normalized cosine similarity, or other functions that can measure semantic closeness. The candidate recall submodule can combine with a vector retrieval engine to quickly select several multi-granularity text blocks with high similarity scores from the knowledge base as a set of candidate text blocks without traversing all text blocks.

[0084] For example, when the search query is "What impact does the current tightening of monetary policy have on stock market valuations?", the candidate recall submodule will recall multiple parent and child text blocks related to "monetary policy", "interest rate adjustment" and "valuation level" from the financial research report knowledge base. These include paragraphs about interest rate policy in the macroeconomic analysis chapter and discussions on valuation changes in the market strategy chapter, thus forming a preliminary set of candidate text blocks.

[0085] To further improve the relevance accuracy between the candidate text block set and the retrieval query, and to take into account the relevance judgment capabilities of different models, the relevance ranking submodule uses at least one relevance judgment model to rank the aforementioned candidate text block set. In one implementation, the relevance judgment model may include a basic relevance model based on vector similarity and a re-ranking model based on deep matching structure. The basic relevance model directly uses the similarity score obtained in the candidate recall stage as the initial relevance score, while the re-ranking model jointly models the original text of the retrieval query and candidate text blocks, outputting a more granular relevance score. When multiple relevance judgment models exist, the relevance ranking submodule normalizes and weights the relevance scores output by each model according to a preset aggregation strategy to obtain an aggregated relevance score for each candidate text block, and then ranks the candidate text block set according to the aggregated relevance score, outputting the ranked candidate text block sequence.

[0086] For example, taking the industrial manufacturing field as an example, when the generated search query in the current embodiment is "How to troubleshoot the frequent tripping of the frequency converter", the semantic encoding submodule encodes the search query into a first semantic representation and compares it with the second semantic representation of the multi-granularity text blocks in the equipment technical manual and fault diagnosis guide. The candidate recall submodule can recall multiple child-level text blocks including "frequency converter overload protection", "motor current abnormality", "line grounding fault check" and their corresponding parent-level text blocks, forming a candidate text block set.

[0087] Subsequently, the relevance ranking submodule calls a customized relevance determination model for industrial scenarios to re-evaluate and rank the semantic matching degree between these candidate text blocks and the search query. This prioritizes text blocks describing "the meaning of overload fault alarm codes" and "on-site current measurement methods," while relegating text blocks that only involve general equipment maintenance to the back, thus providing high-quality ranking input for subsequent positive sample supplementation and difficult negative sample mining.

[0088] Through the above embodiments, after inputting the retrieval query into the semantic retrieval system, the system uses a unified semantic encoding method to represent the retrieval query and multi-granularity text blocks. By combining vector similarity recall and multi-model relevance ranking, it efficiently obtains a sequence of candidate text blocks covering parent and child granularities and spanning multiple chapters from the knowledge base. This ranked candidate text block sequence ensures semantic relevance to the retrieval query while preserving a multi-granularity, multi-source candidate space. This lays a reliable foundation for subsequent positive sample supplementation, hard negative sample identification, and false negative sample filtering based on the ranking results, thereby improving the accuracy and richness of candidate sample acquisition during the embedded data synthesis process.

[0089] In some embodiments, based on the ranking results and the inference model, positive sample text blocks for supplementation and difficult negative sample text blocks that are semantically similar to the retrieval query but do not belong to positive samples are mined from the candidate text block set, including:

[0090] Based on the sorted candidate text block sequence, multiple candidate text blocks are selected according to the first preset ranking interval to form the first context. The first context and the retrieval query are input into the inference model. According to the output of the inference model and the preset positive sample determination rules, positive sample text blocks for supplementation are determined from the candidate text blocks corresponding to the first context.

[0091] Based on the sorted candidate text block sequence, multiple candidate text blocks are selected according to the second preset ranking interval to form the second context. The second context and the retrieval query are input into the inference model. According to the output of the inference model and the preset negative sample determination rules, difficult negative sample text blocks that are semantically similar to the retrieval query but do not belong to positive samples are determined from the candidate text blocks corresponding to the second context.

[0092] Specifically, the positive sample mining unit first selects multiple candidate text blocks from the sorted candidate text block sequence according to a first preset ranking interval to form a first context. The first preset ranking interval can be set to a ranking range higher than the actual recall threshold according to business needs. For example, when downstream online retrieval usually uses Top 20 recall, the first preset ranking interval can be set to the candidate text blocks ranked 21st to 50th, which is used to focus on mining potential positive samples that have been missed by the basic retrieval model but still have high relevance. The system inputs this first context along with the current retrieval query into the long context inference model. The inference model combines the semantic intent of the retrieval query and the content of each candidate text block in the first context to make a comprehensive judgment on the semantic association between each candidate text block and the retrieval query.

[0093] To facilitate automated annotation, the system pre-defines positive sample determination rules. These rules are determined based on a combination of information, including the relevance labels output by the inference model, confidence scores, and the position of candidate text blocks within the first context. For example, the inference model can output category labels such as "highly relevant," "generally relevant," and "irrelevant" along with corresponding confidence scores for each candidate text block in the first context. The positive sample determination rules can mark candidate text blocks that meet the criteria of "highly relevant and with a confidence score higher than a preset threshold" as supplementary positive sample text blocks. Furthermore, when the retrieval query pertains to a multi-hop inference problem, multiple candidate text blocks labeled "highly relevant" can be merged into a single cross-block positive sample.

[0094] For example, in a financial research report scenario, if the search query is "What impact does the current tightening of monetary policy have on stock market valuations?", the 25th and 30th ranked text blocks in the sorted candidate text block sequence describe the impact of interest rate increases on the discount rate and the analysis of the downward shift in the valuation center, respectively. After comprehensively considering the search query and these two pieces of content, the inference model will mark them as "highly relevant". Based on this, the positive sample determination rule will add these two text blocks as supplementary positive sample text blocks to the positive sample set corresponding to the current search query, thereby making up for the insufficient coverage of the initial positive samples in cross-chapter analysis.

[0095] After completing the supplementary positive sample mining, the hard negative sample mining unit selects multiple candidate text blocks to form the second context based on the ranked candidate text block sequence according to a second preset ranking interval. The second preset ranking interval generally selects candidate text blocks that are ranked high and have a high initial similarity to the retrieval query, such as the candidate text blocks ranked 1st to 20th. This is used to focus on mining hard negative samples that are semantically highly similar to the retrieval query but should not be considered as answers in terms of business logic. The system inputs the second context and the retrieval query into the inference model. The inference model analyzes the semantic content of each candidate text block in the second context at a higher resolution semantic level to determine whether it truly answers the question involved in the retrieval query, or whether it only shares some terms and topics but has deviations in key information.

[0096] In some examples, to distinguish between genuine positive samples and difficult negative samples, the system pre-defines negative sample determination rules. These rules can be based on a comprehensive assessment of the "answer matching degree," "key information consistency," and comparison with the content of the initial and supplementary positive samples already identified by the inference model. For instance, when the inference model determines that a candidate text block is highly similar to the retrieval query in terms of surface terminology, but exhibits significant mismatches in key constraints such as time range, applicable objects, and indicator direction, the negative sample determination rules will mark this candidate text block as a difficult negative sample that is "semantically similar but business-mismatched."

[0097] For example, in the industrial manufacturing field, if the search query is "how to troubleshoot the frequent tripping of frequency converters", the top-ranked candidate text blocks contain both "troubleshooting steps for frequency converter overload protection" and "daily maintenance and cleaning of frequency converters". The inference model will determine that the former is highly relevant to troubleshooting the cause of the fault and should be regarded as a positive sample or potential positive sample. The latter, although containing the common term "frequency converter", mainly describes the maintenance process and does not match the question of "troubleshooting the cause of tripping". The negative sample judgment rule can mark the text block as a typical difficult negative sample.

[0098] In some implementations, to avoid mixing simple negative samples with difficult negative samples, the difficult negative sample mining unit can further combine the ranking position and the semantic similarity score output by the inference model to filter candidate text blocks in the second context. For example, it can select difficult negative sample text blocks only from candidate text blocks that the inference model determines are "partially related but do not meet the answer conditions" and whose basic similarity scores are higher than a certain threshold. This ensures that difficult negative samples are both semantically similar and mutually exclusive at the answer level, providing high-value negative samples for subsequent comparative learning.

[0099] Through the above embodiments, the system utilizes the sorted candidate text block sequence and the long context reasoning capability of the reasoning model to automatically mine supplementary positive sample text blocks missed by the basic retrieval process within the first preset ranking interval, and automatically identify difficult negative sample text blocks that are semantically similar to the retrieval query but do not belong to positive samples in terms of business within the second preset ranking interval. Thus, while maintaining the integrity of the semantic structure of the knowledge base, it significantly improves the recall integrity of positive samples and the distinguishing value of negative samples, providing a more accurate, rich, and structurally reasonable sample foundation for subsequent false negative sample filtering and comparative learning training. This is conducive to improving the quality of embedded data synthesis and the discrimination capability of the representation model in actual retrieval tasks.

[0100] In some embodiments, false negative sample text blocks are filtered based on the correlation between text blocks to form corresponding positive sample sets and negative sample sets for each retrieval query, including:

[0101] Based on the pre-established hierarchical and sequential association relationships of multi-granularity text blocks in the knowledge base, the association relationships between each difficult negative sample text block, the initial positive sample text block, and the positive sample text blocks used for supplementation are determined.

[0102] Difficult negative sample text blocks that have a parent-child inclusion predetermined relationship with any initial positive sample text block or a supplementary positive sample text block are identified as false negative sample text blocks, and false negative sample text blocks are removed from the difficult negative sample text blocks or merged into the positive sample text blocks. The predetermined relationship includes parent-child inclusion relationship, index continuation relationship and / or cross-block combination relationship.

[0103] Based on the elimination of false negative sample text blocks, the initial positive sample text blocks and the positive sample text blocks used for supplementation are grouped together to form the positive sample set corresponding to the retrieval query, and the remaining difficult negative sample text blocks and other optional negative sample text blocks are grouped together to form the negative sample set corresponding to the retrieval query.

[0104] Specifically, the false negative sample identification subunit first reads the hierarchical and sequential relationships of multi-granular text blocks related to the current retrieval query from the knowledge base. The hierarchical relationship is used to represent the subordinate relationship between parent text blocks and child text blocks, while the sequential relationship is used to represent the index continuity relationship of child text blocks within the same parent text block and the chapter order relationship across parent text blocks.

[0105] In the aforementioned process, the initial positive sample text blocks originate from context text blocks explicitly labeled by the generation model. The supplementary positive sample text blocks originate from candidate text blocks deemed highly relevant by the inference model within a first preset ranking interval. The difficult negative sample text blocks originate from candidate text blocks within a second preset ranking interval that are semantically similar but do not match at the answer level, as determined by the inference model. The false negative sample identification subunit, centered on each difficult negative sample text block, queries its parent text block identifier, child text block identifier, and index position in the original document, comparing these with the parent-child identifiers and index positions corresponding to the initial positive sample text blocks and the supplementary positive sample text blocks.

[0106] During the comparison process, the system determines the association between difficult negative sample text blocks and positive sample text blocks based on pre-defined relationships. These pre-defined relationships include at least three types: parent-child containment relationships, index continuation relationships, and cross-block combination relationships. Parent-child containment relationships are used to identify whether difficult negative sample text blocks and positive sample text blocks belong to the same parent text block, or whether one is a parent text block and the other is its child text block. For example, a difficult negative sample text block may share the same parent text block identifier with an initial positive sample text block or be located within that parent text block.

[0107] Index continuation is used to identify whether a difficult negative sample text block is adjacent to a positive sample text block in the original document. For example, the index of a difficult negative sample text block may be located before or after the index of a positive sample text block. Cross-block combination is used to identify situations where multiple text blocks need to be semantically combined to fully answer a retrieval query. If a difficult negative sample text block is adjacent to one or more positive sample text blocks at the parent level and has continuous indexes, and has been determined to have partial relevance to the retrieval query in the aforementioned inference phase, it can be considered as part of a cross-block combination relationship.

[0108] When the false negative sample identification subunit determines that a certain difficult negative sample text block satisfies any of the above-mentioned predetermined relationships with any initial positive sample text block or a positive sample text block used for supplementation, the system identifies the difficult negative sample text block as a false negative sample text block and, according to business needs, chooses to remove it from the set of difficult negative sample text blocks or directly merge it into the set of positive sample text blocks.

[0109] For example, in a financial research report scenario, if the search query is "What impact does the current tightening of monetary policy have on stock market valuations?", a difficult negative sample text block is located in the section before the "interest rate adjustment" paragraph in the macroeconomics chapter. The content is "Background analysis of the recent increase in the benchmark interest rate by the central bank". This text block is within the Top 10 in the sorting stage, but it is marked as a difficult negative sample in the inference model judgment because it does not directly involve the keyword "valuation".

[0110] By comparing hierarchical and sequential relationships, the system found that the text block shared the same parent text block identifier with a positive sample text block used for supplementation, and the indices were adjacent. Furthermore, the parent text block as a whole corresponded to the chapter content of "Monetary Policy and Valuation Relationship". Based on this, the false negative sample identification subunit determined the difficult negative sample text block as a false negative sample and merged it into the positive sample text block set to supplement the complete semantic coverage of the retrieval query.

[0111] After identifying and processing false negative sample text blocks, the sample aggregation subunit, after removing false negative sample text blocks, aggregates the initial positive sample text blocks and the positive sample text blocks used for supplementation to form the positive sample set corresponding to the current search query. Simultaneously, it aggregates difficult negative sample text blocks that were not identified as false negative samples, as well as a subset of randomly selected negative sample text blocks, to form the negative sample set corresponding to the current search query. The positive sample set can retain parent-child hierarchy information and index information of text blocks to support subsequent long-context positive sample concatenation or cross-block summarization generation. The negative sample set, while retaining semantically similar tags, removes text blocks that semantically belong to the answer range, ensuring that the samples within the set are clearly distinguishable from the search query in terms of business semantics.

[0112] Through the above embodiments, the system utilizes the pre-established multi-granularity text block hierarchical and sequential association relationships in the knowledge base to perform secondary filtering on text blocks in the candidate samples that have a close structural relationship with positive samples but were misclassified as difficult negative samples in the previous stage. These text blocks are identified as false negative sample text blocks and either removed or merged into the positive sample set. This effectively reduces the proportion of false negative samples in the contrastive learning training, avoiding abnormal amplification of the loss function denominator and incorrect gradient direction. The resulting positive and negative sample sets have more accurate labels and clearer boundaries, which helps improve the quality of sample annotation, stabilizes the training process of the contrastive learning model, and further enhances the discriminative ability of the embedding vectors in actual retrieval tasks.

[0113] In some embodiments, a semantic representation model is trained using contrastive learning training samples to generate embedding vectors for a retrieval task, including:

[0114] The comparative learning training samples are organized into sample groups according to training batches, consisting of a retrieval query, the corresponding positive sample text block, and the negative sample text block.

[0115] Each retrieval query and its corresponding positive and negative sample text blocks are input into the semantic representation model to obtain a first vector representation of the semantic features of the retrieval query and a second vector representation of the semantic features of each text block.

[0116] Based on the similarity relationship between the first vector representation and the second vector representation of the corresponding positive sample text block, as well as the second vector representation of each negative sample text block, the parameters of the semantic representation model are updated according to the preset contrastive learning loss function, so that the semantic representation model can encode the text blocks in the retrieval query and knowledge base after convergence, and output the embedding vector for the retrieval task.

[0117] Specifically, the sample organization submodule first converts the positive and negative sample sets obtained for each retrieval query into contrastive learning training samples. For each retrieval query, the sample organization submodule selects one or more positive sample text blocks from the corresponding positive sample set and several difficult negative sample text blocks and a small number of random negative sample text blocks from the negative sample set to form a contrastive learning sample record, ensuring that the sample record simultaneously contains the three elements of "query, positive sample, and negative sample". During batch construction, the sample organization submodule shuffles the order of multiple sample records from different domains and documents and organizes them into training batches according to a preset batch size. For example, each training batch contains several financial research report retrieval samples, several industrial manufacturing manual retrieval samples, and several construction document retrieval samples, thereby ensuring that the model can simultaneously encounter sample distributions from multiple vertical domains during a single parameter update, avoiding excessive bias towards a single domain.

[0118] After obtaining the training batch, the encoding submodule inputs each retrieval query and its corresponding positive and negative sample text blocks into the same semantic representation model for encoding. The semantic representation model can employ a dual-tower structure or other structures suitable for text semantic encoding, with one side encoding the retrieval query and the other side encoding the text block. During a forward computation, the encoding submodule generates a first vector representation for each retrieval query and second vector representations for its positive and negative sample text blocks, ensuring that all query vectors and text vectors within the same batch reside in the same vector space. For multi-granularity text blocks, the encoding submodule can directly encode individual child text blocks, or it can concatenate consecutively indexed positive sample blocks in the previous stage, using the concatenated long text positive sample as the encoding input, enabling the semantic representation model to learn the complete semantic expression across blocks.

[0119] After obtaining the first and second vector representations, the contrastive learning training submodule calculates the loss value for each training batch based on a preset contrastive learning loss function. For each retrieval query, the contrastive learning training submodule considers the similarity between the first vector representation of the retrieval query and the corresponding second vector representation of the positive sample text block as a target that should be "brought closer," and the similarity between it and the second vector representations of all negative sample text blocks as a target that should be "distanced."

[0120] The loss function considers both the interference from positive sample text blocks corresponding to the current query and the interference from other sample text blocks within the same batch. By reinforcing the constraint that "the similarity between the query and positive samples is significantly higher than that with negative samples," the weights of the semantic representation model are continuously adjusted during parameter backpropagation. Since the aforementioned sample construction stage has already ensured the effectiveness of negative samples in the negative sample set through hard negative sample mining and false negative sample filtering mechanisms, the contrastive learning training submodule in this embodiment can more effectively utilize these high-value negative samples to strengthen the model's discriminative ability.

[0121] During training, the contrastive learning training submodule iteratively updates the parameters of the semantic representation model according to the optimization algorithm. Within each training cycle, the system sequentially retrieves multiple training batches and executes the steps of "forward encoding → loss calculation → backpropagation → parameter update." To prevent overfitting on a few frequently occurring retrieval patterns, the sample organization submodule can dynamically adjust the positive and negative sample sampling ratio during training. For example, it can appropriately increase the proportion of difficult negative samples in the negative sample set, or switch the weights of samples from different domains at different training stages, enabling the semantic representation model to better meet the retrieval needs of various vertical domains such as finance, healthcare, industrial manufacturing, and construction in the later stages of training. After several training cycles, the system determines whether the semantic representation model has converged based on metrics such as recall and precision on the validation set. When the model performance reaches a preset standard, the model parameters are fixed and it is used as the embedding generation model for online retrieval services.

[0122] Through the above embodiments, the system utilizes structured retrieval queries, positive sample text blocks, and negative sample text blocks during the contrastive learning training phase to perform targeted parameter optimization on the semantic representation model. This enables the model to simultaneously narrow the vector distance between the query and positive samples and widen the vector distance between the query and negative samples within a unified semantic space. Building upon the prior completion of difficult negative sample mining and false negative sample filtering, this training method significantly reduces the risk of gradient direction interference by erroneous samples during contrastive learning, improves the stability of model training, and enhances the model's ability to distinguish semantically similar but business-mismatched segments. Ultimately, this yields embedding vectors more suited to vertical domain retrieval tasks, improving recall and ranking accuracy in practical retrieval scenarios.

[0123] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0124] Figure 2 This is a schematic diagram of the embedded data synthesis device for fusion retrieval and large model distillation provided in an embodiment of this application. Figure 2 As shown, the fusion retrieval and large model distillation embedded data synthesis device includes:

[0125] The preprocessing module 201 is used to preprocess unstructured documents in the vertical domain, divide the unstructured documents into multi-granularity text blocks with hierarchical relationships, and build a knowledge base based on the multi-granularity text blocks and their relationships.

[0126] The generation module 202 is used to form a context based on the combination of multi-granularity text blocks in the knowledge base, inject the perturbation information corresponding to the prior knowledge of the vertical domain into the context, call the generation model to generate a retrieval query based on the context, and determine the target text block corresponding to the retrieval query as the initial positive sample.

[0127] The retrieval module 203 is used to input the retrieval query into the semantic retrieval system, retrieve a set of candidate text blocks from the knowledge base, and sort the set of candidate text blocks according to relevance.

[0128] The mining module 204 is used to mine positive sample text blocks for supplementation and difficult negative sample text blocks that are semantically similar to the retrieval query but do not belong to positive samples from the candidate text block set based on the ranking results and inference model, and to filter false negative sample text blocks according to the correlation between text blocks, so as to form a corresponding positive sample set and negative sample set for each retrieval query.

[0129] Training module 205 is used to construct contrastive learning training samples based on positive and negative sample sets, and to train a semantic representation model using the contrastive learning training samples to generate embedding vectors for retrieval tasks.

[0130] In some embodiments, Figure 2 The preprocessing module 201 performs format cleaning and content extraction on the unstructured document to obtain continuous text to be processed; based on the table of contents and natural paragraph boundaries, the continuous text is divided into parent text blocks as the first granularity; according to preset length and structure rules, the parent text blocks are further divided to obtain at least one child text block of different granularities, and a subordinate relationship is established between the parent text block and the corresponding child text block; based on the subordinate relationship and the order information of each text block in the original document, association information representing the hierarchy and order of the multi-granularity text blocks is generated; and the multi-granularity text blocks and association information are stored in a knowledge base.

[0131] In some embodiments, Figure 2 The generation module 202 selects multiple multi-granular text blocks with related relationships from the knowledge base according to preset window rules, and combines the multiple multi-granular text blocks into a context for generating retrieval queries; based on the prior knowledge of the target vertical domain, it determines the perturbation type corresponding to the prior knowledge, and generates perturbation information according to the perturbation type; and injects the perturbation information into the context according to a preset strategy to obtain a perturbation context carrying the features of the vertical domain prior knowledge.

[0132] In some embodiments, Figure 2 The generation module 202 constructs a set of generation models including multiple generation models, and selects a target generation model from the set of generation models according to a preset selection rule; the perturbation-processed context is input into the target generation model to generate at least one retrieval query corresponding to the context, and obtains association information representing the correspondence between the retrieval query and each multi-granularity text block in the context; according to the association information and the preset association determination rule, the target text block corresponding to the retrieval query is determined from the multi-granularity text blocks contained in the context, and the target text block is used as the initial positive sample of the retrieval query.

[0133] In some embodiments, Figure 2 The mining module 204 selects multiple candidate text blocks according to a first preset ranking interval based on the sorted candidate text block sequence to form a first context. The first context and the retrieval query are input into the inference model. Based on the output of the inference model and the preset positive sample determination rules, positive sample text blocks for supplementation are determined from the candidate text blocks corresponding to the first context. Based on the sorted candidate text block sequence, multiple candidate text blocks are selected according to a second preset ranking interval to form a second context. The second context and the retrieval query are input into the inference model. Based on the output of the inference model and the preset negative sample determination rules, difficult negative sample text blocks that are semantically similar to the retrieval query but do not belong to positive samples are determined from the candidate text blocks corresponding to the second context.

[0134] In some embodiments, Figure 2The mining module 204 determines the association between each difficult negative sample text block and the initial positive sample text block and the positive sample text block used for supplementation, based on the multi-granularity text block hierarchical association relationship and sequential association relationship pre-established in the knowledge base. Difficult negative sample text blocks that have a parent-child inclusion predetermined relationship with any initial positive sample text block or the positive sample text block used for supplementation are identified as false negative sample text blocks, and false negative sample text blocks are removed from the difficult negative sample text blocks or merged into the positive sample text blocks. The predetermined relationship includes parent-child inclusion relationship, index continuation relationship and / or cross-block combination relationship. Based on the removal of false negative sample text blocks, the initial positive sample text blocks and the positive sample text blocks used for supplementation are collected to form the positive sample set corresponding to the retrieval query, and the remaining difficult negative sample text blocks and other optional negative sample text blocks are collected to form the negative sample set corresponding to the retrieval query.

[0135] In some embodiments, Figure 2 The training module 205 organizes the contrastive learning training samples into sample groups consisting of retrieval queries, corresponding positive sample text blocks, and negative sample text blocks according to training batches. Each retrieval query and its corresponding positive and negative sample text blocks are input into the semantic representation model to obtain a first vector representation of the semantic features of the retrieval query and a second vector representation of the semantic features of each text block. Based on the similarity relationship between the first vector representation and the second vector representation of the corresponding positive sample text block and the second vector representation of each negative sample text block, the parameters of the semantic representation model are updated according to the preset contrastive learning loss function, so that the semantic representation model can encode the retrieval query and the text blocks in the knowledge base after convergence, and output the embedding vector for the retrieval task.

[0136] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0137] Figure 3 This is a schematic diagram of the electronic device 3 provided in an embodiment of this application. Figure 3 As shown, the electronic device 3 of this embodiment includes: a processor 301, a memory 302, and a computer program 303 stored in the memory 302 and executable on the processor 301. When the processor 301 executes the computer program 303, it implements the steps in the various method embodiments described above. Alternatively, when the processor 301 executes the computer program 303, it implements the functions of each module / unit in the various device embodiments described above.

[0138] Electronic device 3 can be a desktop computer, laptop, handheld computer, cloud server, or other electronic device. Electronic device 3 may include, but is not limited to, processor 301 and memory 302. Those skilled in the art will understand that... Figure 3 This is merely an example of electronic device 3 and does not constitute a limitation on electronic device 3. It may include more or fewer components than shown, or different components.

[0139] The processor 301 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0140] The memory 302 can be an internal storage unit of the electronic device 3, such as a hard disk or memory of the electronic device 3. The memory 302 can also be an external storage device of the electronic device 3, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the electronic device 3. The memory 302 can also include both internal and external storage units of the electronic device 3. The memory 302 is used to store computer programs and other programs and data required by the electronic device.

[0141] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0142] If integrated modules / units are implemented as software functional units and sold or used as independent products, they can be stored in a readable storage medium (e.g., a computer-readable storage medium). Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program may include computer program code, which may be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable storage medium may include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0143] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for embedding data synthesis that integrates retrieval and large-model distillation, characterized in that, include: Unstructured documents in a vertical domain are preprocessed, and the unstructured documents are divided into multi-granularity text blocks with hierarchical relationships. A knowledge base is then constructed based on the multi-granularity text blocks and their relationships. A context is formed by combining multi-granularity text blocks in the knowledge base. Perturbation information corresponding to prior knowledge of the vertical domain is injected into the context. The generative model is called to generate a retrieval query based on the context, and the target text block corresponding to the retrieval query is determined as the initial positive sample. The retrieval query is input into the semantic retrieval system, a set of candidate text blocks is retrieved from the knowledge base, and the set of candidate text blocks is sorted according to relevance. Based on the ranking results and inference model, positive sample text blocks for supplementation and difficult negative sample text blocks that are semantically similar to the retrieval query but do not belong to positive samples are mined from the candidate text block set. False negative sample text blocks are filtered according to the correlation between text blocks to form a corresponding positive sample set and negative sample set for each retrieval query. Contrastive learning training samples are constructed based on the positive sample set and the negative sample set. A semantic representation model is trained using the contrastive learning training samples to generate embedding vectors for retrieval tasks.

2. The method according to claim 1, characterized in that, The step of dividing the unstructured document into multi-granularity text blocks with hierarchical relationships, and constructing a knowledge base based on the multi-granularity text blocks and their relationships, includes: The unstructured document is cleaned up and its content is extracted to obtain continuous text to be processed; Based on the directory headings and natural paragraph boundaries, the continuous text is divided into parent text blocks as the first granularity. According to preset length and structure rules, the parent text block is divided to obtain at least one child text block with different granularities, and a subordinate relationship is established between the parent text block and the corresponding child text block. Based on the subordinate relationships and the order information of each text block in the original document, association information representing the hierarchy and order of multi-granular text blocks is generated. The multi-granularity text blocks and the associated information are stored in the knowledge base.

3. The method according to claim 1, characterized in that, The process of forming a context based on the combination of multi-granularity text blocks in the knowledge base, and injecting perturbation information corresponding to prior knowledge of the vertical domain into the context, includes: Multiple multi-granular text blocks with related relationships are selected from the knowledge base according to preset window rules, and the multiple multi-granular text blocks are combined into a context for generating a retrieval query; Based on prior knowledge of the target vertical domain, the perturbation type corresponding to the prior knowledge is determined, and perturbation information is generated according to the perturbation type; The perturbation information is injected into the context according to a preset strategy to obtain a perturbation context carrying features of prior knowledge in the vertical domain.

4. The method according to claim 3, characterized in that, The call generation model generates a retrieval query based on the context and determines the target text block corresponding to the retrieval query as the initial positive sample, including: Construct a generative model set that includes multiple generative models, and select a target generative model from the generative model set according to a preset selection rule; The perturbated context is input into the target generation model to generate at least one retrieval query corresponding to the context, and to obtain association information characterizing the correspondence between the retrieval query and each multi-granularity text block in the context; Based on the association information and the preset association determination rules, a target text block corresponding to the retrieval query is determined from the multi-granularity text blocks contained in the context, and the target text block is used as the initial positive sample of the retrieval query.

5. The method according to claim 1, characterized in that, The process of mining positive sample text blocks for supplementation from the candidate text block set based on the ranking results and inference model, as well as difficult negative sample text blocks that are semantically similar to the retrieval query but do not belong to positive samples, includes: Based on the sorted candidate text block sequence, multiple candidate text blocks are selected according to the first preset ranking interval to form a first context. The first context and the retrieval query are input into the inference model. According to the output result of the inference model and the preset positive sample determination rule, positive sample text blocks for supplementation are determined from the candidate text blocks corresponding to the first context. Based on the sorted candidate text block sequence, multiple candidate text blocks are selected according to the second preset ranking interval to form a second context. The second context and the retrieval query are input into the inference model. According to the output result of the inference model and the preset negative sample determination rule, difficult negative sample text blocks that are semantically similar to the retrieval query but do not belong to positive samples are determined from the candidate text blocks corresponding to the second context.

6. The method according to claim 5, characterized in that, The step of filtering false negative sample text blocks based on the correlation between text blocks to form a corresponding positive sample set and negative sample set for each search query includes: Based on the pre-established hierarchical and sequential association relationships of multi-granularity text blocks in the knowledge base, the association relationships between each difficult negative sample text block, the initial positive sample text block, and the positive sample text blocks used for supplementation are determined. Difficult negative sample text blocks that have a predetermined parent-child inclusion relationship with any of the initial positive sample text blocks or the supplementary positive sample text blocks are identified as false negative sample text blocks, and the false negative sample text blocks are removed from the difficult negative sample text blocks or merged into the positive sample text blocks, wherein the predetermined relationship includes parent-child inclusion relationship, index continuation relationship and / or cross-block combination relationship; Based on the elimination of false negative sample text blocks, the initial positive sample text blocks and the positive sample text blocks used for supplementation are grouped together to form the positive sample set corresponding to the search query, and the remaining difficult negative sample text blocks and other optional negative sample text blocks are grouped together to form the negative sample set corresponding to the search query.

7. The method according to claim 1, characterized in that, The step of training a semantic representation model using the contrastive learning training samples to generate embedding vectors for the retrieval task includes: The contrastive learning training samples are organized into sample groups according to training batches, consisting of a retrieval query, a positive sample text block corresponding to the retrieval query, and a negative sample text block. Each of the search queries and the corresponding positive and negative sample text blocks are input into the semantic representation model to obtain a first vector representation of the semantic features of the search query and a second vector representation of the semantic features of each text block. Based on the similarity relationship between the first vector representation and the second vector representation of the corresponding positive sample text block and the second vector representation of each negative sample text block, the parameters of the semantic representation model are updated according to the preset contrastive learning loss function, so that the semantic representation model can encode the text blocks in the retrieval query and knowledge base after convergence, and output the embedding vector for the retrieval task.

8. An embedded data synthesis device that integrates retrieval and large-scale model distillation, characterized in that, include: The preprocessing module is used to preprocess unstructured documents in the vertical domain, divide the unstructured documents into multi-granularity text blocks with hierarchical relationships, and build a knowledge base based on the multi-granularity text blocks and their relationships. The generation module is used to form a context based on the combination of multi-granularity text blocks in the knowledge base, inject perturbation information corresponding to the prior knowledge of the vertical domain into the context, call the generation model to generate a retrieval query based on the context, and determine the target text block corresponding to the retrieval query as the initial positive sample. The retrieval module is used to input the retrieval query into the semantic retrieval system, retrieve a set of candidate text blocks from the knowledge base, and sort the set of candidate text blocks according to relevance. The mining module is used to mine positive sample text blocks for supplementation and difficult negative sample text blocks that are semantically similar to the retrieval query but do not belong to positive samples from the candidate text block set based on the ranking results and inference model, and to filter false negative sample text blocks according to the correlation between text blocks, so as to form a corresponding positive sample set and negative sample set for each retrieval query. The training module is used to construct contrastive learning training samples based on the positive sample set and the negative sample set, and to train a semantic representation model using the contrastive learning training samples to generate embedding vectors for retrieval tasks.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images