Embedded data synthesis method and device integrating retrieval and large model distillation and medium

Abstract

The invention provides an embedded data synthesis method and device fusing retrieval and large model distillation and a medium. The method comprises the following steps of: preprocessing an unstructured document in a vertical field, and dividing the unstructured document into multi-granularity text blocks with a hierarchical association relationship; forming a context based on the combination of the multi-granularity text blocks, injecting disturbance information corresponding to the priori knowledge in the vertical field into the context, calling a generative model to generate a retrieval query according to the context, and determining a target text block corresponding to the retrieval query as an initial positive sample; false negative sample text blocks are filtered according to the incidence relation between the text blocks, and a positive sample set and a negative sample set are formed; and constructing a comparative learning training sample, and training the semantic representation model by using the comparative learning training sample to generate an embedded vector for the retrieval task. According to the method, the retrieval task construction efficiency and authenticity can be improved, the positive sample coverage integrity is improved, and the contrast learning training stability and retrieval precision are enhanced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to an embedded data synthesis method and device fusing retrieval and large model distillation and a medium. BACKGROUND

[0002] In the field of large models and agents, especially in the field of embedded data synthesis technology for vertical fields such as finance, medicine, and industrial manufacturing. With the acceleration of digital transformation in various industries, a large number of unstructured documents such as research reports, medical records, technical manuals, and construction files are continuously accumulating, which need to be converted into machine-processable semantic representations through embedded data to support retrieval-enhanced generation, question-and-answer decision-making, and other applications.

[0003] In the prior art, the document is usually divided, a retrieval query is constructed, and a semantic representation model is trained using contrastive learning to obtain an embedded vector for retrieval. However, on the one hand, the existing method lacks an efficient retrieval task construction tool chain when facing a large amount of unstructured documents, and the generated query relies on a general template or a simple prompt, which is difficult to reflect the professional terms, jargon, and real use scenarios of the vertical field, resulting in a disconnection between the synthesized embedded data and the actual retrieval requirements.

[0004] On the other hand, in the contrastive learning sample construction process, the existing technology is based on fixed-length division to obtain text blocks, and selects positive and negative samples through simple similarity or random strategy. This method easily leads to the fragmentation of associated information across text blocks and chapters, and the incomplete recall of positive samples; the negative samples are mostly random negative examples that are easy to distinguish, lacking systematic mining of difficult negative samples that are similar in semantics but do not match in business. At the same time, since the false negative samples are not effectively filtered, it is easy to introduce samples that are actually related in semantics but are labeled as negative examples in the contrastive learning loss, affecting the convergence stability and retrieval accuracy of the model, making it more difficult to obtain high-quality embedded data in data-scarce vertical fields. SUMMARY

[0005] Therefore, the embodiments of the present application provide an embedded data synthesis method and device fusing retrieval and large model distillation and a medium to solve the problems of low retrieval task construction efficiency and poor business adaptation, incomplete positive sample recall, difficult negative sample mining, and insufficient false negative sample filtering in the prior art.

[0006] In a first aspect, an embedding data synthesis method combining retrieval and large model distillation is provided. The method includes: preprocessing unstructured documents in a vertical field, dividing the unstructured documents into multi-granularity text blocks having hierarchical association relationships, and constructing a knowledge base based on the multi-granularity text blocks and the association relationships; combining the multi-granularity text blocks in the knowledge base to form a context, injecting disturbance information corresponding to prior knowledge in the vertical field into the context, calling a generation model to generate a retrieval query according to the context, and determining a target text block corresponding to the retrieval query as an initial positive sample; inputting the retrieval query into a semantic retrieval system, retrieving a candidate text block set in the knowledge base, and sorting the candidate text block set according to relevance; based on a sorting result and an inference model, mining a positive sample text block for supplementing from the candidate text block set, and a difficult negative sample text block that is similar in semantics to the retrieval query but does not belong to the positive sample, and filtering false negative sample text blocks according to association relationships between the text blocks, to form a corresponding positive sample set and a negative sample set for each retrieval query; constructing contrast learning training samples based on the positive sample set and the negative sample set, training a semantic representation model using the contrast learning training samples, and generating embedding vectors for a retrieval task.

[0007] In a second aspect, an embedding data synthesis device combining retrieval and large model distillation is provided. The device includes: a preprocessing module configured to preprocess unstructured documents in a vertical field, divide the unstructured documents into multi-granularity text blocks having hierarchical association relationships, and construct a knowledge base based on the multi-granularity text blocks and the association relationships; a generation module configured to combine the multi-granularity text blocks in the knowledge base to form a context, inject disturbance information corresponding to prior knowledge in the vertical field into the context, call a generation model to generate a retrieval query according to the context, and determine a target text block corresponding to the retrieval query as an initial positive sample; a retrieval module configured to input the retrieval query into a semantic retrieval system, retrieve a candidate text block set in the knowledge base, and sort the candidate text block set according to relevance; a mining module configured to, based on a sorting result and an inference model, mine a positive sample text block for supplementing from the candidate text block set, and a difficult negative sample text block that is similar in semantics to the retrieval query but does not belong to the positive sample, and filter false negative sample text blocks according to association relationships between the text blocks, to form a corresponding positive sample set and a negative sample set for each retrieval query; and a training module configured to construct contrast learning training samples based on the positive sample set and the negative sample set, train a semantic representation model using the contrast learning training samples, and generate embedding vectors for a retrieval task.

[0008] In a third aspect, an electronic device is provided. The electronic device includes a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, the steps of the above method are implemented.

[0009] In a fourth aspect, the present application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above method.

[0010] The above at least one technical solution adopted by the embodiments of the present application can achieve the following beneficial effects: By preprocessing the unstructured document in the vertical field, the unstructured document is divided into multi-granularity text blocks with hierarchical association relationship, and a knowledge base is constructed based on the multi-granularity text blocks and the association relationship; based on the multi-granularity text blocks in the knowledge base, a context is formed, perturbation information corresponding to prior knowledge in the vertical field is injected into the context, a generation model is called to generate a retrieval query according to the context, and a target text block corresponding to the retrieval query is determined as an initial positive sample; the retrieval query is input into a semantic retrieval system, a candidate text block set is retrieved in the knowledge base, and the candidate text block set is sorted according to relevance; based on the sorting result and an inference model, a positive sample text block for supplement is mined from the candidate text block set, and a difficult negative sample text block that is similar in semantics to the retrieval query but does not belong to the positive sample is mined, and false negative sample text blocks are filtered according to the association relationship between the text blocks, so as to form a corresponding positive sample set and a negative sample set for each retrieval query; based on the positive sample set and the negative sample set, a contrast learning training sample is constructed, a semantic representation model is trained using the contrast learning training sample, and an embedding vector for a retrieval task is generated. The present application can improve the retrieval task construction efficiency and authenticity, improve the positive sample coverage integrity, enhance the contrast learning training stability and retrieval precision. BRIEF DESCRIPTION OF DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0012] Figure 1 is a flowchart of the embedding data synthesis method provided by the embodiments of the present application, which fuses retrieval and large model distillation; Figure 2 is a structural diagram of the embedding data synthesis device provided by the embodiments of the present application, which fuses retrieval and large model distillation; Figure 3 is a structural diagram of an electronic device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0013] In the following description, for purposes of explanation and not limitation, specific details are set forth such as particular architectures, techniques, etc. in order to provide a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application can be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known methods, devices, circuits, and

[0014] The present application mainly relates to the vertical field of large models and Agent applications, namely, embedding data synthesis technology, which is the core basis for supporting vertical field retrieval enhancement generation (RAG) and the like, and the quality of the generated embedding data directly determines the retrieval accuracy and business effect of the representation model.

[0015] With the acceleration of digital transformation of financial, medical, industrial manufacturing and other vertical fields, a large amount of unstructured documents such as research reports, medical records, technical manuals and the like have been accumulated in the industry. These documents contain very high business value and need to be converted into machine-understandable semantic representations through embedding data synthesis to support subsequent large model application landing tasks such as retrieval and analysis.

[0016] The current embedding data synthesis in the vertical field faces two core pain points: first, a large amount of unstructured text is difficult to efficiently convert into a retrieval task, and the generated Query is of low quality. The large amount of unstructured private "deposited" knowledge accumulated in the industry lacks an efficient conversion path - constructing a Query is essentially constructing a retrieval task, and existing technologies lack an adaptive tool chain, which cannot efficiently generate a retrieval task based on a large amount of data; at the same time, the generated Query has multiple quality problems: the generated Query is seriously out of line with the real business scenario and is of low quality, so that the embedding data cannot match the actual retrieval demand, and the effect after model deployment is significantly poor.

[0017] Second, the quality of contrastive learning samples is low and is not combined with efficient tools. In the existing technology, when samples are mined, positive samples are prone to incomplete recall due to too fine Chunk division or scattered in different chapters, and cannot cover cross-Chunk and cross-chapter associated information, and negative samples are mostly randomly selected, lacking effective mining of "difficult" negative samples. This makes the contrastive learning training effect poor, the model unstable, and it is difficult to balance the recall rate and precision of retrieval. Especially in the scenario of data scarcity in the vertical field, the lack of high-quality samples further exacerbates the difficulty of embedding data synthesis.

[0018] Therefore, the technical problem mainly solved by the present application focuses on the core pain points of embedding data synthesis in the vertical field, as follows: 1. Efficient and real search task generation: Existing methods cannot efficiently convert massive data into queries, and the generated queries lack vertical field characteristics, are not adapted to professional terms, jargon, and actual business disturbance scenarios, have low quality, and cannot cover multi-granularity and multi-expression search requirements, resulting in low matching degree between embedded data and real business scenarios.

[0019] 2. Incomplete positive sample recall and false sample problem in contrast learning: Traditional methods cannot effectively mine positive and negative samples in contrast learning, and the Chunk segmentation method causes cross-Chunk and cross-chapter related information to be fragmented, making it difficult to effectively mine all positive samples and filter false samples. Single-granularity positive sample mining cannot cover the complete semantics of the query, resulting in missing positive sample information and affecting the semantic representation integrity of embedded data.

[0020] 3. Insufficient difficult negative sample mining and excessive false negative samples: Existing sample mining methods cannot accurately mine "difficult" negative samples, and lack effective false negative sample filtering mechanisms, which is fatal to contrast learning training based on infoNCE loss.

[0021] In view of the problems existing in the prior art, the present application proposes an embedded data synthesis method combining retrieval and large model distillation, which efficiently processes industry unstructured private "deposited" knowledge by using RAG as a data synthesis tool, deeply combines the internal knowledge and reasoning ability of large models, and uses multi-granularity and diversified query generation and accurate sample mining strategies to significantly improve the quality of embedded data and adapt to real business needs in vertical fields.

[0022] The core of the technical solution of the present application is to combine retrieval mechanism and large model distillation technology to build an embedded data synthesis method and system suitable for vertical fields, solving the core pain points of high-quality embedded data synthesis in vertical fields. The core steps of the technical solution of the present application include: 1. Knowledge base construction: Preprocess a large number of unstructured documents in the vertical field, and construct a knowledge base through incremental persistence. Process the documents into multi-granularity structures when they are stored in the database, providing support for subsequent multi-granularity query generation and sample mining.

[0023] 2. Multi-granularity diversification Query generation: traverse the Chunk blocks in the knowledge base as large model context with 3-5 unequal sliding windows, select important knowledge from these chunks to generate retrieval tasks through the reasoning ability of large models, avoid generating low-quality queries through mutual comparison, and inject noise templates (including spelling disturbance, professional term synonym replacement, multi-hop reasoning, etc.) to generate queries with moonshot, GPT-4o, Qwen, DeepSeek, etc. Multiple types of large models are called at random, and the Chunk on which the generated Query is based is recorded as a positive example to ensure the authenticity, diversity and domain adaptability of the Query.

[0024] 3. Multi-granularity retrieval and rearrangement: input the generated Query into the retrieval system, dynamically return relevant Chunk blocks (including different length fragments) of different granularities and different chapters to achieve preliminary acquisition of multi-length cross-chapter samples; call multiple rearrangement models to randomly switch and rearrange, and use the correlation determination ability of different models to improve the diversity and accuracy of the results.

[0025] 4. Sample mining with large model distillation: select the top-k after rearrangement 30 Chunk as context to mine positive samples to enhance recall; select the Chunk in Top-k as context, and use large model reasoning to identify and mine "difficult" negative samples, while filtering false positive samples, and distill large model knowledge to the sample mining process.

[0026] 5. Long context positive example merging and embedded data generation: use the long context reasoning ability of the large model to dynamically splice the relevant positive samples across Chunk blocks into complete samples to cover complete semantics; enhance sample integrity and diversity.

[0027] The technical solutions of the present application will be described in detail below in combination with the drawings and specific embodiments.

[0028] Figure 1 is the flowchart of the embedded data synthesis method combining retrieval and large model distillation provided by the embodiments of the present application. As shown in Figure 1 , the embedded data synthesis method combining retrieval and large model distillation can specifically include: S101, preprocessing the unstructured documents of the vertical field, dividing the unstructured documents into multi-granularity text blocks with hierarchical association relationship, and constructing a knowledge base based on the multi-granularity text blocks and the association relationship; S102, based on the multi-granularity text blocks in the knowledge base, form a context, inject disturbance information corresponding to the prior knowledge of the vertical field into the context, call a generation model to generate a retrieval query according to the context, and determine the target text block corresponding to the retrieval query as an initial positive sample; S103, input the retrieval query into the semantic retrieval system, retrieve the candidate text block set from the knowledge base, and sort the candidate text block set according to relevance; S104. Based on the ranking results and inference model, positive sample text blocks for supplementation and difficult negative sample text blocks that are semantically similar to the retrieval query but do not belong to positive samples are mined from the candidate text block set. False negative sample text blocks are filtered according to the correlation between text blocks to form a corresponding positive sample set and negative sample set for each retrieval query. S105: Contrastive learning training samples are constructed based on positive and negative sample sets. The semantic representation model is trained using the contrastive learning training samples to generate embedding vectors for retrieval tasks.

[0029] In some embodiments, unstructured documents are divided into multi-granularity text blocks with hierarchical relationships, and a knowledge base is constructed based on the multi-granularity text blocks and their relationships, including: Unstructured documents are cleaned of formatting and their content extracted to obtain continuous text to be processed; Based on the table of contents title and natural paragraph boundaries, continuous text is divided into parent text blocks as the first granularity. Based on preset length and structure rules, the parent text block is divided to obtain at least one child text block of different granularities, and a subordinate relationship is established between the parent text block and the corresponding child text block. Based on the subordinate relationships and the order information of each text block in the original document, generate association information that represents the hierarchy and order of multi-granular text blocks; Multi-granular text blocks and related information are stored in a knowledge base.

[0030] Specifically, the system first receives raw document files from fields such as finance, healthcare, industrial manufacturing, or construction, including financial research reports, medical records, equipment technical manuals, and construction organization designs. Files from different sources are then uniformly input into the preprocessing module. The preprocessing module cleans up formatting issues in the raw files, such as layout tags, headers and footers, watermarks, and redundant blank lines. Combining optical character recognition (OCR) results with encoding conversion, it extracts text within images and various formatted texts into a single continuous text segment. Simultaneously, it retains table of contents information, heading level markers, and page number information, providing foundational data for subsequent structure-based segmentation.

[0031] After the continuous text extraction is completed, the chunking module performs a first level of structure division on the continuous text based on the existing directory headings and natural paragraph boundaries in the document. For example, the first level headings, second level headings or chapter headings in the document are parsed, and the continuous text segment corresponding to each chapter is divided into a parent text block. Each parent text block generally corresponds to a chapter in a research report, a functional chapter in a technical manual, or a sub-part engineering instruction in a construction file, ensuring that each parent text block has a relatively complete topic range in terms of semantics. On this basis, the chunking module further performs fine-grained division on the parent text block according to preset length rules and structure rules, for example, the parent text block is divided into one or more child text blocks based on the number of characters, the number of sentences or the number of natural paragraphs, so that the length of each child text block falls within the preset interval, and the division is performed as much as possible without cutting in the middle of a sentence, formula or professional term.

[0032] To facilitate multi-granularity scheduling and cross-block merging during subsequent sample mining, the association building module synchronously records the parent-child relationship between the parent text blocks and the child text blocks and the sequential relationship between the child text blocks during the chunking process. For example, a unique parent block identifier is assigned to each parent text block, and a unique child block identifier is assigned to each child text block generated inside it, and a one-to-many mapping relationship between the parent block identifier and the corresponding child block identifier is maintained inside. At the same time, according to the starting position and ending position of each text block in the original document, the index continuity relationship between adjacent child text blocks is recorded, and sequential information such as chapter number, page number and title level is marked, which is used to represent the arrangement order of the child text blocks inside the same parent text block and the order of the different parent text blocks in the full text.

[0033] After the parent-child hierarchical relationship and the sequential relationship are generated, the association building module unifies the above multi-granularity text blocks and the association information into structured data entries. Each entry includes at least the document identifier, the parent block identifier, the child block identifier, the block type marker, the text content, the start and end positions in the original document, the parent-child affiliation marker and the adjacent block sequential marker, and metadata such as source business type, document date and language category can be attached if necessary. The knowledge base storage module writes these structured data entries into the knowledge base, which can use a storage structure that supports the hybrid management of vector indexes and relationship indexes. On the one hand, it supports subsequent semantic retrieval of text blocks of various granularities, and on the other hand, it supports fast recovery of chapter structure and cross-block context based on parent-child relationship and index continuity relationship in the knowledge base.

[0034] In some specific examples, taking a financial research report as an example, the system receives a research report file containing three chapters of macroeconomic analysis, industry analysis and company analysis. The preprocessing module first removes the cover, disclaimer page, header and footer, and redundant separator, and parses the text part into three chapters according to the table of contents. The blocking module divides each chapter into a parent text block, and divides a number of child text blocks in each parent text block according to the upper limit of about one thousand characters and the sentence boundary.

[0035] For example, the macroeconomic analysis chapter is divided into a parent text block P1, which generates child text blocks C11, C12, C13, etc. inside. The equipment specification of industrial manufacturing enterprises can be divided into parent text blocks according to the function modules of the equipment, and further divided into child text blocks according to the operation steps or fault types. The association construction module records the subordinate relationship between P1 and C11, C12, C13, records the index continuous relationship between C11 and C12, C12 and C13, and stores these blocks and their relationship information into the knowledge base. In this way, when generating a retrieval query and mining samples subsequently, a single child text block can be directly called as a fine-grained semantic unit, or the context of the entire chapter can be restored through the parent-child relationship and index continuous relationship.

[0036] Through the above embodiments, the original unstructured vertical field document is converted into multi-granularity text blocks with parent-child hierarchical relationship and sequential relationship, and is uniformly stored in the knowledge base. When generating a retrieval query, performing semantic retrieval and mining positive and negative samples subsequently, the multi-granularity text blocks can be flexibly switched and combined between different granularities and different levels, maintaining the integrity and continuity of the text semantic structure, and providing a structured foundation for subsequent embedding data synthesis.

[0037] In some embodiments, based on the combination of the multi-granularity text blocks in the knowledge base, the perturbation information corresponding to the prior knowledge of the vertical field is injected into the context, including: Selecting a plurality of multi-granularity text blocks with an association relationship from the knowledge base according to a preset window rule, and combining the plurality of multi-granularity text blocks into a context for generating a retrieval query; Based on the prior knowledge of the target vertical field, determining a perturbation type corresponding to the prior knowledge, and generating perturbation information according to the perturbation type; According to a preset strategy, the perturbation information is injected into the context to obtain a perturbation context carrying the characteristics of the prior knowledge of the vertical field.

[0038] Specifically, the context selection submodule first selects a combination of text blocks with associated relationships as the context according to a preset window rule based on the multi-granularity text blocks in the knowledge base and their hierarchical and sequential associated relationships. The window rule can be set according to the business scenario as a sliding window with a sub-level text block as a basic unit, for example, the window length is 3-5 sub-level text blocks, and the window is allowed to span adjacent sub-level text blocks within the same parent-level text block or, if necessary, the boundaries of adjacent parent-level text blocks to cover the continuous semantic range across paragraphs or sections.

[0039] In actual implementation, the context selection submodule can start from the first sub-level text block of a parent-level text block, sequentially select C1, C2, C3 to form a first window, and then slide one sub-block backward to form a second window of C2, C3, C4, and so on. If a chapter boundary is encountered and there is a sequential associated relationship between adjacent chapters, the last sub-block of the previous chapter and the starting sub-block of the next chapter can be jointly included in the window to cover the semantic fragment that is continuous in business but spans chapters.

[0040] After obtaining the base context for generating a retrieval query, the perturbation injection submodule determines the perturbation type according to the prior knowledge of the target vertical field. The prior knowledge includes common abbreviations and industry jargon in the financial field, commonly used abbreviations and colloquial expressions in the medical field, device colloquial names and operation passwords in the industrial manufacturing field, and internal agreed language in the construction field. The system abstracts and organizes these prior knowledge into multiple perturbation types in advance, such as spelling perturbation type, professional term synonym replacement type, multi-hop reasoning prompt type, and jargon conversion type. After receiving a context and its corresponding field label, the perturbation injection submodule selects one or more suitable perturbation types based on the prior knowledge base of the field, and generates perturbation information according to the corresponding perturbation type.

[0041] For example, in the financial research report scenario, when the target vertical field is the financial field, the perturbation injection submodule can select the "professional term synonym replacement" and "minor spelling error" two types of perturbations from the financial prior knowledge base. For the "professional term synonym replacement" type, the system extracts the corresponding relationship from the pre-constructed term mapping table, for example, replacing "price-earnings ratio" with "PE multiple" and "return on equity" with "ROE", while keeping other key information in the context unchanged; for the "minor spelling error" type, the system introduces a small amount of letter order reversal, similar word replacement, etc. perturbation in part of the ordinary words or index names according to the preset error template, for example, "macro economy" is miswritten as "macro economic", to simulate the possible typing errors or non-standard expressions of real users when inputting queries.

[0042] For the field of industrial manufacturing, the term "concrete" can be replaced with the site-specific term "cement" according to the jargon commonly found in equipment maintenance manuals. Alternatively, the term "high-voltage switch cabinet" can be replaced with "high-voltage cabinet." Additionally, contextual cues such as "as commonly referred to by on-site technicians" can be added to make the subsequent generated search queries more closely aligned with the actual expressions used by engineers on the front line.

[0043] After generating the specific perturbation information, the perturbation injection submodule injects the perturbation information into the context according to a preset strategy to obtain a perturbed context that carries the characteristics of prior knowledge in the vertical field. The preset strategy can include a perturbation position selection strategy and a perturbation intensity control strategy. The perturbation position selection strategy is used to determine which terms, sentences, or paragraphs in the context to replace or insert, such as preferentially implementing perturbation on key business domain terms, index names, device names, process steps, etc., to avoid disrupting the overall semantic coherence of the context.

[0044] The perturbation intensity control strategy is used to control the number and type combination of perturbations applied in a single context, such as injecting only one type of perturbation in most windows and injecting two types of perturbations in some windows to introduce sufficient diversity while ensuring readability. In the implementation process, the perturbation injection submodule can randomly select one or two text segments to be perturbed for each context, replace or insert the previously generated perturbation information into the corresponding positions to form a new perturbed context, while retaining the block index information and domain label of the original context to facilitate subsequent tracing of the corresponding original text blocks after generating the search query.

[0045] For example, in the field of building construction, when the construction organization design file of a certain project is stored in the knowledge base, one of the parent text blocks describes the "bottom plate concrete pouring process," and after segmentation, multiple child text blocks are obtained, including concrete mix ratio requirements, vibration methods, curing time, etc. The context selection submodule can select three consecutive child text blocks to form a context according to the sliding window rule, covering the complete process from batching to curing of concrete pouring. The perturbation injection submodule can replace "concrete" with the site-specific term "cement" based on the prior knowledge in the field of building construction, rewrite "curing" as "damp curing" or "moist curing," and introduce contextual cues such as "as commonly referred to by on-site technicians" to make the overall expression in the perturbed context more closely aligned with the actual language habits of construction site technicians.

[0046] Through the above embodiments, the system uses the multi-granularity text blocks in the knowledge base to construct a context that can reflect cross-section, cross-chapter semantics before generating a retrieval query, and injects multi-type perturbation information based on vertical domain prior knowledge, so that the context itself carries domain terminology, jargon, spelling errors and multi-hop reasoning and other characteristics, thereby naturally generating retrieval expression forms with diverse styles and close to real business scenarios when a generation model is subsequently called to generate a retrieval query. This processing method ensures the integrity of the context semantics while introducing the modeling capability of domain knowledge and real input noise, which is beneficial to better cover various retrieval habits and expression forms in the vertical domain for the synthesized retrieval query, and provides more rich and representative retrieval task inputs for subsequent embedding data synthesis and contrastive learning training.

[0047] In some embodiments, the calling generation model generates a retrieval query according to the context, and determines a target text block corresponding to the retrieval query as an initial positive sample, comprising: building a generation model set comprising a plurality of generation models, and selecting a target generation model from the generation model set according to a preset selection rule; inputting the perturbed context into the target generation model to generate at least one retrieval query corresponding to the context, and obtaining association information representing the corresponding relationship between the retrieval query and each multi-granularity text block in the context; determining a target text block corresponding to the retrieval query from the multi-granularity text blocks contained in the context according to the association information and a preset association determination rule, and taking the target text block as an initial positive sample of the retrieval query.

[0048] Specifically, the generation model scheduling submodule first builds a generation model set based on system deployment requirements. The generation model set can include a plurality of large language models with different training corpus and style characteristics, such as a first type of model optimized for general scenarios, a second type of model enhanced on Chinese financial corpus, a third type of model enhanced on industrial technical documents, etc. For each model, its model identifier, supported maximum context length, adapted vertical domain label, and resource consumption level, etc. meta information are registered in the model set.

[0049] The system pre-sets a generation model selection rule, such as a polling strategy, a random strategy or a vertical domain priority selection strategy. When a perturbed context and its domain label (such as finance, medical care, industrial manufacturing or building construction) are received, the generation model scheduling submodule filters out an adapted model subset in the model set according to the domain label, and then selects a target generation model from it according to the preset selection rule. For example, when the domain label is finance, the model fine-tuned on financial research reports is preferentially selected; when the current model load is high, a standby model of the same domain is switched, so as to ensure the domain adaptability while taking into account the system throughput.

[0050] After determining the target generation model, the generation model scheduling submodule inputs the perturbed context constructed into a standardized input sequence into the target generation model. The standardized input sequence not only contains the aforementioned text content combined by the multi-granularity text blocks, but also contains a prompt for constraining the output format, which is used to guide the generation model to label the text block on which the retrieval query is based while outputting the retrieval query. For example, the prompt can instruct the generation model to “generate 1 retrieval question that a real user may ask based on the following context, and give the text block sequence number on which the question mainly depends in the output”, where the text block sequence number corresponds to the local index assigned to each multi-granularity text block by the system when constructing the context.

[0051] Further, after receiving the perturbed context and the prompt, the target generation model combines the professional terms, jargon expressions and noise patterns in the context, and generates at least 1 retrieval query highly related to the semantic of the context through its language understanding and reasoning ability, and attaches labeling information in the output structure, such as giving 1 or more text block indexes mainly relied on by the retrieval query in the form of a list or special mark. The generation model scheduling submodule passes the output result as the original generation result to the positive sample labeling submodule.

[0052] The positive sample labeling submodule parses the retrieval query text and the text block index set corresponding to the retrieval query from the original generation result. Combined with the block index mapping relationship reserved in the context construction stage, the positive sample labeling submodule can quickly locate the corresponding multi-granularity text blocks in the current context.

[0053] For example, in the financial research report scenario, the perturbed context covers 3 sub-level text blocks about interest rates, inflation and monetary policy in the macroeconomic chapter, the target generation model outputs the retrieval query “What is the impact of the current tightening of monetary policy on stock market valuation”, and labels the text block indexes mainly relied on by the question as C21 and C22. The positive sample labeling submodule finds the sub-level text blocks with indexes C21 and C22 in the context accordingly, and marks them as the target text block set corresponding to the retrieval query.

[0054] In some cases, such as when the generation model only labels 1 text block index or labels more than 3 text block indexes, the positive sample labeling submodule can filter and merge the target text blocks according to the preset association determination rules, such as preferentially retaining two text blocks highly related to the theme of the retrieval query and with consecutive indexes, or merging the text blocks with too many indexes before registering them as single positive sample units.

[0055] For example, in the field of industrial manufacturing or construction, the process of the embodiment is also applicable. Taking the equipment troubleshooting manual as an example, when the disturbance context covers multiple sub-level text blocks of “variable frequency drive overload failure”, and after the jargon disturbance, it contains expressions such as “variable frequency drive old tripping” and “motor old stop”, the target generation model will generate a search query close to the style of the first-line engineer, such as “How to troubleshoot the old tripping of the variable frequency drive”, and mark the main sub-level text block index of the problem in the output. The positive sample labeling submodule analyzes these indexes, and registers the corresponding text blocks as the initial positive samples of the problem. In the subsequent sample mining stage, these initial positive samples can be further expanded to other related text blocks in the knowledge base.

[0056] In some embodiments, the search query is input into a semantic search system, a set of candidate text blocks is retrieved in the knowledge base, and the set of candidate text blocks is sorted according to the relevance, including: Converting the search query into a first semantic representation, and obtaining a second semantic representation of each multi-granularity text block in the knowledge base; Selecting multi-granularity text blocks related to the search query from the knowledge base as a set of candidate text blocks based on the similarity of the first semantic representation and the second semantic representation; Using at least one relevance determination model, calculating the relevance score of each candidate text block based on the semantic representation of the search query and the candidate text block, and when there are multiple relevance determination models, performing aggregation processing on each relevance score, sorting the set of candidate text blocks according to the aggregated relevance score, and obtaining a sequence of sorted candidate text blocks.

[0057] Specifically, the semantic encoding submodule pre-generates a second semantic representation for each multi-granularity text block in the knowledge base during the system initialization stage. The second semantic representation can be obtained by a trained semantic representation model. The semantic representation model can use an encoder structure to encode the input parent text block and sub-level text block respectively, convert them into fixed-dimensional vector representations, and store the vector together with the identification and hierarchical information of the corresponding text block in the vector index structure.

[0058] For newly added documents in the library, the system performs format cleaning, hierarchical division, and association relationship construction at the same time, and performs the same encoding process on the newly generated multi-granularity text blocks, and writes the second semantic representation of the multi-granularity text blocks into the vector index of the knowledge base. When processing online search queries, the semantic encoding submodule receives the search query text output by the generation model, calls the same or compatible semantic representation model, encodes the search query into a first semantic representation, so as to ensure that the search query and the multi-granularity text block can be compared in the same semantic space.

[0059] Further, after obtaining the first semantic representation of the search query and the second semantic representations pre-stored in the knowledge base, the candidate recall sub-module calculates the similarity degree between the first semantic representation and each second semantic representation based on a preset similarity measurement rule. The similarity measurement can adopt vector dot product, normalized cosine similarity or other functions that can measure semantic proximity. The candidate recall sub-module can combine the vector retrieval engine to quickly select several multi-granularity text blocks with higher similarity scores from the knowledge base as the candidate text block set without traversing all the text blocks.

[0060] For example, when the search query is "What impact does the current tightening of monetary policy have on stock market valuation?", the candidate recall sub-module can recall multiple parent text blocks and child text blocks related to "monetary policy", "interest rate adjustment" and "valuation level" from the financial research report knowledge base, including paragraphs about interest rate policy in the macroeconomic analysis chapter and discussions about valuation changes in the market strategy chapter, to form a preliminary candidate text block set.

[0061] To further improve the relevance accuracy between the candidate text block set and the search query, and take into account the relevance determination ability of different models, the relevance ranking sub-module ranks the above-mentioned candidate text block set using at least one relevance determination model. In one implementation, the relevance determination model can include a basic relevance model based on vector similarity and a re-ranking model based on deep matching structure. The basic relevance model directly uses the similarity score obtained in the candidate recall stage as the initial relevance score, and the re-ranking model jointly models the search query and the original text of the candidate text block to output a more fine-grained relevance score. In the case of multiple relevance determination models, the relevance ranking sub-module normalizes and weightedly sums the relevance scores output by each model according to a preset aggregation strategy to obtain the aggregated relevance score of each candidate text block, and then ranks the candidate text block set according to the aggregated relevance score to output a ranked candidate text block sequence.

[0062] For example, in the industrial manufacturing field, when the search query generated in the current embodiment is "How to troubleshoot the frequent tripping of the frequency converter?", the semantic encoding sub-module encodes the search query into a first semantic representation and compares it with the second semantic representations of the multi-granularity text blocks in the device technical manual and troubleshooting guide. The candidate recall sub-module can recall multiple child text blocks including "frequency converter overload protection", "motor current anomaly", "line ground fault check" and corresponding parent text blocks to form a candidate text block set.

[0063] Subsequently, the relevance ranking submodule calls the relevance determination model customized for the industrial scene to reevaluate and rank the semantic matching degree of the candidate text blocks and the search query, so that the text blocks describing the "overload fault alarm code meaning" and "on-site current measurement method" are arranged in the front, and the text blocks related to general equipment maintenance are arranged in the back, thereby providing high-quality ranking input for subsequent positive sample supplement and difficult negative sample mining.

[0064] Through the above embodiment, after the search query is input into the semantic search system, the system represents the search query and the multi-granularity text blocks by using the unified semantic encoding mode, and obtains the candidate text block sequence covering the parent and child granularities and spanning chapters in the knowledge base by combining the vector similarity recall and the multi-model relevance ranking. The ranking candidate text block sequence not only guarantees the semantic relevance with the search query, but also retains the multi-granularity and multi-source candidate space, thereby laying a reliable foundation for subsequent positive sample supplement, difficult negative sample identification and false negative sample filtering based on the ranking result, and improving the accuracy and richness of candidate sample acquisition in the embedding data synthesis process.

[0065] In some embodiments, based on the ranking result and the reasoning model, a positive sample text block for supplement and a difficult negative sample text block similar in semantics to the search query but not belonging to the positive sample are mined from the candidate text block set, including: A plurality of candidate text blocks are selected from the ranked candidate text block sequence according to a first preset ranking interval to form a first context, the first context and the search query are input into the reasoning model, and a positive sample text block for supplement is determined from the candidate text blocks corresponding to the first context according to the reasoning model output result and a preset positive sample determination rule. A plurality of candidate text blocks are selected from the ranked candidate text block sequence according to a second preset ranking interval to form a second context, the second context and the search query are input into the reasoning model, and a difficult negative sample text block similar in semantics to the search query but not belonging to the positive sample is determined from the candidate text blocks corresponding to the second context according to the reasoning model output result and a preset negative sample determination rule.

[0066] Specifically, the positive sample mining unit first selects multiple candidate text blocks from the ranked candidate text block sequence to form a first context according to a first preset ranking interval. The first preset ranking interval can be set to a ranking section higher than the actual recall threshold according to business requirements, for example, when the downstream online retrieval usually adopts Top20 recall, the first preset ranking interval can be set to candidate text blocks ranked from 21st to 50th, which are used to focus on mining potential positive samples that are missed by the basic retrieval model but still have high relevance. The system inputs the first context and the current retrieval query into the long context reasoning model together, and the reasoning model combines the semantic intent of the retrieval query and the content of each candidate text block in the first context to comprehensively judge the semantic association between each candidate text block and the retrieval query.

[0067] To facilitate automatic labeling, the system pre-sets a positive sample determination rule, which can be determined based on the relevance label, confidence score and position of the candidate text block in the first context, etc. output by the reasoning model. For example, the reasoning model can output category labels such as “highly relevant”, “generally relevant” and “irrelevant” and corresponding confidence scores for each candidate text block in the first context. The positive sample determination rule can mark candidate text blocks that meet “highly relevant and confidence higher than a preset threshold” as positive sample text blocks for supplement. When the retrieval query belongs to a multi-hop reasoning problem, the positive sample determination rule can also allow multiple candidate text blocks with “highly relevant” labels to be combined as a cross-block positive sample.

[0068] For example, in the financial research report scenario, if the retrieval query is “What is the impact of the current tightening of monetary policy on stock market valuation?”, the text blocks ranked 25th and 30th in the ranked candidate text block sequence describe the impact of interest rate increase on discount rate and the analysis of valuation pivot moving down, respectively. The reasoning model will mark them as “highly relevant” after comprehensively analyzing the retrieval query and the content of the two text blocks, and the positive sample determination rule will add these two text blocks as positive sample text blocks for supplement to the positive sample set corresponding to the current retrieval query, thereby making up for the insufficient coverage of the initial positive samples in cross-chapter analysis.

[0069] After the completion of the supplementary positive sample mining, the difficult negative sample mining unit selects multiple candidate text blocks according to a second preset ranking interval based on the ranked candidate text block sequence to form a second context. The second preset ranking interval generally selects candidate text blocks with high preliminary similarity to the search query, such as the top 1 to the top 20 candidate text blocks, for focused mining of difficult negative samples that are highly similar in semantics to the search query but should not be answers in business. The system inputs the second context and the search query into the reasoning model, and the reasoning model analyzes the semantic content of each candidate text block in the second context at a higher resolution semantic level to determine whether the candidate text block truly answers the question involved in the search query or only shares some terms and topics but deviates in key information.

[0070] In some examples, to distinguish between true positive samples and difficult negative samples, the system pre-sets a negative sample determination rule. The negative sample determination rule can make a comprehensive judgment based on the “answer matching degree” and “key information consistency” output by the reasoning model, and the comparison result with the content of the initial positive sample and the supplementary positive sample that has been determined. For example, when the reasoning model determines that a candidate text block is highly similar to the search query in surface terms, but there is a significant mismatch in key constraint conditions such as time range, applicable object, and index direction, the negative sample determination rule will mark the candidate text block as a difficult negative sample of “semantically similar but business mismatched”.

[0071] For example, in the industrial manufacturing field, if the search query is “How to troubleshoot the frequent tripping of the frequency converter”, the top-ranked candidate text blocks include both “troubleshooting steps for the overload protection action of the frequency converter” and “paragraphs of daily maintenance and cleaning of the frequency converter”, the reasoning model will determine that the former is highly related to troubleshooting the cause of the fault and should be a positive sample or a potential positive sample, while the latter, although it contains the common term “frequency converter”, mainly describes the maintenance process and does not match the problem of “troubleshooting the tripping reason”, so the negative sample determination rule can mark this text block as a typical difficult negative sample.

[0072] In some implementations, to avoid mixing simple negative samples and difficult negative samples, the difficult negative sample mining unit can further combine the ranking position and the semantic similarity score output by the reasoning model to filter the candidate text blocks in the second context, for example, only selecting difficult negative sample text blocks from candidate text blocks that are determined by the reasoning model to be “partially related but not meet the answer conditions” and have a basic similarity score higher than a certain threshold, so as to ensure that the difficult negative samples have both semantic similarity and answer-level exclusivity, providing high-value negative samples for subsequent contrast learning.

[0073] Through the above embodiments, the system utilizes the ranked candidate text block sequence and the long context reasoning capability of the reasoning model to automatically mine the supplementary positive sample text blocks missed by the basic retrieval process within the first preset ranking interval, and automatically identify the difficult negative sample text blocks that are similar in semantics to the retrieval query but do not belong to the positive sample in business within the second preset ranking interval, thereby significantly improving the recall integrity of the positive sample and the distinguishing value of the negative sample while maintaining the integrity of the semantic structure of the knowledge base, providing more accurate, rich and structurally reasonable sample basis for subsequent false negative sample filtering and contrast learning training, which is conducive to improving the quality of embedded data synthesis and the discriminant ability of the representation model in actual retrieval tasks.

[0074] In some embodiments, the false negative sample text blocks are filtered according to the association relationship between the text blocks to form a corresponding positive sample set and a negative sample set for each retrieval query, comprising: Based on the pre-established multi-granularity text block hierarchical association relationship and sequential association relationship in the knowledge base, the association relationship between each difficult negative sample text block and the initial positive sample text block and the supplementary positive sample text block is determined; The difficult negative sample text blocks that have a predetermined relationship of parent-child inclusion with any initial positive sample text block or supplementary positive sample text block are determined as false negative sample text blocks, and the false negative sample text blocks are removed from the difficult negative sample text blocks or incorporated into the positive sample text blocks, wherein the predetermined relationship includes the parent-child inclusion relationship, the index continuous relationship and / or the cross-block combination relationship; On the basis of removing the false negative sample text blocks, the initial positive sample text blocks and the supplementary positive sample text blocks are collected to form a corresponding positive sample set of the retrieval query, and the remaining difficult negative sample text blocks and optional other negative sample text blocks are collected to form a corresponding negative sample set of the retrieval query.

[0075] Specifically, the false negative sample identification subunit first reads the hierarchical association relationship and sequential association relationship of the multi-granularity text blocks related to the current retrieval query from the knowledge base, wherein the hierarchical association relationship is used to represent the subordinate relationship between the parent text block and the child text block, and the sequential association relationship is used to represent the index continuous relationship of the child text blocks within the same parent text block and the chapter order relationship across the parent text blocks.

[0076] In the foregoing process, the initial positive sample text block is derived from the context text block explicitly labeled by the generation model, the positive sample text block for supplement is derived from the candidate text block within the first preset ranking interval determined by the inference model as highly relevant, and the difficult negative sample text block is derived from the candidate text block within the second preset ranking interval determined by the inference model as semantically similar but not matching at the answer level. The false negative sample identification subunit queries the parent text block identifier, child text block identifier, and index position in the original document for each difficult negative sample text block, and compares them with the parent-child identifiers and index positions corresponding to the initial positive sample text block and the positive sample text block for supplement.

[0077] In the comparison process, the system determines the association between the difficult negative sample text block and the positive sample text block according to a predetermined relationship. The predetermined relationship includes at least three types of parent-child inclusion relationship, index continuity relationship, and cross-block combination relationship. The parent-child inclusion relationship is used to identify whether the difficult negative sample text block and the positive sample text block belong to the same parent text block, or one is a parent text block and the other is a child text block thereof, for example, a certain difficult negative sample text block shares the same parent text block identifier or is located within the parent text block with an initial positive sample text block.

[0078] The index continuity relationship is used to identify whether the difficult negative sample text block is adjacent to the positive sample text block before or after in the original document, for example, the index of a certain difficult negative sample text block is located at the position before or after the index of a certain positive sample text block. The cross-block combination relationship is used to identify the case where multiple text blocks need to be combined in semantics to completely answer the search query. If a certain difficult negative sample text block is adjacent to one or more positive sample text blocks at the parent level and the index is continuous, and it has been determined in the foregoing inference stage that it has partial relevance to the search query, it can be considered as part of the cross-block combination relationship.

[0079] When the false negative sample identification subunit determines that a certain difficult negative sample text block meets any of the above predetermined relationships with any initial positive sample text block or positive sample text block for supplement, the system determines the difficult negative sample text block as a false negative sample text block, and according to business needs, selects to remove it from the difficult negative sample text block set, or directly integrates it into the positive sample text block set.

[0080] For example, in the financial research report scenario, the search query is “What is the impact of the current tightening of monetary policy on stock market valuation”, a certain difficult negative sample text block is located in the previous paragraph of the “interest rate adjustment” paragraph in the macroeconomic chapter, and the content is “background analysis of the recent central bank's increase in the benchmark interest rate”. The text block is within the Top10 in the sorting stage, but is marked as a difficult negative sample in the inference model judgment because it does not directly involve the “valuation” keyword.

[0081] By comparing the hierarchical relationship and the sequential relationship, the system finds that the text block shares the same parent text block identifier with a positive sample text block for supplement, and the indexes are adjacent, and the parent text block as a whole corresponds to the chapter content of “currency policy and valuation relationship”. The false negative sample identification subunit determines the difficult negative sample text block as a false negative sample according to this, and integrates it into the positive sample text block set to supplement the complete semantic coverage of the retrieval query.

[0082] After identifying and processing the false negative sample text block, the sample collection subunit collects the initial positive sample text block and the positive sample text block for supplement on the basis of eliminating the false negative sample text block, to form a positive sample set corresponding to the current retrieval query; at the same time, the difficult negative sample text blocks that are not determined as false negative samples and a part of random negative sample text blocks that can be selected to be introduced are collected to form a negative sample set corresponding to the current retrieval query. The parent-child hierarchical information and index information of the text blocks in the positive sample set can be retained to support subsequent long-context positive sample splicing or cross-block induction generation; the negative sample set eliminates the text blocks that actually belong to the answer range in terms of semantics while retaining their semantic proximity labels, to ensure that the samples in the set are clearly distinguished from the retrieval query in terms of business semantics.

[0083] Through the above embodiments, the system uses the pre-established multi-granularity text block hierarchical association relationship and sequential association relationship in the knowledge base to perform secondary filtering on the text blocks in the candidate samples that have close structural association with the positive samples but were misjudged as difficult negative samples in the previous stage, to identify them as false negative sample text blocks and eliminate or integrate them into the positive sample set, thereby effectively reducing the proportion of false negative samples in the contrast learning training, avoiding abnormal amplification of the loss function denominator and incorrect traction of the gradient direction. On this basis, the positive sample set and the negative sample set formed have more accurate labels and clearer boundaries, which is conducive to improving the sample labeling quality, stabilizing the training process of the contrast learning model, and further improving the discrimination ability of the embedding vector in the actual retrieval task.

[0084] In some embodiments, a semantic representation model is trained using contrast learning training samples to generate embedding vectors for a retrieval task, comprising: The contrast learning training samples are organized into sample groups by training batches, each sample group consisting of a retrieval query, a positive sample text block corresponding to the retrieval query, and a negative sample text block; The semantic representation model is input with each retrieval query and the corresponding positive sample text block and negative sample text block to obtain a first vector representation representing the semantic features of the retrieval query and a second vector representation representing the semantic features of each text block, respectively; The parameters of the semantic representation model are updated according to a preset contrast learning loss function based on a similarity relationship between the first vector representation and the second vector representation of the corresponding positive sample text block and the second vector representation of each negative sample text block, so that the semantic representation model can encode the retrieval query and the text block in the knowledge base after convergence and output an embedding vector for a retrieval task.

[0085] Specifically, the sample organization submodule first converts the positive sample set and the negative sample set obtained for each retrieval query into contrast learning training samples. For each retrieval query, the sample organization submodule selects one or more positive sample text blocks from the corresponding positive sample set and selects several difficult negative sample text blocks and a small number of random negative sample text blocks from the negative sample set to form a contrast learning sample record, so that the sample record contains three types of elements, i.e., “query, positive sample, and negative sample”. In batch construction, the sample organization submodule shuffles the multiple sample records from different fields and different documents, organizes them into training batches according to a preset batch size, for example, each training batch contains several financial research report retrieval samples, several industrial manufacturing manual retrieval samples, and several construction document retrieval samples, so as to ensure that the model can be exposed to sample distribution of multiple vertical fields in one parameter update during the training process and avoid excessive bias towards a single field.

[0086] After obtaining the training batch, the encoding submodule inputs each retrieval query and its corresponding positive sample text block and negative sample text block into the same semantic representation model for encoding. The semantic representation model can adopt a double-tower structure or other structures suitable for text semantic encoding, one side for encoding the retrieval query and the other side for encoding the text block. In one forward calculation process, the encoding submodule generates a corresponding first vector representation for each retrieval query and generates second vector representations for its positive sample text block and negative sample text block, so that all query vectors and text vectors in the same batch are in the same vector space. For multi-granularity text blocks, the encoding submodule can directly encode a single sub-level text block or can input the spliced long text positive sample as the encoding input after the continuous index positive sample block has been spliced in the previous stage, so that the semantic representation model can learn the complete semantic expression across blocks.

[0087] After obtaining the above-mentioned first vector representation and second vector representation, the contrast learning training submodule calculates the loss value of each training batch according to a preset contrast learning loss function. For each retrieval query, the contrast learning training submodule regards the similarity between the first vector representation of the retrieval query and the second vector representation of the corresponding positive sample text block as a target that should be “pulled closer” and regards the similarity between the first vector representation and the second vector representation of all negative sample text blocks as a target that should be “pulled away”.

[0088] The loss function considers the positive sample text block corresponding to the current query and the interference degree of the text blocks corresponding to other samples in the same batch to the query at the time of calculation. Through the constraint of strengthening the similarity between the query and the positive sample being significantly higher than the similarity between the query and the negative sample, the weight of the semantic representation model is continuously adjusted in the parameter back propagation process. Since the difficulty negative sample mining and false negative sample filtering mechanisms in the foregoing sample construction stage have ensured the effectiveness of the negative samples in the negative sample set, the contrast learning training submodule can more concentratedly use these high-value negative samples to strengthen the training of the discrimination ability of the model in the embodiment.

[0089] In the training process, the contrast learning training submodule iteratively updates the parameters of the semantic representation model according to the optimization algorithm. In each training cycle, the system sequentially takes out multiple training batches and performs the steps of “forward encoding → loss calculation → back propagation → parameter update”. To prevent the model from overfitting on a small number of frequently occurring retrieval modes, the sample organization submodule can dynamically adjust the positive and negative sample sampling ratio during the training process, for example, appropriately increasing the proportion of difficult negative samples in the negative sample set, or switching the weights of different domain samples in different training stages, so that the semantic representation model can better balance the retrieval needs of multiple vertical fields such as finance, medicine, industrial manufacturing, and construction in the later training stage. After training for several cycles, the system determines whether the semantic representation model converges according to the recall rate, accuracy, and other indicators on the validation set. When the model performance reaches the preset standard, the model parameters are fixed and used as the embedding generation model for online retrieval services.

[0090] Through the above embodiment, the system uses the structured retrieval query, positive sample text block, and negative sample text block in the contrast learning training phase to perform targeted parameter optimization on the semantic representation model, so that the model simultaneously shortens the vector distance between the query and the positive sample and lengthens the vector distance between the query and the negative sample in the unified semantic space. Based on the difficulty negative sample mining and false negative sample filtering completed in the early stage, this training method can significantly reduce the risk of the gradient direction being disturbed by false samples in the contrast learning process, improve the stability of the model training, and enhance the ability of the model to distinguish between semantically similar but business-mismatched segments. Ultimately, the embedding vector that is more suitable for vertical field retrieval tasks is obtained, and the recall effect and ranking accuracy in the actual retrieval scenario are improved.

[0091] The following is an apparatus embodiment of the present application, which can be used to execute the method embodiments of the present application. For details not disclosed in the apparatus embodiments of the present application, please refer to the method embodiments of the present application.

[0092] Figure 2 is a structural schematic diagram of an embedding data synthesis apparatus for fusion retrieval and large model distillation provided by an embodiment of the present application. As shown in Figure 2 the embedding data synthesis apparatus for fusion retrieval and large model distillation comprises: The preprocessing module 201 is configured to preprocess the unstructured document in the vertical field, divide the unstructured document into multi-granularity text blocks having a hierarchical association relationship, and construct a knowledge base based on the multi-granularity text blocks and the association relationship. The generation module 202 is configured to form a context based on the multi-granularity text blocks in the knowledge base, inject disturbance information corresponding to prior knowledge in the vertical field into the context, call a generation model to generate a retrieval query according to the context, and determine a target text block corresponding to the retrieval query as an initial positive sample. The retrieval module 203 is configured to input the retrieval query into a semantic retrieval system, retrieve a candidate text block set in the knowledge base, and sort the candidate text block set according to relevance. The mining module 204 is configured to mine, based on a sorting result and an inference model, a positive sample text block for supplementing from the candidate text block set and a difficult negative sample text block that is semantically similar to the retrieval query but does not belong to the positive sample, and filter false negative sample text blocks according to an association relationship between the text blocks, to form a corresponding positive sample set and a negative sample set for each retrieval query. The training module 205 is configured to construct a contrast learning training sample based on the positive sample set and the negative sample set, train a semantic representation model by using the contrast learning training sample, and generate an embedding vector for a retrieval task.

[0093] In some embodiments, Figure 2 The preprocessing module 201 of the application performs format cleaning and content extraction on the unstructured document to obtain continuous text to be processed, divides the continuous text into parent text blocks as a first granularity based on directory titles and natural paragraph boundaries, divides the parent text blocks according to preset length rules and structure rules to obtain at least one child text block of different granularities, and establishes a subordinate association relationship between the parent text block and the corresponding child text block, generates association information representing the hierarchy and order of the multi-granularity text blocks based on the subordinate association relationship and the order information of each text block in the original document, and stores the multi-granularity text blocks and the association information in the knowledge base.

[0094] In some embodiments, Figure 2 The generation module 202 of the application selects a plurality of multi-granularity text blocks having an association relationship from the knowledge base according to a preset window rule, combines the plurality of multi-granularity text blocks into a context for generating a retrieval query, determines a disturbance type corresponding to the prior knowledge in the target vertical field based on the prior knowledge in the vertical field, generates disturbance information according to the disturbance type, and injects the disturbance information into the context according to a preset strategy to obtain a disturbance context carrying a feature of the prior knowledge in the vertical field.

[0095] In some embodiments, Figure 2The generation module 202 constructs a generation model set including a plurality of generation models, and selects a target generation model from the generation model set according to a preset selection rule; inputs the context after the disturbance processing into the target generation model to generate at least one search query corresponding to the context, and obtains association information representing the corresponding relationship between the search query and each multi-granularity text block in the context; determines a target text block corresponding to the search query from the multi-granularity text blocks contained in the context according to the association information and a preset association determination rule, and takes the target text block as an initial positive sample of the search query.

[0096] In some embodiments, Figure 2 The mining module 204 selects a plurality of candidate text blocks according to a first preset ranking interval based on the sorted candidate text block sequence to form a first context, inputs the first context and the search query into the reasoning model, and determines a positive sample text block for supplementing from the candidate text blocks corresponding to the first context according to the reasoning model output result and a preset positive sample determination rule; selects a plurality of candidate text blocks according to a second preset ranking interval based on the sorted candidate text block sequence to form a second context, inputs the second context and the search query into the reasoning model, and determines a difficult negative sample text block that is similar in semantics to the search query but does not belong to the positive sample from the candidate text blocks corresponding to the second context according to the reasoning model output result and a preset negative sample determination rule.

[0097] In some embodiments, Figure 2 The mining module 204 determines the association relationship between each difficult negative sample text block and the initial positive sample text block and the positive sample text block for supplementing based on the hierarchical association relationship and the sequential association relationship of the multi-granularity text blocks pre-established in the knowledge base; determines a false negative sample text block from the difficult negative sample text blocks that have a predetermined relationship of parent-child inclusion with any initial positive sample text block or positive sample text block for supplementing, and removes or incorporates the false negative sample text block into the positive sample text block from the difficult negative sample text blocks, wherein the predetermined relationship includes a parent-child inclusion relationship, an index continuous relationship, and / or a cross-block combination relationship; based on the removal of the false negative sample text block, the initial positive sample text block and the positive sample text block for supplementing are collected to form a positive sample set corresponding to the search query, and the remaining difficult negative sample text block and optional other negative sample text block are collected to form a negative sample set corresponding to the search query.

[0098] In some embodiments, Figure 2The training module 205 organizes the contrastive learning training samples into sample groups by training batches, where each sample group is composed of a retrieval query, a positive sample text block corresponding to the retrieval query, and negative sample text blocks; inputs each retrieval query and the corresponding positive sample text block and negative sample text blocks into the semantic representation model to obtain a first vector representation representing semantic features of the retrieval query and second vector representations representing semantic features of the text blocks, respectively; and updates parameters of the semantic representation model according to a preset contrastive learning loss function based on similarity relationships between the first vector representation and the second vector representation of the corresponding positive sample text block and the second vector representations of the negative sample text blocks, so that the semantic representation model can encode the retrieval query and the text blocks in the knowledge base after convergence and output embedding vectors for a retrieval task.

[0099] It should be understood that the size of the serial number of each step in the above embodiments does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0100] Figure 3 is a schematic diagram of an electronic device 3 provided by an embodiment of the present application. As shown in Figure 3 The electronic device 3 of this embodiment includes a processor 301, a memory 302, and a computer program 303 stored in the memory 302 and executable on the processor 301. The processor 301 implements the steps in each of the above method embodiments when executing the computer program 303. Alternatively, the processor 301 implements the functions of each module / unit in each of the above device embodiments when executing the computer program 303.

[0101] The electronic device 3 can be a desktop computer, a notebook computer, a palm computer, a cloud server, or the like. The electronic device 3 can include but is not limited to the processor 301 and the memory 302. Those skilled in the art can understand that Figure 3 The electronic device 3 is merely an example and does not constitute a limitation on the electronic device 3, which can include more or fewer components or different components than those shown.

[0102] The processor 301 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gates or transistor logic components, discrete hardware components, etc.

[0103] The memory 302 can be an internal storage unit of the electronic device 3, for example, a hard disk or a memory of the electronic device 3. The memory 302 can also be an external storage device of the electronic device 3, for example, a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. equipped on the electronic device 3. The memory 302 can also include both the internal storage unit and the external storage device of the electronic device 3. The memory 302 is used to store computer programs and other programs and data required by the electronic device.

[0104] It should be clearly understood by those skilled in the art that, for the convenience and brevity of description, only the above division of functional units and modules is exemplified, and in actual application, the above functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0105] The integrated module / unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a readable storage medium (for example, a computer readable storage medium). Based on this understanding, all or part of the processes in the above-mentioned embodiment methods can also be completed by a computer program instructing related hardware, and the computer program can be stored in a computer readable storage medium. When the processor executes the computer program, the steps of each method embodiment described above can be implemented. The computer program can include computer program code, which can be in the form of source code, object code, executable file or some intermediate form, etc. The computer readable storage medium can include any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier wave signal, telecommunication signal and software distribution medium, etc.

[0106] The above examples are only used to illustrate the technical solutions of the present application, but not limit the same; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that the technical solutions recorded in the foregoing examples can be modified, or some technical features can be replaced by equivalent ones; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. An embedded data synthesis method fusing retrieval with large model distillation, characterized in that, The method comprises the following steps: Preprocessing an unstructured document in a vertical field, dividing the unstructured document into multi-granularity text blocks with hierarchical association relationships, and constructing a knowledge base based on the multi-granularity text blocks and the association relationships; Based on the multi-granularity text block combination in the knowledge base, a context is formed, perturbation information corresponding to prior knowledge in the vertical field is injected into the context, a generation model is called to generate a retrieval query according to the context, and a target text block corresponding to the retrieval query is determined as an initial positive sample; Inputting the retrieval query into a semantic retrieval system, retrieving a candidate text block set in the knowledge base, and sorting the candidate text block set according to the relevance; Based on the sorting result and an inference model, a positive sample text block for supplement is mined from the candidate text block set, and a difficult negative sample text block similar in semantics to the retrieval query but not belonging to the positive sample is mined from the candidate text block set, and false negative sample text blocks are filtered according to the association relationship between the text blocks, so as to form a corresponding positive sample set and a negative sample set for each retrieval query; Based on the positive sample set and the negative sample set, a contrast learning training sample is constructed, and a semantic representation model is trained using the contrast learning training sample to generate an embedding vector for a retrieval task.

2. The method of claim 1, wherein, The method comprises the following steps: Performing format cleaning and content extraction on the unstructured document to obtain continuous text to be processed; Based on directory titles and natural paragraph boundaries, the continuous text is divided into parent text blocks as a first granularity; According to a preset length rule and a structure rule, the parent text blocks are divided to obtain child text blocks of different granularities, and a subordinate association relationship is established between the parent text blocks and the corresponding child text blocks; Based on the subordinate association relationship and the order information of each text block in the original document, association information representing the hierarchy and order of the multi-granularity text blocks is generated; The multi-granularity text blocks and the association information are stored in the knowledge base.

3. The method of claim 1, wherein, The method comprises the following steps: According to a preset window rule, a plurality of multi-granularity text blocks with association relationships are selected from the knowledge base, and the plurality of multi-granularity text blocks are combined into a context for generating a retrieval query; Based on prior knowledge in a target vertical field, a perturbation type corresponding to the prior knowledge is determined, and perturbation information is generated according to the perturbation type; According to a preset strategy, the perturbation information is injected into the context to obtain a perturbed context carrying the characteristics of the prior knowledge in the vertical field.

4. The method of claim 3, wherein, The method comprises the following steps: A generation model set comprising a plurality of generation models is constructed, and a target generation model is selected from the generation model set according to a preset selection rule; inputting the perturbed context into the target generation model, generating at least one search query corresponding to the context, and obtaining association information representing a corresponding relationship between the search query and each multi-granularity text block in the context; determining a target text block corresponding to the search query from the multi-granularity text blocks included in the context according to the association information and a preset association determination rule, and taking the target text block as an initial positive sample of the search query.

5. The method of claim 1, wherein, The mining of the positive sample text blocks for supplement and the difficult negative sample text blocks similar in semantics to the search query but not belonging to the positive samples from the candidate text block set based on the ranking result and the inference model includes: selecting a plurality of candidate text blocks according to a first preset ranking interval based on the ranked candidate text block sequence to form a first context, inputting the first context and the search query into the inference model, and determining the positive sample text blocks for supplement from the candidate text blocks corresponding to the first context according to the inference model output result and a preset positive sample determination rule; selecting a plurality of candidate text blocks according to a second preset ranking interval based on the ranked candidate text block sequence to form a second context, inputting the second context and the search query into the inference model, and determining the difficult negative sample text blocks similar in semantics to the search query but not belonging to the positive samples from the candidate text blocks corresponding to the second context according to the inference model output result and a preset negative sample determination rule.

6. The method of claim 5, wherein, The filtering of the false negative sample text blocks according to the association relationship between the text blocks to form a corresponding positive sample set and a negative sample set for each search query includes: determining the association relationship between each difficult negative sample text block and the initial positive sample text block and the positive sample text block for supplement based on the hierarchical association relationship and the sequential association relationship of the multi-granularity text blocks pre-established in the knowledge base; determining the difficult negative sample text blocks having a predetermined parent-child containing relationship with any initial positive sample text block or positive sample text block for supplement as false negative sample text blocks, and removing or incorporating the false negative sample text blocks from the difficult negative sample text blocks into the positive sample text blocks, wherein the predetermined relationship includes a parent-child containing relationship, an index continuous relationship, and / or a cross-block combination relationship; on the basis of removing the false negative sample text blocks, collecting the initial positive sample text blocks and the positive sample text blocks for supplement to form a positive sample set corresponding to the search query, and collecting the remaining difficult negative sample text blocks and optional other negative sample text blocks to form a negative sample set corresponding to the search query.

7. The method of claim 1, wherein, The training of the semantic representation model using the contrast learning training samples to generate embedding vectors for a retrieval task includes: organizing the contrast learning training samples into sample groups by retrieval queries, positive sample text blocks corresponding to the retrieval queries, and negative sample text blocks according to training batches; inputting each retrieval query and the corresponding positive sample text blocks and negative sample text blocks into the semantic representation model to obtain a first vector representation representing semantic features of the retrieval query and a second vector representation representing semantic features of each text block, respectively; Based on the similarity relationship between the first vector representation and the second vector representation of the corresponding positive sample text block and the second vector representation of each negative sample text block, the parameters of the semantic representation model are updated according to a preset contrast learning loss function, so that the semantic representation model can encode the retrieval query and the text block in the knowledge base after convergence, and output an embedding vector for a retrieval task.

8. An embedded data synthesis apparatus that fuses retrieval with large model distillation, characterized by, Comprise: A preprocessing module for preprocessing unstructured documents in a vertical field, dividing the unstructured documents into multi-granularity text blocks with hierarchical association relationships, and constructing a knowledge base based on the multi-granularity text blocks and the association relationships; A generation module for combining multi-granularity text blocks in the knowledge base to form a context, injecting perturbation information corresponding to prior knowledge in the vertical field into the context, calling a generation model to generate a retrieval query according to the context, and determining a target text block corresponding to the retrieval query as an initial positive sample; A retrieval module for inputting the retrieval query into a semantic retrieval system, retrieving a candidate text block set in the knowledge base, and sorting the candidate text block set according to relevance; A mining module for mining positive sample text blocks for supplement and difficult negative sample text blocks that are semantically similar to the retrieval query but do not belong to positive samples from the candidate text block set based on the sorting result and an inference model, and filtering false negative sample text blocks according to the association relationship between text blocks to form a corresponding positive sample set and a negative sample set for each retrieval query; A training module for constructing contrast learning training samples based on the positive sample set and the negative sample set, training a semantic representation model using the contrast learning training samples to generate an embedding vector for a retrieval task.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and running on the processor, characterized in that, The processor executes the computer program to implement the steps of the method of any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 9. The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 7.

Images