Hard negative example mode based embedding model fine-tuning method, system, device and medium

Abstract

The application relates to a hard negative example mode-based embedded model fine-tuning method, system, device and medium. The method comprises the following steps: collecting, cleaning and semantically slicing vertical industry unstructured text data to obtain a standardized text segment set; generating query text corresponding to the semantic of the text segment in the set, and taking the original text segment as a positive example sample; screening a semantic conflict type hard negative example which is highly similar in literal form but inconsistent in business semantics with the query text from the set by using a keyword matching retrieval algorithm and semantic consistency discrimination; assembling the query text, the positive example and the hard negative example into a structured training sample to construct a data set; and fine-tuning the embedded model based on the data set, and giving the semantic conflict type hard negative example a higher weight during training. By introducing the semantic conflict type hard negative example which is similar in literal form but wrong in business semantics to the query text in the training stage, the vector space semantic distribution is explicitly constrained, so that the embedded model can accurately distinguish the text which is high in similarity but wrong in semantics in the vertical field.

Description

Technical Field

[0001] This invention relates to the field of intelligent coal mining technology; specifically, it relates to a method, system, device, and medium for fine-tuning an embedded model based on hard negative example patterns. Background Technology

[0002] With the development of large language model technology, Retrieval Enhancement Generation (RAG) technology based on "vector retrieval + large model generation" has become an important technical path for scenarios such as industry knowledge question answering, intelligent operation and maintenance, and decision support. This type of system typically vectorizes text data through an embedding model and builds a vector knowledge base. When a user asks a question, relevant knowledge fragments are retrieved based on vector similarity, and then a large model generates a generative answer. For example, in coal production enterprises, this technology is widely used for intelligent retrieval and question answering of coal mining equipment structure descriptions, fault location, maintenance procedures, safety regulations, and operation and maintenance experience.

[0003] However, most existing RAG systems directly adopt general embedding models. These models are usually trained on general corpora and struggle to understand specialized expressions in vertical industries that have highly similar keywords but significant semantic differences, such as the numerous equipment model abbreviations, process terms, and structural component names found in the coal industry. For example, in equipment operation and maintenance scenarios, texts such as "traction unit failure" and "abnormal traction control parameters," and "abnormal gearbox vibration" and "abnormal gearbox lubrication" are highly similar at the literal and keyword level, but they differ fundamentally in actual semantics and processing solutions.

[0004] Therefore, in vertical industry scenarios, models often fail to effectively distinguish highly similar interfering texts, resulting in disordered search results and seriously affecting the reliability and engineering usability of subsequent large-scale model-generated results. Summary of the Invention

[0005] In view of this, the present invention provides a method, system, device and medium for fine-tuning an embedded model based on a hard negative example pattern, thereby solving or at least alleviating one or more of the above-mentioned problems and other problems existing in the prior art.

[0006] To achieve the aforementioned objective, a first aspect of the present invention provides a method for fine-tuning an embedding model based on a hard negative example pattern, comprising the following steps:

[0007] Collect, clean, and semantically segment unstructured text data from vertical industries to obtain a standardized set of text fragments;

[0008] Based on the text fragments in the standardized text fragment set, a query text corresponding to its semantics is generated, and the text fragments are used as positive examples. The query text is used to simulate the way users ask questions in a knowledge retrieval scenario.

[0009] Constructing semantically conflicting hard negative examples includes: using a keyword-matching retrieval algorithm to select a predetermined number of text fragments from the standardized text fragment set that have the highest literal relevance to the query text, forming a set of candidate text fragments; performing semantic filtering on the candidate text fragments based on a semantic consistency discrimination mechanism to identify candidate text fragments that are highly similar to the query text at the literal level but inconsistent with the business semantics, and using them as the semantically conflicting hard negative examples;

[0010] Based on the query text and its corresponding positive and hard negative samples, a structured training sample is assembled to construct a dataset for fine-tuning the embedding model; wherein, the hard negative samples include general hard negative samples and semantically conflicting hard negative samples;

[0011] The embedding model is fine-tuned based on the dataset, and during the contrastive learning training process, the semantically conflicting hard negative examples are given a higher weight than the general hard negative examples.

[0012] In the method described above, optionally, the vertical industry is the coal industry;

[0013] The semantic consistency judgment mechanism combines coal industry knowledge rules or fault semantic relationships to judge the semantics of the text;

[0014] The steps for collecting, cleaning, and semantically segmenting unstructured text data include:

[0015] Collect unstructured text data from coal production enterprises. The unstructured text data includes at least one of the following: equipment manuals, fault cases, maintenance procedures, and operation manual texts.

[0016] The collected text is cleaned and segmented into standardized short text fragments according to semantic integrity.

[0017] In the method described above, optionally, the step of generating query text corresponding to its semantics based on the text fragments in the standardized text fragment set includes:

[0018] The text fragment is semantically understood using a large language model and multiple candidate query texts are generated. The multiple candidate query texts are ranked based on vector similarity or human intervention, and the candidate query texts with the highest ranking are selected as the query texts used to represent the semantics of the text fragment.

[0019] or

[0020] The rule engine is used to extract key information from the text fragment, and the rule engine is used to construct a query text generation template based on vertical industry business semantic rules. Then, a query text is generated based on the query text generation template to represent the semantics of the text fragment.

[0021] In the method described above, optionally, the semantically conflicting hard negative example samples are filtered based on the semantic consistency discrimination mechanism, including: semantic discrimination based on a large language model, or semantic discrimination based on rules or knowledge graphs, or multi-strategy fusion discrimination; wherein,

[0022] The semantic discrimination based on the large language model includes: inputting the query text and the candidate text fragments into the large language model for processing, determining the semantic relationship between the query text and the candidate text fragments, and filtering out text fragments with inconsistent or conflicting semantics as the semantically conflicting hard negative sample, wherein the semantic relationship includes at least one of the following: whether the semantics are consistent, whether the semantics are related, and whether there is a semantic conflict.

[0023] The rule-based or knowledge graph-based semantic discrimination includes: combining vertical industry knowledge rules or semantic relation databases to constrain and judge the semantics of text; when a candidate text fragment violates the knowledge rules or the semantic relation, the candidate text fragment is judged as the semantic conflict type hard negative example sample.

[0024] The multi-strategy fusion discrimination includes: combining keyword matching, vector similarity, and semantic discrimination results for judgment.

[0025] In the method described above, optionally, the keyword matching-based retrieval algorithm is the BM25 algorithm, and the keyword weight and relevance threshold of the BM25 algorithm are adjusted based on the business semantic features of the vertical industry.

[0026] In the method described above, optionally, the general hard negative example samples are obtained based on keyword matching, or based on vector similarity, or based on the candidate text fragment set; wherein,

[0027] Obtaining general hard negative sample based on keyword matching includes: calculating the relevance score between the query text and each text segment in the standardized text segment set using the keyword relevance function; selecting a preset number of text segments with high relevance scores that are not positive sample; and taking the text segments among the selected text segments that are not identified as semantically conflicted by the semantic consistency discrimination mechanism as the general hard negative sample.

[0028] Obtaining general hard negative sample based on vector similarity includes: calculating the vector similarity between the query text and each text segment in the standardized text segment set through an embedding model, and selecting a preset number of text segments with high vector similarity that are not positive samples as the general hard negative sample;

[0029] Obtaining general hard negative examples based on the candidate text fragment set includes: in the candidate text fragment set, selecting candidate text fragments that are neither positive examples nor semantically conflicting hard negative examples as general hard negative examples.

[0030] In the method described above, optionally, the dataset used for fine-tuning the embedded model further includes random negative samples, which are text segments randomly extracted from text segments other than the positive samples; during the contrastive learning training process, the random negative samples are given a weight lower than that of the semantically conflicting hard negative samples and the general hard negative samples.

[0031] In the method described above, optionally, the step of fine-tuning the embedding model includes:

[0032] Input the query text, positive examples, and hard negative examples into the embedding model to obtain the query vector, positive example vector, and hard negative example vector, respectively.

[0033] The loss value is calculated using the multi-negative-example ranking loss function, and the model parameters are optimized so that the distance between the query vector and the positive example vector in the vector space is less than the distance between the query vector and the hard negative example vector.

[0034] In the method described above, optionally, during the fine-tuning of the embedded model, at least one of gradient checkpointing, mixed precision training, or gradient accumulation techniques is used to optimize training efficiency.

[0035] The step size of the gradient accumulation is dynamically adjusted according to the batch screening quantity of the hard negative samples, and the precision threshold of the mixed precision training is set based on the encoding features of the vertical industry text.

[0036] Optionally, in the method described above, the method further includes:

[0037] The performance of the fine-tuned embedding model was evaluated.

[0038] The evaluated embedding models are deployed to vector retrieval or retrieval enhancement generation systems to vectorize text data and build vector knowledge bases.

[0039] To achieve the aforementioned objective, a second aspect of the present invention provides an embedding model fine-tuning system based on hard negative example patterns, comprising:

[0040] The data acquisition and slicing module is used to collect, clean, and semantically slice unstructured text data from vertical industries, generating a standardized set of text fragments.

[0041] The query generation module, connected to the data acquisition and slicing module, is used to automatically generate query text corresponding to the semantics of text fragments in the standardized text fragment set. The query text is used to simulate the way users ask questions in a knowledge retrieval scenario, and the text fragments are used as positive example samples.

[0042] The hard negative example mining module is connected to the data acquisition and slicing module and the query generation module, respectively, and is used to: use a keyword matching-based retrieval algorithm and a semantic consistency discrimination mechanism to select text fragments from the standardized text fragment set that are highly similar to the query text at the literal level but inconsistent at the business semantic level, as semantic conflict type hard negative example samples.

[0043] The training data construction module, connected to the hard negative example mining module, is used to assemble the query text and its corresponding positive example samples, general hard negative example samples and semantically conflicting hard negative example samples into structured training samples, forming a dataset for embedding model fine-tuning.

[0044] An embedded model fine-tuning module is connected to the training data construction module and is used to apply weighted contrastive training to the semantically conflicting hard negative examples.

[0045] To achieve the foregoing objectives, a third aspect of the present invention provides an electronic device including a memory and a processor connected to the memory, wherein the memory stores a computer program executable by the processor, and the processor executes the computer program to implement the method as described in any one of the foregoing first aspects.

[0046] To achieve the foregoing objectives, a fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any one of the first aspects above.

[0047] The embedding model fine-tuning method based on hard negative example pattern of the present invention introduces semantically conflicting hard negative example samples that are literally similar to the query text but have business semantic errors during the training phase, and explicitly constrains the semantic distribution of the vector space, so that the embedding model can accurately distinguish text content with high similarity but semantic errors in the vertical domain.

[0048] The present invention further provides an embedded model fine-tuning system based on hard negative example pattern, an electronic device, and a computer-readable storage medium. The system, device, and medium have the same inventive concept as the aforementioned method, and therefore the system, device, and medium also have the above-mentioned advantages. Attached Figure Description

[0049] The disclosure of this invention will become more apparent from the accompanying drawings. It should be understood that these drawings are for illustrative purposes only and are not intended to limit the scope of protection of this invention. In the drawings:

[0050] Figure 1 This is a flowchart illustrating an embodiment of the embedding model fine-tuning method based on the hard negative example pattern of the present invention.

[0051] Figure 2 This is a schematic block diagram of an embodiment of the embedded model fine-tuning system based on the hard negative example pattern of the present invention.

[0052] Figure labels: 101-Data acquisition and slicing module; 102-Query generation module; 103-Hard negative example mining module; 104-Training data construction module; 105-Embedding model fine-tuning module; 106-Model evaluation and deployment module. Detailed Implementation

[0053] Referring to the accompanying drawings and specific embodiments, the structure, composition, features, and advantages of the embedded model fine-tuning method, system, device, and medium based on the hard negative example pattern of the present invention will be described by way of example below. However, all descriptions should not be construed as limiting the present invention in any way.

[0054] Furthermore, for any single technical feature described or implied in the embodiments mentioned herein, or any single technical feature shown or implied in the various figures, the present invention still allows for any combination or deletion of these technical features (or their equivalents) without any technical obstacle, and thus these further embodiments according to the present invention should also be considered within the scope of this description.

[0055] It should also be noted that, unless otherwise explicitly specified and limited, the terms "connection" and "linked" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; a mechanical connection or an electrical connection; a wired connection or a wireless connection; a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0056] In the description of this invention, "Embedding Model" refers to a technique that maps discrete symbols (such as words, sentences, and documents) to a continuous vector space, making semantically similar texts closer together in the vector space. "RAG" (Retrieval-Augmented Generation) refers to a technical architecture that combines a retrieval system and a large language model, enhancing the generative capabilities of the large model by retrieving external knowledge bases.

[0057] Figure 1 This is a flowchart illustrating an embedding model fine-tuning method based on the hard negative example pattern, provided by an embodiment of the present invention.

[0058] This method is mainly applied to knowledge base construction and retrieval optimization scenarios in vertical industries (such as the coal industry), specifically in the vector retrieval stage of the Large Model Retrieval Enhancement Generation (RAG) system, aiming to solve the problem that general Embedding models cannot distinguish between literally similar but semantically different professional terms.

[0059] like Figure 1 As shown, the method includes:

[0060] Step S1: Collect, clean, and semantically slice unstructured text data from vertical industries to obtain a standardized set of text fragments.

[0061] In this embodiment, the vertical industry is specifically taken as the coal industry. Since the data sources for coal production enterprises are complex, step S1 can specifically include the following steps S11-S13:

[0062] Step S11 involves collecting unstructured text data from coal production enterprises. This data typically includes equipment manuals, failure case studies, maintenance procedures, and operation manuals. These texts contain a large amount of professional knowledge, such as equipment model abbreviations, process terms, and names of structural components.

[0063] Step S12: Clean the collected text to remove garbled characters, irrelevant symbols, headers and footers, and other noisy data.

[0064] Step S13: Segment the cleaned text into standardized short text chunks according to semantic integrity. Segmentation can be based on semantic units such as paragraphs and periods, while ensuring that the context within each chunk is relatively complete, resulting in a set of standardized text chunks, i.e., a set of text slices.

[0065] Step S2: Generate query text corresponding to the semantics of the text fragments in the standardized text fragment set, and use the text fragments as positive samples.

[0066] To construct training pairs for supervised learning, user questions corresponding to text fragments need to be generated. In step S2, the query text is used to simulate the questioning style of real users (such as operations and maintenance personnel or managers) in a knowledge retrieval scenario. The text fragments used as positive examples are the standard answers or the most relevant documents for that query text.

[0067] For example, the methods for generating query text include, but are not limited to, the following two:

[0068] Method 1: Generation using a Large Language Model (LLM). The text fragment is input into the LLM, and the model is instructed by prompt words, such as: "Based on the following coal equipment document fragment, generate a question a user might ask that can be answered by this fragment." This generates query text that characterizes the semantics of the text fragment.

[0069] Optionally, multiple candidate query texts can be generated and sorted. The top-ranked candidate query texts are selected as the actual query texts. The sorting can be based on vector similarity or can be manually intervened. By using a large language model to perform semantic understanding on text fragments and generate multiple candidate query texts, and then sorting the candidate query texts based on similarity, the query texts with the highest semantic matching degree and the most accurate expression can be selected, ensuring that the query texts can accurately represent the semantics of the text fragments. At the same time, using these precise query texts can provide an accurate semantic benchmark for subsequent mining of hard negative examples from standardized text fragments, improving the accuracy and effectiveness of subsequent hard negative example mining, making the mined hard negative examples more likely to form semantic conflicts and more distinguishable.

[0070] Method 2: Utilizing a rule engine. Key information (e.g., equipment name: coal mining machine, fault symptom: high temperature) is extracted from text fragments using Natural Language Processing (NLP) tools, and query text is generated based on a pre-defined question template (e.g., "What should I do if {equipment name} experiences {fault symptom}?"). The question template (i.e., the query text generation template) can be constructed by the rule engine based on business semantic rules specific to a vertical industry (e.g., the coal industry).

[0071] Step S3: Construct semantically conflicting hard negative examples. Exemplarily, this includes the following two stages:

[0072] The first stage of candidate recall involves using a keyword-based retrieval algorithm to select a set of text fragments from a standardized set of text fragments that are highly relevant to the literal meaning of the query text, and using these as the candidate text fragment set.

[0073] The second stage of semantic conflict screening: Based on the semantic consistency discrimination mechanism, the candidate text fragments are semantically screened to identify candidate text fragments that are highly similar to the query text at the literal level (literal similarity is because they have passed the first stage of screening) but are inconsistent with the business semantics, and these are used as hard negative examples of semantic conflict.

[0074] Traditional training often uses randomly sampled text as negative examples, but this leads to overly simplistic model training that fails to learn fine-grained semantic differences. This step aims to discover hard negative examples—samples that are likely to confuse the model. Furthermore, this step goes beyond word matching and introduces semantic constraints, elevating the process from "relevance screening" to "semantic adversarial construction."

[0075] Because this type of keyword-matching retrieval algorithm focuses on literal matching rather than deep semantics, its results often contain a large number of seemingly related but semantically incorrect text fragments. This step uses relevance ranking and deduplication strategies to filter out the top-ranked candidate text fragments, and further filters out semantically conflicting hard negative examples through a semantic consistency discrimination mechanism.

[0076] The aforementioned semantically conflicting hard negative sample is defined as: a text fragment that has a high literal similarity to the query text at the textual expression level, but has a significant semantic inconsistency or semantic conflict with the corresponding positive sample at the business semantic level.

[0077] Specifically, such samples meet the following conditions: (1) they have a high degree of overlap with the query text in terms of keywords or expression; (2) they do not meet the query intent in terms of business semantics, or express meanings that are opposite to or unrelated to positive examples; (3) they are easily misjudged as similar samples by traditional embedding models in semantic space.

[0078] This step also involves general hard negative examples, which are text fragments that are semantically related to the query text but are not the correct answer to the query text. The general hard negative examples described in this embodiment of the invention usually meet the following characteristics: (1) they overlap with the query text at the keyword or topic level; (2) they are semantically related to the query intent but do not constitute a correct matching relationship; (3) they have a certain similarity to the positive example samples in the vector space but do not belong to a semantic conflict or semantic opposition relationship.

[0079] Compared with traditional negative samples, the above samples show that: random negative samples have obvious semantic differences and are easy to distinguish; general hard negative samples only have a certain correlation; while semantically conflicting hard negative samples have the characteristics of "high surface similarity + semantic error", and belong to more adversarial negative samples.

[0080] An exemplary specific screening step for step S3 is as follows:

[0081] Step S31, calculate the relevance score: using the keyword relevance function, calculate the relevance score between the query text and each text fragment in the standardized text fragment set.

[0082] Let the set of standardized text fragments be:

[0083]

[0084] For query text The relevance score between each text fragment in the set and the query text is calculated using the relevance function of the keyword retrieval algorithm.

[0085]

[0086] in, This is a keyword relevance function. For text fragments With query text The literal relevance score.

[0087] In this embodiment, the keyword relevance function can be selected using the BM25 algorithm (Best Matching 25). BM25 is a retrieval algorithm based on a probabilistic retrieval model used to calculate relevance. It primarily measures the literal matching degree between the query term and the document (term frequency, inverse document frequency, etc.), without focusing on deep semantics. The basic formula of the BM25 algorithm is as follows:

[0088]

[0089] Wherein, Score(D,Q) is the relevance score between document D and query Q, which can be used as the keyword relevance function in this step. When applied, document D and query Q correspond to the text fragment and query text mentioned above, respectively. It queries the i-th word in Q; It is a word Frequency in document D; It is a word The inverse document frequency; |D| is the length of document D; avgdl is the average length of all documents; Both 'k1' and 'b' are adjustable parameters that can be set according to specific needs. For example, if k1 is between 1.2 and 2, b can be set to 0.75.

[0090] Optionally, according to some embodiments, when using the BM25 algorithm, it is also specifically adjusted based on the business semantic characteristics of the vertical industry, such as specifically adjusting the keyword weights and relevance thresholds of the BM25 algorithm. Taking the coal industry as an example, according to optional embodiments, higher weights can be assigned to core business keywords such as "fault," "maintenance," and "cutting" in this industry.

[0091] Step S32, constructing a candidate set: Sort each text segment according to its relevance score from high to low, and select a preset number T (e.g., T=100) of the top-ranked text segments to form a candidate set. That is, the set of candidate text fragments:

[0092]

[0093] Since the retrieval algorithm used in step S31 (e.g., BM25) is based on literal matching, these top-ranking documents often contain a large number of keywords from the query text.

[0094] Step S33, Filter semantically conflicting hard negative examples: Remove positive examples corresponding to the query text from the candidate set. (That is, the text fragments used to generate the query text in step S2). Although the remaining text fragments highly overlap with the query text in terms of keywords, they are not the actual answers. In some optional embodiments, these remaining text fragments are directly used as hard negative example samples to obtain a set of hard negative example samples. :

[0095]

[0096] According to some alternative embodiments, semantic consistency discrimination mechanisms can be further used to filter semantically conflicting hard negative examples. This semantic consistency discrimination mechanism combines the characteristics of a vertical industry to judge the semantics of the text. For example, this embodiment targets the coal industry, and the semantic consistency discrimination mechanism combines coal industry knowledge rules or fault semantic relationships to judge the semantics of the text.

[0097] According to some embodiments, the semantic consistency discrimination mechanism described above includes, but is not limited to, the following exemplary implementations:

[0098] (1) Semantic discrimination based on large language model

[0099] The query text and candidate text fragments are jointly input into a large language model for processing. The large language model determines the semantic relationship between the two, which may include whether they are semantically consistent, semantically related, or semantically conflicting. Text fragments that are semantically inconsistent or conflicting are selected as semantically conflicting hard negative examples.

[0100] (2) Semantic discrimination based on rules or knowledge graphs

[0101] By combining vertical industry knowledge rules or semantic relation databases, the semantics of the text are constrained and judged. For example, in the coal industry, the causal relationship between the fault phenomenon and the fault cause, the logical relationship between the operation behavior and the equipment status, and the opposition relationship between the normal state and the abnormal state are considered. When a candidate text fragment violates the above-mentioned semantic relations or knowledge rules, it is judged as a semantically conflicting hard negative example sample.

[0102] (3) Multi-strategy fusion discrimination

[0103] According to optional embodiments, keyword matching, vector similarity, and semantic discrimination results can be combined for comprehensive judgment to improve the accuracy of semantic conflict identification.

[0104] For example, if the query text is "How to troubleshoot a coal mining machine traction unit malfunction?", a positive example sample would be "A coal mining machine traction unit malfunction is usually caused by insufficient hydraulic system pressure. The troubleshooting steps are as follows...". A typical hard negative example sample would be "A coal mining machine traction unit malfunction may be caused by a significant drop in hydraulic system pressure...". A semantically conflicting hard negative example sample would be "Instructions for setting control parameters for a coal mining machine traction unit...". This semantically conflicting hard negative example sample contains "coal mining machine" and "traction unit," but it discusses parameter settings rather than troubleshooting. In this way, the semantically conflicting hard negative examples collected are those literally similar but semantically incorrect distractors.

[0105] Because the semantically conflicting hard negative examples in this step are not randomly selected, but rather are the most easily misjudged distractors, these semantically conflicting hard negative examples are semantically similar to the query text but have key deviations, effectively exposing the model's judgment boundaries. By including such samples in training, the model can be forced to learn to distinguish subtle semantic differences, improving discrimination accuracy.

[0106] As for the general hard negative example introduced in this step, depending on the different embodiments, its acquisition methods include, but are not limited to, the following:

[0107] (1) Elimination method based on candidate text fragment set

[0108] In the process of obtaining semantically conflicting hard negative examples, general hard negative examples are also obtained simultaneously. Specifically, in the candidate text fragment set mentioned above in this step, positive examples and semantically conflicting hard negative examples are removed, and the remaining samples can be used as general hard negative examples.

[0109] (2) Keyword matching-based acquisition

[0110] The relevance score between the query text and standardized text fragments is calculated using a keyword relevance function. Text fragments with higher rankings but not identified as positive examples are selected, and those text fragments that fail to pass the semantic consistency test to identify semantic conflicts are used as general hard negative examples. The keyword relevance function and semantic consistency judgment mechanism in this method can be selected as needed, and can be the same as when obtaining semantic conflict type hard negative examples, or they can be different. This embodiment of the invention does not impose any restrictions on this.

[0111] (3) Obtaining based on vector similarity

[0112] The query text and each standardized text fragment are vectorized using an embedding model, and the similarity between the text vectors is calculated. Text fragments that are close to the query text vector but are not positive examples are selected as general hard negative examples.

[0113] Step S4: Assemble the query text and its corresponding positive samples and hard negative samples (including semantically conflicting hard negatives and general hard negatives) into structured training samples to build a dataset for fine-tuning the Embedding model.

[0114] The constructed dataset can typically be in the form of a quadruple of "query-positive example-general hard negative example-semantic conflict hard negative example".

[0115] Optionally, to further enhance the model's robustness, prevent overfitting to specific hard negative example patterns, and improve generalization ability, random negative examples can be introduced into the dataset in addition to the query text, positive examples, and hard negative examples mentioned above. Random negative examples are text segments randomly selected from a standardized set of text segments. During subsequent contrastive learning training, random negative examples are assigned different weights than hard negative examples.

[0116] Step S5: Fine-tune the Embedding model based on the above dataset.

[0117] This step utilizes the dataset constructed in step S4 to fine-tune the pre-trained general embedding model through contrastive learning. For example, the specific process may include:

[0118] Step S51, Vectorization: Input the query text, positive sample, and hard negative sample (including general hard negative and semantically conflicting hard negative) into the Embedding model to obtain the query vector, positive vector, and hard negative vector (including general hard negative vector and semantically conflicting hard negative vector) respectively.

[0119] Step S52, calculate the loss: Calculate the loss value using the Multiple Negatives Ranking Loss function. The optimization goal is to adjust the model parameters so that, in the vector space, the query vector is closer to the positive vector (e.g., smaller Euclidean distance or greater cosine similarity), while the query vector is farther from hard negative vectors, especially semantically conflicting hard negative vectors, forming a more reasonable semantic distribution structure. Through this explicit constraint, the model is forced to learn to distinguish subtle differences.

[0120] In the aforementioned contrastive learning training process, a differentiated training strategy is introduced for different types of negative samples. Semantically conflicting hard negative samples are weighted, with their weighting coefficients (weights) in the contrastive learning constraints being higher than those of general hard negative samples. This results in a higher contribution to the loss, thereby enhancing the model's ability to distinguish semantically conflicting samples.

[0121] In some embodiments, the structured training samples include not only the query text and its corresponding positive and hard negative samples (including general hard negatives and semantically conflicting hard negatives), but also the aforementioned random negative samples. In this case, during adversarial training, for example, random negative samples can be assigned a base weight, general hard negatives can be assigned a medium weight (higher than the base weight), and semantically conflicting hard negatives can be assigned a weight higher than the medium weight. This approach allows the model to focus more on semantically conflicting samples during training, thereby forming clearer semantic boundaries in the vector space.

[0122] Optionally, to enable large-scale training under single-card (single GPU) or limited computing power conditions, and to suit the scenario of private deployment in coal enterprises, training efficiency can be optimized during fine-tuning. For example, to address potential computing power limitations or large-scale data training needs in coal enterprises, gradient checkpointing can be used to save GPU memory during fine-tuning, mixed precision training (such as FP16) can be used to accelerate computation, or gradient accumulation can be used to simulate large batch size training under limited GPU memory. For example, the step size of gradient accumulation can be dynamically adjusted according to the batch selection quantity of hard negative samples. For example, the precision threshold of mixed precision training can be set based on the encoding features of the vertical industry text, for example, set to FP16.

[0123] Furthermore, the method may also include the following step S6.

[0124] Step S6: Evaluate the performance of the fine-tuned Embedding model (e.g., using metrics such as Recall@K and Mean Reciprocal Rank (MRR)). Deploy the evaluated model to a vector retrieval or RAG system, replacing the original general-purpose model, without intruding on the way the upper-level large model is called.

[0125] In practical operation, the system can use this fine-tuned model to vectorize massive amounts of coal technology documents and build a vector knowledge base, improving its ability to identify texts that are literally similar but semantically incorrect, thereby reducing false recall. When a user asks a question, the system can vectorize the question and retrieve the most relevant knowledge fragments from the knowledge base by calculating vector similarity. Due to training with hard negative examples, the model can effectively filter out literally similar but useless interfering documents, significantly reducing false recall in scenarios such as equipment fault queries and maintenance plan retrieval, improving the ranking quality of search results, and thus providing more accurate and reliable knowledge basis for the generation of upper-level large models. This significantly improves the accuracy of the answers generated by the RAG system and significantly reduces errors such as incorrect knowledge matching and semantic confusion recall.

[0126] Figure 2 This is a schematic block diagram of an embedding model fine-tuning system based on the hard negative example pattern, provided as an embodiment of the present invention. This system can implement the method steps provided in the foregoing embodiments.

[0127] like Figure 2 As shown, the system can be divided into a data layer, a data generation and mining layer, and a model training and application layer, including the following modules:

[0128] Data Acquisition and Segmentation Module 101: Belonging to the data layer, this module is used to uniformly collect, clean, and semantically segment unstructured text data (such as equipment manuals, fault cases, maintenance procedures, and operation manuals) from vertical industries (such as coal production), generating a standardized set of text fragments. This module is responsible for data preprocessing, providing the basic corpus for subsequent processes.

[0129] Query generation module 102: This module belongs to the data generation and mining layer and is connected to the data acquisition and slicing module 101 mentioned above. It utilizes a large language model or rule engine to automatically generate query texts that semantically correspond to text fragments in a standardized text fragment set. These query texts are used to simulate how users ask questions in knowledge retrieval scenarios, while the original text fragments are labeled as positive examples.

[0130] Hard negative example mining module 103: This module belongs to the data generation and mining layer and is connected to the data acquisition and slicing module 101 and the query generation module 102 mentioned above. This module utilizes a keyword-based retrieval algorithm (e.g., BM25) and a semantic consistency discrimination mechanism to filter out text fragments from a standardized text fragment set that are highly similar to the query text at the literal level but inconsistent at the business semantic level. These are then used as semantically conflict-type hard negative example samples. This module introduces highly complex interference terms to address the pain point of traditional models struggling to distinguish similar text.

[0131] Training data construction module 104: This belongs to the data generation and mining layer and is connected to the hard negative example mining module 103 mentioned above. This module is responsible for assembling the query text, the corresponding positive example samples, the mined semantically conflicting hard negative example samples, and the general hard negative example samples into structured training samples (such as the "query-positive example-general hard negative example-semantically conflicting hard negative example" quadruple data), forming the final dataset used for fine-tuning the Embedding model. Optionally, the training samples may also include random negative examples.

[0132] Embedding model fine-tuning module 105: This module belongs to the model training and application layer and is connected to the training data construction module 104 mentioned above. It loads the pre-trained embedding model and, using the constructed dataset, updates and optimizes the model parameters through contrastive learning and a multi-negative-example ranking loss function. This enables the model to narrow the distance between queries and positive examples and widen the distance between queries and hard negative examples in the vector space. Optionally, in contrastive training, weighted contrastive training is applied to semantically conflicting hard negative examples, causing the model to pay more attention to semantically conflicting samples during training, thereby forming clearer semantic boundaries in the vector space.

[0133] Model evaluation and deployment module 106: It belongs to the model training and application layer and is connected to the above-mentioned embedded model fine-tuning module 105. It is used to evaluate the trained Embedding model and deploy the qualified model to the actual vector retrieval or RAG system to replace the original general Embedding model.

[0134] The embedded model fine-tuning system based on hard negative example pattern provided in this embodiment of the invention can execute the embedded model fine-tuning method based on hard negative example pattern provided in this embodiment of the invention. The implementation principle is similar. The actions performed by each module in the system of each embodiment of the invention correspond to the steps in the method of each embodiment of the application. For detailed functional descriptions of each module, please refer to the descriptions in the corresponding methods shown above, which will not be repeated here.

[0135] Furthermore, embodiments of the present invention also provide an electronic device, which may include a memory and a processor connected to the memory. The memory may store a computer program, and the processor may be used to execute the computer program in the memory to implement the steps of the embedded model fine-tuning method based on hard negative example pattern as described in any of the foregoing embodiments.

[0136] Furthermore, embodiments of the present invention also provide a computer-readable storage medium on which a computer program may be stored, wherein when the computer program is executed, it implements the steps of the embedding model fine-tuning method based on the hard negative example pattern as described in any of the foregoing embodiments.

[0137] It should be noted that the computer-readable storage medium described above in this invention can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this invention, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0138] The aforementioned computer-readable storage medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0139] The modules described in the embodiments of the present invention can be implemented in software or hardware. The names of the modules are not, in some cases, limiting the scope of the module itself.

[0140] In summary, some embodiments of the present invention can solve or at least alleviate the problem that existing general embedding models in vertical fields such as the coal industry cannot effectively distinguish text fragments with highly overlapping keywords but different business semantics, resulting in large deviations and high false recall rates in vector retrieval results, which seriously restricts the practical value of RAG systems in key scenarios such as equipment operation and maintenance and fault diagnosis.

[0141] Some embodiments of the present invention may have one or more of the following beneficial effects (but are not limited to):

[0142] (1) Improve semantic discrimination and industry fine-grained distinction capabilities. It can effectively distinguish between texts that are “superficially similar but semantically incorrect”. Through an automated hard negative example mining mechanism, a high-discrimination training dataset is constructed, which significantly improves the model’s ability to perceive industry semantic fine-grained differences and reduces the false recall rate. In vector retrieval or RAG systems, it can significantly reduce errors such as incorrect knowledge matching and semantic confusion recall.

[0143] (2) Optimize the vector space structure. By fine-tuning the Embedding model driven by semantic conflict constraints and hard negative examples, the positive example vectors are more clustered and the semantic conflict samples are effectively pulled apart, making the distribution of positive and negative samples in the vector space more reasonable, thus forming a more reasonable semantic distribution structure, significantly widening the similarity gap between correct documents and interference documents, and enhancing the industry's semantic discrimination ability.

[0144] (3) It is suitable for highly specialized and industrial scenarios, and is applicable to professional fields such as coal and equipment operation and maintenance. It can significantly improve the accuracy of the model in complex semantic scenarios with similar terms but different meanings. It does not require large-scale manual annotation of data and is suitable for rapid deployment in industrial scenarios such as coal where data is scattered and documents are heterogeneous.

[0145] (4) It can be directly embedded into the existing RAG system architecture without intruding on the upper-level large model calling method, and has good engineering implementation and scalability.

[0146] The technical scope of this invention is not limited to the contents of the above specification. Those skilled in the art can make various modifications and variations to the above embodiments without departing from the technical concept of this invention, and all such modifications and variations should fall within the scope of this invention.

Claims

1. A method for fine-tuning an embedding model based on the hard negative example pattern, characterized in that, Includes the following steps: Collect, clean, and semantically segment unstructured text data from vertical industries to obtain a standardized set of text fragments; Based on the text fragments in the standardized text fragment set, a query text corresponding to its semantics is generated, and the text fragments are used as positive examples. The query text is used to simulate the way users ask questions in a knowledge retrieval scenario. Constructing semantically conflicting hard negative examples includes: using a keyword-matching retrieval algorithm to select a predetermined number of text fragments from the standardized text fragment set that have the highest literal relevance to the query text, forming a set of candidate text fragments; performing semantic filtering on the candidate text fragments based on a semantic consistency discrimination mechanism to identify candidate text fragments that are highly similar to the query text at the literal level but inconsistent with the business semantics, and using them as the semantically conflicting hard negative examples; Based on the query text and its corresponding positive and hard negative samples, a structured training sample is assembled to construct a dataset for fine-tuning the embedding model; wherein, the hard negative samples include general hard negative samples and semantically conflicting hard negative samples; The embedding model is fine-tuned based on the dataset, and during the contrastive learning training process, the semantically conflicting hard negative examples are given a higher weight than the general hard negative examples.

2. The method according to claim 1, characterized in that, The vertical industry mentioned is the coal industry; The semantic consistency judgment mechanism combines coal industry knowledge rules or fault semantic relationships to judge the semantics of the text; The steps for collecting, cleaning, and semantically segmenting unstructured text data include: Collect unstructured text data from coal production enterprises. The unstructured text data includes at least one of the following: equipment manuals, fault cases, maintenance procedures, and operation manual texts. The collected text is cleaned and segmented into standardized short text fragments according to semantic integrity.

3. The method according to claim 1, characterized in that, The steps for generating query text corresponding to the semantics of text fragments in the standardized text fragment set include: The text fragment is semantically understood using a large language model and multiple candidate query texts are generated. The multiple candidate query texts are ranked based on vector similarity or human intervention, and the candidate query texts with the highest ranking are selected as the query texts used to represent the semantics of the text fragment. or The rule engine is used to extract key information from the text fragment, and the rule engine is used to construct a query text generation template based on vertical industry business semantic rules. Then, a query text is generated based on the query text generation template to represent the semantics of the text fragment.

4. The method according to claim 1, characterized in that, The semantic consistency discrimination mechanism is used to filter semantically conflicting hard negative examples, including: semantic discrimination based on a large language model, or semantic discrimination based on rules or knowledge graphs, or multi-strategy fusion discrimination; wherein, The semantic discrimination based on the large language model includes: inputting the query text and the candidate text fragments into the large language model for processing, determining the semantic relationship between the query text and the candidate text fragments, and filtering out text fragments with inconsistent or conflicting semantics as the semantically conflicting hard negative sample, wherein the semantic relationship includes at least one of the following: whether the semantics are consistent, whether the semantics are related, and whether there is a semantic conflict. The rule-based or knowledge graph-based semantic discrimination includes: combining vertical industry knowledge rules or semantic relation databases to constrain and judge the semantics of text; when a candidate text fragment violates the knowledge rules or the semantic relation, the candidate text fragment is judged as the semantic conflict type hard negative example sample. The multi-strategy fusion discrimination includes: combining keyword matching, vector similarity, and semantic discrimination results for judgment.

5. The method according to claim 1, characterized in that, The keyword matching-based retrieval algorithm uses the BM25 algorithm, and the keyword weights and relevance thresholds of the BM25 algorithm are adjusted based on the business semantic features of the vertical industry.

6. The method according to claim 1, characterized in that, The general hard negative examples are obtained based on keyword matching, vector similarity, or the candidate text fragment set; wherein, Obtaining general hard negative sample based on keyword matching includes: calculating the relevance score between the query text and each text segment in the standardized text segment set using the keyword relevance function; selecting a preset number of text segments with high relevance scores that are not positive sample; and taking the text segments among the selected text segments that are not identified as semantically conflicted by the semantic consistency discrimination mechanism as the general hard negative sample. Obtaining general hard negative sample based on vector similarity includes: calculating the vector similarity between the query text and each text segment in the standardized text segment set through an embedding model, and selecting a preset number of text segments with high vector similarity that are not positive samples as the general hard negative sample; Obtaining general hard negative examples based on the candidate text fragment set includes: in the candidate text fragment set, selecting candidate text fragments that are neither positive examples nor semantically conflicting hard negative examples as general hard negative examples.

7. The method according to claim 1, characterized in that, The dataset used for fine-tuning the embedded model also includes random negative examples, which are text segments randomly extracted from text segments other than the positive examples. During the contrastive learning training process, the random negative examples are given a weight lower than that of the semantically conflicting hard negative examples and the general hard negative examples.

8. The method according to claim 1, characterized in that, The steps for fine-tuning the embedding model include: Input the query text, positive examples, and hard negative examples into the embedding model to obtain the query vector, positive example vector, and hard negative example vector, respectively. The loss value is calculated using the multi-negative-example ranking loss function, and the model parameters are optimized so that the distance between the query vector and the positive example vector in the vector space is less than the distance between the query vector and the hard negative example vector.

9. The method according to claim 1, characterized in that, During the fine-tuning of the embedded model, at least one of gradient checkpointing, mixed precision training, or gradient accumulation techniques is used to optimize training efficiency. The step size of the gradient accumulation is dynamically adjusted according to the batch screening quantity of the hard negative samples, and the precision threshold of the mixed precision training is set based on the encoding features of the vertical industry text.

10. The method according to claim 1, characterized in that, The method further includes: The performance of the fine-tuned embedding model was evaluated. The evaluated embedding models are deployed to vector retrieval or retrieval enhancement generation systems to vectorize text data and build vector knowledge bases.

11. An embedding model fine-tuning system based on hard negative example patterns, characterized in that, include: The data acquisition and slicing module is used to collect, clean, and semantically slice unstructured text data from vertical industries, generating a standardized set of text fragments. The query generation module, connected to the data acquisition and slicing module, is used to automatically generate query text corresponding to the semantics of text fragments in the standardized text fragment set. The query text is used to simulate the way users ask questions in a knowledge retrieval scenario, and the text fragments are used as positive example samples. The hard negative example mining module is connected to the data acquisition and slicing module and the query generation module, respectively, and is used to: use a keyword matching-based retrieval algorithm and a semantic consistency discrimination mechanism to select text fragments from the standardized text fragment set that are highly similar to the query text at the literal level but inconsistent at the business semantic level, as semantic conflict type hard negative example samples. The training data construction module, connected to the hard negative example mining module, is used to assemble the query text and its corresponding positive example samples, general hard negative example samples and semantically conflicting hard negative example samples into structured training samples, forming a dataset for embedding model fine-tuning. An embedded model fine-tuning module is connected to the training data construction module and is used to apply weighted contrastive training to the semantically conflicting hard negative examples.

12. An electronic device, characterized in that, The method includes a memory and a processor connected to the memory, wherein the memory stores a computer program that can be executed by the processor, and the processor executes the computer program to implement the method as described in any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, It stores a computer program thereon, which, when executed by a processor, implements the method as described in any one of claims 1 to 10.

Images