A natural language hard negative sample mining method, system, device and medium
By combining the BM25 algorithm and a weighted fusion strategy based on cosine similarity, the problems of model dependency and semantic neglect in hard negative sample mining are solved, achieving accurate mining of high-quality hard negative samples and improving the model's discrimination and ranking capabilities.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- TRAVELSKY TECHNOLOGY LIMITED
- Filing Date
- 2026-04-20
- Publication Date
- 2026-08-04
AI Technical Summary
Existing hard negative sample mining methods are not effective in specific domains. General vector models are highly dependent and ignore semantics. Methods based on word frequency statistics ignore semantics, resulting in low quality of hard negative samples.
By combining the BM25 algorithm and cosine similarity, a weighted fusion strategy is used to calculate the comprehensive score of negative samples and select high-quality hard negative samples.
It effectively integrates word-level matching and semantic similarity features, accurately identifies high-difficulty hard negative samples, and improves the model's discrimination and ranking capabilities.
Smart Images
Figure CN122508191A_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of natural language processing technology, and specifically relates to a method, system, device and medium for mining hard negative samples of natural language. Background Technology
[0002] Vector models are essential for RAG applications, but in specific domains such as civil aviation, general vector models are not effective. They usually need to be fine-tuned using specific datasets before use, and the quality of the dataset is crucial to the effectiveness of the fine-tuned model.
[0003] The dataset construction requires both positive and negative samples. Positive samples are those that highly match the semantics of the target query and meet the expected search results; they are the core data for the model to learn correct matching relationships. Negative samples are those that do not match the semantics of the target query and do not belong to the correct search results; they are used to teach the model to distinguish irrelevant content. Hard negative samples are needed to improve the model's discrimination ability and optimize its ranking ability. Hard negative samples refer to high-difficulty negative samples that have a certain semantic or lexical similarity to the target query and are easily misclassified as relevant by the model.
[0004] Hard negative sample mining methods generally fall into two categories: 1. Using a general vector model, the overall semantics of the query are transformed into a high-dimensional vector, and similarity is calculated and ranked. The samples with the highest similarity are selected from the negative samples. However, this method depends entirely on the quality of the vector model and only captures semantic spatial distance.
[0005] 2. Similarity is calculated based on word frequency statistics. This method is sensitive to word-level matching, simple and efficient in calculation, and highly interpretable. However, this method can only consider bag-of-words matching and ignores semantics, resulting in low-quality hard negative samples. Summary of the Invention
[0006] To address the aforementioned issues, this application provides a method, system, device, and medium for mining hard negative samples in natural language. By utilizing the word-level matching advantage of the BM25 algorithm and the semantic representation advantage of cosine similarity through weighted fusion, it is possible to mine high-difficulty, high-quality hard negative samples.
[0007] The following is the technical content of this disclosure: A hard negative sample mining method for natural language processing includes: Obtain the request dataset and the corresponding negative sample dataset based on natural language; The BM25 algorithm is used to calculate the BM25 score of each sample in the negative sample set for the request text. The cosine similarity score is used to calculate the similarity score between each sample in the negative sample set and the request text. For each negative sample, its BM25 score and similarity score are fused using a weighted fusion strategy to obtain a comprehensive score. All negative samples are sorted in descending order of their comprehensive scores, and a preset number of top samples are selected as hard negative samples for the corresponding request text.
[0008] Furthermore, The calculation of the BM25 score for each sample in the negative sample set against the request text using the BM25 algorithm includes: A professional thesaurus for the target domain is constructed, and the Jieba word segmentation method combined with the professional thesaurus is used to segment the request text in the request dataset and all negative sample texts in the negative sample dataset. The segmented request text is used as the query term. The initial BM25 score of each negative sample text and the request text is calculated based on the BM25 algorithm. The initial BM25 scores of all negative sample texts are then subjected to min-max standardization to obtain the standardized BM25 score.
[0009] Furthermore, The calculation of the similarity score between each sample in the negative sample set and the request text using cosine similarity includes: A natural language vectorization model is used to convert the request text in the request dataset and all negative sample text in the negative sample dataset into numerical vectors, respectively. Based on the cosine similarity algorithm, the initial similarity score between the numerical vector corresponding to each negative sample text and the numerical vector corresponding to the request text is calculated. The initial similarity scores of all negative sample texts are subjected to min-max normalization to obtain the normalized similarity scores.
[0010] Furthermore, The weight fusion strategy includes: For unlabeled data: Set a weight coefficient 'a', where the value of 'a' ranges from 0 to 1; calculate the comprehensive score of the negative samples based on the weight coefficient 'a', using the following formula: =a +(1-a)
[0011] in The composite score for negative samples. This is the standardized BM25 score for the negative sample. This is the standardized similarity score for the negative sample. For labeled data: Data is uniformly sampled from the request dataset and the negative sample dataset. For each sampled request text, a corresponding negative sample text is selected from the negative sample dataset. The relevance between the negative sample text and the request text is scored and labeled to construct a labeled dataset. Construct a linear regression model, the model formula is: final_score = w1 bm25_score + w2 vector_score + b Where final_score is the overall prediction score, bm25_score is the standardized BM25 score, vector_score is the standardized similarity score, w1 and w2 are weight coefficients, and b is the bias term; Using the labeled dataset as the training set, with the goal of minimizing the mean square error between the model's predicted comprehensive score and the labeled score, the weight coefficients w1, w2 and the bias term b are obtained through training. The standardized BM25 score and similarity score are substituted into the trained linear regression model to calculate the comprehensive score of the negative samples.
[0012] Furthermore, The weighting coefficient 'a' is set based on the coverage of specialized vocabulary in the target text set.
[0013] Furthermore, The calculation method for the coverage rate of the specialized vocabulary is: the ratio of the total number of specialized vocabulary occurrences in the specialized vocabulary database in the target text set to the total number of words after word segmentation in the target text set.
[0014] A hard negative sample mining system for natural language processing includes: The data acquisition module is used to acquire request datasets and corresponding negative sample datasets based on natural language. The first calculation module is used to calculate the BM25 score of each sample in the negative sample set for the request text using the BM25 algorithm. The second calculation module uses cosine similarity to calculate the similarity score between each sample in the negative sample set and the request text. The sample selection module is used to calculate a comprehensive score for each negative sample by fusing its BM25 score and similarity score through a weighted fusion strategy. The comprehensive scores of all negative samples are sorted in descending order, and a preset number of top samples are selected as hard negative samples for the corresponding request text.
[0015] Furthermore, The sample selection module is configured as follows: For unlabeled data: Set a weight coefficient 'a', where the value of 'a' ranges from 0 to 1; calculate the comprehensive score of the negative samples based on the weight coefficient 'a', using the following formula: =a +(1-a)
[0016] in The composite score for negative samples. This is the standardized BM25 score for the negative sample. This is the standardized similarity score for the negative sample. For labeled data: Data is uniformly sampled from the request dataset and the negative sample dataset. For each sampled request text, a corresponding negative sample text is selected from the negative sample dataset. The relevance between the negative sample text and the request text is scored and labeled to construct a labeled dataset. Construct a linear regression model, the model formula is: final_score = w1 bm25_score + w2 vector_score + b Where final_score is the overall prediction score, bm25_score is the standardized BM25 score, vector_score is the standardized similarity score, w1 and w2 are weight coefficients, and b is the bias term; Using the labeled dataset as the training set, with the goal of minimizing the mean square error between the model's predicted comprehensive score and the labeled score, the weight coefficients w1, w2 and the bias term b are obtained through training. The standardized BM25 score and similarity score are substituted into the trained linear regression model to calculate the comprehensive score of the negative samples. The comprehensive scores of all negative samples are sorted in descending order, and a preset number of top samples are selected as hard negative samples of the corresponding request text.
[0017] A hard negative sample mining device for natural language includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the hard negative sample mining method for natural language.
[0018] A computer storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to implement the method described thereon.
[0019] Compared with the prior art, this application has the following advantages: By acquiring a request dataset and a corresponding negative sample dataset based on natural language, and then using the BM25 algorithm to calculate the BM25 score of each sample in the negative sample set against the request text, and simultaneously using cosine similarity to calculate the similarity score of each sample in the negative sample set against the request text, the two types of scores are fused together using a weighted fusion strategy to obtain a comprehensive score. The comprehensive scores are then sorted in descending order, and a predetermined number of top samples are selected as hard negative samples of the corresponding request text. This approach aims to simultaneously capture word-level matching features and semantic similarity features between the negative samples and the request text, and to conduct precise screening and mining of hard negative samples by combining dual-dimensional features. This approach leverages the advantages of the BM25 algorithm, such as its sensitivity to word-level matching and strong interpretability, while also fully utilizing the ability of cosine similarity to effectively capture deep semantic relationships in text. It overcomes the dual shortcomings of the BM25 algorithm, which only considers bag-of-words matching but ignores textual semantics, and the cosine similarity algorithm, which only captures semantic spatial distance and is entirely dependent on the quality of the vector model. This approach achieves an effective fusion of word-level features and semantic features, and can accurately mine high-difficulty hard negative samples that have both lexical similarity and semantic relevance to the request text and are easily misclassified as positive samples by the model, thereby improving the quality of hard negative sample mining.
[0020] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures pointed out in the description, claims and drawings. Attached Figure Description
[0021] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0022] Figure 1 A schematic diagram of the hard negative sample mining method for natural language disclosed herein is shown. Detailed Implementation
[0023] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0024] like Figure 1 The following is a flowchart of the hard negative sample mining method for natural language in this disclosure. The specific steps are as follows: 1. Step One: Obtain Data Hard negative sample mining requires cleaned and organized data to form a request dataset and a negative sample dataset, as shown in the example below: Example of requesting a dataset: "query": "Requested data 1", "pos": "Positive sample data 1" "query": "Requested data 2", "pos": "Positive sample data 2" Example of a negative sample dataset: "text": "\"Negative Sample Data 1\"" "text": "\"Negative Sample Data 2\"" "text": "\"Negative sample data n\"" The query in the request dataset is the request data, which is a series of independent data (the subsequent pos data is its corresponding positive sample). For the request data, the first n hard negative samples corresponding to it can be mined from the negative sample dataset, which will be used as the final model fine-tuning dataset.
[0025] From the negative sample dataset, the top n negative samples most similar to a given request dataset will be selected; these are the mined hard negative samples. All request data will undergo the following steps during hard negative sample data mining: 2. Step Two: Calculate the BM25 score All data in the request data and negative sample set need to be segmented. Jieba segmentation is used here. Before segmentation, a domain-specific thesaurus needs to be defined to ensure the quality of segmentation.
[0026] The BM25 algorithm was then used to obtain the score for each sample in the negative sample set against the request text. Min-max normalization was then applied to all scores to obtain the final score data.
[0027] =[ ,……, ] 3. Step Three: Calculate the vector similarity score For both the request data and all data in the negative sample set, the text needs to be converted into vectors using a vectorization model. Then, cosine similarity is used to calculate the similarity score between each sample in the negative sample set and the request text. Finally, min-max normalization is applied to all scores to obtain the final similarity score for the negative sample set.
[0028] =[ ,……, ] 4. Step Four: Calculate the comprehensive score by fusing the BM25 score and similarity score using a weighted fusion strategy. 4.1 For unlabeled data, weights are manually assigned for fusion. Set a weight 'a' (0 <= a <= 1), and calculate the overall score as follows: =a +(1-a)
[0029] Regarding the method for setting the weight of 'a': According to the experimental statistics, if the coverage rate of professional vocabulary in the text set (number of professional vocabulary words in the text / total number of words) is >25%, 'a' can be set to 0.6~0.8 to increase the weight of BM25; if the coverage rate of professional vocabulary is <10%, the text is relatively long, and the semantics are closely related, 'a' can be set to 0.1~0.3.
[0030] Foundation The scores are sorted, and the top n negative samples with the largest scores are taken as the required hard negative sample data.
[0031] 4.2 For labeled data, weights are obtained through a linear regression model and then fused. In addition to providing the requested data and negative sample datasets mentioned above, a small amount of data needs to be manually sampled evenly from the requested data sample set, and specific data points need to be selected from the negative sample set for each sampled requested data point and scored on a 5-point scale. This will create a labeled dataset. {"quary":"Requested data 1","neg":"Negative sample data 1","score":"2"} {"quary":"Requested data 2","neg":"Negative sample data 5","score":"4"} Notice: 1) The data in quary and neg must be collected from the actual data samples that need to be mined in order to obtain the weighted score for that sample.
[0032] 2) The meaning of the 5-point system and the sampling ratio should meet the following requirements: high correlation (4-5 points): 30%, medium correlation (2-3 points): 40%, low correlation (0-1 point): 30%.
[0033] The strategy linear regression model disclosed herein is set as follows: final_score = w1 bm25_score + w2 vector_score + b The training objective is to solve for the weights w1, w2 and bias b using labeled data, so as to minimize the mean squared error (MSE) between the model's predicted final_score and the labeled score.
[0034] Using the labeled dataset described above, the linear regression model can learn the optimal weight combination of the two ranking methods, thereby improving the final ranking effect while preserving the interpretability of BM25 and integrating the semantic representation capability of the vectorized model.
[0035] Definitions: 1) Vectors: In Natural Language Processing (NLP), vectors are the core tool for mathematizing text data. By mapping linguistic symbols to numerical vectors, computers are able to process semantic and syntactic relationships.
[0036] 2) RAG: Retrieval-augmented Generation (RAG). RAG models combine language models and information retrieval techniques. Specifically, when the model needs to generate text or answer questions, it first retrieves relevant information from a large collection of documents, and then uses this retrieved information to guide text generation, thereby improving the quality and accuracy of predictions.
[0037] 3) Hard negative samples: These are negative samples, but due to similar features, the model is prone to misclassifying them as positive samples.
[0038] 4) Specialized terminology database: proprietary terms in specific industries and vertical fields, with a low coefficient of variation (number of synonyms and abbreviations of specialized terms).
[0039] 5) BM25 Algorithm: BM25 (BestMatching25) is a ranking algorithm commonly used in information retrieval. It can more accurately calculate the relevance between queries and documents and rank the documents. This algorithm considers factors such as the frequency of query terms in documents, document length, and the number of documents in the corpus containing the query term.
[0040]
[0041] in: TF stands for Term Frequency: it refers to the number of times a word appears in a document, and is used to measure the representativeness of a word to the content of the document.
[0042] IDF stands for Inverse Document Frequency: It measures the "importance" of a word and is inversely proportional to document frequency: the fewer documents in which the word appears, the higher the IDF.
[0043] 6) Jieba Word Segmentation: A Python-based Chinese word segmentation library. Jieba first builds a prefix dictionary (Trie tree structure), organizing words in the dictionary according to prefix relationships to achieve fast word matching. For example, if the dictionary contains "A University" and "A", the Trie tree records the prefix relationship between these two words, accelerating the matching process. Jieba uses a dynamic programming algorithm to calculate possible word segmentation combinations for each position and finds the segmentation path with the highest probability based on word frequency and statistical information in the dictionary. For example, for the text "I am studying at A", the algorithm calculates multiple possible segmentation methods (such as "I / at / A / study" vs "I / at / A University / study") and selects the optimal path. Jieba optimizes word segmentation results by adding custom dictionaries, adjusting word frequencies, and adding stop words, adapting to the needs of specific domains. For example, given the text "The Civil Aviation Distribution GDS System is a product developed by China TravelSky for...", if "Civil Aviation Distribution GDS System, TravelSky" is added as a custom term in the custom thesaurus and given a high weight, the sentence will be split into: Civil Aviation Distribution GDS System / is / China / TravelSky / developed / of / a / product / for / . Otherwise, it will be split into: Civil Aviation / Distribution / GDS / System / is / China / TravelSky / developed / of / a / product / for / .
[0044] 7) Cosine similarity: Vector similarity is an indicator used to measure the degree of similarity between two vectors in space. For text semantic analysis and information retrieval, cosine similarity is used to measure the similarity in direction by calculating the cosine value of the angle between two vectors, which is independent of the vector length.
[0045]
[0046] 8) Min-max Standardization: Min-max standardization is a commonly used data preprocessing technique that maps data to a specified interval (usually [0,1]) through linear transformation. Its core idea is to scale the original data using its maximum and minimum values, ensuring the processed data falls within a uniform range. The original data is X = [X1, ..., Xn], where the minimum value is Xmin and the maximum value is Xmax. The standardization formula for each value in X is:
[0047] The following are embodiments of this disclosure: This technical solution proposes a method for mining hard negative samples in natural language processing. Implementation schemes are provided for both labeled and unlabeled data. The specific implementation process is as follows: Two files were provided: hn_mine_bm25.py: The execution script for hard negative sample mining, executed with parameters.
[0048] Execution example: python hn_mine_bm25.py --embedder_name_or_path "E:\bge-small-zh-v1.5" `# Vectorized model --negative_number 15 # Number of hard negative samples mined --input_file "main_output1.jsonl" # Request a sample data set --candidate_pool "pos_output1.jsonl" # Negative sample data set --output_file "output2.jsonl" # Output file --bm25_weight 0.5# Blending weight dict.txt: A domain-specific thesaurus that maintains specialized vocabulary and weights. Each line contains a word and its weight. Words with higher weights are prioritized for inclusion as complete words.
[0049] Example: Company A 20 TravelSky 10 Company B 20 China Eastern Airlines 10 Data acquisition and processing: I. Unlabeled data 1. The user prepares the dataset for mining, including the request dataset and the negative sample dataset. 1) Example of requesting a dataset: {"query": "What ticketing process should be followed when using the E1 interface for UATP card auto-ticketing?", "pos": ""When using the E1 (domestic auto-ticketing, AirDemandTicket) interface for UATP card auto-ticketing, the payment process must be completed using the AirPayment interface before ticketing."} 2) Example of a negative sample dataset: The automatic ticket refund service only supports tickets used in sequence. If a ticket is not used in sequence, it cannot be refunded through the automatic ticket refund service. {"text":"When using the E1 (Domestic Automatic Ticketing, AirDemandTicket) interface, the EI field cannot be empty; either upload it or delete the EI node."} {"text":"If automatic ticketing for some flight segments is successful, the PNR's ticketing status will be 'Pending Ticketing' and highlighted in red. The 'Pending Ticketing' task query will only look for tasks with a 'Pending Ticketing' or 'Ticketing in Progress' status."} {"text":"When using the A5 interface to perform multi-office queries, the results can be presented in two ways: one is to return the lowest price among the searchable offices and a list of offices that apply that price; the other is to return the price for each office. Which result is returned can be controlled through the user's request parameters."} {"text":"The value 'A' in the request parameters indicates that the user wants to obtain the lowest price among all available cabin classes."} 2. Users maintain a professional thesaurus in the field and add it to dict.txt. Then, select a suitable base vector model, such as bge-m3 or bge-base-zh embedding models.
[0050] 3. Determine the fusion weights as needed. Calculate the coverage of specialized terms: the number of specialized terms in dict.txt that appear in the whole text / the total number of words after word segmentation of the whole text.
[0051] For example, if the coverage rate of specialized vocabulary is calculated to be 27, and the weight 'a' is set to 0.6... Execute the script to obtain the final result. Each request data corresponds to one positive sample data and multiple hard negative sample data.
[0052] Example of final result: {"query": "What ticketing process should be followed when using the E1 interface for UATP card auto-ticketing?", "pos": ""When using the E1 (domestic auto-ticketing, AirDemandTicket) interface for UATP card auto-ticketing, the payment process must be completed using the AirPayment interface before ticketing.", "neg": ["If auto-ticketing for some flight segments is successful, the PNR ticketing status will be 'pending ticketing' and highlighted in red. The 'pending ticketing task query' will only query tasks with ticketing status 'pending ticketing' and 'ticketing in progress'.", "When using the E1 (domestic auto-ticketing, AirDemandTicket) interface, the EI field cannot be empty; either upload it or delete the EI node."} II. Labeled Data 1) The user prepares a dataset for mining, including the request dataset and the negative sample dataset.
[0053] 2) Users collect some data and mark and score it on a 5-point scale.
[0054] A small amount of data was manually selected from the requested dataset, and samples were evenly selected from the negative sample dataset according to high, medium and low similarity, and labeled with a 5-branch scoring system.
[0055] Example of a labeled scoring dataset: {“quary”:“What ticketing process should be followed when using the E1 interface for UATP card automatic ticketing?”,“neg”:“When using the E1 (domestic automatic ticketing, AirDemandTicket) interface, the EI field cannot be empty; either transmit or delete the EI point.”,score”:“5”} {“quary”:“What ticketing process should be followed when using the E1 interface for automatic ticketing of UATP cards?”,”neg”:“If the automatic ticketing of some flight segments is successful, the PNR ticketing status will be “pending ticketing” and highlighted in red. The pending ticketing task query will only look for tasks with the ticketing status of “pending ticketing” and “ticketing in progress”.”,”score”:“5”} {“quary”:“What ticketing process should be followed when using the E1 interface for UATP card automatic ticketing?”,“neg”:“The automatic refund service only supports tickets used in sequence. If the ticket is not used in sequence, it cannot be refunded through the automatic refund service.”,“score”:“3”} {“quary”:“What ticketing process should be followed when using the E1 interface for automatic UATP card ticketing?”,“neg”:“When using the A5 interface for multi-office queries, the returned results can be presented in two ways: one is to return the lowest price among the queryable offices and a list of offices that apply that price; the other is to return the price for each office. Which result is returned can be controlled through user request parameters.”,“score”:“1”} {“quary”:“What ticketing process should be followed when using the E1 interface for automatic ticketing with a UATP card?”,“neg”:“The value A in the request parameters indicates that the user wants to obtain the lowest price for all available cabin classes.”,“score”:“1”} 3) Users maintain a domain-specific thesaurus in dict.txt and select a suitable base vector model, such as bge-m3, bge-base-zh, etc., for embedding.
[0056] 4) Execute the script to obtain the final result.
[0057] The final example is as follows: {"query": "What ticketing process should be followed when using the E1 interface for UATP card auto-ticketing?", "pos": ""When using the E1 (domestic auto-ticketing, AirDemandTicket) interface for UATP card auto-ticketing, the payment process must be completed using the AirPayment interface before ticketing.", "neg": ["When using the E1 (domestic auto-ticketing, AirDemandTicket) interface, the EI field cannot be empty; either upload or delete the EI node.", "After the user submits an order, no TC processing is required, and no pending confirmation email will be sent to the TC. The system automatically issues tickets for the trip. If the TC matches two rules (even if the content is completely identical), the trip follows a semi-automatic ticketing process. The TC can click the "Auto-Ticketing" button on the trip details page, select the desired ticketing channel, and submit for auto-ticketing."} Based on the method of this disclosure, this embodiment also provides a system corresponding to the above method, which includes: The data acquisition module is used to acquire request datasets and corresponding negative sample datasets based on natural language. The first calculation module is used to calculate the BM25 score of each sample in the negative sample set for the request text using the BM25 algorithm. The second calculation module uses cosine similarity to calculate the similarity score between each sample in the negative sample set and the request text. The sample selection module is used to calculate a comprehensive score for each negative sample by fusing its BM25 score and similarity score through a weighted fusion strategy. The comprehensive scores of all negative samples are sorted in descending order, and a preset number of top samples are selected as hard negative samples for the corresponding request text.
[0058] Based on the same inventive concept as disclosed above, embodiments of this disclosure also provide an apparatus corresponding to the above method, comprising at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the above method.
[0059] It should be noted that the electrical connection between the above-mentioned units does not necessarily mean the connection between lines. The indirect connection method can be applied to the embodiments of this disclosure as long as it achieves the purpose of this disclosure.
[0060] Based on the same inventive concept, this disclosure also provides a computer storage medium having executable instructions stored thereon, which are executed by a processor to cause the processor to implement the above-described method.
[0061] Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.
Claims
1. A method for mining hard negative samples in natural language processing, characterized in that, include: Obtain the request dataset and the corresponding negative sample dataset based on natural language; The BM25 algorithm is used to calculate the BM25 score of each sample in the negative sample set for the request text. The cosine similarity score is used to calculate the similarity score between each sample in the negative sample set and the request text. For each negative sample, its BM25 score and similarity score are fused using a weighted fusion strategy to obtain a comprehensive score. All negative samples are sorted in descending order of their comprehensive scores, and a preset number of top samples are selected as hard negative samples for the corresponding request text.
2. The hard negative sample mining method for natural language according to claim 1, characterized in that, The calculation of the BM25 score for each sample in the negative sample set against the request text using the BM25 algorithm includes: A professional thesaurus for the target domain is constructed, and the Jieba word segmentation method combined with the professional thesaurus is used to segment the request text in the request dataset and all negative sample texts in the negative sample dataset. The segmented request text is used as the query term. The initial BM25 score of each negative sample text and the request text is calculated based on the BM25 algorithm. The initial BM25 scores of all negative sample texts are then subjected to min-max standardization to obtain the standardized BM25 score.
3. The hard negative sample mining method for natural language according to claim 1, characterized in that, The calculation of the similarity score between each sample in the negative sample set and the request text using cosine similarity includes: A natural language vectorization model is used to convert the request text in the request dataset and all negative sample text in the negative sample dataset into numerical vectors, respectively. Based on the cosine similarity algorithm, the initial similarity score between the numerical vector corresponding to each negative sample text and the numerical vector corresponding to the request text is calculated. The initial similarity scores of all negative sample texts are subjected to min-max normalization to obtain the normalized similarity scores.
4. The hard negative sample mining method for natural language according to claim 1, characterized in that, The weight fusion strategy includes: For unlabeled data: Set a weight coefficient 'a', where the value of 'a' ranges from 0 to 1; calculate the comprehensive score of the negative samples based on the weight coefficient 'a', using the following formula: =a +(1-a) in The composite score for negative samples. This is the standardized BM25 score for the negative sample. This is the standardized similarity score for the negative sample. For labeled data: Data is uniformly sampled from the request dataset and the negative sample dataset. For each sampled request text, a corresponding negative sample text is selected from the negative sample dataset. The relevance between the negative sample text and the request text is scored and labeled to construct a labeled dataset. Construct a linear regression model, the model formula is: final_score = w1 bm25_score + w2 vector_score + b Where final_score is the overall prediction score, bm25_score is the standardized BM25 score, vector_score is the standardized similarity score, w1 and w2 are weight coefficients, and b is the bias term; Using the labeled dataset as the training set, with the goal of minimizing the mean square error between the model's predicted comprehensive score and the labeled score, the weight coefficients w1, w2 and the bias term b are obtained through training. The standardized BM25 score and similarity score are substituted into the trained linear regression model to calculate the comprehensive score of the negative samples.
5. The hard negative sample mining method for natural language according to claim 4, characterized in that, The weighting coefficient 'a' is set based on the coverage of specialized vocabulary in the target text set.
6. The hard negative sample mining method for natural language according to claim 5, characterized in that, The calculation method for the coverage rate of the specialized vocabulary is: the ratio of the total number of specialized vocabulary occurrences in the specialized vocabulary database in the target text set to the total number of words after word segmentation in the target text set.
7. A hard negative sample mining system for natural language processing, characterized in that, include: The data acquisition module is used to acquire request datasets and corresponding negative sample datasets based on natural language. The first calculation module is used to calculate the BM25 score of each sample in the negative sample set for the request text using the BM25 algorithm. The second calculation module uses cosine similarity to calculate the similarity score between each sample in the negative sample set and the request text. The sample selection module is used to calculate a comprehensive score for each negative sample by fusing its BM25 score and similarity score through a weighted fusion strategy. The comprehensive scores of all negative samples are sorted in descending order, and a preset number of top samples are selected as hard negative samples for the corresponding request text.
8. The system according to claim 7, characterized in that, The sample selection module is configured as follows: For unlabeled data: Set a weight coefficient 'a', where the value of 'a' ranges from 0 to 1; calculate the comprehensive score of the negative samples based on the weight coefficient 'a', using the following formula: =a +(1-a) in The composite score for negative samples. This is the standardized BM25 score for the negative sample. This is the standardized similarity score for the negative sample. For labeled data: Data is uniformly sampled from the request dataset and the negative sample dataset. For each sampled request text, a corresponding negative sample text is selected from the negative sample dataset. The relevance between the negative sample text and the request text is scored and labeled to construct a labeled dataset. Construct a linear regression model, the model formula is: final_score = w1 bm25_score + w2 vector_score + b Where final_score is the overall prediction score, bm25_score is the standardized BM25 score, vector_score is the standardized similarity score, w1 and w2 are weight coefficients, and b is the bias term; Using the labeled dataset as the training set, with the goal of minimizing the mean square error between the model's predicted comprehensive score and the labeled score, the weight coefficients w1, w2 and the bias term b are obtained through training. The standardized BM25 score and similarity score are substituted into the trained linear regression model to calculate the comprehensive score of the negative samples. Sort all negative samples in descending order of their overall scores, and select a preset number of top-ranked samples as hard negative samples for the corresponding request text.
9. A hard negative sample mining device for natural language processing, comprising: At least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the hard negative sample mining method for natural language as described in any one of claims 1 to 6.
10. A computer storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the method of any one of claims 1 to 6.