Difficulty negative sample generation method, intention recognition method, electronic equipment and storage medium

Abstract

The invention relates to a difficult negative sample generation method, an intention recognition method, electronic equipment and a storage medium. The method comprises the following steps: extracting a keyword from a positive sample corresponding to a target intention; a target combination is constructed through the keyword, and the target combination is a phrase used for describing key information of the target intention; inputting a context guide description corresponding to the target intention and the target combination into a large language model (LLM) to obtain a candidate negative sample; and screening the candidate negative samples through semantic similarity to determine difficult negative samples. Through the method and the device, the technical problem of low quality of difficult negative samples dug in related technologies is solved, and the technical effect of improving the quality of the difficult negative samples is achieved.

Description

Technical Field

[0001] The present application relates to the field of natural language processing, and in particular to difficult negative sample generation, intent recognition methods, electronic devices and storage media. Background Art

[0002] Difficult negative samples refer to those negative samples that are easily confused, have low discrimination, and are easily mistakenly predicted as positive samples. Difficult negative samples play a vital role in the model training process, and can effectively improve the generalization ability of the model, prevent overfitting, balance the category distribution, and improve the robustness of the model. However, the difficult negative samples mined in related technologies generally have the technical problem of low quality, and no effective solution has been proposed for this technical problem. Summary of the invention

[0003] The embodiments of the present application provide difficult negative sample generation, intent recognition methods, electronic devices and storage media to at least solve the technical problem that difficult negative samples mined in related technologies generally have low quality.

[0004] According to one aspect of an embodiment of the present application, a method for generating difficult negative samples is provided, comprising: extracting keywords from a positive sample corresponding to a target intent; constructing a target combination through the keywords, wherein the target combination is a phrase used to describe key information of the target intent; inputting a context-guided description corresponding to the target intent and the target combination into a large language model (LLM) to obtain candidate negative samples; and screening the candidate negative samples by semantic similarity to determine difficult negative samples.

[0005] Optionally, constructing the target combination using the keywords includes: determining the part of speech of the keywords; grouping the keywords according to the part of speech; extracting a keyword from each group, and constructing the target combination based on the extracted keywords.

[0006] Optionally, the context-guided description corresponding to the target intent includes at least one of the following: a target task description to be executed by the LLM, a constraint description of the target task, a description of a keyword combination instance, and an output description corresponding to the keyword combination instance.

[0007] Optionally, the candidate negative samples are screened by semantic similarity to determine difficult negative samples, including: extracting a specified number of positive samples from the positive samples corresponding to the target intent as a sample pool; for any candidate negative sample, obtaining the vector representation of the candidate negative sample and the vector representation of all samples in the sample pool; respectively calculating the cosine distances between the vector representation of the candidate negative sample and the vector representation of all samples in the sample pool to obtain multiple cosine distances; when the multiple cosine distances are all higher than a specified threshold, determining that the candidate negative sample is the difficult negative sample.

[0008] Optionally, the method also includes: obtaining a target positive sample and a target negative sample including the difficult negative sample, and performing self-supervised comparative learning on a text classification model through the target positive sample and the target negative sample; training the text classification model through the self-supervised comparative learning loss, the first cross entropy loss of the target positive sample, and the second cross entropy loss of the expanded sample after the target positive sample is expanded, to obtain a trained text classification model.

[0009] Optionally, training a text classification model by using the self-supervised contrastive learning loss, the first cross entropy loss of the target positive sample, and the second cross entropy loss of the expanded sample after the target positive sample is expanded to obtain the trained text classification model includes: training the text classification model by using the following formula to obtain the trained text classification model: ,in, is the first cross entropy loss determined by the target positive sample, is the second cross entropy loss determined after expanding the target positive sample, is the self-supervised contrastive learning loss, To specify weights.

[0010] Optionally, extracting keywords from the positive sample corresponding to the target intent includes: extracting keywords from the positive sample by using a text sorting algorithm based on a graph model.

[0011] According to another aspect of an embodiment of the present application, a method for intent recognition is also provided, including: in response to receiving an intent, using a trained intent recognition model to identify the category of the intent, wherein the trained intent recognition model is obtained by training the intent recognition model with difficult negative samples generated by the above-mentioned difficult negative sample generation method.

[0012] According to another aspect of the embodiments of the present application, an electronic device is provided, including: a processor, and a memory storing a program, wherein the program includes instructions, and when the instructions are executed by the processor, the processor executes the method in any of the above embodiments.

[0013] According to another aspect of the embodiments of the present application, a non-transitory machine-readable medium storing computer instructions is also provided, where the computer instructions are used to enable a computer to execute the method in any of the above embodiments.

[0014] According to another aspect of the embodiments of the present application, a computer program product is provided, including a computer program, which, when executed by a processor of a computer, causes the computer to execute the method in any of the above embodiments.

[0015] Beneficial effects of the embodiments of the present application: In an embodiment of the present application, keywords are extracted from the positive samples corresponding to the target intent; a target combination is constructed through the keywords, wherein the target combination is a phrase used to describe the key information of the target intent; the context-guided description corresponding to the target intent and the target combination are input into the large language model LLM to obtain candidate negative samples; the candidate negative samples are screened by semantic similarity to determine the difficult negative samples. That is to say, the embodiment of the present application first uses keyword extraction technology to extract keywords from the positive samples corresponding to the target intent and construct a target combination, and with the help of the powerful emergence ability of the large language model LLM, the context-guided description corresponding to the target intent and the target combination are input into the LLM to generate a large number of candidate negative samples, and then filtered and screened by semantic similarity to generate a large number of high-quality difficult negative samples, thereby solving the technical problem that the difficult negative samples mined in the related technology generally have low quality.

[0016] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other embodiments can be obtained based on these drawings without creative work.

[0018] Figure 1 It is an example diagram of LLM-based extended text in the related art; Figure 2 is a flowchart of a method for generating difficult negative samples according to an embodiment of the present application; Figure 3 is a schematic diagram of a framework of a difficult negative sample generation method according to an embodiment of the present application; Figure 4 is a structural block diagram of a difficult negative sample generation device according to an embodiment of the present application; Figure 5 is a flow chart of an intention recognition method according to an embodiment of the present application; Figure 6 is a structural block diagram of an intention recognition device according to an embodiment of the present application; Figure 7 Schematic diagram of the structure of the electronic device of this embodiment. DETAILED DESCRIPTION

[0019] Embodiments of the present embodiment will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present embodiment are shown in the accompanying drawings, it should be understood that the present embodiment can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein, which are instead provided for a more thorough and complete understanding of the present embodiment. It should be understood that the drawings and embodiments of the present embodiment are only for exemplary purposes and are not intended to limit the scope of protection of the present embodiment.

[0020] In order to better understand the embodiments of the present application, the technical terms involved in the embodiments of the present application are explained as follows: Positive samples are samples corresponding to the category you want to correctly classify. For example, to classify the sentiment of a text, for "positive" sentiment, all sentences with positive sentiment are positive samples, such as: "The food here is so delicious", "The living environment of this hotel is great".

[0021] Negative samples, according to the above example, all other sentences that are not positive emotions are negative samples, such as: "The hotel environment is very poor" and "The price is too expensive".

[0022] Difficult positive samples are those positive samples that are easy to be confused, have low discrimination, and are easily mistakenly predicted as negative samples. They can also be the positive samples with the highest loss during training.

[0023] Difficult negative samples are those negative samples that are easy to confuse, have low discrimination, and are easily mispredicted as positive samples. They can also be the negative samples with the highest loss during training.

[0024] When training classification models, the importance of using negative samples cannot be ignored. They play a vital role in training data, mainly in the following aspects: 1. The generalization ability of the model: In the real world, there are often more negative samples than positive samples. If the model is only exposed to positive samples and not enough negative samples, the model may perform poorly when facing unseen data because it cannot distinguish between negative and positive samples well.

[0025] 2. Prevent overfitting: During the training process, the model may try to remember every detail in the training data, including noise and outliers, which may be negative samples. By introducing negative samples, the model can be better generalized to new data and reduce the risk of overfitting.

[0026] 3. Balanced category distribution: In classification tasks, category imbalance is a common problem, that is, the number of samples in some categories is significantly less than that in other categories. If there are only positive samples but not enough negative samples, the model may tend to favor the category with a larger number, resulting in a decrease in model performance. By introducing a sufficient number of negative samples, it can help balance the category distribution and improve the performance of the model.

[0027] 4. Improve the robustness of the model: Negative samples can help the model better understand the background information of the data, thereby improving the model's tolerance to noise and outliers and making the model more robust.

[0028] Therefore, in the training of classification models, the importance of negative samples should be fully considered, and ensuring that the training data contains a sufficient number of negative samples is a fundamental requirement for improving model performance and generalization ability. However, more importantly, a high-quality combination of negative samples is required. In addition to a sufficient number of negative samples, a sufficient number of high-quality difficult negative samples are also required. According to a study by Meta, in common classification data sets, the vast majority (about 95%) of negative examples are simple negative samples, which are not semantically similar to positive samples and are very easy to distinguish and learn. Therefore, using simple negative samples alone is not enough to train a good model. Secondly, about 5% of negative examples are difficult negative samples, which are semantically similar to positive samples but belong to different categories. These difficult negative samples almost determine the performance of the model and play a key role in training. Therefore, how to construct high-quality difficult negative samples is a very important task in both academia and industry.

[0029] Large Language Model (LLM), also known as large language model, is based on transformer architecture and usually refers to large language models containing tens of billions (or more) parameters. They are trained on massive amounts of text data and then use reinforcement learning from human feedback (RLHF) to optimize the language model, thereby gaining a deeper understanding of the language. At present, well-known LLMs abroad include ChatGPT, PaLM, Claude, and LLaMA, and domestic ones include Wenxin Yiyan, Tongyi Qianwen, ChatGLM, Baichuan, etc.

[0030] One of the most notable features that distinguish large language models (LLMs) from previous pre-trained language models (PLMs) is their emergent capabilities. Emergent capabilities are a surprising ability that is not obvious in small models but is particularly prominent in large models. Similar to the phase transition phenomenon in physics, the emergent capability is like the rapid improvement of model performance as the scale increases, exceeding the random level, which is what we often call quantitative change leading to qualitative change. Emergent capabilities can be related to certain complex tasks, but we are more concerned with its general capabilities. Next, we briefly introduce three typical emergent capabilities of LLM: Contextual learning: The contextual learning capability was first introduced by GPT-3. This capability allows the language model to perform tasks by understanding the context and generating corresponding outputs when provided with natural language instructions or multiple task examples, without the need for additional training or parameter updates.

[0031] Instruction following: Fine-tuning is performed using multi-task data described in natural language, also known as instruction fine-tuning. LLM is shown to perform well on unseen tasks described formally using instructions. This means that LLM is able to perform tasks according to task instructions without having seen specific examples beforehand, demonstrating its strong generalization ability. This case mainly relies on this ability of LLM to perform text expansion very accurately.

[0032] Step-by-step reasoning: Small language models often have difficulty solving complex tasks involving multiple reasoning steps, such as math problems. However, LLMs solve these tasks by adopting a Chain of Thought (CoT) reasoning strategy, using a prompting mechanism that includes intermediate reasoning steps to arrive at the final answer. It is speculated that this ability may be acquired through training on code.

[0033] These emergent capabilities allow LLMs to excel in a variety of tasks, making them powerful tools for solving complex problems and applying them in multiple fields.

[0034] In related technologies, the following solutions are usually used for difficult sample mining: Solution 1: Data level. Data expansion is performed to increase the amount of data. Solution 2: At the model and algorithm level, such as using loss functions such as focal loss that can handle unbalanced data and using the online hard example mining (OHEM) method. The following describes Solution 1 and Solution 2 respectively.

[0035] Solution 1: Data level. Data expansion is used to increase the amount of data. For text classification tasks, the conventional text expansion solution is to generate new samples to expand the data set by applying different transformation techniques (such as synonym replacement, part-of-speech tagging, sentence reorganization, etc.) to the original data. However, this type of method has several main problems: 1. Unstable quality: The conversion process may introduce problems such as syntax errors and semantic inconsistencies, which may reduce data quality.

[0036] 2. Information loss: Some important information of the original sample may be lost during the conversion process, resulting in the expanded data being inaccurate or incomplete.

[0037] 3. The quality of the generated negative samples is not high: This type of method only expands the existing corpus. Although the semantics are similar, they are still syntactically similar to the existing corpus, and there is no way to generate more diverse high-quality negative samples.

[0038] With the popularity of large models (LLM), solutions for using large models to enhance data for intent classification tasks have also been invented. The usual practice is to use a carefully designed context-guided description (prompt) and use a large model to generate similar extended corpus, such as Figure 1 shown. Figure 1 is an example diagram based on the LLM extended text, in Figure 1 The context prompt used in is: “The following sentences belong to the same category {call_wechat_phone}: Sentence 1: Make a WeChat call to Zhang San Sentence 2: Call mom on WeChat Sentence 3: Call Li Si via WeChat Help me generate more sentences that fall into this category:" LLM (taking ChatGPT as an example) generates the following corpus: 1. Send WeChat calls to friends; 2. Call customers using WeChat; 3. Open WeChat and call colleagues; 4. Talk to family members using WeChat; 5. I usually call my father using WeChat; 6. Call friends using WeChat; 7. Initiate WeChat calls to classmates; 8. I am used to contacting colleagues through WeChat; 9. Called my sister using WeChat today; 10. Called group members using WeChat.

[0039] Then, the corpus generated by the large model is added to the specific category to retrain the model. However, the problem with this method is that it can only be used to generate positive samples. In reality, positive samples are relatively easy to obtain, while high-quality negative samples, especially difficult negative samples, are difficult to obtain. Experiments using this method show that the generated data is not effective for intents with similar semantics, but is effective for intents with greater semantic differentiation.

[0040] Solution 2: At the model and algorithm level. The problem of difficult negative samples is often accompanied by the problem of sample imbalance, so we can use focused loss. Its core idea is to increase the contribution of samples with large losses to the gradient. It adds an optimization to the cross entropy, which can make difficult negative samples contribute more to the loss. In its formula, if If the loss is relatively large, that is, a simple sample, the loss is relatively small. Therefore, focusing the loss on the loss function can make the difficult negative samples contribute more to the loss, thereby making the training effect better for difficult negative samples.

[0041] You can also use the Gradient Harmonizing Mechanism (GHM). The key idea is to dynamically adjust the weights of each sample in the loss function so that each sample contributes relatively evenly to the gradient update. It introduces a "gradient balance index" to measure the importance of each sample and weights the gradient according to this index, making the training process smoother and more efficient. The effect of this solution still depends on the quality of the training samples. It still optimizes the current training data, so the improvement of the model is limited.

[0042] You can also use Online Hard Example Mining (OHEM). The core idea of ​​OHEM is to calculate the loss of each sample after several training rounds (epochs) during training, and then add more difficult samples. In practice, in addition to the training data, there is also a data pool for additional data, which is a pool for screening difficult samples. Of course, you can also use a method similar to resampling. The training data is the same as the data pool, and each time you select difficult samples from the data pool and throw them into the training data. OHEM is a relatively heavy training method. Its training is very inefficient and does not solve the core problem, that is, the quality and quantity of data have a decisive impact on the model effect.

[0043] Therefore, for the second solution at the model and algorithm level mentioned above, when the data quality and quantity are insufficient, the effect of simply improving the model and algorithm is very limited.

[0044] In order to solve the above problems, the embodiments of the present application provide relevant solutions, which are described in detail below. Among them, the application scenarios of the embodiments of the present application include but are not limited to vertical domain classification, intent recognition, rejection, arbitration and other text classification scenarios in the dialogue system, for example: 1. Chatbots and virtual assistants. Accurate dialogue understanding: By using negative samples that are very close to positive samples but have different actual intentions, chatbots can understand user intentions more accurately. For example, when a user enters "book a train ticket" and "book a movie ticket", although the sentence structure is similar, the intention is different. The discriminative power of the model can be improved by training with difficult negative samples. Reduce false triggers: In smart home assistants, it is key to avoid misunderstanding users' ordinary conversations as specific commands (such as "turn on the lights"). Using difficult negative samples, such as similar but unintended ordinary conversations, can improve the robustness of the system.

[0045] 2. Voice assistant. Improving speech understanding: By using negative samples that are very similar to the intent in terms of grammar and pronunciation, the accuracy of understanding various user voices can be improved. For example, for two similar commands, "play music" and "play movie", determining different intents helps the system respond accurately. Handling ambiguous input: In the face of ambiguous or unclear user voice input, using difficult negative sample training can help the system make correct judgments even in ambiguous states.

[0046] 3. Natural language query and search system. Optimize search results: Through difficult negative samples, such as those search results that are highly relevant to the user's query but do not meet their actual needs, it helps to train a better ranking model so that the results returned by the search engine are more in line with the user's intention. Accurately match the query intent: In the question-answering system, through negative samples that are difficult to judge the intent (such as semantically similar but different topics), the system can improve its ability to distinguish different types of queries.

[0047] 4. Customer service system. Classify complex inquiries: Customer inquiries are often complex. By selecting negative samples that are very close to a specific category but do not actually belong to that category, the customer service system can more accurately classify different types of inquiries and provide more accurate answers. Automated troubleshooting: In the automated troubleshooting system, using negative samples with similar problems but different solutions can improve the classification accuracy of the system when facing complex problems.

[0048] 5. E-commerce recommendation system. User intention interpretation: In product recommendation, the system can improve its ability to understand the user's true purchase intention through difficult-to-distinguish negative samples (such as products with similar functions but different categories as the target product). Accurate recommendation: Using negative samples of similar but unrelated products, the recommendation system can more accurately recommend products that meet the user's intention, rather than just based on superficial similarities.

[0049] 6. Financial services and trading systems. Understanding transaction intent: Improve the system's ability to identify different financial operation requests through negative samples with similar semantics but different intents (such as "transfer" and "check balance"). Fraud detection: In fraud detection involving user intent, the model's accuracy in identifying fraudulent behavior can be enhanced by selecting negative samples that appear legitimate but have fraudulent intent.

[0050] Figure 2 is a flowchart of a method for generating difficult negative samples according to an embodiment of the present application. Figure 2 As shown, the method comprises the following steps: Step S202, extracting keywords from positive samples corresponding to the target intent.

[0051] It should be noted that the positive samples corresponding to the above-mentioned target intent may be the corpus belonging to this intent. For example, if the target intent is “making a call”, the positive samples corresponding to this target intent may be “making a call to someone”, etc. The negative samples corresponding to this target intent do not belong to the expectations of this intent, for example, “initiating navigation”, “playing music”, etc.

[0052] Optionally, in an embodiment of the present application, methods for extracting keywords from positive samples corresponding to the target intent include but are not limited to: Method 1, calculated by term frequency (TF) and inverse document frequency (IDF); Method 2, a text sorting algorithm based on a graph model; Method 3, based on a rapid automatic keyword extraction (RAKE) algorithm; 4, extracting keywords from positive samples relying on predefined rules and templates, the rules including part-of-speech tagging, position features, context features, etc.; 5, extracting keywords from positive samples based on deep learning methods; 6. Using sequence labeling technology to convert the keyword extraction problem into a sequence labeling problem, using models such as bidirectional long short-term memory networks and conditional random fields (Bi-LSTM-CRF) or based on Transformer models to implement it.

[0053] Step S204, constructing a target combination through the keywords, wherein the target combination is a phrase used to describe key information of the target intention; It should be noted that the target combination can be regarded as a relatively complete intent, which can simplify the expression of user intent, including but not limited to the following composition: verb + noun, verb + noun + noun. For example, if the positive sample is "call someone", then the corresponding target combination can be "call, phone", "dial, phone", "call, phone", "call, phone, Xiao Ming", etc.

[0054] Step S206: input the context-guided description corresponding to the target intent and the target combination into the large language model LLM to obtain candidate negative samples.

[0055] Optionally, the context-guided description corresponding to the target intent includes at least one of the following: a target task description to be executed by the LLM, a constraint description of the target task, a description of a keyword combination instance, and an output description corresponding to the keyword combination instance.

[0056] The emergence ability of large models is manifested in many aspects: 1. Language generation and understanding: Large language models can produce high-quality texts, such as articles, news reports, stories, poems, etc. They can understand and generate natural language, showing a deep understanding of language structure and semantics. 2. Knowledge acquisition and reasoning: Through training, large models can acquire a wide range of knowledge and are able to make inferences and logical inferences. This enables them to answer a variety of questions, including common sense questions and questions in professional fields. 3. Creative expression: Large models can not only restate known facts, but also generate new perspectives, ideas and creativity. They can write, compose, design images, etc., showing a certain degree of creativity. Therefore, based on this emergence ability, contextual learning (In-Context Learning, ICL) can be carried out. It is a new learning paradigm, which means that without updating parameters, only a few examples are needed in the input to let the model learn, by constructing its own templates, instance descriptions and context-guided descriptions (prompts), to complete its specific tasks, predict or generate the final results. Exemplarily, the above context-guided description can be: “#Goal: Help me generate more sentences #Requirements: It needs to contain at least one keyword, the sentence should be concise, and the thinking should be more divergent #example: Keywords: [send, WeChat] Output: "send love", "how to use WeChat", "post a WeChat Moments", "how many WeChat contacts do you have", "open WeChat applet", "WeChat is Tencent's app" Keyword: [%s] Output: " Among them, "[%s]" corresponds to the above target combination.

[0057] Due to the design of context-guided description and the divergence and uncontrollability of the large model itself, it is easy to generate samples with poor correlation. Although they can also be used as negative samples, they cannot meet the requirements of difficult negative samples. Through semantic similarity, they can be in a fuzzy boundary area. Therefore, after the above step S206, the embodiment of the present application also proposes: Step S208 , screening the candidate negative samples by semantic similarity to determine difficult negative samples.

[0058] Through the above steps S202~S208, keywords are extracted from the positive samples corresponding to the target intent; a target combination is constructed through the keywords, wherein the target combination is a phrase used to describe the key information of the target intent; the context-guided description corresponding to the target intent and the target combination are input into the large language model LLM to obtain candidate negative samples; the candidate negative samples are screened by semantic similarity to determine the difficult negative samples. That is to say, the embodiment of the present application first uses the keyword extraction technology to extract keywords from the positive samples corresponding to the target intent and construct a target combination, and with the help of the powerful emergence ability of the large language model LLM, the context-guided description corresponding to the target intent and the target combination are input into the LLM to generate a large number of candidate negative samples, and then filtered and screened by semantic similarity to generate a large number of high-quality difficult negative samples, thereby solving the technical problem that the difficult negative samples mined in the related technology generally have low quality.

[0059] The embodiments of the present application are described below with reference to specific examples.

[0060] This example first extracts keywords to obtain the "seed" combination of each intent (corresponding to the target combination above), and then uses the "seed" combination to form a specific context-guided description (prompt). After generating candidate negative samples through a large model, they are filtered by semantic similarity. This solution has been proven to be very effective and can generate a large number of high-quality difficult negative samples. At the same time, this solution is not limited to the data set and can be easily migrated to other tasks. The overall architecture diagram is as follows: Figure 3 Examples of positive samples, negative samples, and difficult negative samples in the intent recognition task are shown in Table 1.

[0061]

[0062] In the voice assistant's intent recognition task, "make a call" (Phone_CallOneDial) is an intent, and "positive sample" is the corpus belonging to this intent, indicating that the user "makes a call to someone". "Negative sample" is the corpus that does not belong to this intent, which can be initiating navigation, system settings, playing music, etc., which is relatively easy to distinguish. "Difficult negative sample" is a corpus very similar to "positive sample". They may have overlapping vocabulary or similar semantics, which are easy to confuse. This type of sample is very important for model training. If the model is not learned with difficult samples, there will be a lot of noise and interference, which is very easy to predict errors, resulting in the impact of the model's robustness. Therefore, this example is dedicated to generating high-quality "difficult negative samples" to fundamentally improve the performance of the model. This example mainly includes: using keyword extraction to generate "seeds" for the corpus of each intent; constructing a "seed" combination to generate a prompt exclusive to each intent, and using a large model to generate candidate negative samples; screening candidate negative samples by semantic similarity to select the final difficult negative samples.

[0063] In a possible implementation, the above step S202 may include: S11, extracting keywords from the positive samples through a text sorting algorithm based on a graph model.

[0064] It should be noted that the above-mentioned text sorting algorithm based on the graph model can be an algorithm based on TextRank. TextRank is an improvement of PangeRank. The core idea is to treat the words in the text as nodes in the graph, which are connected to each other through edges. Different nodes will have different scores, and nodes with high scores can be used as keywords. The TextRank algorithm can be represented as a directed weighted graph G=(V, E), which consists of a point set V and an edge set E, where E is a subset of V×V. Any two points , The weight of the edge between , in the present embodiment, it is uniformly regarded as 1. For a given point , is the set of points pointing to this point, For point The collection of points pointed to. Point The TextRank score of is defined as shown in Formula 1:

[0065] Among them, d is the damping coefficient, which ranges from 0 to 1 and represents the probability of pointing from a specific point in the graph to any other point, and its value is 0.85.

[0066] Specifically, keywords can be extracted for each category, and the TextRank algorithm is calculated as follows: S21, a certain intent has n samples, each sample is regarded as a separate sentence , all samples are regarded as a large document D, that is: .

[0067] S22, for each sentence, perform word segmentation and part-of-speech tagging, and filter out stop words, retaining only m words with specified parts of speech (such as nouns, verbs, adjectives), where The candidate keywords after retention are: .

[0068] S23, construct a candidate keyword graph G, whose node set is composed of the candidate keywords generated in S22. The co-occurrence relationship is used to construct the edge between any two nodes. The edge between two nodes exists only when their corresponding words co-occur in a window of length K, where K represents the window size, i.e., a maximum of K words co-occur. In the embodiment of the present application, K is set to 4.

[0069] S24, according to the formula of TextRank, iteratively propagate the scores of each node until convergence.

[0070] S25, sorting the node scores in reverse order to obtain the most important N words as candidate keywords, where N is set to 20.

[0071] S26, the most important N words are obtained from S25, and then marked in the original text. If there are two words that are adjacent in the original text, they can form adjacent phrases, and then can be combined into multi-word keywords. Figure 3 The above is an example result after keyword extraction for a certain intent in the telephone vertical domain, where the keywords include: call, dial, call, phone, call to, number, mobile phone, China Mobile, China Unicom, China Telecom, wife, and husband.

[0072] Optionally, the above step S204 may include: S31, determining the part of speech of the keyword; S32, grouping the keywords according to the part of speech; S33, extracting a keyword from each group, and constructing the target combination based on the extracted keywords. The target combination constructed by the above steps S31 to S33 can express a complete intention.

[0073] For example, assuming that the parts of speech of keywords include verbs and nouns, all verbs can be divided into the first group, all nouns can be divided into the second group, and then a keyword is extracted from the first group and the second group respectively, that is, keyword 1 and keyword 2, and then a target combination is constructed based on keyword 1 and keyword 2.

[0074] Optionally, the above step S208 may include: S41, extracting a specified number of positive samples from the positive samples corresponding to the target intent as a sample pool; S42, for any candidate negative sample, obtaining the vector representation of the candidate negative sample and the vector representation of all samples in the sample pool; S43, respectively calculating the cosine distance between the vector representation of the candidate negative sample and the vector representation of all samples in the sample pool to obtain multiple cosine distances; S44, when the multiple cosine distances are all higher than a specified threshold, determining that the candidate negative sample is the difficult negative sample. The above steps S41~S44, screening candidate negative samples by cosine distance to determine difficult negative samples, can significantly improve the model's ability to distinguish, speed up training, enhance generalization ability, and improve the accuracy of feature learning.

[0075] In an embodiment of the present application, the above-mentioned method of obtaining the vector representation of the candidate negative sample includes but is not limited to: method 1, using a pre-trained language model (such as BERT, ELMo, GPT-3, etc.) to generate a vector representation of the text; method 2, using a word vector model (such as Word2Vec, GloVe, FastText) to represent each word as a vector, and then averaging or other aggregation operations on these word vectors to obtain the vector representation of the sentence; method 3, using a deep neural network (Deep Neural Network, DNN) or a convolutional neural network (Convolutional Neural Network, CNN) to directly extract the vector representation from the text; method 4, using a large model to encode the text.

[0076] For example, Figure 3As shown, 200 corpora are randomly selected from each intent category as the standard sample pool (anchor samples), and then each candidate negative sample is represented by a vector with the samples in all the standard sample pools, and the cosine distance between the vectors is calculated as the text semantic similarity. Samples above a certain threshold are screened out according to the semantic similarity. In the embodiment of the present application, the threshold can be 0.75, that is, the final difficult negative sample is obtained. Therefore, correctly calculating the text similarity between samples is a key link. The pre-trained language model (BERT) can be used to calculate the vector representation of the sentence. First, the sentence needs to be segmented and special tags added. Taking the sentence "the sound of answering the phone" as an example, it is divided into the following tag sequence: tokens=[[CLS], answer, listen, phone, of, voice, sound, [SEP]]. Then, these tags are input into the pre-trained BERT model to obtain the word representation of each tag. Next, the [CLS] tag can be used to obtain the vector representation of the entire sentence. After obtaining the sentence vector representation, the cosine similarity between them is calculated. The closer the cosine similarity value is to 1, the more similar the semantics of the two sentences are; conversely, the closer the cosine similarity value is to 0, the less similar the semantics of the two sentences are. Assume that the dimensions of the two vectors x and y are both d. Then, using the cosine calculation method, their semantic similarity is formula 2:

[0077] In addition, sentence vectors obtained by pre-trained language models such as BERT are often anisotropic, which means that the vectors are unevenly distributed and filled in a narrow conical space, which greatly limits their expressive ability. Therefore, cosine similarity cannot be calculated between anisotropic vectors. Therefore, it is necessary to convert the anisotropic sentence vector into an isotropic sentence vector. The anisotropy problem can be alleviated by contrastive learning methods. Therefore, using BERT trained by unsupervised contrastive learning to vectorize sentences can more reasonably calculate the semantic similarity between sentences.

[0078] Table 2 below lists some examples of difficult negative samples generated by intent. It can be found that the quality of the generated difficult negative samples is very high. They are semantically similar and easy to be confused, but they are not the same intent.

[0079]

[0080] After generating difficult negative samples by the above method, the text classification model can be trained. The embodiment of the present application also proposes: S51, obtaining a target positive sample and a target negative sample including the difficult negative sample, and performing self-supervised comparative learning on the text classification model through the target positive sample and the target negative sample; S52, training the text classification model through the self-supervised comparative learning loss, the first cross entropy loss of the target positive sample, and the second cross entropy loss of the expanded sample after the target positive sample is expanded, to obtain a trained text classification model. Through the above steps S51~S52, after generating a large number of high-quality difficult negative samples, the problems of data missing and data balance can be effectively solved, and the text classification model can be further trained, so that the overall performance of the trained text classification model is significantly improved.

[0081] Optionally, the above step S52 includes S521: training the text classification model using the following formula 3 to obtain the trained text classification model:

[0082] in, is the first cross entropy loss determined by the target positive sample, is the second cross entropy loss determined after expanding the target positive sample, is the self-supervised contrastive learning loss, To specify weights.

[0083] Above It can be determined by the following formula 4:

[0084] Among them, C represents the number of categories in the data set, and N represents the number of data in a training batch. represents the true label of the i-th positive sample in category c in a batch, It represents the probability that the i-th positive sample output by the model belongs to the c-th category.

[0085] Above It can be determined by the following formula 5:

[0086] Among them, C represents the number of categories in the data set, and N represents the number of data in a training batch. represents the true label of the i-th augmented sample in category c in a batch, It represents the probability that the i-th augmented sample output by the model belongs to the c-th category.

[0087] Above It can be determined by the following formula 6:

[0088] in, It represents the vector representation of i positive samples after BERT encoding, that is, the latent vector corresponding to [CLS], Represents the vector representation of the i-th augmented sample after BERT encoding. For the above eigenvector pair, represents the cosine similarity of two feature vectors, It is a temperature constant, which is used to control the degree of attention paid by the text classification model to difficult samples. It is not good to be too large or too small. The subscript i identifies the subscript of the sample.

[0089] In summary, the embodiment of the present application uses the powerful emergence capability of the large language model (LLM) to first use keyword extraction technology to extract keywords from each category to build a "seed" combination, and then use the large model to generate a large number of candidate negative samples through carefully designed prompts, and then filter and screen by semantic similarity to generate a large number of high-quality difficult negative samples. This framework is an effective, complete, easy-to-use, and easy-to-migrate method that can be easily applied to various data sets and similar tasks, significantly improving the overall performance of the model.

[0090] Corresponding to the application scenario and method of the method provided in the embodiment of the present application, the embodiment of the present application also provides a difficult negative sample generation device. Figure 4 The figure shows a structural block diagram of a difficult negative sample generation device according to an embodiment of the present application, including: An extraction module 42, used to extract keywords from positive samples corresponding to the target intent; A construction module 44 is used to construct a target combination through the keyword, wherein the target combination is a phrase used to describe key information of the target intention; An acquisition module 46 is used to input the context-guided description corresponding to the target intent and the target combination into a large language model LLM to obtain a candidate negative sample; The determination module 48 is used to screen the candidate negative samples by semantic similarity to determine the difficult negative samples.

[0091] pass Figure 4The device shown extracts keywords from the positive samples corresponding to the target intent; constructs a target combination through the keywords, wherein the target combination is a phrase used to describe the key information of the target intent; inputs the context-guided description corresponding to the target intent and the target combination into the large language model LLM to obtain candidate negative samples; and screens the candidate negative samples by semantic similarity to determine the difficult negative samples. That is to say, the embodiment of the present application first uses the keyword extraction technology to extract keywords from the positive samples corresponding to the target intent and constructs a target combination. With the help of the powerful emergence ability of the large language model LLM, the context-guided description corresponding to the target intent and the target combination are input into the LLM to generate a large number of candidate negative samples, and then filtered and screened by semantic similarity to generate a large number of high-quality difficult negative samples, thereby solving the technical problem that the difficult negative samples mined in the related technology generally have low quality.

[0092] In one possible implementation, the construction module 44 includes: a first determination unit, used to determine the part of speech of the keyword; a grouping unit, used to group the keywords according to the part of speech; and a construction unit, used to extract a keyword from each group and construct the target combination based on the extracted keywords.

[0093] Optionally, the context-guided description corresponding to the target intent includes at least one of the following: a target task description to be executed by the LLM, a constraint description of the target task, a description of a keyword combination instance, and an output description corresponding to the keyword combination instance.

[0094] The determination module 48 includes: a first extraction unit, used to extract a specified number of positive samples from the positive samples corresponding to the target intent as a sample pool; a first acquisition unit, used to obtain the vector representation of any candidate negative sample and the vector representation of all samples in the sample pool; a second acquisition unit, used to respectively calculate the cosine distance between the vector representation of the candidate negative sample and the vector representation of all samples in the sample pool to obtain multiple cosine distances; a second determination unit, used to determine that the candidate negative sample is the difficult negative sample when the multiple cosine distances are all higher than a specified threshold.

[0095] Optionally, the above-mentioned device also includes: a processing module, used to obtain a target positive sample and a target negative sample including the difficult negative sample, and perform self-supervised comparative learning on the text classification model through the target positive sample and the target negative sample; a training module, used to train the text classification model through the self-supervised comparative learning loss, the first cross entropy loss of the target positive sample and the second cross entropy loss of the expanded sample after the target positive sample is expanded, so as to obtain a trained text classification model.

[0096] The training module includes: a training unit, which is used to train the text classification model using the following formula to obtain the trained text classification model: ,in, is the first cross entropy loss determined by the target positive sample, is the second cross entropy loss determined after expanding the target positive sample, is the self-supervised contrastive learning loss, To specify weights.

[0097] In a possible implementation, the extraction module 42 includes: a second extraction unit, configured to extract keywords from the positive sample by using a text sorting algorithm based on a graph model.

[0098] The present application also provides an intention recognition method. Figure 5 is a flow chart of an intention recognition method according to an embodiment of the present application. Figure 5 As shown, the method comprises the following steps: S502, in response to receiving the intent, using a trained intent recognition model to identify the category of the intent, wherein the trained intent recognition model is obtained by training the intent recognition model by combining the difficult negative samples generated by the above method.

[0099] Through the above step S502, the intent recognition model is trained in combination with the difficult negative samples generated by the above method, which helps the model to quickly learn the boundary. After obtaining the trained intent recognition model, the trained intent recognition model is used to identify the category of intent, which significantly improves the accuracy of intent recognition.

[0100] Corresponding to the application scenario and method of the method provided in the embodiment of the present application, the embodiment of the present application also provides an intention recognition device. Figure 6 The structure block diagram of the intention recognition device according to an embodiment of the present application is shown as follows: The recognition module 62 is used to identify the category of the intent using a trained intent recognition model in response to receiving the intent, wherein the trained intent recognition model is obtained by training the intent recognition model by combining the difficult negative samples generated by the above method.

[0101] pass Figure 6 The device shown, combined with the difficult negative samples generated by the above method, trains the intent recognition model, which helps the model to quickly learn the boundary. After obtaining the trained intent recognition model, the trained intent recognition model is used to identify the category of intent, which significantly improves the accuracy of intent recognition.

[0102] The functions of each module in each device in the embodiments of the present application can be found in the corresponding description in the above method, and have corresponding beneficial effects, which will not be repeated here.

[0103] The embodiment of the present application also provides an electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication. The memory stores a computer program executable by the at least one processor, and the computer program is used to enable the electronic device to perform the method of the embodiment of the present application when executed by the at least one processor.

[0104] An embodiment of the present application also provides a non-transitory machine-readable medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to execute the method of the embodiment of the present application.

[0105] The embodiment of the present application also provides a computer program product, including a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to execute the method of the embodiment of the present application.

[0106] refer to Figure 7 , the structural block diagram of the electronic device that can be used as the server or client of the embodiment of the present application will now be described, which is an example of the hardware device that can be applied to various aspects of the present application. The electronic device is intended to represent various forms of digital electronic computer equipment, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices and other similar computing devices. The components shown herein, their connections and relationships, and their functions are only examples, and are not intended to limit the implementation of the present application described and / or required herein.

[0107] like Figure 7 As shown, the electronic device includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 to a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the electronic device can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 707. An input / output (I / O) interface 705 is also connected to the bus 707.

[0108] Multiple components in the electronic device are connected to the I / O interface 705, including: an input unit 706, an output unit 707, a storage unit 708, and a communication unit 709. The input unit 706 can be any type of device that can input information to the electronic device, and the input unit 706 can receive input digital or character information, and generate key signal input related to user settings and / or function control of the electronic device. The output unit 707 can be any type of device that can present information, and can include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 708 can include but is not limited to a disk, an optical disk. The communication unit 709 allows the electronic device to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and can include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0109] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a CPU, a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above. For example, in some embodiments, the method embodiments of the present application may be implemented as a computer program, which is tangibly contained in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on an electronic device via a ROM 702 and / or a communication unit 709. In some embodiments, the computing unit 701 may be configured to perform the above method in any other appropriate manner (e.g., by means of firmware).

[0110] The computer program for implementing the method of the embodiment of the present application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0111] In the context of the embodiments of the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or apparatus. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable signal medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, or infrared system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0112] It should be noted that the term "including" and its variations used in the embodiments of the present application are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". The modifications of "one" and "multiple" mentioned in the embodiments of the present application are illustrative and not restrictive. Those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0113] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of the present application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0114] The various steps described in the method implementation methods provided in the embodiments of the present application can be performed in different orders and / or in parallel. In addition, the method implementation methods may include additional steps and / or omit the steps shown. The scope of protection of the present application is not limited in this respect.

[0115] The term "embodiment" in this specification refers to specific features, structures or characteristics described in conjunction with the embodiment that can be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily mean the same embodiment, nor does it mean that it is mutually exclusive with other embodiments and is independent or optional. The various embodiments in this specification are described in a related manner, and the same or similar parts between the various embodiments refer to each other. In particular, for the device, equipment, and system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts refer to the partial description of the method embodiment.

[0116] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of patent protection. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the scope of protection of the present application. Therefore, the scope of protection of the present application shall be subject to the attached claims.

Claims

1. Difficult negative sample generation method, including: Extract keywords from the positive samples corresponding to the target intent; Constructing a target combination through the keywords, wherein the target combination is a phrase used to describe key information of the target intention; Inputting the context-guided description corresponding to the target intent and the target combination into the large language model LLM to obtain a candidate negative sample; The candidate negative samples are screened by semantic similarity to determine difficult negative samples.

2. The method according to claim 1, constructing a target combination through the keywords, comprising: Determining the part of speech of the keyword; Grouping the keywords according to the parts of speech; A keyword is extracted from each group, and the target combination is constructed based on the extracted keyword.

3. According to the method according to claim 1, the context-guided description corresponding to the target intent includes at least one of the following: a description of the target task to be executed by the LLM, a description of the constraints of the target task, a description of a keyword combination instance, and an output description corresponding to the keyword combination instance.

4. The method according to claim 1, screening the candidate negative samples by semantic similarity to determine the difficult negative samples, comprising: Extracting a specified number of positive samples from the positive samples corresponding to the target intent as a sample pool; For any candidate negative sample, obtain a vector representation of the candidate negative sample and a vector representation of all samples in the sample pool; Calculating the cosine distances between the vector representation of the candidate negative sample and the vector representations of all samples in the sample pool respectively to obtain multiple cosine distances; When the multiple cosine distances are all higher than a specified threshold, the candidate negative sample is determined to be the difficult negative sample.

5. The method according to claim 1, further comprising: Obtaining a target positive sample and a target negative sample including the difficult negative sample, and performing self-supervised comparative learning on a text classification model through the target positive sample and the target negative sample; The text classification model is trained by using the self-supervised contrastive learning loss, the first cross entropy loss of the target positive sample, and the second cross entropy loss of the expanded sample after the target positive sample is expanded to obtain a trained text classification model.

6. The method according to claim 5, training a text classification model by using the self-supervised contrastive learning loss, the first cross entropy loss of the target positive sample, and the second cross entropy loss of the expanded sample after the target positive sample is expanded, to obtain a trained text classification model, comprising: The text classification model is trained using the following formula to obtain the trained text classification model: ,in, is the first cross entropy loss determined by the target positive sample, is the second cross entropy loss determined after expanding the target positive sample, is the self-supervised contrastive learning loss, To specify weights.

7. According to the method of claim 1, extracting keywords from positive samples corresponding to the target intent comprises: Keywords are extracted from the positive samples through a text sorting algorithm based on a graph model.

8. Intent recognition methods, including: In response to receiving an intent, a category of the intent is identified using a trained intent recognition model, wherein the trained intent recognition model is obtained by training the intent recognition model using difficult negative samples generated by the method described in any one of claims 1 to 7.

9. An electronic device comprising a memory, a processor and a computer program stored in the memory, wherein the processor implements the method according to any one of claims 1 to 8 when executing the computer program.

10. A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images