Data enhancement method and device, medium and equipment
Through the methods of text replacement and intent tag filtering, the problem of insufficient data generalization ability of small sample in human-computer voice interaction is solved, the accuracy of question-and-answer and data quality is improved, and more efficient data expansion and robustness is achieved.
Patent Information
- Application Number
- CN202310769440.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-27
- Publication Date
- 2024-12-27
AI Technical Summary
In human-computer voice interaction scenarios, there are many forms of expression of the same problem, resulting in insufficient generalization ability of small sample data, difficulty in improving the accuracy of question-and-answer and data quality, low effectiveness of existing data enhancement methods, and poor robustness of the model.
By obtaining the original problem sample marked with intent labels, using text replacement to generate candidate problem samples, and identify the intent labels of the candidate problem samples through the preset text intent classification model, filtering candidate problem samples consistent with the original problem intention as the expanded Target problem sample.
The robustness and Q&A quality of small sample data expansion are improved, ensuring that the expanded target question sample is consistent with the original question sample intention, and improving the Q&A matching accuracy and data quality.
Smart Images
Figure CN119204006A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the technical field of natural language processing, and in particular to a data enhancement method, device, medium and equipment. Background Art
[0002] In the current human-computer voice interaction scenario, questions and their corresponding answers are often used as training data to train the corresponding algorithm model, and then the trained algorithm model is relied upon for human-computer voice interaction. In practice, there are many kinds of questions, and even the same question may have various forms of expression, which leads to very few samples of the expression form of some questions, that is, such questions become small sample data. In this case, when using existing methods to generate answers for these small sample data, the answers are often not 100% accurate and the generalization ability is not strong. Therefore, how to improve the quality of questions and the accuracy of questions and answers, there is an urgent need for a solution to effectively expand the question data. Summary of the invention
[0003] In order to solve the above technical problems, the present disclosure provides a data enhancement method, device, medium and equipment to improve the data quality of the expanded target problem sample.
[0004] The present disclosure provides a data enhancement method, comprising:
[0005] Obtain original question samples annotated with first intention labels;
[0006] Performing data enhancement on the original question sample by using text replacement to obtain multiple candidate question samples;
[0007] Identify a second intent label for each of the candidate question samples;
[0008] The candidate question sample corresponding to the second intention label that is the same as the first intention label is determined as the expanded target question sample.
[0009] In some embodiments, the data enhancement of the original question sample by using text replacement to obtain multiple candidate question samples includes:
[0010] Determining at least two enhancement modes and a distribution ratio between the enhancement modes;
[0011] The original question samples are data enhanced according to each of the enhancement methods to obtain a plurality of candidate question samples under each of the enhancement methods; wherein the number of candidate question samples under each of the data enhancement methods conforms to the distribution ratio.
[0012] In some embodiments, the data enhancement of the original question sample by using text replacement to obtain multiple candidate question samples includes:
[0013] Performing word segmentation processing on the original question sample to obtain multiple word segments and the part of speech of each word segment;
[0014] Determining non-keywords from the plurality of segmented words according to the part of speech;
[0015] The non-keywords in the original question sample are replaced with text to obtain a plurality of candidate question samples.
[0016] In some embodiments, the text replacement of the non-keywords in the original question sample includes:
[0017] A preset mask is used to perform text replacement on part of the non-keywords in the original question sample.
[0018] In some embodiments, the text replacement of the non-keywords in the original question sample includes:
[0019] The non-keyword in the original question sample is replaced with a text word that meets preset conditions; wherein the preset conditions include at least one of the following: the text word has the same part of speech as the non-keyword, and the semantic similarity between the text word and the non-keyword reaches a preset similarity threshold.
[0020] In some embodiments, the data enhancement of the original question sample by using text replacement to obtain multiple candidate question samples includes:
[0021] Convert the original problem sample into an original vector sample;
[0022] Performing text enhancement processing on the original vector sample to obtain multiple candidate vector samples;
[0023] The candidate vector samples are converted into candidate question samples.
[0024] In some embodiments, the step of identifying the second intent label of each of the candidate question samples includes:
[0025] Identifying the second intent label of each of the candidate question samples through a preset text intent classification model;
[0026] The text intent classification model is obtained by training based on training question samples marked with benchmark intent labels; the benchmark intent labels include: function description, usage, comparison intent and yes / no judgment.
[0027] In some embodiments, it also includes:
[0028] Obtaining an answer sample to the original question sample;
[0029] The original question sample and the expanded target question sample are used as a question combination, and the question combination and the answer sample form a question-answer pair;
[0030] Human-computer voice interaction is performed according to the question-answer pair.
[0031] The present disclosure provides a data enhancement device, comprising:
[0032] A sample acquisition module, used to acquire an original question sample annotated with a first intention label;
[0033] A data enhancement module, used to perform data enhancement on the original question sample by using text replacement to obtain multiple candidate question samples;
[0034] An intention recognition module, used to identify a second intention label of each candidate question sample;
[0035] The effective sample determination module is used to determine the candidate question sample corresponding to the second intention label that is the same as the first intention label as the expanded target question sample.
[0036] The present disclosure also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a program or instruction, wherein the program or instruction enables a computer to execute the steps of any of the above methods.
[0037] The present disclosure also provides an electronic device, comprising:
[0038] one or more processors;
[0039] A memory for storing one or more programs or instructions;
[0040] The processor is used to execute the steps of any of the above methods by calling the program or instruction stored in the memory.
[0041] Compared with the prior art, the technical solution provided by the embodiments of the present disclosure has the following advantages:
[0042] The technical solution provided by the disclosed embodiment first obtains the original question sample marked with the first intent label; uses text replacement to perform data enhancement on the original question sample to obtain multiple candidate question samples; then identifies the second intent label of each candidate question sample; and determines the candidate question sample corresponding to the second intent label that is the same as the first intent label as the expanded target question sample. This technical solution performs data enhancement on the original question sample, and on the basis of ensuring the diversity of the enhanced data, it also uses the intent label to perform validity screening on the enhanced candidate question samples, so that the final determined target question sample is more likely to have a difference in intent with the original question sample. Figure 1This improves the data quality of the expanded target question samples and the robustness of small sample data expansion of the original question samples, which can also help improve the quality of question answering. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.
[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0045] Figure 1 A flowchart of a data enhancement method provided in an embodiment of the present disclosure;
[0046] Figure 2 A flowchart of another data enhancement method provided in an embodiment of the present disclosure;
[0047] Figure 3 A structural block diagram of a data enhancement device provided in an embodiment of the present disclosure;
[0048] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION
[0049] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.
[0050] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.
[0051] Human-computer voice interaction has a wide range of applications in many fields, such as Internet of Vehicles, smart home, smart education, industry and medical fields. Taking the human-computer voice interaction scenario in the field of Internet of Vehicles as an example, users often ask questions about vehicle operation, after-sales service, etc. in IM (Instant Messenger) or voice system, and query answers for these questions in the question and answer database. In the standard question and answer database, the answer data is standard, standardized, and limited in quantity. Correspondingly, the question data that corresponds to the answer data one by one can be considered as typical small sample data. However, in practice, users ask questions in a variety of ways, which are very colloquial. The same question can have a variety of expressions, which leads to a small number of samples of the expression form of some questions, that is, such questions become small sample data. In view of this situation, when using the existing method to generate answers for these small sample data, the answers are often not 100% accurate and the generalization ability is not strong; in other words, the small number of question data in the question and answer database is difficult to accurately match questions with various question forms. Therefore, the scarcity of question data samples will lead to inaccurate question and answer responses. Therefore, effective expansion of question data is a key factor in improving question quality and question and answer accuracy. Currently, data augmentation is generally performed directly to expand small sample data. However, the effectiveness of this method is too low, resulting in poor model robustness and failure to improve the quality of question and answering.
[0052] In view of the shortage of question data under the above small sample conditions, which leads to low accuracy of question and answer, the embodiments of the present disclosure provide a data enhancement method, device, medium and equipment, which can effectively enhance question data based on intent to improve the accuracy of question and answer matching in question and answer scenarios. For ease of understanding, the embodiments of the present disclosure are described in detail below.
[0053] Figure 1 A flow chart of a data enhancement method provided for an embodiment of the present disclosure. The method can be performed by a data enhancement device, which can be implemented in software and / or hardware, such as an electronic device or a server. Among them, the electronic device may include a mobile phone, a tablet computer, a desktop computer, a laptop computer and other devices with communication functions. The server may be a cloud server or a server cluster and other devices with storage and computing functions. It should be noted that the following embodiments are explained exemplarily with electronic devices as the execution subject.
[0054] like Figure 1 As shown, the data enhancement method includes the following steps:
[0055] S102, obtaining an original question sample marked with a first intention label.
[0056] In some embodiments, the original question sample may be a question sample obtained from a public website or the like. When the original question sample is obtained, the correct answer corresponding to the original question sample is also obtained as the corresponding answer sample.
[0057] In other embodiments, for specific fields such as Internet of Vehicles and smart home, original question samples and corresponding answer samples can be obtained through relevant institutions in the specific fields. For example, in the field of Internet of Vehicles, the corresponding original question samples and corresponding answer samples can be obtained from the internal database of the vehicle.
[0058] The original question sample may be in the form of text and / or audio, and the answer sample may be in the form of at least one of text, image, audio and video.
[0059] In this embodiment, the intent category of each original question sample is identified by manual labeling or model classification, and the first intent label of each original question sample is labeled. The model used for intent identification of original question samples can be constructed by machine learning models such as TextCNN and FastText, and is trained based on training question samples labeled with baseline intent labels.
[0060] The first intention label annotated with the original question sample is used to represent the classification result of the question intent. Taking the question-answering scenario in the automotive field as an example, the first intention labels can be divided into the following five categories: function description, usage, comparison intention, yes or no judgment, and others. The first intention label can be text information; alternatively, the text information can be first mapped into machine codes such as numbers and letters that are easy for the computer to recognize, and then the machine code can be used as the first intention label. Exemplarily, the first intention label is "function description". Then, the original question sample can be directly labeled with "function description" in the form of text information; alternatively, the "function description" can be first mapped to the number "0" according to the preset mapping relationship between the text information and the number, and then the original question sample can be labeled with the number "0".
[0061] S104, using text replacement to perform data enhancement on the original question sample to obtain multiple candidate question samples.
[0062] This embodiment may use a single enhancement method to perform data enhancement on the original sample data to increase the diversity of samples and simulate data changes encountered in real scenarios.
[0063] This embodiment can also form a data enhancement strategy by combining different enhancement methods and setting the distribution ratio between each enhancement method; that is, the data enhancement strategy includes multiple enhancement methods with distribution ratios. The above distribution ratio is used to represent the quantitative ratio of the candidate question samples obtained under each enhancement method in the overall candidate question samples. The original question samples are data enhanced by the data enhancement strategy to obtain multiple candidate question samples after enhancement. This embodiment performs data enhancement on the original question samples through multiple enhancement methods to achieve more diverse data changes and expand samples.
[0064] S106, identifying the second intention label of each candidate question sample.
[0065] In this embodiment, the second intent label of each candidate question sample is identified by a preset text intent classification model; the text intent classification model may be the above-mentioned model for identifying the first intent label of the original question sample.
[0066] S108, determining the candidate question sample corresponding to the second intention label that is the same as the first intention label as the expanded target question sample.
[0067] It is understandable that the intention of the candidate question sample after data enhancement may change from that of the original question sample, so the candidate question sample cannot accurately match the answer sample of the original question sample. Based on this, this embodiment can refer to the following method to judge the validity of the expanded candidate question sample:
[0068] Determine whether the second intent label of the candidate question sample is consistent with the first intent label of the original question sample. If they are consistent, the candidate question sample is determined as a valid target question sample, and the target question sample can be added to the database for storing the original question sample. If they are inconsistent, the candidate question sample is discarded. The above method can improve the robustness of small sample data expansion such as the original question sample by screening out valid target question samples from multiple candidate question samples based on the consistency of the intent labels.
[0069] The disclosed embodiment provides a data enhancement method, which first obtains the original question sample marked with a first intent label; uses text replacement to perform data enhancement on the original question sample to obtain multiple candidate question samples; then identifies the second intent label of each candidate question sample; and determines the candidate question sample corresponding to the second intent label that is the same as the first intent label as the expanded target question sample. This technical solution performs data enhancement on the original question sample, and on the basis of ensuring the diversity of the enhanced data, it also uses the intent label to perform validity screening on the enhanced candidate question samples, so that the final determined target question sample is more likely to be different from the original question sample in terms of intent. Figure 1 This improves the data quality of the expanded target question samples and the robustness of small sample data expansion of the original question samples, which can also help improve the quality of question answering.
[0070] Based on the above embodiments, Figure 2 This is a flowchart of another data enhancement method in an embodiment of the present disclosure. Optionally, the method may include the following contents.
[0071] S201, obtaining an original question sample marked with a first intention label.
[0072] In an exemplary embodiment, in a question-and-answer scenario in the field of Internet of Vehicles, a question-and-answer database about vehicle use can be obtained from automobile-related organizations. Obtain one-to-one corresponding original question samples and answer samples from the database, and mark the first intention label of the original question sample. The first intention label can be divided into the following five categories: function description, usage, comparison intention, yes or no judgment, and others; set the mapping relationship between the five categories of first intention labels and numbers, as shown below: {0: function description; 1: usage; 2: comparison intention; 3: yes or no judgment; 4: others}; map the first intention label to a digital expression according to the above mapping relationship.
[0073] S202: Determine at least two enhancement modes and distribution ratios between the enhancement modes.
[0074] S203, performing data enhancement on the original question sample according to each enhancement method, to obtain a plurality of candidate question samples under each enhancement method; wherein the number of candidate question samples under each data enhancement method conforms to a distribution ratio.
[0075] The following provides a description of this step in a variety of enhanced ways.
[0076] In some embodiments, performing data enhancement on the original question sample by using text replacement to obtain multiple candidate question samples may include:
[0077] (1) Perform word segmentation on the original question sample to obtain multiple word segments and the part of speech of each word segment.
[0078] In some specific ways of word segmentation processing, the original question sample can be matched with a preset dictionary to obtain multiple word segmentations in the original question sample, and the part of speech of each word segmentation can be determined at the same time. Parts of speech include: nouns, verbs, pronouns, numerals, quantifiers, prepositions, etc. Alternatively, a statistical machine learning model such as a hidden Markov model and a maximum entropy model can be used to perform word segmentation processing on the original question sample to obtain multiple word segmentations and the part of speech of each word segmentation. Of course, the above are only two specific examples of word segmentation processing and should not be understood as limitations.
[0079] (2) Determine non-keywords from multiple word segments according to part of speech. It can be understood that part of speech generally classifies words into content words with actual meanings that can independently act as sentence components, and function words that do not represent real meanings, cannot be used as grammatical components alone, but have grammatical meanings or functions. Based on this, in this embodiment, it can be determined whether a word segment has an actual meaning and can act as a sentence component according to part of speech, and the word segments with actual meanings and capable of acting as sentence components are determined as keywords. Correspondingly, the word segments without actual meanings and / or incapable of acting as sentence components are determined as non-keywords. Exemplarily, referring to the original question sample in Table 1 below as "Can you introduce ACC?", where ACC is the abbreviation of Adaptive Cruise Control, representing an adaptive cruise control system. The word segments and their parts of speech of this original question sample can be specifically referred to in Table 1, and the words "Can (auxiliary verb)", "a bit (adverbial)", "ma (particle)", and "? (punctuation mark)" can be considered non-keywords.
[0080] (3) Perform text replacement on the non-keywords in the original question sample to obtain multiple candidate question samples.
[0081] In this embodiment, text replacement such as masking operation and word replacement can be performed on the non-keywords in the original question sample. Specifically, the following several methods can be referred to.
[0082] Text replacement method 1: Use a preset mask to perform text replacement on some non-keywords in the original question sample to obtain multiple candidate question samples. Specifically, the preset mask
mask
mask
mask
mask
[0083] Table 1
[0084]
[0085] Referring to the above method, any one or several non-keywords can be randomly masked to obtain multiple candidate question samples.
[0086] Text replacement method 2: Replace the non-keywords in the original question sample with text words that meet preset conditions to obtain multiple candidate question samples; where the preset conditions include at least one of the following: the text words have the same part of speech as the non-keywords, and the semantic similarity between the text words and the non-keywords reaches a preset similarity threshold.
[0087] Specifically, in this embodiment, the following preset conditions are determined by considering the part of speech and semantics: the part of speech of the text word is the same as that of the non-keyword, and the semantic similarity between the text word and the non-keyword reaches a preset similarity threshold. Among them, the same part of speech can make the meaning and components of the words before and after text replacement remain similar or unchanged, and try to avoid the change of the problem intention. Regarding the semantic similarity reaching the preset similarity threshold, it is also to ensure that the information such as the meaning of the words, the essence of the described matters, and the logical relationship changes little before and after text replacement. The similarity threshold can be set according to the actual situation. For example, if more candidate problem samples are needed, the similarity threshold can be set lower; if candidate problem samples with higher similarity are needed, the similarity threshold can be set higher.
[0088] In this embodiment, at least one non-keyword to be replaced can be randomly determined from the original problem samples first, and then according to the above preset conditions, the non-keyword is replaced with a text word with the same part of speech and similar semantics. For example, "can" is replaced with "be able to", and "ma" is replaced with "me", so as to obtain the following candidate problem sample "Be able to introduce ACC me?"
[0089] Text replacement method three: Mask the non-keywords in the original problem samples through a pre-trained masked language model, and then perform text prediction on the masked part to obtain multiple candidate problem samples. Masked language models such as BERT, Robeta and other models. When using a masked language model for text enhancement, it is difficult to determine which part of the original text to mask, which may result in the generated new text not being able to retain the meaning of the original text. In contrast, in this embodiment, the maskable non-keywords are first determined, and then the masked language model is directly used to mask and predict the non-keywords, which can better overcome the above problems and better maintain the semantic similarity between the candidate problem samples and the original problem samples.
[0090] In some other embodiments, using text replacement to perform data augmentation on the original problem samples to obtain multiple candidate problem samples may include: converting the original problem samples into original vector samples; performing text augmentation processing on the original vector samples to obtain multiple candidate vector samples; converting the candidate vector samples into candidate problem samples.
[0091] Specifically, in order to obtain more candidate question samples and achieve better text enhancement effect, this embodiment first converts the original question sample into a vector form through text representation models such as Word2Vec (word to vector, a model for generating word vectors) and Glove (Global Vectors for Word Representation, a word representation tool based on global word frequency statistics) to obtain the original vector sample. Then, the original vector sample is subjected to text enhancement processing, and the text enhancement processing is, for example, replacing the original vector sample with a plurality of candidate vector samples whose spatial distance is within a preset distance range, wherein the candidate vector sample can be obtained in advance using an existing method, and the candidate vector sample has the same dimension as the original vector sample, so the candidate vector sample and the original vector sample can be grouped into one space (or coordinate system), and then the spatial distance between the two vectors can be obtained by using methods such as cosine angle. Then, the candidate vector sample is converted into a text form to obtain a candidate question sample.
[0092] In other embodiments, synonym replacement, random insertion, random exchange, random deletion, etc. are performed on non-keywords in the original question samples to obtain multiple candidate question samples.
[0093] The above embodiments provide a variety of data enhancement methods, and the original question sample can be enhanced by using any one of the methods alone, or the original question sample can be enhanced separately according to a variety of enhancement methods to obtain multiple candidate question samples under each enhancement method, and the number of candidate question samples under each data enhancement method conforms to a preset distribution ratio.
[0094] Exemplarily, the original question sample is segmented to obtain multiple segmented words and the part of speech of each segmented word; non-keywords are determined from the multiple segmented words according to the part of speech; a preset mask is used to perform text replacement on some non-keywords in the original question sample to obtain multiple candidate question samples, and the multiple candidate question samples can be represented as A{A1, A2, A3...A n}.
[0095] The original question sample is converted into an original vector sample; the original vector sample is subjected to text enhancement processing to obtain multiple candidate vector samples; the candidate vector samples are converted into candidate question samples. The multiple candidate question samples can be represented as B{B1, B2, B3...B m}.
[0096] According to the preset distribution ratio, samples are selected from the first candidate question sample and the second candidate question sample to obtain the final candidate question sample. For example, assuming that 100 candidate question samples are ultimately needed, then according to the preset 2:3 distribution ratio, samples are selected from the candidate question samples A{A1, A2, A3...A n} select 100×2 / 5=40 candidate question samples (which can be expressed as A'), and select candidate question samples B{B1, B2, B3...B m} selects 100×3 / 5=60 candidate question samples (which can be expressed as B'), and determines 40 candidate question samples A' and 60 candidate question samples B' as the final candidate question samples for data enhancement.
[0097] It can be understood that in the above embodiments, the candidate question samples obtained by using different enhancement methods have different characteristics; for example: the above method 1 uses a preset mask to perform text replacement, which only covers some non-keywords and does not add new words, and has almost no impact on the semantics; the above method 2 performs text replacement based on preset conditions, which will replace non-keywords with other words with similar semantics; the above vector-based text enhancement processing can perform text enhancement at different stages of encoding and decoding from text to vector and then to text; (not listed one by one here).
[0098] In this case, this embodiment combines different enhancement methods and sets the distribution ratio between different enhancement methods to make up for the shortcomings of a single enhancement method, integrate the advantages of multiple enhancement methods, achieve better text enhancement effects, and make the expanded candidate question samples more rich and diverse.
[0099] S204, identifying the second intent label of each candidate question sample through a preset text intent classification model. The text intent classification model is trained based on training question samples marked with baseline intent labels; the baseline intent labels include: function description, usage, comparison intent and yes / no judgment. In order to enhance the pertinence of the text intent classification model in the field of Internet of Vehicles, the training question samples in this embodiment can be automobile data specially provided by automobile organizations, which can improve the generalization ability of the model.
[0100] S205 , judging whether the candidate question sample is valid according to whether the second intention label of the candidate question sample is consistent with the first intention label of the original question sample.
[0101] S206: If they are consistent, the candidate question sample is determined to be valid, and the candidate question sample is determined as the expanded target question sample.
[0102] S207: If they are inconsistent, the candidate question sample is determined to be invalid and the candidate question sample is discarded.
[0103] For example, see Table 2 below. The first row indicates that the original question sample is "Can you introduce ACC?", and its first intent label is 0; the second row and subsequent rows indicate multiple candidate question samples and their second intent labels.
[0104] Assuming that the number of candidate question samples obtained through the above step S203 is 100, then after the effectiveness screening of steps S205 to S207, the final effective target question samples will be less than 100, such as 80. In the above embodiment, according to the consistency of the second intent label and the first intent label, the effective target question samples are screened out from multiple candidate question samples, which can improve the robustness of small sample data expansion such as the original question sample.
[0105] Table 2
[0106]
[0107] S208, performing human-computer voice interaction based on the original question sample, the target question sample and the answer sample.
[0108] In this embodiment, the answer sample of the original question sample can be obtained first; then the original question sample and the expanded target question sample are used as a question combination, and the question combination and the answer sample are combined to form a question-answer pair; finally, human-computer voice interaction is performed based on the question-answer pair.
[0109] In summary, in view of the scarcity of original question samples under small sample conditions, the disclosed embodiment provides a data enhancement method, which identifies the second intent label of each candidate question sample after data enhancement of the original question sample, and then selects the effective target question sample from the candidate question sample according to the consistency between the first intent label and the second intent label, thereby ensuring the consistency between the target question sample and the original question sample. Figure 1 This improves the data quality of the expanded target question samples and the robustness of the small sample data expansion of the original question samples, which can help improve the question-answer matching accuracy and the question-answer quality in the question-answer scenario.
[0110] Corresponding to the data enhancement method provided by the embodiment of the present disclosure, the embodiment of the present disclosure also provides a data enhancement device. Figure 3 A structural block diagram of a data enhancement device provided in an embodiment of the present disclosure, such as Figure 3 As shown, the data enhancement device includes:
[0111] A sample acquisition module 310 is used to acquire an original question sample marked with a first intention label;
[0112] A data enhancement module 320 is used to perform data enhancement on the original question sample by using text replacement to obtain multiple candidate question samples;
[0113] An intention identification module 330, used to identify a second intention label of each candidate question sample;
[0114] The valid sample determination module 340 is configured to determine the candidate question sample corresponding to the second intention label that is the same as the first intention label as the expanded target question sample.
[0115] In some embodiments, the data enhancement module 320 is specifically used to:
[0116] Determining at least two enhancement modes and a distribution ratio between the enhancement modes;
[0117] The original question samples are data enhanced according to each of the enhancement methods to obtain a plurality of candidate question samples under each of the enhancement methods; wherein the number of candidate question samples under each of the data enhancement methods conforms to the distribution ratio.
[0118] In some embodiments, the data enhancement module 320 is specifically used to:
[0119] Performing word segmentation processing on the original question sample to obtain multiple word segments and the part of speech of each word segment;
[0120] Determining non-keywords from the plurality of segmented words according to the part of speech;
[0121] The non-keywords in the original question sample are replaced with text to obtain a plurality of candidate question samples.
[0122] In some embodiments, the data enhancement module 320 is specifically used to:
[0123] A preset mask is used to perform text replacement on part of the non-keywords in the original question sample.
[0124] In some embodiments, the data enhancement module 320 is specifically used to:
[0125] The non-keyword in the original question sample is replaced with a text word that meets preset conditions; wherein the preset conditions include at least one of the following: the text word has the same part of speech as the non-keyword, and the semantic similarity between the text word and the non-keyword reaches a preset similarity threshold.
[0126] In some embodiments, the data enhancement module 320 is specifically used to:
[0127] Convert the original problem sample into an original vector sample;
[0128] Performing text enhancement processing on the original vector sample to obtain multiple candidate vector samples;
[0129] The candidate vector samples are converted into candidate question samples.
[0130] In some embodiments, the intention recognition module 330 is specifically used to:
[0131] Identifying the second intent label of each of the candidate question samples through a preset text intent classification model;
[0132] The text intent classification model is obtained by training based on training question samples marked with benchmark intent labels; the benchmark intent labels include: function description, usage, comparison intent and yes / no judgment.
[0133] In some embodiments, a voice interaction module is further included, which is used to:
[0134] Obtaining an answer sample to the original question sample;
[0135] The original question sample and the expanded target question sample are used as a question combination, and the question combination and the answer sample form a question-answer pair;
[0136] Human-computer voice interaction is performed according to the question-answer pair.
[0137] The data enhancement device disclosed in the above embodiments can execute the data enhancement method disclosed in the above embodiments, and has the same or corresponding beneficial effects. To avoid repetition, it will not be described again here.
[0138] The embodiment of the present disclosure also provides a computer-readable storage medium, which stores a program or instruction, and the program or instruction enables a computer to execute the steps of any of the above methods.
[0139] Obtain original question samples annotated with first intention labels;
[0140] Performing data enhancement on the original question sample by using text replacement to obtain multiple candidate question samples;
[0141] Identify a second intent label for each of the candidate question samples;
[0142] The candidate question sample corresponding to the second intention label that is the same as the first intention label is determined as the expanded target question sample.
[0143] Optionally, when the computer executable instructions are executed by a computer processor, they can also be used to execute the technical solutions of any of the above-mentioned data enhancement methods provided in the embodiments of the present disclosure to achieve corresponding beneficial effects.
[0144] Through the above description of the implementation methods, the technicians in the relevant field can clearly understand that the embodiments of the present disclosure can be implemented with the help of software and necessary general hardware, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the embodiments of the present disclosure is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment of the present disclosure.
[0145] The embodiment of the present disclosure also provides an electronic device, including: one or more processors; a memory for storing one or more programs or instructions; the processor calls the programs or instructions stored in the memory to execute the steps of any of the above methods to achieve corresponding beneficial effects.
[0146] Figure 4 Schematic diagram of the hardware structure of the electronic device provided by the embodiment of the present disclosure. Figure 4 As shown, the electronic device includes one or more processors 401 and a memory 402 .
[0147] The processor 401 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.
[0148] The memory 402 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 401 may run the program instructions to implement the data enhancement method of the embodiment of the present disclosure described above, and / or other desired functions. Various contents such as input signals, signal components, noise components, etc. may also be stored in the computer-readable storage medium.
[0149] In one example, the electronic device may further include: an input device 403 and an output device 404, and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).
[0150] In addition, the input device 403 may also include, for example, a keyboard, a mouse, and the like.
[0151] The output device 404 can output various information to the outside, including the determined distance information, direction information, etc. The output device 404 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.
[0152] Of course, to simplify, Figure 4 Only some of the components related to the present disclosure in the electronic device are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device may further include any other appropriate components.
[0153] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0154] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A data enhancement method, characterized in that: include: Obtain original question samples annotated with first intention labels; Performing data enhancement on the original question sample by using text replacement to obtain multiple candidate question samples; Identify a second intent label for each of the candidate question samples; The candidate question sample corresponding to the second intention label that is the same as the first intention label is determined as the expanded target question sample.
2. The method according to claim 1, characterized in that The data enhancement of the original question sample is performed by using text replacement to obtain multiple candidate question samples, including: Determining at least two enhancement modes and a distribution ratio between the enhancement modes; The original question samples are data enhanced according to each of the enhancement methods to obtain a plurality of candidate question samples under each of the enhancement methods; wherein the number of candidate question samples under each of the data enhancement methods conforms to the distribution ratio.
3. The method according to claim 1 or 2, characterized in that: The data enhancement of the original question sample is performed by using text replacement to obtain multiple candidate question samples, including: Performing word segmentation processing on the original question sample to obtain multiple word segments and the part of speech of each word segment; Determining non-keywords from the plurality of segmented words according to the part of speech; The non-keywords in the original question sample are replaced with text to obtain a plurality of candidate question samples.
4. The method according to claim 3, characterized in that The step of performing text replacement on the non-keywords in the original question sample includes: A preset mask is used to perform text replacement on part of the non-keywords in the original question sample.
5. The method according to claim 3, characterized in that: The step of performing text replacement on the non-keywords in the original question sample includes: The non-keyword in the original question sample is replaced with a text word that meets preset conditions; wherein the preset conditions include at least one of the following: the text word has the same part of speech as the non-keyword, and the semantic similarity between the text word and the non-keyword reaches a preset similarity threshold.
6. The method according to claim 1 or 2, characterized in that: The data enhancement of the original question sample is performed by using text replacement to obtain multiple candidate question samples, including: Convert the original problem sample into an original vector sample; Performing text enhancement processing on the original vector sample to obtain multiple candidate vector samples; The candidate vector samples are converted into candidate question samples.
7. The method according to claim 1, characterized in that The identifying the second intention label of each candidate question sample includes: Identifying the second intent label of each of the candidate question samples through a preset text intent classification model; The text intent classification model is obtained by training based on training question samples marked with benchmark intent labels; the benchmark intent labels include: function description, usage, comparison intent and yes / no judgment.
8. The method according to claim 1, characterized in that Also includes: Obtaining an answer sample to the original question sample; The original question sample and the expanded target question sample are used as a question combination, and the question combination and the answer sample form a question-answer pair; Human-computer voice interaction is performed according to the question-answer pair.
9. A data enhancement device, characterized in that: include: A sample acquisition module, used to acquire an original question sample annotated with a first intention label; A data enhancement module, used to perform data enhancement on the original question sample by using text replacement to obtain multiple candidate question samples; An intention recognition module, used to identify a second intention label of each candidate question sample; The effective sample determination module is used to determine the candidate question sample corresponding to the second intention label that is the same as the first intention label as the expanded target question sample.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program or instruction, and the program or instruction enables a computer to execute the steps of the method according to any one of claims 1 to 8.
11. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more programs or instructions; The processor is used to execute the steps of the method according to any one of claims 1 to 8 by calling the program or instruction stored in the memory.