A sample data distribution optimization method, device and storage medium

Abstract

Embodiments of the present application provide a sample data distribution optimization method, device and storage medium. The method includes an end-to-end optimization method, which adaptively finds target keywords with high overfitting risk and uses them as optimization targets for downstream negative sample sampling tasks. Secondly, an improved term frequency-inverse document frequency calculation method is constructed to clearly represent the co-occurrence relationship of keywords within and between multi-intent corpora. Finally, by processing the target keywords and screening logic of new training texts containing the target keywords, a matching negative sample sampling method is obtained to optimize the training text distribution by adding new negative sample corpora. Using the embodiments of the present application, the entire modeling can be optimized for overfitting at the data source.

Description

Technical Field

[0001] This application relates to the Internet field, and in particular to a method, apparatus and storage medium for optimizing sample data distribution. Background Technology

[0002] Text, as one of the most important information carriers today, can enter the internet through various social media platforms, news media, and other means. These text messages vary greatly in format, topic, content, and length, making the proper application and processing of this text information a pressing need. Text classification is a crucial task in intent recognition, with a wide range of applications.

[0003] In practical applications, multi-class models used for intent recognition may be overly complex, fitting noise in the training sample set, or have too few or unrepresentative training samples, or be subject to noise interference in the training samples. This can lead to the model fitting these noises, resulting in overfitting of positive samples of certain data types with large amounts of data. Consequently, too much of that type of content will be generated in the model's prediction results, reducing the richness of the ranking or the classification effect of other types of content.

[0004] Generally, solving the problem of model overfitting involves two aspects: the first aspect is to optimize the model network architecture, such as adding regularization, reducing model training parameters, and optimizing the number of training epochs; the second aspect is to optimize the training data, such as the positive and negative ratio of labeled training samples and the richness of the semantics of labeled data text.

[0005] Optimization from the model perspective takes a long time and may not solve the problem. Optimization from the training data perspective requires manual text annotation to increase richness, or adjustment of the ratio or content of positive and negative text in the training data, and resampling. However, since it is random sampling, the result is unlikely to be exactly what is missing in the training data.

[0006] Therefore, how to solve the problem of model overfitting is an urgent issue that needs to be addressed. Summary of the Invention

[0007] This application provides a sample data distribution optimization method, apparatus, and storage medium, which can perform overfit optimization on the entire modeling process at the data source.

[0008] Firstly, embodiments of this application provide a method for optimizing sample data distribution. This method is a versatile approach capable of optimizing training text in multi-class classification tasks. The method includes:

[0009] Obtain a first training sample set, wherein the first training sample set includes multiple training texts, and each of the multiple training texts is labeled with an intent;

[0010] Extract keywords from multiple training texts in the first training sample set to obtain a keyword set;

[0011] Obtain the word frequency and inverse document frequency of the first keyword in the keyword set in the second training sample set, wherein the first keyword is any keyword in the keyword set, and the second training sample set is a set of training texts with the same intent as the training text to which the first keyword belongs; the word frequency is the number of training texts in the second training sample set that contain the first keyword, and the proportion of the number of training texts in the second training sample set to the number of training texts in the second training sample set; the inverse document frequency is used to characterize the frequency of the first keyword in the second training sample set.

[0012] Based on the term frequency and inverse document frequency of each keyword in the keyword set, the concentration score of each keyword corresponding to each intent is calculated. The concentration score of a keyword corresponding to any intent is used to characterize the degree of concentration of the keyword in the training text containing the intent.

[0013] Target keywords are determined based on the concentration score of each keyword in the keyword set corresponding to each intent.

[0014] Intent labeling is performed on the third training sample set containing the target keywords but without labeled intent, to obtain a negative sample set;

[0015] The first training sample set is processed according to the negative sample set to obtain an updated first training sample set, wherein the first training sample set is used to train a multi-classification model, and the model is used to predict the intent classification of the input sample.

[0016] The key aspect of this application's embodiments lies in optimizing the training data of the classification model to address the problem of overfitting. Common causes of overfitting include: the model itself is too complex, causing it to fit noise in the training sample set. In this case, a simpler model needs to be selected, or the model needs to be pruned; the training samples are too few or lack representativeness. In this case, the number of samples needs to be increased, or the diversity of samples needs to be increased; interference from noise in the training samples causes the model to fit this noise. In this case, noisy data needs to be removed, or a model that is less sensitive to noise needs to be used.

[0017] In multi-class classification tasks, if a word (keyword) appears almost exclusively in only one intent, the model will assume that text containing this word is highly likely to contain that intent. This imbalanced distribution of training data leads to severe overfitting and a significant decrease in recognition performance. Essentially, an extremely flexible model can greatly reduce the training data error (training MSE), but this extreme flexibility comes at the cost of incorporating the training data error into the model estimation process, thus affecting its predictive ability on new data (validation data) (test MSE). Similarly, when the model doesn't adequately fit the patterns in the data, the test MSE will also be relatively large, which is known as underfitting.

[0018] Generally, solving the problem of model overfitting involves two aspects:

[0019] The first is to optimize the training data, such as the positive and negative ratio of labeled training samples and the richness of the semantics of labeled data text.

[0020] The second is to optimize the model network architecture, such as adding regularization processing, reducing model training parameters, and optimizing training epochs.

[0021] Given the same training sample set, a better model and more effective training tuning will result in a better algorithm model. However, during the modeling process, the training data determines the upper limit of the model's performance. Good training text allows the model to converge faster, learn more knowledge, and reduce overfitting.

[0022] This application proposes a general method for optimizing training texts in multi-class classification tasks. This method optimizes overfitting at the data source. The key is to find the source of overfitting in the training samples. Overfitting training samples are generally those where the labels corresponding to a certain word are too concentrated. For example, the word "profit" is generally labeled or intended as "finance," but training text containing "profit" may also be labeled or intended as "internet" or "real estate." If the number of training texts containing "profit" with the label or intention of "finance" is much greater than the number of training texts containing "profit" with the label of "internet" or "real estate," then the model trained on such training texts may directly label input text containing "profit" as "finance." This situation is called model overfitting. Therefore, to address this situation, training texts containing keywords like "profit" are labeled with labels other than "finance," so that when the model encounters input text containing "profit," it can determine results other than "finance."

[0023] Specifically, the existing labeled first training sample set is split to decompose all training texts in the first training sample set into words, resulting in a keyword set. The concentration of each keyword in the keyword set in training texts corresponding to a certain intent is determined. This concentration is used to determine whether there is a situation similar to the above, where the number of training texts containing "profit" with the model's label or intent "finance" is much greater than the number of training texts containing "profit" with the label "internet" or "real estate". If this occurs, it means that the keyword is the target keyword, and corresponding negative texts containing the target keyword need to be generated to maintain the ratio of positive to negative training texts containing the target keyword in the first training sample set, thus solving the overfitting problem of the model.

[0024] In another possible implementation of the first aspect, the step of extracting keywords from multiple training texts in the first training sample set to obtain a keyword set includes:

[0025] The first training sample set is segmented into words using a word segmentation tool to obtain a word set.

[0026] Construct a business keyword library, which includes multiple business-related keywords;

[0027] The keyword set is obtained by filtering the word set based on the business keyword library.

[0028] The key point of this implementation is to determine the keywords in the first training sample set. These keywords are used to characterize the semantics of the training text corresponding to the keywords. It is understood that this step is essential. While the method provided in this embodiment searches for words that appear frequently in the training text corresponding to certain tags within the first training text set, considering that certain modal particles or connecting phrases appear frequently, and that processing all the relevant data for all words in the training text would be time-consuming and inefficient, the training text in the first training sample set is screened to obtain keywords that can characterize the semantics of the text, thereby improving the efficiency of subsequent operations. Furthermore, by constructing a business keyword library, keywords strongly related to the business in the first training sample set are obtained, saving time.

[0029] In another possible implementation of the first aspect, determining the target keywords based on the concentration score corresponding to each intent for each keyword in the keyword set includes:

[0030] The concentration scores of the first keyword in the keyword set corresponding to each intent are sorted to obtain the target intent with the highest concentration score corresponding to the first keyword;

[0031] Based on the concentration score of the intent corresponding to the first keyword, the target score percentage is obtained. The target score percentage is the proportion of the concentration score of the first keyword in the target intent to the total concentration scores of the first keyword corresponding to each intent.

[0032] The first keyword whose target score percentage is higher than a preset threshold is identified as the target keyword.

[0033] In this embodiment, the concentration score is based on the training text of the intent label and can characterize the concentration of the first keyword in the training text corresponding to the intent label. The higher the concentration score, the more frequently the first keyword appears in the training text corresponding to the intent label. Therefore, this application also constructs an improved word frequency-inverse document frequency calculation method for the co-occurrence relationship of keywords within and between classes in multi-intent corpora. The calculation formula is as follows:

[0034] CR = freq * idf

[0035] Where CR is the concentration score, freq is the term frequency, and idf is the inverse document frequency.

[0036] However, it should be noted that the high concentration of the first keyword in the training text corresponding to the intent label does not necessarily mean that it is the source of model overfitting. It is also necessary to look at the concentration of the first keyword in the training text corresponding to other intent labels. If the concentration of the first keyword in the training text corresponding to other intent labels is lower than that in the training text corresponding to the intent label, that is, if the target score ratio exceeds the preset threshold, then it may be the source of model overfitting. Therefore, the process of determining the target keyword is the key to the implementation of this application.

[0037] In another possible implementation of the first aspect, the step of intention-labeling the third training sample set containing the target keyword without labeled intent to obtain a negative sample set includes:

[0038] Construct an unlabeled third training sample set, which includes multiple training samples;

[0039] Search the unlabeled third training sample set for training texts containing the target keywords;

[0040] The training text is input into a multi-classification model trained based on the first training sample set for prediction.

[0041] The prediction results of the multi-classification model are verified to obtain a negative sample set.

[0042] After determining the target keywords, the method filters the target keywords in a historical corpus, which serves as the third training sample set, to obtain training texts containing the target keywords. It should be noted that the training texts containing the target keywords in the unlabeled third training sample set do not have corresponding labels. Manual labeling would be extremely resource-intensive. Therefore, the training texts are input into a multi-classification model trained on the first training sample set for prediction. The prediction results are then checked, and those that do not match the intent labels of the training texts containing the target keywords in the first training sample set are identified as negative samples, which are then aggregated into a negative sample set and input into the first training sample set. This process is repeated until all negative sample sets for the target keywords are determined, thus completing the optimized distribution of positive and negative samples in the first training sample set.

[0043] In another possible implementation of the first aspect, after processing the first training sample set according to the negative sample set to obtain an updated first training sample set, the method further includes:

[0044] Based on the updated first training sample set, the multi-classification model trained on the first training sample set is trained to obtain the updated multi-classification model.

[0045] The updated multi-class classification model is tested based on the test data to obtain test results. The test results include a mean squared error value, which is the magnitude of the mean squared error between the result output by the multi-class classification model based on the test data and the result output based on the training data.

[0046] Based on the preset threshold and the mean squared error, determine whether the updated first training sample set causes the updated multi-classification model to overfit.

[0047] If the multi-classification model experiences overfitting, the sample distribution of the updated first training sample set will be optimized.

[0048] In this implementation, the mean squared error is used to characterize the overfitting degree of the model. This is used to verify the multi-classification model trained on the first training sample set with optimized distribution. This helps to identify and correct any deficiencies in the optimized distribution of the first training sample set. If any problems are found, they are corrected in a timely manner.

[0049] In yet another possible implementation of the first aspect,

[0050] The formula for calculating the inverse document frequency is:

[0051]

[0052] Where idf is the inverse document frequency, m is the number of intents, and Y... j This represents the total number of training texts for the j-th intention. This represents the number of training texts containing keywords in the j-th intent.

[0053] The inverse document frequency in this embodiment differs from the common inverse document frequency calculation method. The inverse document frequency of the keyword for intent i is calculated by the following logic: calculate the inverse document frequency for each of the n-1 intents excluding intent i and take the average to obtain the inverse document statistics between classes.

[0054] In yet another possible implementation of the first aspect, the formula for calculating the target score percentage is:

[0055] The formula for calculating the target score percentage is as follows:

[0056]

[0057] Where score is the percentage of the target score, n is the number of intents in the first training sample set, and max(CR1,CR2,...,CRn) is the highest concentration score of the first keyword in the corresponding intent.

[0058] In this implementation, n intentions are traversed and the operation is repeated n times to obtain the concentration score of the first keyword for each intention. Then, they are sorted from largest to smallest, and the top 1 is the intention with the highest concentration of the first keyword.

[0059] The concentration score of the highest-scoring pair <intent-first keyword> is divided by the sum of the concentration scores of the n intents corresponding to the first keyword. This is the proportion of the top 1 score to the total concentration score, which gives the target score percentage. This indicates whether the first keyword appears in only one intent in the vast majority of cases.

[0060] Secondly, embodiments of this application provide a sample data distribution optimization apparatus, which includes at least a first acquisition unit, an extraction unit, a second acquisition unit, a calculation unit, a determination unit, a labeling unit, and a processing unit. This sample data distribution optimization apparatus is used to implement the method described in any embodiment of the first aspect, wherein the first acquisition unit, extraction unit, second acquisition unit, calculation unit, determination unit, labeling unit, and processing unit are described below:

[0061] The first acquisition unit is used to acquire a first training sample set, wherein the first training sample set includes multiple training texts, and each of the multiple training texts is labeled with an intent;

[0062] The extraction unit is used to extract keywords from multiple training texts in the first training sample set to obtain a keyword set;

[0063] The second acquisition unit is used to acquire the word frequency and inverse document frequency of the first keyword in the keyword set in the second training sample set, wherein the first keyword is any keyword in the keyword set, the second training sample set is a set of training texts with the same intent as the training text to which the first keyword belongs; the word frequency is the number of training texts in the second training sample set that contain the first keyword, and the proportion of the number of training texts in the second training sample set to the number of training texts in the second training sample set; the inverse document frequency is used to characterize the frequency of the first keyword in the second training sample set.

[0064] The calculation unit is used to calculate the concentration score of each keyword corresponding to each intent based on the word frequency and inverse document frequency of each keyword in the keyword set. The concentration score of the keyword corresponding to any intent is used to characterize the degree of concentration of the keyword in the training text containing the intent.

[0065] The determining unit is used to determine the target keywords based on the concentration score of each keyword in the keyword set corresponding to each intent;

[0066] The annotation unit is used to annotate the third training sample set containing the target keyword without annotated intent, so as to obtain a negative sample set;

[0067] The processing unit is configured to process the first training sample set according to the negative sample set to obtain an updated first training sample set, wherein the first training sample set is used to train a multi-classification model, and the model is used to predict the intent classification of the input sample.

[0068] The key aspect of this application's embodiments lies in optimizing the training data of the classification model to address the problem of overfitting. Common causes of overfitting include: the model itself is too complex, causing it to fit noise in the training sample set. In this case, a simpler model needs to be selected, or the model needs to be pruned; the training samples are too few or lack representativeness. In this case, the number of samples needs to be increased, or the diversity of samples needs to be increased; interference from noise in the training samples causes the model to fit this noise. In this case, noisy data needs to be removed, or a model that is less sensitive to noise needs to be used.

[0069] In multi-class classification tasks, if a word (keyword) appears almost exclusively in only one intent, the model will assume that text containing this word is highly likely to contain that intent. This imbalanced distribution of training data leads to severe overfitting and a significant decrease in recognition performance. Essentially, an extremely flexible model can greatly reduce the training data error (training MSE), but this extreme flexibility comes at the cost of incorporating the training data error into the model estimation process, thus affecting its predictive ability on new data (validation data) (test MSE). Similarly, when the model doesn't adequately fit the patterns in the data, the test MSE will also be relatively large, which is known as underfitting.

[0070] Generally, solving the problem of model overfitting involves two aspects:

[0071] The first is to optimize the training data, such as the positive and negative ratio of labeled training samples and the richness of the semantics of labeled data text.

[0072] The second is to optimize the model network architecture, such as adding regularization processing, reducing model training parameters, and optimizing training epochs.

[0073] Given the same training sample set, a better model and more effective training tuning will result in a better algorithm model. However, during the modeling process, the training data determines the upper limit of the model's performance. Good training text allows the model to converge faster, learn more knowledge, and reduce overfitting.

[0074] This application proposes a general method for optimizing training texts in multi-class classification tasks. This method optimizes overfitting at the data source. The key is to find the source of overfitting in the training samples. Overfitting training samples are generally those where the labels corresponding to a certain word are too concentrated. For example, the word "profit" is generally labeled or intended as "finance," but training text containing "profit" may also be labeled or intended as "internet" or "real estate." If the number of training texts containing "profit" with the label or intention of "finance" is much greater than the number of training texts containing "profit" with the label of "internet" or "real estate," then the model trained on such training texts may directly label input text containing "profit" as "finance." This situation is called model overfitting. Therefore, to address this situation, training texts containing keywords like "profit" are labeled with labels other than "finance," so that when the model encounters input text containing "profit," it can determine results other than "finance."

[0075] Specifically, the existing labeled first training sample set is split to decompose all training texts in the first training sample set into words, resulting in a keyword set. The concentration of each keyword in the keyword set in training texts corresponding to a certain intent is determined. This concentration is used to determine whether there is a situation similar to the above, where the number of training texts containing "profit" with the model's label or intent "finance" is much greater than the number of training texts containing "profit" with the label "internet" or "real estate". If this occurs, it means that the keyword is the target keyword, and corresponding negative texts containing the target keyword need to be generated to maintain the ratio of positive to negative training texts containing the target keyword in the first training sample set, thus solving the overfitting problem of the model.

[0076] In another possible implementation of the second aspect, the extraction unit is specifically used for:

[0077] The first training sample set is segmented into words using a word segmentation tool to obtain a word set.

[0078] Construct a business keyword library, which includes multiple business-related keywords;

[0079] The keyword set is obtained by filtering the word set based on the business keyword library.

[0080] The key point of this implementation is to determine the keywords in the first training sample set. These keywords are used to characterize the semantics of the training text corresponding to the keywords. It is understood that this step is essential. While the method provided in this embodiment searches for words that appear frequently in the training text corresponding to certain tags within the first training text set, considering that certain modal particles or connecting phrases appear frequently, and that processing all the relevant data for all words in the training text would be time-consuming and inefficient, the training text in the first training sample set is screened to obtain keywords that can characterize the semantics of the text, thereby improving the efficiency of subsequent operations. Furthermore, by constructing a business keyword library, keywords strongly related to the business in the first training sample set are obtained, saving time.

[0081] In another possible implementation of the second aspect, the determining unit is specifically used for:

[0082] The concentration scores of the first keyword in the keyword set corresponding to each intent are sorted to obtain the target intent with the highest concentration score corresponding to the first keyword;

[0083] Based on the concentration score of the intent corresponding to the first keyword, the target score percentage is obtained. The target score percentage is the proportion of the concentration score of the first keyword in the target intent to the total concentration scores of the first keyword corresponding to each intent.

[0084] The first keyword whose target score percentage is higher than a preset threshold is identified as the target keyword.

[0085] In this embodiment, the concentration score is based on the training text of the intent label and can characterize the concentration of the first keyword in the training text corresponding to the intent label. The higher the concentration score, the more frequently the first keyword appears in the training text corresponding to the intent label. Therefore, this application also constructs an improved word frequency-inverse document frequency calculation method for the co-occurrence relationship of keywords within and between classes in multi-intent corpora.

[0086] However, it should be noted that the high concentration of the first keyword in the training text corresponding to the intent label does not necessarily mean that it is the source of model overfitting. It is also necessary to look at the concentration of the first keyword in the training text corresponding to other intent labels. If the concentration of the first keyword in the training text corresponding to other intent labels is lower than that in the training text corresponding to the intent label, that is, if the target score ratio exceeds the preset threshold, then it may be the source of model overfitting. Therefore, the process of determining the target keyword is the key to the implementation of this application.

[0087] In another possible implementation of the second aspect, the processing unit is specifically used for:

[0088] Construct an unlabeled third training sample set, which includes multiple training samples;

[0089] Search the unlabeled third training sample set for training texts containing the target keywords;

[0090] The training text is input into a multi-classification model trained based on the first training sample set for prediction.

[0091] The prediction results of the multi-classification model are verified to obtain a negative sample set.

[0092] After determining the target keywords, the method filters the target keywords in a historical corpus, which serves as the third training sample set, to obtain training texts containing the target keywords. It should be noted that the training texts containing the target keywords in the unlabeled third training sample set do not have corresponding labels. Manual labeling would be extremely resource-intensive. Therefore, the training texts are input into a multi-classification model trained on the first training sample set for prediction. The prediction results are then checked, and those that do not match the intent labels of the training texts containing the target keywords in the first training sample set are identified as negative samples, which are then aggregated into a negative sample set and input into the first training sample set. This process is repeated until all negative sample sets for the target keywords are determined, thus completing the optimized distribution of positive and negative samples in the first training sample set.

[0093] In yet another possible implementation of the second aspect, the device further includes:

[0094] The training unit is used to train the multi-classification model trained on the first training sample set according to the updated first training sample set, so as to obtain the updated multi-classification model.

[0095] The testing unit is used to test the updated multi-classification model based on test data and obtain test results. The test results include a mean squared error value, which is the magnitude of the mean squared error between the result output by the multi-classification model based on the test data and the result output based on the training data.

[0096] The judgment unit is used to determine whether the updated first training sample set causes the updated multi-classification model to overfit based on a preset threshold and the mean square error value.

[0097] An optimization unit is used to optimize the sample distribution of the updated first training sample set if the multi-classification model experiences overfitting.

[0098] In this implementation, the mean squared error is used to characterize the overfitting degree of the model. This is used to verify the multi-classification model trained on the first training sample set with optimized distribution. This helps to identify and correct any deficiencies in the optimized distribution of the first training sample set. If any problems are found, they are corrected in a timely manner.

[0099] Thirdly, embodiments of this application provide a sample data distribution optimization device, which includes a processor, a memory, and a communication interface; the memory stores a computer program; when the processor executes the computer program, the communication interface is used to send and / or receive data, and the sample data distribution optimization device can perform the method described in the first aspect or any possible implementation of the first aspect.

[0100] It should be noted that the processor included in the sample data distribution optimization device described in the third aspect above can be a processor specifically designed to execute these methods (referred to as a dedicated processor for distinction), or a processor that executes these methods by calling a computer program, such as a general-purpose processor. Optionally, at least one processor may include both dedicated and general-purpose processors.

[0101] Optionally, the computer program described above can be stored in memory. For example, the memory can be a non-transitory memory, such as read-only memory (ROM), which can be integrated with the processor on the same device or disposed on different devices. This application does not limit the type of memory or the arrangement of the memory and the processor.

[0102] In one possible implementation, the at least one memory is located outside the sample data distribution optimization device.

[0103] In yet another possible implementation, the at least one memory is located within the sample data distribution optimization device.

[0104] In another possible implementation, a portion of the memory of the at least one memory is located within the sample data distribution optimization device, while another portion of the memory is located outside the sample data distribution optimization device.

[0105] In this application, the processor and memory may also be integrated into a single device, that is, the processor and memory can be integrated together.

[0106] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed on at least one processor, implements the method described in the first aspect or any of the optional solutions of the first aspect.

[0107] Fifthly, this application provides a computer program product comprising a computer program that, when run on at least one processor, implements the method described in the first aspect or any of the optional solutions of the first aspect.

[0108] Optionally, the computer program product can be a software installation package, which can be downloaded and executed on a computing device when the aforementioned method is required.

[0109] The beneficial effects of the technical solutions provided in the third to fifth aspects of this application can be referred to the beneficial effects of the technical solutions in the first and second aspects, and will not be repeated here. Attached Figure Description

[0110] The accompanying drawings used in the description of the embodiments will be briefly introduced below.

[0111] Figure 1 This is a schematic diagram of the architecture of a sample data distribution optimization system provided in an embodiment of this application;

[0112] Figure 2 This is a flowchart illustrating a sample data distribution optimization method provided in an embodiment of this application;

[0113] Figure 3 This is a flowchart illustrating a sample data optimization and verification method provided in an embodiment of this application;

[0114] Figure 4 This is a schematic diagram of the structure of a sample data distribution optimization device provided in an embodiment of this application;

[0115] Figure 5 This is a schematic diagram of the structure of a sample data distribution optimization device provided in an embodiment of this application. Detailed Implementation

[0116] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0117] The terms "first," "second," "third," and "fourth," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such processes, methods, products, or apparatus.

[0118] The system architecture used in the embodiments of this application is described below. It should be noted that the system architecture and business scenarios described in this application are for the purpose of more clearly illustrating the technical solutions of this application, and do not constitute a limitation on the technical solutions provided in this application. As those skilled in the art will know, with the evolution of system architecture and the emergence of new business scenarios, the technical solutions provided in this application are also applicable to similar technical problems.

[0119] First, the terminology used in one or more embodiments of the present invention will be explained.

[0120] Term frequency, as defined in this paper, is the percentage of a word that appears in all annotated corpora corresponding to a particular intent. For example, if intent A has 2000 annotated texts, the keyword 'a' appears in 1600 of these 2000 texts, with a term frequency of 0.8.

[0121] Inverse Document Frequency (IDF) represents the frequency of a word in all texts (in this case, the text set corresponding to the intent). If a word appears in many texts, its IDF value should be low.

[0122] Overfitting occurs when the training data contains sampling errors, and the complex model incorporates these errors into the model's parameter fitting process. This results in the algorithm performing well on the training set but poorly on the test set, exhibiting poor generalization performance.

[0123] A classification model is a type of supervised learning model that divides samples into different categories. Essentially, the model learns a series of prior features to predict the label of a sample.

[0124] Please see Figure 1 , Figure 1 This is a schematic diagram of the architecture of a sample data distribution optimization system provided in an embodiment of this application. The system includes a server 101 and a terminal 102, wherein the terminal 102 communicates with the server 101 through a network.

[0125] Server 101 can be implemented using a standalone server or a server cluster consisting of multiple servers; this application does not impose specific limitations on this. After receiving an instruction from terminal 102, server 101 first retrieves a first training sample set from the database, performs word segmentation and filtering on each training text in the first training sample set, and obtains a keyword set. Then, server 101 calculates the importance of each keyword in the corresponding second training sample set based on the word frequency and inverse document frequency of any keyword in the keyword set. This importance is used to assess whether the keyword is a target keyword. If the keyword is a target keyword, unlabeled corpus containing the keyword is downloaded from the database, and the corpus is input into a multi-classification model trained on the first training sample set. The model output is processed to obtain a negative sample set. The number of training texts containing each target keyword in the negative sample set is adjusted, and the adjusted negative sample set is combined with the first training sample set to obtain an updated first training sample set. Finally, the updated first training sample set is sent to the database.

[0126] Training samples, such as the first training sample set, and corpora, such as the third training sample set, are stored in a database, which may be located on server 101 or exist independently of server 101.

[0127] Terminal 102 may be, but is not limited to, various personal computers, laptops, smartphones, tablets and portable wearable devices. This application does not impose specific limitations on these. It is mainly used to control server 101. Optionally, it is used to verify the output results obtained after inputting the corpus into the multi-classification model trained by the first training sample set.

[0128] Please see Figure 2 , Figure 2 This is a flowchart illustrating a sample data distribution optimization method provided in an embodiment of this application. This sample data distribution optimization method can be based on... Figure 1 The system architecture diagram shown can also be implemented based on other architectures. This method includes, but is not limited to, the following steps:

[0129] Step S201: Obtain the first training sample set.

[0130] The first training sample set includes multiple training texts, each of which is labeled with an intent.

[0131] Understandably, the first training sample training set can be a collection of texts from different scenarios. For example, in an intelligent multi-turn dialogue scenario, this sample text training set can be dialogue information between the user and the robot. In an intelligent multi-turn dialogue scenario, the training text can be a single sentence of dialogue information between the user and the robot, such as "Change bank card password," "I want to change my phone number," "Change linked phone number," etc. Each sentence segment is a corpus sample. This training text has been pre-annotated manually, meaning that one training text is associated with one corpus tag, and the corpus tag represents the intent of the training text.

[0132] Since the embodiments of this application address the problem of model overfitting, there may be a problem with the ratio of positive to negative samples in the first training sample set.

[0133] Step S202: Extract keywords from multiple training texts in the first training sample set to obtain a keyword set.

[0134] The keywords are words in the training text that can represent the training text. The keyword set refers to the collective term or set of one or more keywords. Text, on the other hand, refers to the written form of language, usually a sentence or a combination of sentences with a complete and systematic meaning. A text can be a sentence, a paragraph, or a chapter, all of which belong to text.

[0135] In an optional implementation, the keyword set is obtained using a word segmentation tool and a keyword database, as follows:

[0136] First, multiple training texts in the first training sample set are segmented using a word segmentation tool to obtain a word set. In this embodiment, the word segmentation tool used is jieba, a commonly used Chinese word segmentation library in the NLP field. Jieba can perform simple word segmentation, parallel word segmentation, and command-line word segmentation, and also supports keyword extraction, part-of-speech tagging, and word position lookup. Optionally, the segmented words and their corresponding training texts are set as keys to facilitate backtracking to their corresponding training texts later and to avoid finding multiple keywords in a single training text. The segmented words are then summarized to obtain the word set.

[0137] Secondly, a business keyword library is constructed according to the requirements. The business keyword library includes multiple business-related keywords. Taking banking business as an example, the corresponding business keyword library contains a variety of banking business terms, such as: bank card, password, withdrawal, remittance, ID card, mobile phone number, etc., which can improve the efficiency of keyword extraction and reduce the model running time.

[0138] Finally, the set of words is filtered according to the business keyword library to obtain the keyword set.

[0139] During this process, the word set is compared one by one with the business keyword library. Words that are successfully compared are added to the keyword set until all keywords in the word set have been compared.

[0140] Step S203: Obtain the word frequency and inverse document frequency of the first keyword in the keyword set in the second training sample set.

[0141] Wherein, the first keyword is any keyword in the keyword set, the second training sample set is a set of training texts with the same intent as the training text to which the first keyword belongs; the word frequency is the number of training texts in the second training sample set that contain the first keyword, and the proportion of the second training sample set to the number of training texts in the second training sample set; and the inverse document frequency is used to characterize the frequency of the first keyword in the second training sample set.

[0142] It should be noted that in step S203, the term frequency and inverse document frequency of each keyword in the keyword set need to be calculated. The second training sample set is associated with the keywords, and each keyword corresponds to at least one second training sample set, and the second training sample sets corresponding to any keyword are different.

[0143] It should also be noted that in this application, the calculation of term frequency and inverse document frequency is based on the intent corresponding to the keyword, in order to calculate the concentration of the keyword in a specific intent and multiple corresponding intents. This is illustrated by an example: the training text corresponding to the keyword "profit" has multiple intent labels, including "finance," "real estate," and "internet." The term frequency is calculated as the percentage of training texts containing "profit" corresponding to the intent "finance." This is done by iterating through each intent label and calculating the percentage of the keyword "profit" in each intent label. The formula is as follows:

[0144]

[0145] Where freq is the word frequency, X w X represents the training text containing keywords in the i-th intent, and X represents the total number of training texts corresponding to the i-th intent. The i-th intent can be any intent labeled on the training text corresponding to the keywords.

[0146] Taking the keyword "profit" as an example, the calculation logic of inverse document frequency is to calculate the inverse document frequency for each of the m-1 intents excluding intent i and take the average to obtain the inverse document frequency of the keyword "profit" among multiple intents other than intent i.

[0147] Understandably, the higher the word frequency, the more important the keyword is to the training text.

[0148] The formula for calculating the inverse document frequency is:

[0149]

[0150] Where idf is the inverse document frequency, m is the number of intents, and Y... j This represents the total number of training texts for the j-th intention. This represents the number of training texts containing keywords in the j-th intent.

[0151] Understandably, the higher the inverse document frequency, the less important the keyword is to the training text.

[0152] When calculating the word frequency of the keyword "profit", the corresponding inverse document frequency is also calculated. The two are inseparable.

[0153] Optionally, after the term frequency and inverse document frequency of the keyword "profit" have been calculated across all intents, the calculation of the term frequency and inverse document frequency of the next keyword can be initiated.

[0154] Generally speaking, TF-IDF (term frequency–inverse document frequency) is a statistical method commonly used in text processing to assess the importance of a word in a document. Simply put, it can be used to extract keywords from documents, calculated based on term frequency and inverse document frequency.

[0155] However, in the embodiments of this application, the calculation method of the inverse document frequency is different from the common inverse document frequency calculation method. The common inverse document frequency represents the frequency of keyword occurrence in all training texts, but the inverse document frequency of this application is calculated based on the training text of a single intent, and each keyword belongs to multiple intents, that is, each keyword will have multiple inverse document frequencies, which are used to determine the concentration of keywords in the first training text of this embodiment.

[0156] Step S204: Calculate the concentration score of each keyword corresponding to each intent based on the word frequency and inverse document frequency of each keyword in the keyword set.

[0157] The concentration score of the keyword corresponding to any intent is used to characterize the degree of concentration of the keyword in training text containing that intent, thereby illustrating the co-occurrence relationship of the keyword within and between classes; the formula for calculating the concentration score is as follows:

[0158] CR = freq * idf

[0159] Where CR is the concentration score, freq is the term frequency, and idf is the inverse document frequency.

[0160] Understandably, a higher concentration score suggests that the keyword is more important to the training text for that intent, but less important to other training texts. To address the characteristics of multi-class training datasets, a concentration calculation method based on the intent-keyword dimension is constructed, improving the overfitting problem caused by uneven distribution in the original training dataset. In actual testing, the training set obtained using this method improves the final test accuracy of the model by approximately 2% under the same classification algorithm.

[0161] Optionally, each keyword in the keyword set has multiple concentration scores, which are related to the intent corresponding to the keyword. Taking the keyword "profit" as an example, the intent labels of the training text corresponding to the keyword "profit" are set to three, including "finance", "real estate" and "internet". Then the concentration score of the keyword "profit" has three, which correspond to the above three intent labels respectively.

[0162] Step S205: Determine the target keywords based on the concentration score of each keyword in the keyword set corresponding to each intent.

[0163] Specifically, the concentration scores of the first keyword in the keyword set corresponding to each intent are sorted to obtain the target intent corresponding to the first keyword with the highest concentration score. Taking the keyword "profit" as an example, the intent labels of the training text corresponding to the keyword "profit" are set to three, including "finance", "real estate" and "internet". The concentration score of the keyword "profit" in the intent "finance" is 90 points, the concentration score of the keyword "profit" in the intent "real estate" is 5 points, and the concentration score of the keyword "profit" in the intent "internet" is 5 points. After sorting and comparison, it can be found that the keyword "profit" has the highest concentration score in the intent "finance", and the target intent is "finance".

[0164] Based on the concentration score of the intent corresponding to the first keyword, the target score percentage is obtained. The target score percentage is the proportion of the concentration score of the first keyword in the target intent to the total concentration scores of the first keyword corresponding to each intent.

[0165] The formula for calculating the target score percentage is as follows:

[0166] The formula for calculating the target score percentage is as follows:

[0167]

[0168] Where score is the percentage of the target score, n is the number of intents in the first training sample set, and max(CR1,CR2,...,CRn) is the highest concentration score of the first keyword in the corresponding intent.

[0169] Continuing with the example above, the target score percentage of the keyword "profit" is 90 / 100 = 0.9. In this embodiment, the preset threshold is 0.85. Therefore, the target score percentage of the keyword "profit" exceeds the preset threshold, so it can be determined as the target keyword. The specific value of the preset threshold needs to be flexibly set according to different task requirements. The target keyword is a word or phrase that leads to a high risk of model fitting.

[0170] Step S206: Perform intent annotation on the third training sample set containing the target keywords but without labeled intent, to obtain a negative sample set.

[0171] The third training sample set includes multiple training samples, all of which contain the target keyword; the target keyword can be one or multiple, and if there are multiple, each target keyword has a corresponding third training sample set.

[0172] In one alternative implementation, an unlabeled third training sample set is first constructed.

[0173] In practical applications, there are various ways to obtain the third training sample set. For example, the operator can send relevant instructions to the execution entity, or send an instruction to obtain the third training sample set. Accordingly, the execution entity, such as the server, will begin obtaining the third training sample set after receiving the instruction. Alternatively, the server can automatically obtain the third training sample set at preset intervals, for example, after a preset interval, the server will automatically obtain the third training sample set. Or, after a preset interval, a terminal with raw corpus extraction capabilities will automatically obtain the third training sample set. This specification does not limit the method of obtaining the third training sample set.

[0174] Furthermore, the third training sample set can be documents in any format, including DOC (Document) format documents, txt format documents, image format documents, and PDF (Portable Document Format) format documents. This specification does not limit the types of documents that can be used.

[0175] After obtaining the third training sample set, the text content of the third training sample set can be extracted: select the corresponding text box extraction tool according to the format of the third training sample set, and then use the text box extraction tool to extract text boxes from the third training sample set. The text boxes contain the text that makes up the text content, or the text that makes up the text content. In this way, selecting the text box extraction tool that corresponds to the format of the third training sample set and extracting the text boxes can improve the accuracy and speed of text content extraction.

[0176] For example, if the obtained third training sample set is in PDF format, then the corresponding pdfominer tool is selected to extract the text from the third training sample set, thereby extracting at least one text box containing text content, thus obtaining the text content of the third training sample set. As another example, if the obtained third training sample set is in image format, then the corresponding Optical Character Recognition (OCR) tool is selected to extract the text from the third training sample set, thereby extracting at least one text box containing text content, thus obtaining the text content of the third training sample set.

[0177] Search the unlabeled third training sample set for training text containing the target keyword. If there is only one target keyword, no search is required. If there are multiple target keywords, the third training sample set is reorganized so that each target keyword has a corresponding third training sample set consisting of training text containing itself.

[0178] The training text containing the target keywords is input into a multi-classification model trained on the first training sample set for prediction, and the prediction results of the multi-classification model are verified. Since the multi-classification model is trained on the first training sample set, the model's output results for the training text containing the target keywords may still exhibit overfitting. Therefore, the output results are verified, and the results identified as containing the target intent are extracted and labeled for further verification. This verification process can be performed manually or by following a pre-set procedure.

[0179] After final review, the corrected training texts are compiled into a negative sample set, while the uncorrected training texts can be used as a supplement to the positive texts in the first training sample set, making it convenient for later expansion of the corpus.

[0180] Step S207: Process the first training sample set according to the negative sample set to obtain the updated first training sample set.

[0181] The first training sample set is used to train a multi-classification model, which is used to predict the intent classification of the input sample.

[0182] Based on the ratio of positive to negative text in the first training sample set, the negative sample set is injected into the first training sample set to obtain an updated first training sample set. Generally, the ratio of positive to negative text in the training data is 1:3, with negative samples far exceeding the number of positive samples. If the number of samples in the negative sample set is insufficient, the negative sample set is padded using methods such as word replacement and noise addition.

[0183] In one optional implementation, after processing the first training sample set according to the negative sample set to obtain an updated first training sample set, the first training sample set after distribution optimization is verified. Figure 3 To explain, Figure 3 This is a flowchart illustrating a sample data optimization and verification method provided in an embodiment of this application. The method includes:

[0184] Step S301: Based on the updated first training sample set, train the multi-classification model trained on the first training sample set to obtain the updated multi-classification model.

[0185] Step S302: Test the updated multi-classification model based on the test data to obtain the test results.

[0186] The test data is other training texts that contain the target keywords and are different from the training texts in the first training sample set. The test results include a mean squared error value, which is the magnitude of the mean squared error between the result output by the multi-classification model based on the test data and the result output based on the training data.

[0187] The formula for calculating the mean square error (MSE) is as follows:

[0188]

[0189] Step S303: Based on the preset threshold and the mean square error, determine whether the updated first training sample set causes the updated multi-classification model to overfit.

[0190] Mean-square error (MSE) is a loss function commonly used in machine learning to represent the degree of difference between predicted and actual values. MSE consists of two parts: the variance of the point estimate and the square of the deviation. Generally speaking, in model training, a small deviation and a large variance can easily lead to overfitting.

[0191] Step S304: If the multi-classification model experiences overfitting, the sample distribution of the updated first training sample set is optimized.

[0192] Optionally, if overfitting occurs again, a negative sample set is obtained, and it is determined whether to inject the negative sample set into the first training sample set or replace the negative samples in the first training sample set, depending on the actual situation. The training text of the negative sample set here should be consistent with the above. Figure 2 The training texts in the negative sample set are different.

[0193] In summary, this application proposes an end-to-end optimization method. First, it adaptively identifies target keywords with high overfitting risk and uses them as optimization targets in downstream negative sample sampling tasks. Second, it constructs an improved word frequency-inverse document frequency calculation method for keyword co-occurrence relationships within and between classes of multi-intent corpora to explicitly represent these relationships. Finally, it obtains a matching negative sample sampling method by processing target keywords and using a new training text filtering logic that includes target keywords, thereby optimizing the training text distribution by adding new negative sample corpora.

[0194] The methods of the embodiments of this application have been described in detail above, and the apparatus of the embodiments of this application is provided below.

[0195] Please see Figure 4 , Figure 4 This is a schematic diagram of the structure of a sample data distribution optimization device 40 provided in this application embodiment. The device 40 can be a device in the server mentioned above. The device 40 may include a first acquisition unit 401, an extraction unit 402, a second acquisition unit 403, a calculation unit 404, a determination unit 405, a labeling unit 406, and a processing unit 407. The detailed description of each unit is as follows.

[0196] The first acquisition unit 401 is used to acquire a first training sample set, wherein the first training sample set includes multiple training texts, and each of the multiple training texts is labeled with an intent.

[0197] Extraction unit 402 is used to extract keywords from multiple training texts in the first training sample set to obtain a keyword set;

[0198] The second acquisition unit 403 is used to acquire the word frequency and inverse document frequency of the first keyword in the keyword set in the second training sample set, wherein the first keyword is any keyword in the keyword set, the second training sample set is a set of training texts with the same intent as the training text to which the first keyword belongs; the word frequency is the proportion of the number of training texts in the second training sample set that contain the first keyword to the second training sample set, and the inverse document frequency is used to characterize the frequency of the first keyword in the second training sample set.

[0199] The calculation unit 404 is used to calculate the concentration score of each keyword corresponding to each intent based on the word frequency and inverse document frequency of each keyword in the keyword set. The concentration score of the keyword corresponding to any intent is used to characterize the degree of concentration of the keyword in the training text containing the intent.

[0200] The determining unit 405 is used to determine the target keyword based on the concentration score of each keyword in the keyword set corresponding to each intent;

[0201] The annotation unit 406 is used to annotate the third training sample set containing the target keyword without annotated intent to obtain a negative sample set;

[0202] Processing unit 407 is configured to process the first training sample set according to the negative sample set to obtain an updated first training sample set, wherein the first training sample set is used to train a multi-classification model, and the model is used to predict the intent classification of the input sample.

[0203] In one possible implementation, the extraction unit 402 is specifically used for:

[0204] The first training sample set is segmented into words using a word segmentation tool to obtain a word set.

[0205] Construct a business keyword library, which includes multiple business-related keywords;

[0206] The keyword set is obtained by filtering the word set based on the business keyword library.

[0207] In one possible implementation, the determining unit 405 is specifically used for:

[0208] The concentration scores of the first keyword in the keyword set corresponding to each intent are sorted to obtain the target intent with the highest concentration score corresponding to the first keyword;

[0209] Based on the concentration score of the intent corresponding to the first keyword, the target score percentage is obtained. The target score percentage is the proportion of the concentration score of the first keyword in the target intent to the total concentration scores of the first keyword corresponding to each intent.

[0210] The first keyword whose target score percentage is higher than a preset threshold is identified as the target keyword.

[0211] In one possible implementation, the processing unit 407 is specifically used for:

[0212] Construct an unlabeled third training sample set, which includes multiple training samples;

[0213] Search the unlabeled third training sample set for training texts containing the target keywords;

[0214] The training text is input into a multi-classification model trained based on the first training sample set for prediction.

[0215] The prediction results of the multi-classification model are verified to obtain a negative sample set.

[0216] In one possible implementation, the device 40 further includes:

[0217] The training unit is used to train the multi-classification model trained on the first training sample set according to the updated first training sample set, so as to obtain the updated multi-classification model.

[0218] The testing unit is used to test the updated multi-classification model based on test data and obtain test results. The test results include a mean squared error value, which is the magnitude of the mean squared error between the result output by the multi-classification model based on the test data and the result output based on the training data.

[0219] The judgment unit is used to determine whether the updated first training sample set causes the updated multi-classification model to overfit based on a preset threshold and the mean square error value.

[0220] An optimization unit is used to optimize the sample distribution of the updated first training sample set if the multi-classification model experiences overfitting.

[0221] Please see Figure 5 , Figure 5This is a schematic diagram of the structure of a sample data distribution optimization device 50 provided in an embodiment of this application. The sample data distribution optimization device 50 includes a processor 501, a communication interface 502, and a memory 503. The processor 501, communication interface 502, and memory 503 can be connected via a bus or other means; this embodiment takes a bus connection as an example.

[0222] The processor 501 is the computing and control core of the sample data distribution optimization device 50. It can parse various instructions and data within the sample data distribution optimization device 50. For example, the processor 501 can be a Central Processing Unit (CPU), which can transmit various interactive data between internal structures of the sample data distribution optimization device 50, and so on. The communication interface 502 can optionally include a standard wired interface or a wireless interface (such as Wi-Fi, mobile communication interface, etc.), which can be used to send and receive data under the control of the processor 501. The communication interface 502 can also be used for the transmission and interaction of internal signaling or instructions of the sample data distribution optimization device 50. The memory 503 is the storage device in the sample data distribution optimization device 50, used to store programs and data. It is understood that the memory 503 here can include the built-in memory of the sample data distribution optimization device 50, or it can include the extended memory supported by the sample data distribution optimization device 50. The memory 503 provides storage space for storing the operating system of the sample data distribution optimization device 50. The storage space also stores program code or instructions required for the processor to perform corresponding operations. Optionally, the storage space may also store relevant data generated after the processor performs the corresponding operation.

[0223] In this embodiment, the processor 501 runs executable program code in the memory 503 to perform the following operations:

[0224] Obtain a first training sample set, wherein the first training sample set includes multiple training texts, and each of the multiple training texts is labeled with an intent;

[0225] Extract keywords from multiple training texts in the first training sample set to obtain a keyword set;

[0226] Obtain the word frequency and inverse document frequency of the first keyword in the keyword set in the second training sample set, wherein the first keyword is any keyword in the keyword set, and the second training sample set is a set of training texts with the same intent as the training text to which the first keyword belongs; the word frequency is the number of training texts in the second training sample set that contain the first keyword, and the proportion of the number of training texts in the second training sample set to the number of training texts in the second training sample set; the inverse document frequency is used to characterize the frequency of the first keyword in the second training sample set.

[0227] Based on the term frequency and inverse document frequency of each keyword in the keyword set, the concentration score of each keyword corresponding to each intent is calculated. The concentration score of a keyword corresponding to any intent is used to characterize the degree of concentration of the keyword in the training text containing the intent.

[0228] Target keywords are determined based on the concentration score of each keyword in the keyword set corresponding to each intent.

[0229] Intent labeling is performed on the third training sample set containing the target keywords but without labeled intent, to obtain a negative sample set;

[0230] The first training sample set is processed according to the negative sample set to obtain an updated first training sample set, wherein the first training sample set is used to train a multi-classification model, and the model is used to predict the intent classification of the input sample.

[0231] In one alternative embodiment, regarding the extraction of keywords from multiple training texts in the first training sample set to obtain a keyword set, the processor 501 is specifically configured to:

[0232] The first training sample set is segmented into words using a word segmentation tool to obtain a word set.

[0233] Construct a business keyword library, which includes multiple business-related keywords;

[0234] The keyword set is obtained by filtering the word set based on the business keyword library.

[0235] In one alternative, in determining the target keywords based on the concentration score corresponding to each intent for each keyword in the keyword set, the processor 501 is specifically configured to:

[0236] The concentration scores of the first keyword in the keyword set corresponding to each intent are sorted to obtain the target intent with the highest concentration score corresponding to the first keyword;

[0237] Based on the concentration score of the intent corresponding to the first keyword, the target score percentage is obtained. The target score percentage is the proportion of the concentration score of the first keyword in the target intent to the total concentration scores of the first keyword corresponding to each intent.

[0238] The first keyword whose target score percentage is higher than a preset threshold is identified as the target keyword.

[0239] In one alternative, regarding the intent annotation of the third training sample set containing the target keyword and lacking labeled intent, to obtain a negative sample set, the processor 501 is specifically configured to:

[0240] Construct an unlabeled third training sample set, which includes multiple training samples;

[0241] Search the unlabeled third training sample set for training texts containing the target keywords;

[0242] The training text is input into a multi-classification model trained based on the first training sample set for prediction.

[0243] The prediction results of the multi-classification model are verified to obtain a negative sample set.

[0244] In one alternative embodiment, the processor 501 is further configured to:

[0245] Based on the updated first training sample set, the multi-classification model trained on the first training sample set is trained to obtain the updated multi-classification model.

[0246] The updated multi-class classification model is tested based on the test data to obtain test results. The test results include a mean squared error value, which is the magnitude of the mean squared error between the result output by the multi-class classification model based on the test data and the result output based on the training data.

[0247] Based on the preset threshold and the mean squared error, determine whether the updated first training sample set causes the updated multi-classification model to overfit.

[0248] If the multi-classification model experiences overfitting, the sample distribution of the updated first training sample set will be optimized.

[0249] This application provides a computer-readable storage medium storing a computer program, the computer program including program instructions, which, when executed by a processor, causes the processor to perform the operations performed by the server in this embodiment.

[0250] This application also provides a computer program product that, when run on a processor, implements the operations performed by the server in the embodiment.

[0251] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

Claims

1. A method for optimizing sample data distribution, characterized in that, The method comprises: obtaining a first training sample set, wherein the first training sample set comprises a plurality of training texts, and each of the plurality of training texts is labeled with an intent; extracting keywords of the plurality of training texts in the first training sample set to obtain a keyword set; obtaining a term frequency and an inverse document frequency of a first keyword in the keyword set in a second training sample set, wherein the first keyword is any keyword in the keyword set, the second training sample set is a set of training texts with the same intent as the training text to which the first keyword belongs; the term frequency is the number of training texts containing the first keyword in the second training sample set, and the proportion of the second training sample set, and the inverse document frequency is used to represent the frequency of the first keyword appearing in the second training sample set; calculating a concentration score of each keyword corresponding to each intent according to the term frequency and the inverse document frequency of each keyword in the keyword set, wherein the concentration score of the keyword corresponding to any intent is used to represent the concentration degree of the keyword in the training text containing the any intent; determining a target keyword according to the concentration score of each keyword in the keyword set corresponding to each intent; annotating a third training sample set containing unmarked intents with the target keyword to obtain a negative sample set; processing the first training sample set according to the negative sample set to obtain an updated first training sample set, wherein the first training sample set is used to train a multi-classification model, and the model is used to predict the intent classification of an input sample, wherein the annotating the third training sample set containing unmarked intents with the target keyword to obtain a negative sample set comprises: constructing an unmarked third training sample set, wherein the third training sample set comprises a plurality of training samples; searching for training texts containing the target keyword in the unmarked third training sample set; inputting the training texts into a multi-classification model trained according to the first training sample set for prediction; verifying the prediction result of the multi-classification model to obtain a negative sample set; the calculation formula of the inverse document frequency is: wherein, df is the inverse document frequency, m is the number of intents, represents the total number of training texts of the jth intent, represents the number of training texts containing the keyword in the jth intent.

2. The method of claim 1, wherein, the extracting the keywords of the plurality of training texts in the first training sample set to obtain a keyword set comprises: performing word segmentation processing on the plurality of training texts in the first training sample set by using a word segmentation tool to obtain a word set; constructing a business keyword library, wherein the business keyword library comprises a plurality of business-related keywords; screening the word set according to the business keyword library to obtain the keyword set.

3. The method of claim 1, wherein, the determining a target keyword according to the concentration score of each keyword in the keyword set corresponding to each intent comprises: sorting the concentration scores of the first keyword in the keyword set corresponding to each intent to obtain a target intent corresponding to the first keyword with the highest concentration score. obtaining a target score proportion according to the concentration score of the intent corresponding to the first keyword, the target score proportion being a proportion of the concentration score of the first keyword in the target intent in a total of the concentration scores of the first keyword corresponding to each intent; determining the first keyword with the target score proportion higher than a preset threshold as the target keyword.

4. The method of claim 1, wherein, After the first training sample set is processed according to the negative sample set to obtain an updated first training sample set, the method further includes: training a multi-classification model trained according to the first training sample set according to the updated first training sample set to obtain an updated multi-classification model; testing the updated multi-classification model according to test data to obtain a test result, the test result including a mean square error value, the mean square error value being a numerical value of a mean square error between a result output by the multi-classification model according to the test data and a result output according to training data; judging whether the updated first training sample set causes the updated multi-classification model to have an overfitting problem according to a preset threshold and the numerical value of the mean square error; if the multi-classification model has the overfitting problem, optimizing sample distribution of the updated first training sample set.

5. The method of claim 3, wherein a calculation formula of the target score proportion is: Wherein, the score is the target score proportion, n is the number of intents in the first training sample set, is the highest concentration score of the first keyword in the corresponding intent.

6. A sample data distribution optimization apparatus characterized by comprising: The apparatus includes: a first obtaining unit, configured to obtain a first training sample set, wherein the first training sample set includes a plurality of training texts, and each of the plurality of training texts is labeled with an intent; an extracting unit, configured to extract keywords of the plurality of training texts in the first training sample set to obtain a keyword set; a second obtaining unit, configured to obtain a term frequency and an inverse document frequency of a first keyword in the keyword set in a second training sample set, wherein the first keyword is any keyword in the keyword set, the second training sample set is a set of training texts with the same intent as the training text to which the first keyword belongs, the term frequency is a proportion of a number of training texts containing the first keyword in the second training sample set to the second training sample set, and the inverse document frequency is used to represent a frequency of occurrence of the first keyword in the second training sample set; a calculating unit, configured to calculate concentration scores of each keyword corresponding to each intent according to the term frequency and the inverse document frequency of each keyword in the keyword set, the concentration score of the keyword corresponding to any intent being used to represent a concentration degree of the keyword in the training text containing the any intent; a determining unit, configured to determine a target keyword according to the concentration scores of each keyword in the keyword set corresponding to each intent; a labeling unit, configured to label a third training sample set containing an unmarked intent of the target keyword to obtain a negative sample set. The processing unit is configured to process the first training sample set according to the negative sample set to obtain an updated first training sample set, wherein the first training sample set is used to train a multi-classification model, and the model is used to predict an intention classification of an input sample. The labeling unit is specifically configured to: construct an unlabeled third training sample set, the third training sample set including a plurality of training samples; search for training texts containing the target keyword in the unlabeled third training sample set; input the training texts into the multi-classification model trained according to the first training sample set for prediction; verify the prediction result of the multi-classification model to obtain a negative sample set; The calculation formula of the inverse document frequency is: wherein, df is the inverse document frequency, m is the number of intents, denotes the total number of training texts for the jth intent, denotes the number of training texts that contain the keyword in the jth intent.

7. A sample data distribution optimization device, characterized by, The sample data distribution optimization device includes at least one processor, a communication interface, and a memory. The communication interface is configured to send and / or receive data. The memory is configured to store a computer program. The at least one processor is configured to call the computer program stored in the at least one memory to implement the method according to any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and when the computer program runs on the processor, the method according to any one of claims 1-5 is implemented.

Images