Sample processing method, classification model training method, computing device, and storage medium

Abstract

The present disclosure relates to a sample processing method, a classification model training method, a computing device and a storage medium. The sample processing method comprises: obtaining a training sample set; determining that a positive sample quantity distribution of a plurality of vertical domains in the training sample set is unbalanced, determining at least one vertical domain in which a distribution probability of a positive sample is less than a probability distribution threshold from the plurality of vertical domains; and determining at least part of negative samples of the at least one vertical domain, and updating a label value of the at least one vertical domain in the at least part of negative samples to a negative sample label value which is not zero. The method makes the at least part of negative samples be partially fitted on the small sample vertical domain by changing the label value about the vertical domain in the at least part of negative samples of the small sample vertical domain to a non-zero value, thereby increasing the recall rate of the small sample vertical domain.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of artificial intelligence, and in particular, to a sample processing method, a classification model training method, a computing device and a storage medium. BACKGROUND

[0002] In the field of artificial intelligence, a neural network model can be used to perform a classification task, which includes multi-label classification. Multi-label classification refers to the classification of a sample into several categories at the same time, i.e., the labels of several categories of the sample can be 1 or more than the decision threshold (threshold for determining whether to belong to a certain category) at the same time, for example, a given picture can be predicted to be a dog or a cat, and its fur can be long or short. SUMMARY

[0003] Embodiments of the present disclosure provide a sample processing method, a classification model training method, and corresponding computing devices and non-transitory machine-readable storage media for executing these methods.

[0004] According to a first aspect of embodiments of the present disclosure, a sample processing method is provided, including: obtaining a training sample set; determining that the number of positive samples in a plurality of vertical domains in the training sample set is unevenly distributed, determining at least one vertical domain from the plurality of vertical domains in which the distribution probability of the positive samples is less than a probability distribution threshold; and determining at least a part of the negative samples in the at least one vertical domain, and updating the label value of the at least one vertical domain in the at least a part of the negative samples to a non-zero negative sample label value.

[0005] Optionally, determining that the number of positive samples in a plurality of vertical domains in the training sample set is unevenly distributed, and determining at least one vertical domain from the plurality of vertical domains in which the distribution probability of the positive samples is less than a probability distribution threshold, includes: calculating the information entropy of the probability distribution of the positive samples in each vertical domain, and determining that the number of positive samples in a plurality of vertical domains in the training sample set is unevenly distributed if the information entropy is less than an information entropy threshold; and regarding the vertical domain in which the distribution probability of the positive samples is less than the probability distribution threshold as the at least one vertical domain.

[0006] Optionally, the label value of other vertical domains in the at least a part of the negative samples remains unchanged.

[0007] Optionally, the sample processing method further includes: randomly generating a positive value less than 0.1 as the negative sample label value.

[0008] Optionally, the negative sample label value is inversely proportional to the number of positive samples in the at least one vertical domain.

[0009] Optionally, the training samples in the training sample set comprise inquiries received by the dialogue system, and obtaining the training sample set comprises obtaining, as at least part of the training sample set, samples with only one positive sample label value.

[0010] Optionally, the sample processing method further comprises: accessing natural language understanding services of each vertical domain of the dialogue system using the unlabeled sample, and determining a recall result for the unlabeled sample; and for the unlabeled sample, setting label values of vertical domains with the recall result as positive sample label values, setting label values of vertical domains without the recall result as negative sample label values, and adding the sample with the set label values to the training sample set.

[0011] According to a second aspect of the embodiments of the present disclosure, a classification model training method is provided, comprising: obtaining a training sample set; determining that a number of positive samples in a plurality of vertical domains in the training sample set is unevenly distributed, determining at least one vertical domain from the plurality of vertical domains, a distribution probability of a positive sample in the at least one vertical domain being less than a probability distribution threshold; determining at least part of negative samples in the at least one vertical domain, updating label values of the at least one vertical domain in the at least part of negative samples to non-zero negative sample label values; and obtaining the training sample set after the at least part of negative samples are updated, and training a classification model.

[0012] Optionally, the classification model training method further comprises: updating the training sample set to obtain an updated training sample set after each training of the classification model, and iteratively training the classification model until a trained classification model satisfying a training stop condition is obtained.

[0013] Optionally, the training stop condition comprises that a loss function no longer decreases or a preset number of training iterations is reached, and the loss function of the classification model is determined by label values of each vertical domain in all training samples in the training sample set and probability values of each vertical domain output by the classification model.

[0014] According to a third aspect of the embodiments of the present disclosure, a computing device is provided, comprising: a processor; and a memory having executable code stored thereon, when the executable code is executed by the processor, causing the processor to perform the method of the first aspect or the second aspect.

[0015] According to a fourth aspect of the embodiments of the present disclosure, a non-transitory machine-readable storage medium is provided, having executable code stored thereon, when the executable code is executed by a processor of an electronic device, causing the processor to perform the method of the first aspect or the second aspect. BRIEF DESCRIPTION OF DRAWINGS

[0016] The above and other objects, features and advantages of the present disclosure will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings in which like reference characters refer to like parts throughout the figures, and in which:

[0017] Figure 1 A schematic flowchart of a method of training a classification model according to at least one embodiment of the present disclosure is shown.

[0018] Figure 2A An example of label values of training samples according to the prior art is shown.

[0019] Figure 2B An example of label values of training samples according to at least one embodiment of the present disclosure is shown.

[0020] Figure 2C An example of label values of training samples according to at least one embodiment of the present disclosure is shown.

[0021] Figure 3 A schematic flowchart of a method of obtaining a set of training samples according to at least one embodiment of the present disclosure is shown.

[0022] Figure 4 A structural schematic diagram of a computing device according to at least one embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0023] Preferred embodiments of the present disclosure will be described herein below with reference to the accompanying drawings. While preferred embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the present disclosure to those skilled in the art.

[0024] In some dialog systems, the dialog with a user is usually divided into multiple domains, and a specialized natural language understanding (NLU, natural language understanding, hereinafter referred to as NLU) service is provided for each domain, so that the dialog with the user can be better implemented. In this case, in order to ensure the recall rate, improve the accuracy, or reduce the access to the NLU service of each domain to meet the service performance requirements, etc., before the NLU processing of the query input by the user, the query is usually classified to determine which domain or domains the query belongs to, so that the NLU service of which domain or domains should be called, and unnecessary calling of the NLU service of the irrelevant domain is avoided. This is particularly applicable in the case of many domains. The query can be classified by using a general multi-classification scheme or a multi-label classification scheme. Compared with the general multi-classification scheme, the multi-label classification scheme can ensure that the probability value of each class is distributed between 0 and 1.

[0025] However, the present disclosure finds that when classification is performed using a classification model, in the case of an unbalanced distribution of training samples provided for the model, the problem of under-recall of small-sample domains (small-sample classes) is prone to occur. The unbalanced distribution of training samples refers to a large difference in the number of positive samples of each domain / class. For example, in some cases, the navigation domain has more function points, and the number of positive samples of its training data may be 200,000, while the system setting domain has fewer function points, and the number of positive samples for training may be only 10,000, so there is an unbalanced distribution of training samples among each domain / class. In addition, in the present disclosure, the expression "small sample" refers to a small proportion of positive samples, and accordingly, the expressions "small-sample domain" or "small-sample class" refer to a domain / class with a much smaller number of positive samples than other domains / classes, as will be described in detail later, various methods can be used to identify small-sample domains or small-sample classes.

[0026] Therefore, the present application proposes an improved sample processing method and a corresponding classification model training method, which increases the learning of the classification model for a small-sample class by changing the label value (for example Figures 2B-2C Although the foregoing discusses the specific application scenario of classifying the query of the dialog system into a domain, and the application scenario will be taken as an example to help understand the present disclosure hereinafter, those skilled in the art can understand that, unless otherwise specified, the embodiments of the present application are applicable to all application scenarios of the classification scheme, and are not limited to the above-mentioned domain classification scenario. In addition, although the multi-label classification is taken as an example to help understand the present disclosure hereinafter, those skilled in the art can understand that, unless otherwise specified, the embodiments of the present application are also applicable to all classification schemes or classification models.

[0027] At least some embodiments of the solutions of the present disclosure will be described below in detail with reference to the accompanying drawings. Figures 1-3

[0028] A schematic flowchart of a method for training a classification model is shown according to at least one embodiment of the present disclosure. Figure 1 As

[0029] shown in FIG. 10, in step S110, a training sample set is obtained. Figure 1 In some embodiments, the classification model can be any model in the field of machine learning that can be used for classification, including various neural network models. The so-called training sample set refers to a set of training samples used when training the model, which is not intended to be limited to the entire set of samples used for training in this article, but can also be a set of part of the training samples, such as a set of training samples used in a training phase or a training batch, etc. In order to achieve the classification task, the training sample set usually contains positive samples of various categories, and for each category, the positive samples of other categories are equivalent to the negative samples of the category. In the case of multi-label classification, a training sample can have positive sample label values of multiple categories, for example, the label values of multiple categories are all 1 (positive sample label value), which means that the training sample can belong to multiple categories and is a positive sample of the multiple categories.

[0030] Various ways can be used to obtain the training sample set of the classification model. For example, in the case of the aforementioned vertical domain classification of the inquiry of the dialogue system, the NLU service of each vertical domain can be used to automatically label samples, thereby generating training samples, such as

[0031] shown in FIG. 11. Figure 3

[0032] A schematic flowchart of a method for obtaining a training sample set is shown according to at least one embodiment of the present disclosure. As Figure 3 shown in FIG. 12, at least two ways (one can include steps S210-S230, and the other can include step S240) can be used to obtain training samples respectively. Figure 3 The method of Figure 3 is applicable to the aforementioned vertical domain classification scenario, in which the training samples in the training sample set include inquiries received by the dialogue system, and the multi-label classification model is used to determine the vertical domain to which the inquiry belongs.

[0033] As Figure 3 shown in FIG. 12, in the branch shown in steps S210-S230, the NLU service of each vertical domain of the dialogue system can be used to automatically label samples, thereby generating training samples.

[0034] Specifically, in step S210, an unlabeled query can be acquired, and then in step S220, the unlabeled query is used to access the NLU services of the various vertical domains of the dialogue system, and it is determined whether the NLU services of the various vertical domains have recall results for the query. In this context, the expression "have recall results" means that the NLU service of a vertical domain recognizes the query and the query includes the intent related to the vertical domain, and the expression "have no recall results" means that the NLU services of all vertical domains have no recognition results for the query, or the NLU service of a vertical domain recognizes the query but the query does not include the intent related to the vertical domain.

[0035] Then, in step S230, the label values of the query with respect to the various vertical domains are set according to whether the NLU services of the vertical domains have recall results. Specifically, for the query, the label values of the vertical domains with recall results can be set as positive sample label values, and the label values of the vertical domains without recall results can be set as negative sample label values. The query thus labeled (i.e., with label values set) can become a training sample and be added to the training sample set. In this context, the expression "positive sample label value" means a label value indicating that it is a positive sample, and the expression "negative sample label value" means a label value indicating that it is a negative sample. Usually, "1" and "0" are used as "positive sample label value" and "negative sample label value", respectively, but are not limited thereto. For example, in some cases, a probability value can be used as a label value, and a threshold value is set, and a label value greater than or equal to the threshold value is a positive sample label value, and a label value less than the threshold value is a negative sample label value.

[0036] As described above, through the operations of steps S210-S230, training samples can be automatically labeled and generated. However, in some cases, this approach can have some room for improvement: for example, in the case of many vertical domains, there can be many vertical domain NLU services that have recall results, so that a training sample is divided into too many vertical domains, causing the trained model to classify too many vertical domains when classifying related queries in practice, resulting in too much vertical domain data being issued and not achieving the desired effect.

[0037] For example, an unlabeled query might be "nursery rhyme phone call." Assuming the current dialogue system has three vertical categories: phone, music, and navigation, the label obtained by accessing the vertical category's NLU service as described above might be: Phone: 1, Music: 1, Navigation: 0 (the part before the colon indicates the vertical category, and the part after indicates the corresponding label value). However, the actual vertical category for this query should be music, meaning the actual label should be: Phone: 0, Music: 1, Navigation: 0. If only steps S210-S230 are used for automatic labeling and generating training samples, the label values ​​of the training samples may be inaccurate. This could lead to a query being assigned to an unnecessary vertical category during subsequent classification, increasing the number of vertical categories assigned. Furthermore, in some cases, NLU services from different vertical categories might be able to handle the query. In such cases, it's sufficient to assign the query to only one vertical category's NLU service; there's no need to assign it to multiple vertical categories. In other words, it's best to assign the query to only one vertical category.

[0038] Therefore, this application embodiment proposes another method for generating training samples, such as step S240. In step S240, samples with only one positive label value can be obtained as at least a part of the training sample set. The expression "sample with only one positive label value" means that the label value of the sample belongs to only one category. Since training samples with a more singular label (i.e., only one vertical domain has a positive label value) are used, the number of vertical domains to which the model will assign queries during actual classification can be reduced, thereby correspondingly reducing the average number of vertical domains sent and reducing the pressure on the online NLU service.

[0039] Various methods can be used to obtain samples with only one positive label value. For example, in some embodiments, samples can be labeled manually or semi-manually (machine-assisted) to ensure that the labeled training samples have only one category label value as a positive sample label value. For example, in some embodiments, machines can use some simple natural language processing rules to perform preliminary screening and classification of queries, and then humans confirm the final label values. Additionally, such as... Figure 3 As shown by the dotted lines, some or all of the queries labeled in step S230 can be further processed in step S240 to become samples with only one positive sample label value, thus becoming training samples. For example, the vertical domain of the queries labeled in step S230 can be manually reconfirmed, so that the query is classified into only one vertical domain category. Alternatively, in some cases, machines can use certain rules to filter and ensure that the query is classified into only one vertical domain category. Therefore, although Figure 3It is shown that two ways are used to generate training samples, but those skilled in the art should understand that only the way of S240 can be selected to generate training samples as needed, and in some cases, the arrow of steps S230 to the training sample set (i.e. not directly generating training samples) can be removed, and training samples are generated as shown by the dashed line as pre-step of step S240.

[0040] Returning to Figure 1 After obtaining the training sample set in step S110, in step S120, it is determined that the positive sample quantity distribution of the plurality of vertical domains in the training sample set is unbalanced, and at least one vertical domain whose distribution probability of positive samples is less than a probability distribution threshold is determined from the plurality of vertical domains.

[0041] In some embodiments, information entropy can be used to determine whether the positive sample quantity distribution of the plurality of vertical domains in the training sample set is balanced (or whether there is a "small sample vertical domain"). In some embodiments, the information entropy of the probability distribution of the positive samples of each vertical domain can be calculated, and in the case where the information entropy is less than an information entropy threshold, it is determined that the positive sample quantity distribution of the plurality of vertical domains in the training sample set is unbalanced (i.e. there is a small sample vertical domain).

[0042] Take the two vertical domains mentioned earlier when talking about small sample vertical domains as an example (i.e. assuming that there are only two vertical domains of navigation and system settings, and the number of positive samples of the navigation vertical domain and the system settings vertical domain is 200,000 and 10,000 respectively): the probability of positive samples of the navigation vertical domain is P(navigation) = 200,000 / (200,000 + 10,000) = 0.95, and the probability of positive samples of the system settings vertical domain is P(system settings) = 10,000 / (200,000 + 10,000) = 0.05. The information entropy of this probability distribution is

[0043] -sum(P*log(P)) = -(0.95*log(0.95) + 0.05*log(0.05)) ≈ 0.2, where the logarithm calculation in this paper is calculated with e as the base.

[0044] If the information entropy is larger, it indicates that the sample distribution is more balanced. For example, if the positive sample quantities of two vertical domains are the same, the probabilities are both 0.5, and the information entropy is about 0.7. Therefore, an information entropy threshold can be set according to actual conditions to determine whether there is an unbalanced sample distribution, that is, whether there is a small sample vertical domain. In some embodiments, one or more vertical domains with the smallest positive sample quantity can be set as the small sample vertical domain when the information entropy is less than the information entropy threshold, or alternatively, a threshold of the positive sample quantity (which can be a threshold of the quantity or a threshold of the probability distribution) can be set as needed to determine the vertical domain smaller than the threshold as the small sample vertical domain. For example, a vertical domain with a probability distribution of the positive sample smaller than a probability distribution threshold can be set as the small sample vertical domain.

[0045] In addition, in some embodiments, the information entropy can not be used to determine the small sample vertical domain, but a threshold of the positive sample quantity (which can be a threshold of the quantity or a threshold of the proportion or probability distribution) can be directly set to determine the vertical domain smaller than the threshold as the small sample vertical domain.

[0046] Then, after determining the small sample vertical domain, in step S130, at least part of the negative samples of the at least one vertical domain is determined, and the label value of the at least one vertical domain in the at least part of the negative samples is updated to a non-zero negative sample label value. The prior art usually uses 0 to represent the negative sample label value, and the present disclosure strengthens the fitting of the classification model to the small sample vertical domain by changing the negative sample label value of the small sample vertical domain to a non-zero value, and improves the recall rate of the small sample vertical domain.

[0047] In some embodiments, the label value of the other vertical domain than the small sample vertical domain in the at least part of the negative samples is unchanged.

[0048] In some embodiments, the non-zero negative sample label value can be set to less than 0.1. Setting a smaller value can avoid the label value affecting the judgment of normal positive samples. In some embodiments, the non-zero negative sample label value can be set according to the size of the positive sample quantity of the small sample vertical domain. For example, if the positive sample quantity is smaller, the non-zero negative sample label value is set to be larger, that is, the negative sample label value is inversely proportional to the positive sample quantity, but the negative sample label value is usually not more than 0.1.

[0049] In some embodiments, the label values of all or a randomly sampled portion of the negative samples in the small sample vertical domain can be set to a non-zero negative sample label value. For example, the label values of the negative samples can be set to the same value, or to a randomly generated smaller value (e.g., a random number generated in the range of greater than 0 to less than 0.1) respectively. In some cases, by setting the non-zero negative sample label value to a random number, the impact on the judgment of the positive samples can be reduced, and the fitting effect of the small sample vertical domain is enhanced, further improving the recall rate of the small sample vertical domain.

[0050] In addition, in some embodiments, in the case where there is more than one small sample vertical domain, the negative sample label value can also be changed for one or several or all of the small sample vertical domains. For example, in some embodiments, for each small sample vertical domain, the label value of at least a portion or all of the negative samples in the small sample vertical domain with respect to the small sample vertical domain can be set to a non-zero negative sample label value. The same negative sample label value can be set between these small sample vertical domains, or different negative sample label values can be set, for example, the negative sample label value set for the vertical domain with the least number of positive samples is greater. For example, the vertical domain with the least number of samples can set the negative sample label value to 0.08, the second least to 0.05, the third least to 0.03, and so on.

[0051] In order to enhance the understanding of the present disclosure, the following Figures 2A-2C As a comparison, an example of the label value of the training sample according to an embodiment of the present disclosure is given.

[0052] In Figures 2A-2C In the example, it is assumed that the query as the training sample is "navigate to Zhongguancun", and the current dialogue system has four vertical domains: music, phone call, navigation, and system setting, wherein the system setting vertical domain and the phone call vertical domain are determined to be small sample vertical domains according to the various manners described above, and the number of positive samples in the phone call vertical domain is less than that in the system setting vertical domain.

[0053] Figure 2A An example of the label value of the query is given. As shown in the figure, since the query "navigate to Zhongguancun" belongs to the navigation vertical domain, the label value of the navigation vertical domain is 1, and the label values of the other vertical domains are all 0.

[0054] Figure 2B An example of the label value of the query according to at least one embodiment of the present disclosure is given. Compared with Figure 2AIn comparison, the disclosure changes the label value of the system setting vertical domain, which is one of the small sample vertical domains, to 0.02, and the label values of other vertical domains remain unchanged, especially the label value of the positive sample vertical domain remains unchanged. Thus, the negative samples of the system setting vertical domain are partially fitted to the system setting vertical domain, thereby increasing the recall rate of the model for the system setting vertical domain.

[0055] Figure 2C Another example of the label value of the query according to at least one embodiment of the disclosure is given. Compared with the example in the first aspect of the disclosure, Figure 2B In comparison, Figure 2C The label values of all small sample vertical domains, i.e., the telephone vertical domain and the system setting vertical domain, are changed to non-zero values, and the label values of other vertical domains remain unchanged, especially the label value of the positive sample vertical domain remains unchanged. Since the number of positive samples of the telephone vertical domain is less than that of the system setting vertical domain, the label value of the negative sample of the telephone vertical domain is set to a value larger than 0.02, i.e., 0.05. Thus, the negative samples of the small sample category with less positive samples are more fitted to the small sample category, thereby increasing the recall rate of the model for the small sample category.

[0056] In addition, when updating the label value of the small sample vertical domain in the embodiment of the disclosure, the label value of the positive sample vertical domain is not smoothed to the label value of the small sample vertical domain, and the label value of the positive sample vertical domain remains unchanged. When the number of small sample vertical domains is large, the label value of the positive sample vertical domain will not be affected, so that the classification model trained by using the above samples can always maintain high accuracy when processing queries of the positive sample vertical domain.

[0057] Through the above steps S110-S130 and other operations, the processing of the sample for training the classification model according to the disclosure (i.e., the sample processing method) is realized, so that the updated training sample set of at least part of the negative samples of the small sample vertical domain is obtained in step S140 to train the classification model.

[0058] In some embodiments, after training the classification model each time, the training sample set is updated to obtain an updated training sample set, and the classification model is iteratively trained until a trained classification model satisfying a training stop condition is obtained. For example, each training iteration can only update part of the negative samples of the small sample vertical domain. For example, a batch size of 8 is set, i.e., 8 negative samples are randomly taken each time for updating, the negative sample label values of the 8 negative samples are processed, and then the model is trained. In the next round of training, the batch size of negative samples is taken for updating, and then the model is trained. The iteration is continuously performed until the training stops (e.g., the loss function no longer decreases or the iteration reaches a preset number of rounds).

[0059] In some embodiments, the training stopping condition comprises that the loss function no longer decreases or the iteration reaches a preset number of training times, and the loss function of the classification model is determined by the label value of each vertical domain and the probability value of each vertical domain output by the classification model for all training samples in the training sample set.

[0060] For example, in some embodiments, due to the negative sample label value according to the present disclosure becoming a non-zero value, the loss function used by the classification model considers the label value of each category and the probability value of each category output by the classification model for one training sample.

[0061] For example, an example of a loss function available for vertical domain classification is given below (this is expressed in terms of a single sample, if the loss function considering multiple samples (e.g. all samples in a batch) is considered, the sum of each sample can be summed and then divided by the total number of samples. In addition, the formula is not only applicable to vertical domain classification, but is applicable to various classification scenarios, and the vertical domain in the formula is replaced by the category accordingly) :

[0062]

[0063] where Loss represents the loss value, n represents the number of vertical domains, i is the subscript of the vertical domain, represents the i-th vertical domain, Pdomain i represents the label value of the i-th vertical domain, Qdomain i represents the probability value of the i-th vertical domain output by the classification model, and the base of the logarithm is e. 0 <= i <= n in the above formula means that all vertical domains are traversed, for example, according to the sample example in the above Figure 2B , the formula is expanded and the label value of each vertical domain is substituted as follows:

[0064] Loss = -P_music*log(Q_music) - (1-P_music)*log(1-Q_music)

[0065] - P_phone*log(Q_phone) - (1-P_phone)*log(1-Q_phone)

[0066] - P_navigation*log(Q_navigation) - (1-P_navigation)*log(1-Q_navigation)

[0067] - P_system settings*log(Q_system settings) - (1-P_system settings)*log(1-Q_system settings)

[0068] = -0*log(Q_music) - (1-0)*log(1-Q_music)

[0069] -0*log(Q_phone) - (1-0)*log(1-Q_phone)

[0070] -1 * log(Q_navigational) - (1 - 1) * log(1 - Q_navigational)

[0071] -0.02 * log(Q_system setting) - (1 - 0.02) * log(1 - Q_system setting)

[0072] Those skilled in the art should understand that the loss functions described above are given only as examples, and are not intended to limit the present disclosure, which can employ various suitable loss functions as needed.

[0073] The sample processing method and the classification model training method according to the present disclosure have been described in detail above with reference to the accompanying drawings, which increase the recall rate of the small sample vertical domain by changing the label value about the vertical domain in at least part of the negative samples in the small sample vertical domain to a non-zero value, so that the at least part of the negative samples are partially fitted on the small sample vertical domain.

[0074] Figure 4 A structural schematic diagram of a computing device according to an embodiment of the present disclosure is shown, which can be used to implement the sample processing method or the classification model training method described above.

[0075] Referring to Figure 4 , the computing device 400 includes a memory 410 and a processor 420.

[0076] The processor 420 can be a multi-core processor, or can include multiple processors. In some embodiments, the processor 420 can include a general-purpose main processor and one or more special-purpose coprocessors, such as a graphics processor (GPU), a digital signal processor (DSP), and the like. In some embodiments, the processor 420 can be implemented using custom circuitry, such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0077] The memory 410 can include various types of storage units, such as a system memory, a read-only memory (ROM), and a permanent storage device. Among them, the ROM can store static data or instructions required by the processor 420 or other modules of the computer. The permanent storage device can be a read-write storage device. The permanent storage device can be a non-volatile storage device that does not lose stored instructions and data even after the computer is powered off. In some embodiments, the permanent storage device uses a mass storage device (such as a magnetic or optical disk, a flash memory) as a permanent storage device. In some other embodiments, the permanent storage device can be a removable storage device (such as a floppy disk, an optical drive). The system memory can be a read-write storage device or a volatile read-write storage device, such as a dynamic random access memory. The system memory can store some or all of the instructions and data required by the processor during runtime. In addition, the memory 410 can include a combination of any computer readable storage media, including various types of semiconductor storage chips (DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), magnetic disks and / or optical disks. In some embodiments, the memory 410 can include a read and / or write removable storage device, such as a compact disc (CD), a read-only digital versatile disc (such as DVD-ROM, double-layer DVD-ROM), a read-only Blu-ray disc, an ultra-density optical disc, a flash memory card (such as an SD card, a min SD card, a Micro-SD card, etc.), a magnetic floppy disk, etc. The computer readable storage medium does not include a carrier wave and an instantaneous electronic signal transmitted by wireless or wired transmission.

[0078] The memory 410 stores executable code, which, when processed by the processor 420, can cause the processor 420 to perform the sample processing method or the classification model training method described above.

[0079] In addition, the sample processing method or the classification model training method according to the present disclosure can also be implemented as a computer program or a computer program product, which includes computer program code instructions for performing each step defined in the above method of the present disclosure.

[0080] Alternatively, the present disclosure can also be implemented as a non-transitory machine readable storage medium (or computer readable storage medium, or machine readable storage medium) having stored executable code (or computer program, or computer instruction code) thereon, which, when executed by a processor of an electronic device (or computing device, server, etc.), causes the processor to perform each step of the above method according to the present disclosure.

[0081] Those skilled in the art will further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein can be implemented as electronic hardware, computer software, or combinations of both.

[0082] The flow diagrams and the block diagrams in the drawings are presented to illustrate the architecture, functionality, and operation of possible implementations of systems and methods in accordance with the present disclosure. In this regard, each block in the flow diagrams and the block diagrams can represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the block can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may

[0083] Embodiments of the present disclosure have been described above, and the description is intended to be illustrative of the embodiments and not exhaustive, and is not limited to the embodiments disclosed. Numerous modifications and adaptations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The selection of the terms to be used in the description is intended to best describe the principles of the embodiments, practical application, or improvement to the technology in the art, or to enable other ordinary skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for training a classification model, comprising: obtaining a training sample set; determining that a number of positive samples in each vertical domain of the training sample set is unevenly distributed, and determining at least one vertical domain in which a probability of distribution of positive samples is less than a probability distribution threshold; and determining at least one part of negative samples in the at least one vertical domain, and updating a label value of the at least one part of negative samples in the at least one vertical domain to a non-zero negative sample label value, while label values of other vertical domains in the at least one part of negative samples remain unchanged; wherein each training sample in the training sample set comprises an inquiry received by a dialogue system, and obtaining the training sample set comprises obtaining, as at least part of the training sample set, a sample having only one positive sample label value. 2.The method of claim 1, wherein determining that a number of positive samples in each vertical domain of the training sample set is unevenly distributed, and determining at least one vertical domain in which a probability of distribution of positive samples is less than a probability distribution threshold comprises: calculating an information entropy of the probability of distribution of positive samples in each vertical domain, and determining that a number of positive samples in each vertical domain of the training sample set is unevenly distributed when the information entropy is less than an information entropy threshold; and determining the at least one vertical domain in which the probability of distribution of positive samples is less than the probability distribution threshold. 3.The method of claim 1, further comprising: randomly generating a positive value less than 0.1 as the negative sample label value. 4.The method of claim 1, wherein the negative sample label value is inversely proportional to a number of positive samples in the at least one vertical domain. 5.The method of claim 1, further comprising: accessing natural language understanding services of each vertical domain of the dialogue system by using an unlabeled sample, and determining a recall result for the unlabeled sample; and for the unlabeled sample, setting a label value of a vertical domain having the recall result to a positive sample label value, setting a label value of a vertical domain without the recall result to a negative sample label value, and adding the sample with the set label value to the training sample set. 6.A method for training a classification model, comprising: obtaining a training sample set; determining that a number of positive samples in each vertical domain of the training sample set is unevenly distributed, and determining at least one vertical domain in which a probability of distribution of positive samples is less than a probability distribution threshold; determining at least one part of negative samples in the at least one vertical domain, and updating a label value of the at least one part of negative samples in the at least one vertical domain to a non-zero negative sample label value, while label values of other vertical domains in the at least one part of negative samples remain unchanged; and obtaining the training sample set after the at least one part of negative samples is updated, and training the classification model; wherein each training sample in the training sample set comprises an inquiry received by a dialogue system, and obtaining the training sample set comprises obtaining, as at least part of the training sample set, a sample having only one positive sample label value. Each time the classification model is trained, the training sample set is updated to obtain an updated training sample set, and the classification model is iteratively trained until a trained classification model satisfying a training stop condition is obtained. ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ 7. The method of claim 6, further comprising: ​ 8. The method of claim 7, wherein, The training stop condition comprises that a loss function no longer decreases or a preset number of training iterations is reached, and the loss function of the classification model is determined by label values of each vertical domain and probability values of each vertical domain output by the classification model.

9. A computing device comprising: a processor; and a memory having stored thereon executable code that, when executed by the processor, causes the processor to perform the method of any of claims 1-5, or the method of any of claims 6-8.

10. A non-transitory machine-readable storage medium having stored thereon executable code that, when executed by a processor of an electronic device, causes the processor to perform the method of any of claims 1-5, or the method of any of claims 6-8.

Images