Sample generation method, model training method, and text classification method and device

Abstract

The present disclosure provides a sample generation method, a model training method, a text classification method, an apparatus, a device, a storage medium and a program product, relates to the technical field of data processing, and in particular to the technical field of big data and deep learning. The specific implementation scheme of the sample generation method of the present disclosure is: inputting a first mixed sample set into a first pre-training model to obtain a first reference label of each sample in the first mixed sample set; selectively labeling the samples of the first mixed sample set according to the first reference label to obtain a first confidence sample set; inputting a second mixed sample set into a second pre-training model to obtain a second reference label of each sample in the second mixed sample set; and selectively labeling the samples of the second mixed sample set according to the second reference label to generate a second confidence sample set.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and more particularly to the fields of big data and deep learning technology, specifically to a sample generation method, a model training method, a text classification method, an apparatus, a device, a storage medium, and a program product. Background Technology

[0002] Samples are crucial for model training and testing. For example, training a model requires a large number of samples with accurate labels. Therefore, how to generate a large number of high-confidence samples is a problem that urgently needs to be solved. Summary of the Invention

[0003] This disclosure provides a sample generation method, a model training method, a text classification method, an apparatus, a device, a storage medium, and a program product.

[0004] According to one aspect of this disclosure, a sample generation method is provided, comprising: inputting a first mixed sample set into a first pre-trained model to obtain a first reference label for each sample in the first mixed sample set, wherein the first mixed sample set includes a labeled sample set and a first unlabeled sample set, and the first pre-trained model is trained from the labeled sample set; selectively labeling the samples in the first mixed sample set according to the first reference label to obtain a first confidence sample set; inputting a second mixed sample set into a second pre-trained model to obtain a second reference label for each sample in the second mixed sample set, wherein the second mixed sample set includes the first confidence sample set and a second unlabeled sample set, and the second pre-trained model is trained from the first confidence sample set; and selectively labeling the samples in the second mixed sample set according to the second reference label to generate a second confidence sample set.

[0005] According to another aspect of this disclosure, a model training method is provided, comprising: training an initial text classification model based on a confidence sample set to obtain a target text classification model, wherein the initial text classification model includes a convolutional neural network, and the confidence sample set is obtained using the sample generation method of embodiments of this disclosure.

[0006] According to another aspect of this disclosure, a text classification method is provided, comprising: inputting text information into a target text classification model to obtain the category of the text information, wherein the target text classification model is trained using the model training method of the embodiments of this disclosure, and the text information includes the subject text information.

[0007] According to another aspect of this disclosure, a sample generation apparatus is provided, comprising: a first reference label acquisition module, a first confidence sample set determination module, a second reference label acquisition module, and a second confidence sample set generation module. The first reference label acquisition module is used to input a first mixed sample set into a first pre-trained model to obtain a first reference label for each sample in the first mixed sample set, wherein the first mixed sample set includes a labeled sample set and a first unlabeled sample set, and the first pre-trained model is trained from the labeled sample set. The first confidence sample set determination module is used to selectively label the samples in the first mixed sample set according to the first reference label to obtain a first confidence sample set. The second reference label acquisition module is used to input a second mixed sample set into a second pre-trained model to obtain a second reference label for each sample in the second mixed sample set, wherein the second mixed sample set includes the first confidence sample set and a second unlabeled sample set, and the second pre-trained model is trained from the first confidence sample set. The second confidence sample set generation module is used to selectively label the samples in the second mixed sample set according to the second reference label to generate a second confidence sample set.

[0008] According to another aspect of this disclosure, a model training apparatus is provided, comprising: a target text classification model determination module, configured to train an initial text classification model based on a confidence sample set to obtain a target text classification model, wherein the initial text classification model includes a convolutional neural network, and the confidence sample set is obtained using a sample generation apparatus according to an embodiment of this disclosure.

[0009] According to another aspect of this disclosure, a text classification apparatus is provided, comprising: a text information category determination module, configured to input text information into a target text classification model to obtain the category of the text information, wherein the target text classification model is trained using a model training apparatus according to an embodiment of this disclosure, and the text information includes main text information.

[0010] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor and a memory communicatively connected to said at least one processor. The memory stores instructions executable by said at least one processor, which, when executed by said at least one processor, enables the at least one processor to perform the methods of embodiments of this disclosure.

[0011] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions for causing the computer to perform the methods of embodiments of this disclosure.

[0012] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the methods of embodiments of this disclosure.

[0013] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0014] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0015] Figure 1 This schematically illustrates a system architecture diagram of a sample generation method, a model training method, a text classification method, and a corresponding apparatus according to embodiments of the present disclosure.

[0016] Figure 2 A flowchart illustrating a sample generation method according to an embodiment of the present disclosure is shown schematically;

[0017] Figure 3 A schematic diagram of a sample generation method according to an embodiment of the present disclosure is shown;

[0018] Figure 4 The illustration shows a schematic diagram of obtaining a first confidence sample set according to an embodiment of the present disclosure;

[0019] Figure 5 The illustration shows a schematic diagram of obtaining a first set of samples to be labeled according to an embodiment of the present disclosure;

[0020] Figure 6 This illustration schematically shows an example diagram of obtaining a second reference label and generating a second confidence sample set according to an embodiment of the present disclosure;

[0021] Figure 7 A schematic diagram of a sample generation method according to an embodiment of the present disclosure is shown;

[0022] Figure 8 A flowchart illustrating a model training method according to an embodiment of the present disclosure is shown schematically.

[0023] Figure 9 A flowchart illustrating a text classification method according to an embodiment of the present disclosure is shown schematically;

[0024] Figure 10 A block diagram of a sample generation apparatus according to an embodiment of the present disclosure is shown schematically;

[0025] Figure 11 A block diagram of a model training apparatus according to an embodiment of the present disclosure is shown schematically;

[0026] Figure 12 A block diagram of a text classification apparatus according to an embodiment of the present disclosure is schematically shown; and

[0027] Figure 13 A block diagram of an electronic device that can implement the sample generation method, model training method, and text classification method of the embodiments of this disclosure is shown schematically. Detailed Implementation

[0028] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0029] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0030] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0031] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0032] Text classification is a current research area in deep learning technology, and it can be applied to various aspects of life and production. The interpretation of text information varies greatly, making accurate text classification a significant challenge.

[0033] The following will use the sample generation method, model training method, and text classification method of the present disclosure as an example to illustrate the sample generation method, model training method, and text classification method of the present disclosure in the specific scenario of department classification.

[0034] The subject's chief complaint can be used as input to a text classification model, which then outputs a predicted department category. The subject's chief complaint can characterize the subject's symptoms.

[0035] In some embodiments, a text classification model based on traditional deep learning is used to classify text information. Examples of such models include TextCNN (Text Convolutional Neural Network), LSTM (Long Short-Term Memory) network models, and BoW (Bag of Word) models. Traditional deep learning-based text classification models are suitable for simple tasks or scenarios with a large amount of high-confidence sample data, while department classification is a complex task. Furthermore, obtaining a large amount of high-confidence sample data by having relevant professionals label samples is costly, resulting in low sample generation efficiency and low text classification accuracy.

[0036] Figure 1 The illustration schematically depicts the system architecture of a sample generation method, a model training method, a text classification method, and an apparatus according to an embodiment of the present disclosure. It should be noted that... Figure 1 The examples shown are merely examples of system architectures that can be applied to the embodiments of this disclosure, in order to help those skilled in the art understand the technical content of this disclosure, but do not mean that the embodiments of this disclosure cannot be used in other devices, systems, environments or scenarios.

[0037] like Figure 1 As shown, the system architecture 100 according to this embodiment may include: a terminal 101 for generating a second confidence sample set, a terminal 102 for model training, and a terminal 103 for text classification.

[0038] In this embodiment of the disclosure, terminal 101 can be used to execute a sample generation method to obtain a second confidence sample set for model training. Terminal 102 can execute a corresponding model training method based on the second confidence sample set obtained by terminal 101 to achieve corresponding model training. Terminal 103 can perform text classification on text information based on the model obtained by terminal 102.

[0039] It should be noted that the generation of the second confidence sample set and model training can be implemented on the same terminal or on different terminals.

[0040] Terminal 101, Terminal 102 and Terminal 103 can be servers or server clusters.

[0041] It should be understood that Figure 1 The number of terminals 101, 102, and 103 is merely illustrative. Depending on implementation needs, there can be any number of terminals 101, 102, and 103.

[0042] It should be noted that the collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0043] In the technical solution disclosed herein, the user's authorization or consent is obtained before acquiring or collecting the user's personal information.

[0044] This disclosure provides a sample generation method, a model training method, and a text classification method, which are described below in conjunction with... Figure 1 The system architecture, referencing Figures 2-9 This describes a sample generation method, a model training method, and a text classification method according to exemplary embodiments of the present disclosure.

[0045] Figure 2 A flowchart illustrating a sample generation method according to an embodiment of the present disclosure is shown schematically.

[0046] like Figure 2 As shown, the sample generation method 200 of this disclosure embodiment may include, for example, operations S210 to S240.

[0047] In operation S210, the first mixed sample set is input into the first pre-trained model to obtain the first reference label for each sample in the first mixed sample set.

[0048] The first mixed sample set includes a labeled sample set and a first unlabeled sample set, and the first pre-trained model is trained from the labeled sample set.

[0049] For example, labeled sample sets can be manually labeled by relevant professionals, which has a higher confidence level.

[0050] The following example uses samples from embodiments of this disclosure for departmental classification. It should be understood that samples from embodiments of this disclosure can also be applied to text classification in other scenarios.

[0051] In a departmental classification scenario, a labeled sample set can be understood as a collection of samples pre-labeled, and the samples in the labeled sample set can be understood as the chief complaint text information labeled with departmental category tags. The labels on the labeled sample set can be provided by relevant professionals, thus making the labels on the labeled sample set more accurate and increasing its confidence level.

[0052] A pre-trained model can be understood as a model trained on a large dataset, which can be applied to similar datasets and fine-tuned to apply knowledge learned in one domain to different and related domains. The first pre-trained model in this embodiment is trained on a set of labeled samples with higher confidence, and can accurately classify the department of the chief complaint text information by combining the semantics of the chief complaint text information. For example, the first pre-trained model may include an ERNIE (Enhanced Representation through kNowledge IntEgration) model or a BERT (Bidirectional Encoder Representation from Transformer) model.

[0053] In operation S220, the samples in the first mixed sample set are selectively labeled according to the first reference label to obtain the first confidence sample set.

[0054] The first mixed sample set includes a labeled sample set and a first unlabeled sample set. Therefore, when the first mixed sample set is input into the first pre-trained model, the first reference labels for the samples LS of the labeled sample set and the first reference labels for the samples NLS of the first unlabeled sample set can be obtained. The first reference labels for the samples LS of the labeled sample set can be compared with pre-labeled labels, and the comparison result can represent, for example, the classification accuracy of the first pre-trained model. Similarly, the first reference labels for the samples NLS of the first unlabeled sample set can be compared with the first reference labels for the samples LS of the labeled sample set, and the comparison result can represent, for example, the classification accuracy of the first unlabeled sample set by the first pre-trained model. For example, samples with lower classification accuracy in the first unlabeled sample set can be selected for labeling to obtain a first confidence sample set.

[0055] For example, the annotation samples may include text classification annotation samples, such as department classification annotation samples.

[0056] It should be understood that the first confidence sample set has a larger number of samples than the labeled sample set, and its confidence level is higher than that of the first unlabeled sample set.

[0057] In operation S230, the second mixed sample set is input into the second pre-trained model to obtain the second reference label for each sample in the second mixed sample set.

[0058] The second mixed sample set includes the first confidence sample set and the second unlabeled sample set, and the second pre-trained model is trained from the first confidence sample set.

[0059] The second pre-trained model may be the same as or different from the first pre-trained model. For example, the second pre-trained model may include a BERT model and / or an ERNIE model.

[0060] The first confidence sample set has a higher confidence level, therefore, the classification accuracy of the second pre-trained model trained on the first confidence sample set is also higher.

[0061] In operation S240, based on the second reference label, the samples in the second mixed sample set are selectively labeled to generate the second confidence sample set.

[0062] It should be understood that the second confidence sample set has a larger number of samples than the first confidence sample set, and the second confidence sample set has a higher confidence level than the second unlabeled sample set.

[0063] The sample generation method of this disclosure adds a first unlabeled sample set to the labeled sample set to obtain a first mixed sample set with a larger number of samples. Then, selectively labels the first mixed sample set to obtain a first confidence sample set with higher confidence. Furthermore, a second unlabeled sample set is added to the first confidence sample set to obtain a second mixed sample set with a larger number of samples. Then, selectively labels the second mixed sample set to obtain a second confidence sample set with even higher confidence. Therefore, the sample generation method of this disclosure balances high sample confidence and a large sample quantity, resulting in higher sample generation efficiency.

[0064] The second confidence sample set obtained using the sample generation method of this disclosure can, for example, be used to train a text classification model that has high requirements for the number of samples and sample confidence, thereby improving the accuracy of text classification. "Generating the second confidence sample set" in this disclosure can be understood as selectively labeling the samples in the second mixed sample set, resulting in a computer-generated sample set.

[0065] Figure 3 A schematic diagram of a sample generation method according to an embodiment of the present disclosure is shown.

[0066] Operations S310 to S340 of the sample generation method 300 according to embodiments of this disclosure correspond to operations S210 to S240 described above. Specifically, in operation S310, a first mixed sample set 301, including a labeled sample set and a first unlabeled sample set, is input to a first pre-trained model 302 trained from the labeled sample set, resulting in a first reference label 303 for each sample in the first mixed sample set 301. In operation S320, samples in the first mixed sample set 301 are selectively labeled based on the first reference labels 303, resulting in a first confidence sample set 304. In operation S330, a second mixed sample set 305, including the first confidence sample set and a second unlabeled sample set, is input to a second pre-trained model 306 trained from the first confidence sample set, resulting in a second reference label 307 for each sample in the second mixed sample set 305. In operation S340, samples in the second mixed sample set 305 are selectively labeled based on the second reference labels 307, generating a second confidence sample set 308.

[0067] Figure 4 The illustration schematically depicts a diagram of obtaining a first confidence sample set according to an embodiment of the present disclosure. This can be achieved, for example... Figure 2 The operation S210 shown is executed afterward. Figure 4 The example method shown selectively labels samples in the first mixed sample set according to the first reference label to obtain the first confidence sample set.

[0068] like Figure 4 As shown, selective labeling of samples in the first mixed sample set according to the first reference label to obtain the first confidence sample set may include, for example, operations S421 to S423.

[0069] In operation S421, at least one sample in the first unlabeled sample set 401 is selected according to the active learning selection algorithm to obtain the first unlabeled sample set 402.

[0070] Active learning can be understood as a process where, in certain situations, there are many labeled samples but few unlabeled samples, and manual labeling is costly. In such cases, an algorithm can be selected to proactively determine which samples to label, and then these samples are labeled by relevant professionals.

[0071] In operation S423, a first annotation confirmation sample set 404 is obtained based on the corrected first unannotated sample set 403.

[0072] For example, the first sample set to be labeled 402 can be manually corrected by relevant professionals to obtain the corrected first sample set to be labeled 403.

[0073] In operation S423, a first confidence sample set 406 is obtained based on the first mixed sample set 405 and the first labeled confirmation sample set 404.

[0074] The first confidence sample set can be understood as: the first mixed sample set obtained after the first unlabeled sample set selected from the first mixed sample set is updated by the first labeled confirmed sample set.

[0075] The sample generation method of this disclosure selects a first sample set to be labeled from a first unlabeled sample set, which can reduce the number of samples that need to be labeled, reduce the sample labeling cost, and improve the sample labeling efficiency. By correcting the first sample set to be labeled, a first confidence sample set with high confidence can be obtained.

[0076] Figure 5 The illustration schematically depicts a first sample set to be labeled according to an embodiment of the present disclosure. This can be achieved, for example... Figure 2 The operation S210 shown is executed afterward. Figure 5 The example method shown selects at least one sample from the first unlabeled sample set according to the active learning selection algorithm to obtain the first unlabeled sample set.

[0077] like Figure 5 As shown, the labeled sample set 501 may include multiple labeled samples, and the first unlabeled sample set 502 may include multiple unlabeled samples. According to the active learning selection algorithm, at least one sample in the first unlabeled sample set is selected to obtain the first unlabeled sample set. For example, it may include: operation S511 to operation S512.

[0078] In operation S511, based on the first reference label of the labeled sample, the label probability parameters of the reference label error sample and the reference label error sample are determined.

[0079] Taking the first pre-trained model for departmental classification as an example, the sample labels obtained by the first pre-trained model can include departmental category labels such as gynecology, pediatrics, and neurosurgery. By inputting labeled samples into the first pre-trained model, the model can obtain the probability values ​​of each departmental category label for the labeled samples.

[0080] For example, inputting labeled sample Li into the first pre-trained model yields the following classification results: the probability of labeled sample Li being "Gynecology" is 30%, the probability of it being "Pediatrics" is 65%, and the probability of it being "Neurosurgery" is 5%. The department category label with the highest probability can be used as the first reference label for labeled sample Li; therefore, in this example, the first reference label for labeled sample Li is "Pediatrics." Furthermore, labeled samples also have pre-labeled labels; for example, in the above example, the pre-labeled label for labeled sample Li is "Gynecology." Therefore, since the pre-labeled label for labeled sample Li should have a higher confidence level, it can be considered that the first reference label for labeled sample Li, "Pediatrics," is incorrect. Thus, labeled sample Li can be determined to be a sample ELi with an incorrect reference label.

[0081] The label probability parameter may include at least one of the following: label probability distribution, label probability value, and label probability difference.

[0082] For example, when the label probability parameter is a label probability value, in the above example, the label probability parameter of the reference label mislabeled sample ELi may include: a probability value of 30% for the label "gynecology", a probability value of 65% for the label "pediatrics", and a probability value of 5% for the label "neurosurgery".

[0083] For example, when there are multiple incorrectly labeled reference samples, the label probability values ​​of these samples can be statistically analyzed to obtain the label probability distribution, a label probability parameter. The label probability distribution represents the pattern of the label probability values ​​of the incorrectly labeled reference samples. For instance, if the label probability values ​​of the incorrectly labeled reference samples are divided into a first probability interval, a second probability interval, and a third probability interval, the label probability distribution is as follows: among all incorrectly labeled reference samples, 70% of the label probability values ​​fall into the first probability interval, 20% fall into the second probability interval, and 10% fall into the third probability interval.

[0084] For example, the label probability difference can represent the difference between the label probability value of the first reference label obtained by inputting the labeled sample into the first pre-trained model and the label probability values ​​of other category labels. In the example above, the difference between the 65% probability value of the labeled sample Li being the reference label "Pediatrics" and the 30% probability value of the labeled sample Li being the category label "Gynecology" is 35%, and the difference between the 65% probability value of the labeled sample Li being the reference label "Pediatrics" and the 5% probability value of the labeled sample Li being the category label "Neurosurgery" is 60%. That is, the label probability difference includes 35% and 60%.

[0085] In operation S512, based on the label probability parameters of the reference label error sample and the label probability parameters of the first reference label of the unlabeled sample, at least one unlabeled sample in the first unlabeled sample set is selected to obtain the first unlabeled sample set 503.

[0086] The label probability parameters for obtaining the first reference label of unlabeled samples are similar to those for obtaining the first reference label of labeled samples, and will not be repeated here.

[0087] Taking the label probability values ​​in the above example as an example of the label probability distribution. For instance, if the label probability value of the first reference label of the unlabeled sample Ni falls into the first probability interval, it can be considered that the confidence level of the unlabeled sample Ni is low and it needs to be relabeled, that is, the unlabeled sample Ni is selected as a sample in the first set of samples to be labeled.

[0088] Taking the label probability parameter in the above example as the label probability value, for example, a first reference threshold Th1 can be set. The comparison result between the label probability value of the first reference label of the unlabeled sample Ns and the first reference threshold Th1 determines whether to select the unlabeled sample Ns as a sample in the first set of samples to be labeled.

[0089] Taking the label probability parameter in the above example as the label probability difference, for example, a second reference threshold Th2 can be set. The comparison result between the label probability parameter difference of the first reference label of the unlabeled sample Nu and the second reference threshold Th2 determines whether to select the unlabeled sample Nu as a sample in the first set of samples to be labeled.

[0090] It should be understood that the pre-determined labels of the labeled samples have a higher confidence level. The pre-determined labels of the labeled samples can be used as a basis for comparison. The first reference label of the labeled sample can be compared with the pre-determined label of the labeled sample to accurately identify the reference label error samples in the labeled sample set where the first reference label is incorrect.

[0091] The sample generation method of this disclosure uses the label probability parameters of the reference label error sample as a comparison basis. It compares the label probability parameters of the unlabeled samples in the first unlabeled sample set with the label probability parameters of the reference label error sample, which can accurately determine the unlabeled samples in the first unlabeled sample set whose first reference label is incorrect, and obtain the first sample set to be labeled.

[0092] Figure 5An illustrative example shows that the labeled sample set 501 may include n labeled samples, n first reference labels determined by the n labeled samples, s reference label error samples determined by the n first reference labels, and label probability parameters of the s reference label error samples, where s is less than or equal to n. The first unlabeled sample set 502 may include m unlabeled samples, m first reference labels determined by the m unlabeled samples, and label probability parameters of the m first reference labels.

[0093] Figure 5 The label probability parameters pL1 of the first reference label RL1, the reference label error sample EL1, and the reference label error sample EL1 of the labeled sample L1 are also schematically shown, as are the label probability parameters pLs of the first reference label RLn, the reference label error sample ELs, and the reference label error sample ELs of the labeled sample Ln.

[0094] Figure 5 The diagram also schematically illustrates the first reference label RN1 of the unlabeled sample N1, the label probability parameter pN1 of the first reference label RN1, the first reference label RNm of the first unlabeled sample Nm, and the label probability parameter pNm of the first reference label RNm.

[0095] Figure 6 This schematically illustrates a diagram of obtaining a second reference label for each sample in a second mixed sample set according to an embodiment of the present disclosure. For example, it can be... Figure 2 The operation S220 shown is executed afterward. Figure 6 The example method shown inputs the second mixed sample set into the second pre-trained model to obtain the second reference label for each sample in the second mixed sample set.

[0096] like Figure 6 As shown, the second mixed sample set 601 may include multiple mixed samples, and the second pre-trained model may include multiple different second pre-trained sub-models. Inputting the second mixed sample set 601 into the second pre-trained model to obtain the second reference label for each sample in the second mixed sample set 601 may include, for example, operation S631.

[0097] In operation S631, multiple mixed samples are input into multiple second pre-trained sub-models respectively, and multiple second reference labels are obtained for each mixed sample.

[0098] A pre-trained model may have superior performance in one aspect but poor performance in another, while multiple pre-trained models with complementary performance characteristics can achieve good overall performance. The sample generation method of this disclosure uses multiple different second pre-trained sub-models. For each mixed sample, the resulting multiple second reference labels can respectively reflect the advantages of the multiple second pre-trained sub-models. For example, subsequently, based on the multiple second reference labels, a second mixed sample set can be comprehensively selected to obtain a second set of samples to be labeled.

[0099] For example, the second pre-trained sub-model may include at least one BERT model, at least one ERNIE1.0 model, and at least one ERNIE2.0 model.

[0100] For example, the second pre-trained sub-model may include three BERT models, three ERNIE 1.0 models, and three ERNIE 2.0 models.

[0101] For example, each mixed sample in the second mixed sample set can be input into multiple second pre-trained sub-models to obtain multiple second reference labels for each mixed sample in the second mixed sample set. For instance, the second mixed sample set may include 10,000 mixed samples, and in the example of the nine second pre-trained sub-models mentioned above, 90,000 second reference labels can be obtained.

[0102] For example, the second mixed sample set can also be evenly divided into multiple second mixed sample subsets, each of which corresponds to a second pre-trained sub-model. For instance, the second mixed sample set may include 90,000 mixed samples, and can be divided into nine second mixed sample subsets. The nine second mixed sample subsets are then input into nine second pre-trained sub-models, and each second mixed sample subset can obtain 10,000 second reference labels, resulting in a total of 90,000 second reference labels for the second mixed sample set.

[0103] Figure 6 The illustration shows that the second mixed sample set 601 includes q mixed samples, such as mixed sample M1 and mixed sample Mq. The second pre-trained model includes r second pre-trained sub-models, such as second pre-trained sub-models V1 and Vr. Each mixed sample is input to each second pre-trained sub-model to obtain multiple second reference labels. For example, mixed sample M1 is input to second pre-trained sub-model V1 to obtain second reference label L11, mixed sample M1 is input to second pre-trained sub-model Vr to obtain second reference label L1r, mixed sample Mq is input to second pre-trained sub-model V1 to obtain second reference label Lq1, and mixed sample Mq is input to second pre-trained sub-model Vr to obtain second reference label Lqr.

[0104] Figure 6 The illustration schematically depicts the generation of a second confidence sample set according to an embodiment of the present disclosure. This can be achieved, for example... Figure 2 The operation S230 shown is executed afterward. Figure 6 The example method shown selectively labels samples in the second mixed sample set according to the second reference label to generate a second confidence sample set.

[0105] like Figure 6 As shown, selectively labeling samples in the second mixed sample set according to the second reference label to generate the second confidence sample set may include, for example, operations S641 to S643.

[0106] In operation S641, based on the multiple second reference labels of each mixed sample, at least one mixed sample in the second mixed sample set is selected to obtain the second unlabeled sample set 602.

[0107] Similar to the above embodiments, the second reference label obtained by the second pre-trained sub-model can also be understood as the category label with the highest probability value among all category labels.

[0108] In operation S642, a second annotation confirmation sample set 604 is obtained based on the corrected second unannotated sample set 603.

[0109] For example, the second sample set 602 to be labeled can be manually corrected by relevant professionals to obtain the corrected second sample set 603 to be labeled.

[0110] In operation S643, a second confidence sample set 605 is generated based on the second mixed sample set 601 and the second labeled confirmation sample set 604.

[0111] The sample generation method of this disclosure selects low-confidence mixed samples from the second mixed sample set to obtain a second sample set to be labeled, which can reduce the number of samples that need to be labeled, reduce the sample labeling cost, and improve the sample labeling efficiency. By correcting the second sample set to be labeled, a high-confidence second sample set can be obtained.

[0112] According to the sample labeling method of this disclosure, selecting at least one mixed sample from the second mixed sample set based on the plurality of second reference labels of each mixed sample to obtain the second sample set to be labeled may include, for example, selecting at least one mixed sample from the second mixed sample set based on the voting results of the plurality of second reference labels or based on the label probability values ​​of the plurality of second reference labels to obtain the second sample set to be labeled.

[0113] The sample generation method of this disclosure can accurately select low-confidence mixed samples as samples to be labeled from the second mixed samples based on multiple second reference labels of each mixed sample, thereby obtaining a second sample set to be labeled.

[0114] The following example, using the departmental classification example above, illustrates how to select at least one mixed sample from the second mixed sample set based on the voting results of multiple second reference labels to obtain the second unlabeled sample set. For example, if the nine second reference labels corresponding to the mixed sample Mi are: Gynecology, Pediatrics, Pediatrics, Gynecology, Gynecology, Gynecology, Pediatrics, Pediatrics, Pediatrics, Pediatrics, then the four votes for Gynecology and the five votes for Pediatrics can be taken as the voting results. A third comparison threshold Th3 can be set, and the comparison result between the voting results and the third comparison threshold Th3 determines whether to select the mixed sample Mi as a sample in the second unlabeled sample set.

[0115] For example, the third comparison threshold Th3 can represent the ratio of the vote count of a second reference label to the sum of the vote counts of the other second reference labels among multiple second reference labels. For example, when the voting result is greater than the third comparison threshold Th3, the mixed sample can be selected as a sample in the second set of samples to be labeled. For example, when the third comparison threshold Th3 is 2:1, in the example of the mixed sample Mi above, among the nine second reference labels, the second reference label with the highest vote count has 5 votes, and the sum of the vote counts of the other second reference labels is 4. Therefore, the voting result is 5:4. The voting result 5:4 is greater than the third comparison threshold 2:1, so the mixed sample Mi can be selected as a sample in the second set of samples to be labeled.

[0116] In another example, at least one mixed sample from the second mixed sample set can be selected based on the label probability values ​​of multiple second reference labels to obtain the second unlabeled sample set. For example, a fourth comparison threshold Th4 can be set, and the selection of the mixed sample Mf as a sample in the second unlabeled sample set can be determined by comparing the label probability values ​​of multiple second reference labels with the fourth comparison threshold Th4.

[0117] For example, the nine second reference labels corresponding to the mixed sample Mf are: Gynecology (for example, among multiple department category labels, the probability value of the mixed sample Mf being the label of Gynecology is 0.4, and the probability value of the second highest is Pediatrics is 0.38, with a difference of 0.02), Gynecology (similarly, for example, Gynecology: 0.39, and the second highest probability value is Pediatrics: 0.37, with a difference of 0.02), Pediatrics, Gynecology (similarly, for example, Gynecology: 0.4, and the second highest probability value is Pediatrics: 0.39, with a difference of 0). 01) Gynecology (similarly, for example, Gynecology: 0.38, Pediatrics with the second highest probability value: 0.37, the difference between the two is 0.01), Gynecology (similarly, for example, Gynecology: 0.4, Pediatrics with the second highest probability value: 0.38, the difference between the two is 0.02), Gynecology (similarly, for example, Gynecology: 0.4, Pediatrics with the second highest probability value: 0.37, the difference between the two is 0.03), Gynecology (similarly, for example, Gynecology: 0.4, Pediatrics with the second highest probability value: 0.39, the difference between the two is 0.01), Pediatrics.

[0118] For example, the fourth comparison threshold Th4 can represent the difference between the probability value of each second reference label and the probability value of the category label with the second highest probability value among multiple second reference labels. For example, when the probability values ​​of multiple second reference labels are less than the fourth comparison threshold Th4, the mixed sample can be selected as a sample in the second set of samples to be labeled. For instance, when the fourth comparison threshold Th4 is 0.2, in the example of the mixed sample Mf above, among the nine second reference labels, the one with the highest number of votes is gynecology. The difference between the probability value of each second reference label "gynecology" and the probability value of the other category label with the second highest probability value is less than 0.2. Therefore, the mixed sample Mf can be selected as a sample in the second set of samples to be labeled.

[0119] Figure 7 A schematic diagram of a sample generation method according to an embodiment of the present disclosure is shown.

[0120] like Figure 7 As shown, the sample generation method of this embodiment can train the initial pre-trained model 702 using a labeled sample set 701 with higher confidence to obtain a first pre-trained model 703. Since the number of samples in the labeled sample set is relatively small, for example, several thousand samples, the classification accuracy of the first pre-trained model may not be high due to the small number of training samples.

[0121] The sample generation method of this disclosure can add a first unlabeled sample set to the labeled sample set 701 to obtain a first mixed sample set 704 with a larger number of samples. Inputting the first mixed sample set 704 into the first pre-trained model 703 yields a sample set 705 with a first reference label, corresponding to the first mixed sample set 704. Each sample in this sample set 705 carries a first reference label. Since the accuracy of the first pre-trained model is not high, the confidence level of the sample set 705 obtained by inputting the first mixed sample set 704 into the first pre-trained model is not high. Therefore, the sample generation method of this disclosure can determine whether the label probability parameters of the samples in the sample set 705 meet the requirements, and select at least one sample from the first unlabeled sample set to obtain a first unlabeled sample set 707. For example, each sample in the first unlabeled sample set can be manually labeled to obtain a first confidence sample set 706 with a larger number of samples and higher sample confidence. For example, the number of samples in the first confidence sample set 706 can reach tens of thousands.

[0122] The sample generation method of this disclosure embodiment can train a second pre-trained sub-model using a first confidence sample set 706. Since the first confidence sample set 706 has a higher confidence level than the first mixed sample set and has a larger number of samples than the labeled sample set, the second pre-trained sub-model has higher classification accuracy than the first pre-trained model.

[0123] The sample generation method of this disclosure can further add a second unlabeled sample set to the first confidence sample set 706 to obtain a second mixed sample set 708 with a larger number of samples. The samples in the second mixed sample set 708 are input into multiple second pre-trained sub-models. For example, three second pre-trained sub-models V1, three second pre-trained sub-models V2, and three second pre-trained sub-models V3, thereby obtaining a sample set 709 with second reference labels.

[0124] The sample generation method of this disclosure can further determine whether the voting results or label probability values ​​of sample set 709 meet the requirements, and select at least one sample from the second mixed sample set 708 to obtain a second unlabeled sample set 711. For example, each sample in the second unlabeled sample set can be manually labeled, or the second unlabeled sample set can be deleted to obtain a second confidence sample set 710. The second confidence sample set 710 has a larger number of samples than the first confidence sample set 706. For example, the number of samples in the second confidence sample set 710 can reach the millions. The second confidence sample set 710 has a higher confidence level than the second mixed sample set 708.

[0125] In summary, the sample generation method of this disclosure can efficiently generate a second confidence sample set with a larger number of samples and higher sample confidence by adding an unlabeled sample set twice and selecting a sample set to be labeled twice, and labeling the samples in the sample set to be labeled, using a smaller number of labeled sample sets.

[0126] According to embodiments of this disclosure, a model training method is also disclosed.

[0127] like Figure 8 As shown, the model training method 800 according to an embodiment of the present disclosure includes: operation S810.

[0128] When operating the S810, the initial text classification model is trained based on the confidence sample set to obtain the target text classification model.

[0129] The confidence sample set is obtained according to the sample generation method of the embodiments of this disclosure.

[0130] For example, the initial text classification model may include a convolutional neural network.

[0131] The confidence sample set obtained by the sample generation method according to the embodiments of this disclosure has a higher confidence level and a sufficient number of samples. Therefore, the model training method of the embodiments of this disclosure obtains a target text classification model with higher accuracy and better performance through training the confidence sample set.

[0132] According to embodiments of this disclosure, a text classification method is also disclosed.

[0133] like Figure 9 As shown, a text classification method 900 according to an embodiment of the present disclosure includes: operation S910.

[0134] In operation S910, text information is input into the target text classification model to obtain the category of the text information.

[0135] The target text classification model is trained according to the model training method of the embodiments of this disclosure.

[0136] For example, the text information may include the main complaint text information.

[0137] The target text classification model obtained by the model training method according to the embodiments of this disclosure has better performance and higher accuracy. Therefore, the text classification method of the embodiments of this disclosure classifies text information through the target text classification model, resulting in higher classification accuracy. The target classification model used in the text classification method of the embodiments of this disclosure is trained with a sufficiently large sample set and high confidence. Therefore, it has learned the features of different types of text information and text information with different expressions very well, thus having high text classification accuracy. It can be applied to complex and highly specialized application scenarios such as department classification. In the department classification application scenario, the text information may include the chief complaint text information.

[0138] Figure 10 A block diagram of a sample generation apparatus according to an embodiment of the present disclosure is shown schematically.

[0139] like Figure 10 As shown, the sample generation apparatus 1000 of this embodiment includes, for example, a first reference label acquisition module 1010, a first confidence sample set determination module 1020, a second reference label acquisition module 1030, and a second confidence sample set generation module 1040.

[0140] The first reference label acquisition module 1010 is used to input the first mixed sample set into the first pre-trained model to obtain the first reference label for each sample in the first mixed sample set. The first mixed sample set includes a labeled sample set and a first unlabeled sample set, and the first pre-trained model is trained from the labeled sample set. In one embodiment, the first reference label acquisition module 1010 can be used to perform the above operation S210, which will not be described in detail here.

[0141] The first confidence sample set determination module 1020 is used to selectively label samples in the first mixed sample set according to the first reference label to obtain a first confidence sample set. In one embodiment, the first confidence sample set determination module 1020 can be used to perform the above-described operation S220, which will not be described in detail here.

[0142] The second reference label acquisition module 1030 is used to input the second mixed sample set into the second pre-trained model to obtain a second reference label for each sample in the second mixed sample set. The second mixed sample set includes a first confidence sample set and a second unlabeled sample set, and the second pre-trained model is trained using the first confidence sample set. In one embodiment, the second reference label acquisition module 1030 can be used to perform the above operation S230, which will not be described in detail here.

[0143] The second confidence sample set generation module 1040 is used to selectively label samples in the second mixed sample set according to the second reference label to generate a second confidence sample set. In one embodiment, the second confidence sample set generation module 1040 can be used to perform the above-described operation S240, which will not be described in detail here.

[0144] According to the sample generation apparatus of the present disclosure, the first confidence sample set determination module may include: a first unlabeled sample set determination submodule, a first labeled determined sample set acquisition submodule, and a first confidence sample set determination submodule.

[0145] The first unlabeled sample set determination submodule can be used to select at least one sample from the first unlabeled sample set according to the active learning selection algorithm to obtain the first unlabeled sample set.

[0146] The first annotation determination sample set acquisition submodule can be used to obtain the first annotation confirmation sample set based on the corrected first annotation sample set.

[0147] The first confidence sample set determination submodule can be used to obtain the first confidence sample set based on the first mixed sample set and the first labeled confirmation sample set.

[0148] According to the sample generation apparatus of the present disclosure, the labeled sample set may include multiple labeled samples, the first unlabeled sample set may include multiple unlabeled samples, and the first sample set to be labeled determination submodule may include: a reference label error sample determination unit and a first sample set to be labeled determination unit.

[0149] The reference label error sample determination unit can be used to determine the reference label error sample and the label probability parameter of the reference label error sample based on the first reference label of the labeled sample.

[0150] The first unlabeled sample set determination unit can be used to select at least one unlabeled sample from the first unlabeled samples based on the label probability parameters of the reference label error samples and the label probability parameters of the first reference labels of the unlabeled samples, thereby obtaining the first unlabeled sample set. The label probability parameters include at least one of the following: label probability distribution, label probability value, and label probability difference.

[0151] According to the sample generation apparatus of the present disclosure, the second mixed sample set may include multiple mixed samples, the second pre-trained model may include multiple different second pre-trained sub-models, and the second reference label acquisition module may include a second reference label acquisition sub-module.

[0152] The second reference label acquisition submodule can be used to input multiple mixed samples into multiple second pre-trained sub-models, and obtain multiple second reference labels for each mixed sample.

[0153] According to the sample generation apparatus of the present disclosure, the second confidence sample set generation module may include: a second unlabeled sample set determination submodule, a second labeled confirmed sample set acquisition submodule, and a second confidence sample set generation submodule.

[0154] The second unlabeled sample set determination submodule can be used to select at least one mixed sample from the second mixed sample set based on multiple second reference labels of each mixed sample to obtain the second unlabeled sample set.

[0155] The second annotation confirmation sample set acquisition submodule can be used to obtain the second annotation confirmation sample set based on the corrected second annotation sample set.

[0156] The second confidence sample set generation submodule can be used to generate a second confidence sample set based on the second mixed sample set and the second labeled confirmation sample set.

[0157] According to the sample generation apparatus of the present disclosure, the second sample set determination submodule may include: a second sample set determination unit.

[0158] The second unlabeled sample set determination unit can be used to select at least one mixed sample from the second mixed sample set based on the voting results of multiple second reference labels or based on the label probability values ​​of multiple second reference labels, so as to obtain the second unlabeled sample set.

[0159] Figure 11 A block diagram of a model training apparatus according to an embodiment of the present disclosure is shown schematically.

[0160] like Figure 11 As shown, the model training apparatus 1100 of this embodiment includes, for example, a target text classification model determination module 1110.

[0161] The target text classification model determination module 1110 is used to train the initial text classification model based on the confidence sample set to obtain the target text classification model. In one embodiment, the target text classification model determination module 1110 can be used to perform the above operation S810, which will not be described in detail here.

[0162] The initial text classification model includes a convolutional neural network, and the confidence sample set is obtained using the sample generation apparatus of this disclosure.

[0163] Figure 12 A block diagram of a text classification apparatus according to an embodiment of the present disclosure is shown schematically.

[0164] like Figure 12As shown, the text classification device 1200 of this embodiment includes: a text information category determination module 1210.

[0165] The text information category determination module 1210 is used to input text information into a target text classification model to obtain the category of the text information. In one embodiment, the text information category determination module 1210 can be used to perform the above operation S910, which will not be described in detail here.

[0166] The target text classification model is trained using a model training device according to an embodiment of the present disclosure, and the text information includes the main complaint text information.

[0167] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0168] Figure 13 A schematic block diagram of an example electronic device 1300 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0169] like Figure 13 As shown, device 1300 includes a computing unit 1301, which can perform various appropriate actions and processes according to a computer program stored in read-only memory (ROM) 1302 or a computer program loaded from storage unit 1308 into random access memory (RAM) 1303. The RAM 1303 may also store various programs and data required for the operation of device 1300. The computing unit 1301, ROM 1302, and RAM 1303 are interconnected via bus 1304. Input / output (I / O) interface 1305 is also connected to bus 1304.

[0170] Multiple components in device 1300 are connected to I / O interface 1305, including: input unit 1306, such as keyboard, mouse, etc.; output unit 1307, such as various types of monitors, speakers, etc.; storage unit 1308, such as disk, optical disk, etc.; and communication unit 1309, such as network card, modem, wireless transceiver, etc. Communication unit 1309 allows device 1300 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0171] The computing unit 1301 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1301 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1301 performs the various methods and processes described above, such as sample generation methods, model training methods, and text classification methods. For example, in some embodiments, the sample generation methods, model training methods, and text classification methods can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 1308. In some embodiments, part or all of the computer program can be loaded and / or installed on device 1300 via ROM 1302 and / or communication unit 1309. When the computer program is loaded into RAM 1303 and executed by the computing unit 1301, one or more steps of the sample generation methods, model training methods, and text classification methods described above can be performed. Alternatively, in other embodiments, the computing unit 1301 may be configured in any other suitable manner (e.g., by means of firmware) to perform a sample generation method, a model training method, or a text classification method.

[0172] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0173] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0174] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0175] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0176] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0177] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.

[0178] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0179] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A sample generation method, comprising: inputting a first mixed sample set to a first pre-training model to obtain a first reference label of each sample in the first mixed sample set, wherein the first mixed sample set comprises a labeled sample set and a first unlabeled sample set, the first pre-training model is trained by the labeled sample set; wherein the labeled sample set comprises a plurality of labeled samples, the first unlabeled sample set comprises a plurality of unlabeled samples, and the labeled sample comprises a text classification labeled sample; selectively labeling samples in the first mixed sample set according to the first reference label to obtain a first confidence sample set; inputting a second mixed sample set to a second pre-training model to obtain a second reference label of each sample in the second mixed sample set, wherein the second mixed sample set comprises the first confidence sample set and a second unlabeled sample set, the second pre-training model is trained by the first confidence sample set; and generating a second confidence sample set by selectively labeling samples in the second mixed sample set according to the second reference label; the selectively labeling samples in the first mixed sample set according to the first reference label to obtain a first confidence sample set comprises: selecting at least one sample in the first unlabeled sample set according to a selection algorithm of active learning to obtain a first to-be-labeled sample set; obtaining a first labeled confirmation sample set according to the corrected first to-be-labeled sample set; and obtaining the first confidence sample set according to the first mixed sample set and the first labeled confirmation sample set; the selecting at least one sample in the first unlabeled sample set according to the selection algorithm of active learning to obtain a first to-be-labeled sample set comprises: determining a reference label error sample and a label probability parameter of the reference label error sample according to the first reference label of the labeled sample; and selecting at least one unlabeled sample in the first unlabeled sample set according to the label probability parameter of the reference label error sample and the label probability parameter of the first reference label of the unlabeled sample to obtain the first to-be-labeled sample set; the label probability parameter comprises at least one of the following: a label probability distribution, a label probability value, and a label probability difference value.

2. The method of claim 1, wherein, the second mixed sample set comprises a plurality of mixed samples, the second pre-training model comprises a plurality of different second pre-training sub-models, and the inputting the second mixed sample set to the second pre-training model to obtain a second reference label of each sample in the second mixed sample set comprises: inputting the plurality of mixed samples into a plurality of second pre-training sub-models respectively, and obtaining a plurality of second reference labels for each mixed sample.

3. The method of claim 2, wherein, the selectively labeling samples in the second mixed sample set according to the second reference label to generate a second confidence sample set comprises: selecting at least one mixed sample in the second mixed sample set according to a plurality of second reference labels of each mixed sample to obtain a second to-be-labeled sample set; obtaining a second labeled confirmation sample set according to the corrected second to-be-labeled sample set; and According to the second mixed sample set and the second labeled confirmation sample set, the second confidence sample set is generated.

4. The method of claim 3, wherein, The second confidence sample set is generated according to the second reference label of each of the mixed samples in the second mixed sample set. The second confidence sample set is generated according to the second reference label of each of the mixed samples in the second mixed sample set.

5. A model training method, comprising: training an initial text classification model according to a confidence sample set to obtain a target text classification model, the initial text classification model comprising a convolutional neural network, wherein the confidence sample set is obtained by using the sample generation method in any one of claims 1 to 4.

6. A text classification method, comprising: inputting text information into a target text classification model to obtain a category of the text information, wherein the target text classification model is trained by using the model training method in claim 5, the text information comprises chief complaint text information.

7. A sample generation apparatus, comprising: a first reference label obtaining module configured to input a first mixed sample set into a first pre-training model to obtain a first reference label of each sample in the first mixed sample set, wherein the first mixed sample set comprises a labeled sample set and a first unlabeled sample set, the first pre-training model is trained by using the labeled sample set; wherein the labeled sample set comprises a plurality of labeled samples, the first unlabeled sample set comprises a plurality of unlabeled samples, and the labeled sample comprises a text classification labeled sample; a first confidence sample set determining module configured to selectively label samples in the first mixed sample set according to the first reference label to obtain a first confidence sample set; a second reference label obtaining module configured to input a second mixed sample set into a second pre-training model to obtain a second reference label of each sample in the second mixed sample set, wherein the second mixed sample set comprises the first confidence sample set and a second unlabeled sample set, and the second pre-training model is trained by using the first confidence sample set; and a second confidence sample set generating module configured to selectively label samples in the second mixed sample set according to the second reference label to generate a second confidence sample set; the first confidence sample set determining module comprises: a first unlabeled sample set determining submodule configured to select at least one sample in the first unlabeled sample set according to a selection algorithm of active learning to obtain a first unlabeled sample set; a first labeled confirmation sample set obtaining submodule configured to obtain a first labeled confirmation sample set according to the corrected first unlabeled sample set; a first confidence sample set determining submodule configured to obtain the first confidence sample set according to the first mixed sample set and the first labeled confirmation sample set; the first unlabeled sample set determining submodule comprises: The reference label error sample determination unit is configured to determine a reference label error sample and a label probability parameter of the reference label error sample according to the first reference label of the labeled sample. The first unlabeled sample set determination unit is configured to select at least one unlabeled sample in the first unlabeled sample set according to the label probability parameter of the reference label error sample and the label probability parameter of the first reference label of the unlabeled sample, to obtain the first to-be-labeled sample set. The label probability parameter includes at least one of the following: a label probability distribution, a label probability value, and a label probability difference value.

8. The apparatus of claim 7, wherein, The second mixed sample set includes a plurality of mixed samples, the second pre-training model includes a plurality of different second pre-training sub-models, and the second reference label obtaining module includes: The second reference label obtaining sub-module is configured to input the plurality of mixed samples into a plurality of second pre-training sub-models respectively, and obtain a plurality of second reference labels for each mixed sample.

9. The apparatus of claim 8, wherein, The second confidence sample set generation module includes: The second to-be-labeled sample set determination sub-module is configured to select at least one mixed sample in the second mixed sample set according to the plurality of second reference labels of each mixed sample, to obtain a second to-be-labeled sample set. The second labeled confirmation sample set obtaining sub-module is configured to obtain a second labeled confirmation sample set according to the corrected second to-be-labeled sample set. The second confidence sample set generation sub-module is configured to generate the second confidence sample set according to the second mixed sample set and the second labeled confirmation sample set.

10. The apparatus of claim 9, wherein, The second to-be-labeled sample set determination sub-module includes: The second to-be-labeled sample set determination unit is configured to select at least one mixed sample in the second mixed sample set according to a voting result of the plurality of second reference labels, or according to label probability values of the plurality of second reference labels, to obtain the second to-be-labeled sample set.

11. A model training apparatus, comprising: a target text classification model determination module configured to train an initial text classification model according to a confidence sample set to obtain a target text classification model, the initial text classification model including a convolutional neural network, wherein the confidence sample set is obtained by the sample generation apparatus of any one of claims 7 to 10.

12. A text classification apparatus, comprising: a text information category determination module configured to input text information into a target text classification model to obtain a category of the text information, wherein the target text classification model is trained by the model training apparatus of claim 11, the text information includes chief complaint text information.

13. An electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-6.

14. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to enable the computer to perform the method of any one of claims 1-6.

15. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1-6.

Images