A sample screening method, system and neural network model training method

Abstract

The present invention relates to the field of artificial intelligence technology, and provides a sample screening method, system and neural network model training method. The method calculates the distance between an unlabeled sample set and a labeled sample set, and sorts them according to the distance to obtain a sorting result; extracts characterization parameters of the unlabeled sample set to obtain a corresponding activity value; based on the sorting result and the activity value, selects some samples from the unlabeled sample set for labeling to obtain a new labeled sample set; while selecting samples related to the training task, the present invention also maintains a low sampling rate for irrelevant data and avoids collecting outliers irrelevant to the training task, thereby ensuring that valuable samples are sampled. Therefore, during the training process of the neural network model, the influence of samples that are completely unrelated to the task on the training results is avoided, thereby improving the recognition accuracy of the neural network model.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a sample screening method, system and neural network model training method. Background Art

[0002] Active learning refers to the use of uncertainty sampling to find the data with the highest information content in the unlabeled dataset for labeling, thereby avoiding over-labeling of low-information data. The quality of labeled samples will affect the training effect of the model and thus affect the application performance of the model.

[0003] Traditional active learning methods such as entropy sampling and edge sampling will oversample some samples that are very similar but contain a lot of information, resulting in a large amount of duplication in the labeled data.

[0004] To avoid the above situation, the traditional diversity sampling strategy calculates the cosine similarity between existing samples as a judgment indicator. When the similarity in the data set is higher than a certain threshold, the system automatically gives up labeling and extracting this type of data. Although this solution ensures that the data is no longer repeated, the extracted unlabeled data are all outliers in the invalid data area. These examples are often isolated cases in the entire unlabeled pool and have no effect on model training. In addition, the valid data area of ​​the unlabeled pool represents sample points that cannot be sampled. Although the amount of information in these data is not very high, they are often the centroid of a sample group and are very representative. These points are often ignored because they are generally similar to valid data. Summary of the Invention

[0005] In view of the shortcomings of the prior art described above, the purpose of the present invention is to provide a sample screening method, system and neural network model training method to solve the problem in the prior art that the active learning method will over-sampling, resulting in a large amount of duplication of labeled data, which in turn affects the low recognition accuracy of the trained model.

[0006] A first aspect of the present invention provides a sample screening method, comprising:

[0007] Constructing a first training set, wherein the first training set includes an unlabeled sample set and a labeled sample set;

[0008] Calculating the distance between the unlabeled sample set and the labeled sample set, and sorting them in ascending order to obtain a sorting result;

[0009] Inputting the unlabeled sample set into a pre-trained neural network model, extracting corresponding representation parameters, and calculating the activity value of each representation parameter;

[0010] Selecting some samples from the unlabeled sample set for labeling according to the ranking result and the activity value, and adding the labeled samples to the labeled sample set to form a new labeled sample set;

[0011] The remaining samples in the unlabeled sample set and the new labeled sample set are combined to form a first filtered training set.

[0012] In one embodiment of the present invention, the step of calculating the distance between the unlabeled sample set and the labeled sample set includes:

[0013] Using the KM algorithm, cluster analysis is performed on the unlabeled sample set and the labeled sample set to obtain a first cluster group and a second cluster group;

[0014] Selecting a preset number of clusters from each of the first cluster group and the second cluster group, and extracting the centroid of each cluster;

[0015] The sum of the Euclidean distances between each centroid from the first cluster group and all centroids from the second cluster group is calculated to obtain the distance between the unlabeled sample set and the labeled sample set.

[0016] In one embodiment of the present invention, the step of inputting the unlabeled sample set into a pre-trained neural network model, extracting corresponding characterization parameters, and calculating the activity value of each characterization parameter includes:

[0017] Inputting each cluster selected from the first cluster group into a pre-trained neural network model to extract corresponding representation parameters;

[0018] Calculate the average value of each characterization parameter and determine the corresponding activity value based on the average value; the larger the average value, the higher the corresponding activity value.

[0019] In one embodiment of the present invention, the step of selecting some samples from the unlabeled sample set for labeling according to the ranking result includes:

[0020] According to the sorting results, select multiple position intervals;

[0021] Assigning a sampling rate to each position interval, wherein the sampling rate of the position interval with a higher ranking is greater than the sampling rate of the position interval with a lower ranking;

[0022] According to the sampling rate, clusters from the first cluster group corresponding to each position interval are extracted, and all samples in each extracted cluster are labeled.

[0023] In one embodiment of the present invention, the step of selecting some samples from the unlabeled sample set for labeling according to the activity value includes:

[0024] From each cluster selected from the first cluster group, a cluster whose activity value is less than a preset activity value and whose sorting order is before the preset order is selected, and all samples in each selected cluster are labeled.

[0025] In one embodiment of the present invention, the unlabeled sample set is input into a pre-trained neural network model to obtain corresponding classification probabilities;

[0026] Calculate the entropy value of each classification probability, and eliminate unlabeled samples with entropy values ​​greater than a preset entropy value.

[0027] A second aspect of the present invention further provides a method for training a language recognition model, comprising:

[0028] Step 1: Using the sample screening method described in any one of the first aspects, screen the samples in the second training set to obtain a new second training set;

[0029] Step 2: Use the new second training set to train the initial language recognition model under the constraint of the cross entropy loss function;

[0030] During the training process, it is determined whether the initial language recognition model has converged.

[0031] If convergence occurs, stop training and obtain the trained language recognition model;

[0032] Otherwise, return to step 1 and re-screen the samples in the second training set.

[0033] A third aspect of the present invention further provides a sample screening system, comprising:

[0034] A sample construction module, configured to construct a first training set, wherein the first training set includes an unlabeled sample set and a labeled sample set;

[0035] A distance calculation module is used to calculate the distance between the unlabeled sample set and the labeled sample set, and sort them in ascending order to obtain a sorting result;

[0036] An activity value calculation module, configured to input the unlabeled sample set into a pre-trained neural network model, extract corresponding characterization parameters, and calculate the activity value of each characterization parameter;

[0037] a processing module, configured to select some samples from the unlabeled sample set for labeling based on the sorting result and the activity value, and add the labeled samples to the labeled sample set to form a new labeled sample set;

[0038] The sample construction module is further configured to combine the remaining samples in the unlabeled sample set and the new labeled sample set to form a screened first training set.

[0039] The fourth aspect of the present invention also provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method steps described in the sample screening method of the first aspect of the present invention are implemented, or when the processor executes the computer program, the method steps described in the language recognition model training method of the second aspect of the present invention are implemented.

[0040] The fifth aspect of the present invention also provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements the method steps described in a sample screening method of the first aspect of the present invention, or, when executed by a processor, implements the method steps described in a language recognition model training method of the second aspect of the present invention.

[0041] As described above, the sample screening method, system, and neural network model training method of the present invention have the following beneficial effects:

[0042] The present invention calculates the distance between an unlabeled sample set and a labeled sample set, and sorts them according to the distance to obtain a sorting result; extracts the characterization parameters of the unlabeled sample set to obtain a corresponding activity value; based on the sorting result and the activity value, selects some samples from the unlabeled sample set for labeling to obtain a new labeled sample set; while selecting samples related to the training task, the present invention also maintains a low sampling rate for irrelevant data and avoids collecting outliers irrelevant to the training task, thereby ensuring that valuable samples are sampled. Therefore, during the training process of the neural network model, the influence of samples that are irrelevant to the task on the training results is avoided, thereby improving the recognition accuracy of the neural network model. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0044] Figure 1 It is a schematic flow chart of the sample screening method according to the first embodiment of the present invention.

[0045] Figure 2 Shown is a schematic diagram of data sample distribution in the first embodiment of the present invention.

[0046] Figure 3 It is a flowchart of the training method of the neural network model in the second embodiment of the present invention.

[0047] Figure 4 Shown is a structural block diagram of a sample screening system according to a third embodiment of the present invention.

[0048] Figure 5 Shown is a flow chart of a method for detecting violations of life insurance customer service in a fourth embodiment of the present invention.

[0049] Figure 6 It is a schematic diagram of a computer device according to a fifth embodiment of the present invention. DETAILED DESCRIPTION

[0050] The following describes the embodiments of the present invention through specific examples. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments. The details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other unless they conflict.

[0051] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention. Therefore, the figures only show components related to the present invention and are not drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component can be changed at will, and the component layout type may also be more complicated.

[0052] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to achieve optimal results.

[0053] Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0054] See also Figure 1 The first embodiment of the present invention relates to a sample screening method for screening training samples of a neural network model in active learning, specifically comprising:

[0055] Step 101: Construct a first training set.

[0056] Specifically, during active learning, an original sample set for training a neural network model is typically constructed based on the recognition task. This original sample set generally uses a publicly available data source on the Internet and includes an unlabeled sample set and a labeled sample set. The labeled sample set can be selected from labeled samples from publicly available data sources on the Internet, or a small number of samples can be randomly selected from the unlabeled sample set for manual labeling based on project needs. The initial neural network model is then iteratively trained based on the training set in the original sample set, ultimately yielding a trained neural network model. During manual labeling, in order to improve the recognition accuracy of the neural network model, the most informative unlabeled data should be selected for manual labeling to complete the data augmentation for this task.

[0057] The sample screening method of this embodiment is applicable to various neural network models. For the convenience of description, the following takes the language recognition model as an example to provide a detailed explanation:

[0058] For example, when detecting violations in the sales talk of life insurance agents, it is necessary to determine whether the sales talk of the agents violates a series of quality inspection points. Before training the language recognition model, the first training set is constructed based on the life insurance sales corpus. The first training set includes an unlabeled sample set and a labeled sample set. It should be noted that when screening labeled samples from the unlabeled sample set, sample data related to the quality inspection points should be selected as much as possible.

[0059] See also Figure 2 The outer circle is the entire data sample space of the insurance sales corpus, and the inner circle is the data points related to the quality inspection point A in the entire data sample space. It should be understood that each data point is a sentence or paragraph obtained by converting the agent's sales speech into text and then processing the text. Among them,

[0060] The first type of data points are clustered data points related to quality inspection point A. These are the data points required in the detection process. In order to avoid a large amount of data duplication, oversampling is not allowed.

[0061] The second type of data points are outlier data points related to quality inspection point A. These are the data points required in the detection process and need to be sampled as much as possible;

[0062] The third type of data points are clustered data points that are not related to quality inspection point A. These are not required data points in the detection process, but these samples are often interference samples. It is necessary to find some representative sample points for each cluster.

[0063] The fourth type of data points are outlier data points that are irrelevant to quality inspection point A. This type of data is not needed in the detection process, and the selection strategy of active learning should try to avoid this type of data.

[0064] Based on the above principles for selecting sample data, this embodiment uses an active learning approach to select samples that meet preset conditions from an unlabeled sample set and label them.

[0065] It should be understood that when training a language recognition model, a first test set is also included. The sample screening method of this embodiment is used to screen the data sources publicly available on the Internet, and then the screened data sources are randomly divided into a first training set and a first test set according to a preset ratio. The first test set is used to test the recognition accuracy of the trained language recognition model. If the recognition accuracy reaches the expected value, training is stopped; otherwise, training of the language recognition model continues.

[0066] Step 102: Calculate the distance between the unlabeled sample set and the labeled sample set, and sort them in ascending order to obtain a sorting result.

[0067] Specifically, in order to filter out the sample data with the most information, this embodiment calculates the distance between the unlabeled sample set and the labeled sample set. The closer the distance, the more information the sample data contains. The step of calculating the distance includes:

[0068] The Kmeans algorithm, referred to as KM algorithm, is used to perform unsupervised cluster analysis on the unlabeled sample set and the labeled sample set, respectively, to obtain the first cluster group and the second cluster group.

[0069] The KM algorithm is a commonly used clustering algorithm in the prior art, and its clustering analysis logic is:

[0070] Step 1: Randomly select k initial centroids from the observation sample set;

[0071] Step 2: Traverse all sample data in the observation sample set, find the initial centroid closest to each sample data, and divide each sample data into the same class as the initial centroid closest to it;

[0072] Step 3: Calculate the average value of all sample data in each category to obtain the new centroid;

[0073] Step 4: Repeat steps 2 and 3 until the k centroids no longer change or the number of iterations is reached to obtain the cluster analysis results.

[0074] To continue the explanation, a preset number of clusters are selected from the first cluster group and the second cluster group respectively, the centroid of each cluster is extracted, and the sum of the Euclidean distances between each centroid from the first cluster group and all centroids from the second cluster group is calculated to obtain the distance between the unlabeled sample set and the labeled sample set.

[0075] It should be understood that the first cluster group and the second cluster group each include multiple clusters, from which a preset number of clusters are selected. This number can be set as needed. In this embodiment, 50 clusters are selected from the first cluster group and 20 clusters are selected from the second cluster group. The centroid of each cluster is extracted, and the centroids from the first cluster group are marked as A1 to A50, and the centroids from the second cluster group are marked as B1 to B20. For the centroids A1 to A50, the sum of the Euclidean distances from each centroid to all centroids from the second cluster group is calculated, and all the sums are sorted in ascending order. The smaller the sum, the higher the ranking, that is, the closer the centroid is to the cluster centroid of the inner circle, and the more sampling rate should be allocated to it.

[0076] It should be noted that the sample data screening process is an iterative process. The screened sample data is used as the training set of the language recognition model. According to the recognition results of the language recognition model, the parameters involved in the sample screening method are fine-tuned, such as adjusting the number of clusters selected from the first cluster group and the second cluster group, the sampling rate, etc., until the language recognition model converges.

[0077] To explain further, as the language recognition model iterates, the above sampling rate should change dynamically with the iterative process of active learning and is a continuously changing value. However, in general, a ratio can be preset based on experience and sorting results to allocate a sampling rate for each unlabeled sample.

[0078] Step 103: Input the unlabeled sample set into the pre-trained neural network model, extract the corresponding representation parameters, and calculate the activity value of each representation parameter.

[0079] Specifically, the pre-trained neural network model in this embodiment is a language recognition model, which is obtained by constructing a training set based on the recognition task and pre-training on the basis of the original language recognition model. For example, in the violation detection of life insurance agent speech, a sample set is constructed based on the insurance sales corpus, and a small number of original samples are randomly selected from the sample set for annotation. The sample set is then randomly divided into a training set and a test set according to a preset ratio. The training set is used to train the original language recognition model under the constraint of the loss function. When the original language recognition model converges or reaches a preset number of iterations, the pre-trained language recognition model can be obtained. The language recognition model in this embodiment can directly use the BERT model; it can also be obtained by self-training based on the BERT model in combination with the recognition task.

[0080] It should be understood that the BERT model is a language encoder. The large-scale corpus on which the BERT model is based covers various knowledge domains. Pre-trained language models trained on this corpus can be used to solve natural language problems in different fields. The BERT model introduces two pre-training tasks: MLM (Masked Language Model) and NSP (Next Sentence Prediction). The language model is first pre-trained on a large amount of unsupervised corpus, and then fine-tuned using a small amount of annotated corpus to complete downstream NLP tasks such as text classification, sequence labeling, machine translation, and reading comprehension.

[0081] To explain further, the unlabeled sample can be a sentence or a paragraph. Each cluster selected from the first cluster group is input into the pre-trained BERT model to extract the corresponding representation parameter, which is a 768-dimensional vector output by the relu layer, the activation function of the BERT model. The average value of the 768-dimensional vector is calculated, and the corresponding activity value is determined based on the average value. The larger the average value, the higher the corresponding activity value.

[0082] Step 104: Select some samples from the unlabeled sample set for labeling based on the sorting results and the activity value, and add the labeled samples to the labeled sample set to form a new labeled sample set.

[0083] Specifically, the higher the ranking result, the more information the sample data corresponding to that position is considered to contain. Therefore, during screening, a higher sampling rate can be assigned to the sample data at the top. At the same time, to avoid large amounts of data duplication, an appropriate sampling rate can be assigned to the sample data at the middle position, thereby completing the sampling of the first category of data in step 101. Although the sample data at the back position contains less information, this type of data is usually interference samples, so a lower sampling rate can be assigned to retain some representative sample data, thereby completing the sampling of the third category of data in step 101. Overall, the sampling rate of the top ranking is greater than the sampling rate of the bottom ranking.

[0084] Specifically applied to this embodiment, according to the sorting results, three position intervals are selected, for example, the top five, 20-25, and last five position intervals are selected, and the sampling rate allocated to each position interval is 100:10:1 respectively; according to the sampling rate, the cluster from the first cluster group corresponding to each position interval is extracted, and all samples in the extracted cluster are marked.

[0085] To continue, the second and fourth types of data in step 101 are both outlier data, wherein the second type of data is the sample data required for training the language recognition model, and the sampling rate should be allocated as much as possible; while the fourth type of data is the sample data not required for training the language recognition model, and such data should be avoided as much as possible. In order to identify these outlier data, in step 103, this embodiment uses the pre-trained BERT model to extract the characterization parameters of each cluster selected from the first cluster group, calculates the average value of the characterization parameters, and then determines the activity value corresponding to the average value. For samples whose activity value is less than the preset activity value, it is recorded as a cooling point. The cooling points with a higher sorting order are considered to be the required second type of data; the cooling points with a lower sorting order are considered to be the unnecessary fourth type of data.

[0086] Specifically applied to this embodiment, from each cluster selected from the first cluster group, a cluster whose activity value is less than the preset activity value and whose sorting order is before the preset order is selected, and all samples in each selected cluster are marked; for the cluster whose activity value is less than the preset activity value and whose sorting order is after the preset order, a sampling rate of 0% is assigned.

[0087] It is further explained that in order to improve the screening accuracy, the sample screening method of the present invention can also screen samples by judging the stability of the sample data, and input each unlabeled sample in the unlabeled sample set into the pre-trained BERT model to obtain the corresponding classification probability; the entropy value of each classification probability is calculated, and the smaller the entropy value, the better the stability of the information included in the sample data. The unlabeled samples with entropy values ​​greater than the preset entropy value are eliminated, and the sample data with good stability are retained.

[0088] Among them, the entropy value of each classification probability is calculated using the following formula:

[0089]

[0090] Among them, x i For each unlabeled sample, P(x i ) is the classification probability of each unlabeled sample.

[0091] It should be understood that the above-mentioned preset activity value, preset order and preset entropy value can be set according to actual needs, and this solution does not limit the specific data.

[0092] The samples screened by the above scheme are labeled, and the labeled samples are added to the labeled sample set to generate a new labeled sample set.

[0093] Step 105: Combine the remaining samples in the unlabeled sample set and the new labeled sample set to form a filtered first training set.

[0094] Specifically, the above scheme is to select and eliminate samples in the unlabeled sample set, and combine the remaining samples in the unlabeled sample set and the new labeled sample set generated in step 104 to form a first filtered training set. The first filtered training set is used to train the language recognition model and is also the basis for the next sample screening.

[0095] It can be seen that this embodiment obtains the sorting result by calculating the distance between the unlabeled sample set and the labeled sample set, and sorting them according to the distance; obtains the corresponding activity value by extracting the characterization parameters of the unlabeled sample set; based on the sorting result and the activity value, selects some samples from the unlabeled sample set for labeling to obtain a new labeled sample set; while selecting samples related to the training task, the present invention also maintains a low sampling rate for irrelevant data and avoids collecting outliers irrelevant to the training task, thereby ensuring that valuable samples are sampled. Therefore, in the training process of the neural network model, the influence of samples that have nothing to do with the task on the training results is avoided, thereby improving the recognition accuracy of the neural network model.

[0096] See also Figure 3 The second embodiment of the present invention relates to a training method for a neural network model, specifically comprising:

[0097] Step 301: Filter samples in the second training set to obtain a new second training set; wherein the screening method adopts the sample screening method disclosed in the first embodiment.

[0098] Specifically, for the convenience of description, a language recognition model is used for illustration in this embodiment. The language recognition model is trained based on the BERT model. The second training set can use the corpus sample set that comes with the BERT model, or it can be constructed by itself according to the recognition field.

[0099] The second training set includes an unlabeled sample set and a labeled sample set.

[0100] It should be understood that the sample set of this embodiment also includes a second test set. The sample screening method in the first embodiment is used to screen the sample set of this embodiment, and then the screened sample set is randomly divided into a second training set and a second test set according to a preset ratio. The second test set is used to test the recognition accuracy of the trained language recognition model. If the recognition accuracy reaches the expected value, the training is stopped; otherwise, the language recognition model continues to be trained.

[0101] Step 302: Using the new second training set, train the initial language recognition model under the constraint of the cross-entropy loss function. During the training process, determine whether the initial language recognition model has converged. If so, stop training to obtain the trained language recognition model. Otherwise, return to step 301 and re-screen the samples in the second training set.

[0102] Specifically, the new second training set is input into the BERT model, the classification probability is calculated, and the cross-entropy loss function is used to calculate the loss value of the classification probability and the true value, where the true value is obtained based on the labeled sample set; based on the loss value, the network parameters of the BERT model are adjusted until the BERT model converges or completes the preset number of iterations, and the trained speech recognition model is obtained. During the training process, if the BERT model does not converge or does not complete the preset number of iterations, the process returns to step 301, and based on the second training set previously screened, the unlabeled samples therein are re-screened, and the second training set that was screened this time is used as the training set for the next training of the BERT model.

[0103] Furthermore, the training method of the neural network model in this embodiment also includes the following testing steps:

[0104] The second test set is input into the trained speech recognition model to test its recognition accuracy. If the recognition accuracy is greater than the preset accuracy, the trained speech recognition model is used as the final speech recognition model. Otherwise, steps 301 to 302 are repeated.

[0105] It can be seen that this embodiment adopts the sample screening method of the first embodiment to screen the training set of the neural network model. The screened samples ensure relevance to the training task while maintaining a low sampling rate for irrelevant data, and avoid collecting outliers irrelevant to the training task, thereby ensuring that valuable samples are sampled. Therefore, in the training process of the neural network model, the influence of samples that have nothing to do with the task on the training results is avoided, thereby improving the recognition accuracy of the neural network model.

[0106] See also Figure 4 A third embodiment of the present invention relates to a sample screening system, specifically comprising:

[0107] The sample construction module is used to construct the first training set.

[0108] Specifically, the first training set can adopt a data source publicly available on the Internet, or can be constructed independently according to the recognition field; wherein the first training set includes an unlabeled sample set and a labeled sample set.

[0109] It should be understood that the sample set of this embodiment also includes a first test set, which is used to test the recognition accuracy of the trained neural network model. If the recognition accuracy reaches the expected value, training is stopped; otherwise, the neural network model continues to be trained.

[0110] The distance calculation module is used to calculate the distance between the unlabeled sample set and the labeled sample set, and sort them in ascending order to obtain the sorting result.

[0111] Specifically, in order to filter out the sample data with the most information, this embodiment calculates the distance between the unlabeled sample set and the labeled sample set. The closer the distance, the more information the sample data contains. The step of calculating the distance includes:

[0112] The Kmeans algorithm, referred to as KM algorithm, is used to perform unsupervised cluster analysis on the unlabeled sample set and the labeled sample set, respectively, to obtain the first cluster group and the second cluster group.

[0113] The KM algorithm is a commonly used clustering algorithm in the prior art, and its clustering analysis logic is:

[0114] Step 1: Randomly select k initial centroids from the observation sample set;

[0115] Step 2: Traverse all sample data in the observation sample set, find the initial centroid closest to each sample data, and divide each sample data into the same class as the initial centroid closest to it;

[0116] Step 3: Calculate the average value of all sample data in each category to obtain the new centroid;

[0117] Step 4: Repeat steps 2 and 3 until the k centroids no longer change or the number of iterations is reached to obtain the cluster analysis results.

[0118] To continue the explanation, a preset number of clusters are selected from the first cluster group and the second cluster group respectively, the centroid of each cluster is extracted, and the sum of the Euclidean distances between each centroid from the first cluster group and all centroids from the second cluster group is calculated to obtain the distance between the unlabeled sample set and the labeled sample set.

[0119] It should be understood that the first cluster group and the second cluster group each include multiple clusters, from which a preset number of clusters are selected. This number can be set as needed. In this embodiment, 50 clusters are selected from the first cluster group and 20 clusters are selected from the second cluster group. The centroid of each cluster is extracted, and the centroids from the first cluster group are marked as A1 to A50, and the centroids from the second cluster group are marked as B1 to B20. For the centroids A1 to A50, the sum of the Euclidean distances from each centroid to all centroids from the second cluster group is calculated, and all the sums are sorted in ascending order. The smaller the sum, the higher the ranking, that is, the closer the centroid is to the cluster centroid of the inner circle, and the more sampling rate should be allocated to it.

[0120] It should be noted that the sample data screening process is an iterative process. The screened sample data is used as the training set of the neural network model. According to the recognition results of the neural network model, the parameters involved in the sample screening method are fine-tuned, such as adjusting the number of clusters selected from the first cluster group and the second cluster group, the sampling rate, etc., until the neural network model converges.

[0121] To explain further, as the neural network model iterates, the above sampling rate should change dynamically with the iterative process of active learning and is a continuously changing value. However, in general, a ratio can be preset based on experience and sorting results to assign a sampling rate to each unlabeled sample.

[0122] The activity value calculation module is used to input the unlabeled sample set into the pre-trained neural network model, extract the corresponding representation parameters, and calculate the activity value of each representation parameter.

[0123] Specifically, the pre-trained neural network model in this embodiment is obtained by constructing a training set based on the recognition task and pre-training on the basis of the original neural network model. For example, in the violation detection of life insurance agent speech, a sample set is constructed based on the insurance sales corpus, and a small number of original samples are randomly selected from the sample set for annotation. The sample set is then randomly divided into a training set and a test set according to a preset ratio. The training set is used to train the original language recognition model under the constraint of the loss function. When the original language recognition model converges or reaches a preset number of iterations, the pre-trained language recognition model can be obtained. The language recognition model in this embodiment can directly use the BERT model; it can also be obtained by self-training based on the BERT model in combination with the recognition task.

[0124] To explain further, the unlabeled sample can be a sentence or a paragraph. Each cluster selected from the first cluster group is input into the pre-trained BERT model to extract the corresponding representation parameter, which is a 768-dimensional vector output by the relu layer, the activation function of the BERT model. The average value of the 768-dimensional vector is calculated, and the corresponding activity value is determined based on the average value. The larger the average value, the higher the corresponding activity value.

[0125] The processing module is used to select some samples from the unlabeled sample set for labeling according to the sorting result and the activity value, and add the labeled samples to the labeled sample set to form a new labeled sample set.

[0126] Specifically, the closer the ranking result is, the more information the sample data corresponding to that position is considered to contain. Therefore, during screening, a higher sampling rate can be assigned to the sample data at the front. At the same time, to avoid large amounts of data duplication, an appropriate sampling rate can be assigned to the sample data at the middle position. This completes the sampling of the first category of data in step 101 of the first embodiment. Although the sample data at the back position contains less information, this type of data is usually interference samples. Therefore, a lower sampling rate can be assigned to retain some representative sample data, thereby completing the sampling of the third category of data in step 101. Overall, the sampling rate of the top ranking is greater than the sampling rate of the bottom ranking.

[0127] Specifically applied to this embodiment, according to the sorting results, three position intervals are selected, for example, the top five, 20-25, and last five position intervals are selected, and the sampling rate allocated to each position interval is 100:10:1 respectively; according to the sampling rate, the cluster from the first cluster group corresponding to each position interval is extracted, and all samples in the extracted cluster are marked.

[0128] Continuing to explain, the second category of data and the fourth category of data in step 101 of the first embodiment are both outlier data, wherein the second category of data is the sample data required for training the language recognition model, and the sampling rate should be allocated as much as possible; while the fourth category of data is the sample data not required for training the language recognition model, and such data should be avoided as much as possible. In order to identify these outlier data, in step 103 of the first embodiment of this embodiment, the characterization parameters of each cluster selected from the first cluster group are extracted through the pre-trained BERT model, the average value of the characterization parameters is calculated, and the activity value corresponding to the average value is determined. For samples with an activity value less than the preset activity value, it is recorded as a cooling point. For cooling points with a higher sorting order, it is considered to be the required second category of data; for cooling points with a lower sorting order, it is considered to be the unnecessary fourth category of data.

[0129] Specifically applied to this embodiment, from each cluster selected from the first cluster group, a cluster whose activity value is less than the preset activity value and whose sorting order is before the preset order is selected, and all samples in each selected cluster are marked; for the cluster whose activity value is less than the preset activity value and whose sorting order is after the preset order, a sampling rate of 0% is assigned.

[0130] It is further explained that in order to improve the screening accuracy, the processing module can also screen samples by judging the stability of the sample data, and input each unlabeled sample in the unlabeled sample set into the pre-trained BERT model to obtain the corresponding classification probability; calculate the entropy value of each classification probability, the smaller the entropy value, the better the stability of the information included in the sample data, eliminate the unlabeled samples with entropy values ​​greater than the preset entropy value, and retain the sample data with good stability.

[0131] Among them, the entropy value of each classification probability is calculated using the following formula:

[0132]

[0133] Among them, x i For each unlabeled sample, P(x i ) is the classification probability of each unlabeled sample.

[0134] The sample construction module is also used to combine the remaining samples in the unlabeled sample set and the new labeled sample set to form the first training set after screening.

[0135] As can be seen, this embodiment calculates the distance between the unlabeled sample set and the labeled sample set and sorts them according to the distance to obtain a sorting result; extracts the characterization parameters of the unlabeled sample set to obtain the corresponding activity value; and based on the sorting result and the activity value, selects some samples from the unlabeled sample set for labeling to obtain a new labeled sample set. While selecting samples related to the training task, this embodiment also maintains a low sampling rate for irrelevant data and avoids collecting outliers that are irrelevant to the training task, thereby ensuring that valuable samples are sampled. Therefore, during the training process of the neural network model, the influence of samples that are irrelevant to the task on the training results is avoided, thereby improving the recognition accuracy of the neural network model.

[0136] See also Figure 5 A fourth embodiment of the present invention relates to a method for detecting violations in life insurance customer service, specifically comprising:

[0137] Step 501: Obtain text information of the agent's sales promotion.

[0138] Specifically, the call audio between the agent and the customer is obtained from the call log library. In this embodiment, the call audio is automatically recorded during the call between the agent and the customer, and the audio is saved to the call log library. The call audio used for illegal speech detection can be either quasi-real-time or historical, depending on actual needs. Optionally, when storing the call audio, further information related to the call audio can be stored, such as the agent's name, agent ID, call start time, end time, customer name, customer phone number, and type of business involved.

[0139] To further illustrate, the acquired call audio is input into the speech-to-text model to extract sound features. Each extracted sound feature is matched with standard sound data in the sound library, and the text corresponding to the standard sound data with the highest matching degree is selected as the converted text information. The speech-to-text model is trained based on the DNN-HMM neural network. The deep neural network DNN is used to model the observation probability of the input audio information, and the hidden Markov model HMM is used to model the jump relationship between different states to characterize the temporal changes of the speech signal and achieve the best match between the unknown speech form and a model in the model library.

[0140] Step 502: construct a third training set based on the life insurance sales corpus, and screen the third training set to obtain a new third training set; wherein the screening method adopts the sample screening method disclosed in the first embodiment.

[0141] Specifically, a life insurance test sample set is constructed based on a life insurance sales corpus, and the life insurance test sample set includes an unlabeled sample set and a labeled sample set; the sample screening method in the first embodiment is used to screen the life insurance test sample set, and then the screened life insurance test sample set is randomly divided into a third training set and a third test set according to a preset ratio.

[0142] Step 503: Use the new third training set to train the language recognition model to obtain a model for identifying illegal speech.

[0143] Specifically, the training method in the second embodiment can be used to train the language recognition model in this embodiment. The main steps of the training include:

[0144] A new third training set is used to train the language recognition model under the constraint of the cross-entropy loss function. During the training process, whether the language recognition model has converged is determined. If so, training is stopped to obtain a model for identifying illegal speech. Otherwise, the samples in the third training set are rescreened, and the rescreened samples are used to retrain the language recognition model.

[0145] To further explain, the training steps also include:

[0146] The third test set is input into the illegal speech recognition model to test its recognition accuracy. If the recognition accuracy is greater than the preset accuracy, the illegal speech recognition model is used as the trained illegal speech recognition model, otherwise, it is retrained.

[0147] Step 504: Input the text information into the illegal speech recognition model to obtain a recognition result; based on the recognition result, determine whether the agent has used illegal speech.

[0148] Specifically, the text information is divided into multiple paragraphs or sentences as needed, and each paragraph or sentence is input into the illegal speech recognition model for recognition to obtain a recognition result, which is a probability value. If the probability value is greater than the preset probability value, it is considered that illegal speech exists.

[0149] Furthermore, the service quality of the agent can be scored based on the recognition results, and an evaluation form can be generated based on the score and information related to the call audio.

[0150] It can be seen that this embodiment adopts the sample screening method of the first embodiment to screen the training set of the illegal speech recognition model. The screened samples ensure relevance to the training task while maintaining a low sampling rate for irrelevant data, and avoid collecting outliers irrelevant to the training task, thereby ensuring that valuable samples are sampled. Therefore, in the training process of the neural network model, the influence of samples that have nothing to do with the task on the training results is avoided, thereby improving the recognition accuracy of the illegal speech recognition model.

[0151] See also Figure 6 A fifth embodiment of the present invention relates to a computer device, comprising a memory 601, a processor 602, and a computer program stored in the memory 601 and executable on the processor 602. When the processor 602 executes the computer program, the following steps are implemented:

[0152] Constructing a first training set, where the first training set includes an unlabeled sample set and a labeled sample set;

[0153] Calculate the distance between the unlabeled sample set and the labeled sample set, and sort them in ascending order to obtain the sorting result;

[0154] Input the unlabeled sample set into the pre-trained neural network model, extract the corresponding representation parameters, and calculate the activity value of each representation parameter;

[0155] According to the ranking results and activity values, some samples are selected from the unlabeled sample set for labeling, and the labeled samples are added to the labeled sample set to form a new labeled sample set;

[0156] The remaining samples in the unlabeled sample set and the new labeled sample set are combined to form the first training set after screening.

[0157] When the processor 602 executes the computer program, the following steps may also be implemented:

[0158] Step 1: Using any one of the sample screening methods described in the first embodiment, screen the samples in the second training set to obtain a new second training set;

[0159] Step 2: Use the new second training set to train the initial neural network model under the constraint of the cross entropy loss function;

[0160] During the training process, it is determined whether the initial neural network model has converged.

[0161] If convergence occurs, stop training and obtain the trained neural network model;

[0162] Otherwise, return to step 1 and re-screen the samples in the second training set.

[0163] Memory 601 and processor 602 are connected using a bus. The bus can include any number of interconnected buses and bridges, connecting various circuits of one or more processors 602 and memory 601. The bus can also connect various other circuits such as peripheral devices 603, voltage regulators 604, and power management circuits. These are all well known in the art and are therefore not described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single component or multiple components, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by processor 602 is transmitted over a wireless medium via an antenna. Furthermore, the antenna receives data and transmits it to processor 602.

[0164] The processor 602 is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. The memory 601 can be used to store data used by the processor 602 when performing operations.

[0165] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0166] A sixth embodiment of the present invention relates to a storage medium having a computer program stored thereon, which, when executed by a processor, implements the following steps:

[0167] Constructing a first training set, where the first training set includes an unlabeled sample set and a labeled sample set;

[0168] Calculate the distance between the unlabeled sample set and the labeled sample set, and sort them in ascending order to obtain the sorting result;

[0169] Input the unlabeled sample set into the pre-trained neural network model, extract the corresponding representation parameters, and calculate the activity value of each representation parameter;

[0170] According to the ranking results and activity values, some samples are selected from the unlabeled sample set for labeling, and the labeled samples are added to the labeled sample set to form a new labeled sample set;

[0171] The remaining samples in the unlabeled sample set and the new labeled sample set are combined to form the first training set after screening.

[0172] When the computer program is executed by a processor, the following steps may be further implemented:

[0173] Step 1: Using any one of the sample screening methods described in the first embodiment, screen the samples in the second training set to obtain a new second training set;

[0174] Step 2: Use the new second training set to train the initial neural network model under the constraint of the cross entropy loss function;

[0175] During the training process, it is determined whether the initial neural network model has converged.

[0176] If convergence occurs, stop training and obtain the trained neural network model;

[0177] Otherwise, return to step 1 and re-screen the samples in the second training set.

[0178] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0179] In summary, the present invention provides a sample screening method, system and neural network model training method, which calculates the distance between the unlabeled sample set and the labeled sample set and sorts them according to the distance to obtain a sorting result; extracts the characterization parameters of the unlabeled sample set to obtain the corresponding activity value; based on the sorting result and the activity value, selects some samples from the unlabeled sample set for labeling to obtain a new labeled sample set; while ensuring that the selected samples are outliers in each cluster, the present invention also maintains a low sampling rate for each cluster of irrelevant data and avoids collecting outliers that are irrelevant to the training task, thereby ensuring that valuable samples are sampled. Therefore, in the training process of the neural network model, the influence of samples that are irrelevant to the task on the training results is avoided, thereby improving the recognition accuracy of the neural network model. Therefore, the present invention effectively overcomes the various shortcomings of the prior art and has a high industrial utilization value.

[0180] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the present invention. Anyone skilled in the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by one of ordinary skill in the art without departing from the spirit and technical principles disclosed herein are intended to be covered by the claims of the present invention.

Claims

1. A sample screening method, characterized in that: include: Constructing a first training set, wherein the first training set includes an unlabeled sample set and a labeled sample set; Performing cluster analysis on the unlabeled sample set and the labeled sample set respectively to obtain a first cluster group and a second cluster group, calculating the distance between the unlabeled sample set and the labeled sample set based on the first cluster group and the second cluster group, and sorting them in ascending order to obtain a sorting result; Inputting the unlabeled sample set into the pre-trained language recognition model, extracting the corresponding representation parameters through the activation function, and calculating the activity value of each representation parameter; Selecting some samples from the unlabeled sample set for labeling according to the ranking result and the activity value, and adding the labeled samples to the labeled sample set to form a new labeled sample set; Combining the remaining samples in the unlabeled sample set and the new labeled sample set to form a filtered first training set; Furthermore, the step of selecting some samples from the unlabeled sample set for labeling according to the ranking result includes: According to the sorting results, select multiple position intervals; Assigning a sampling rate to each position interval, wherein the sampling rate of the position interval with a higher ranking is greater than the sampling rate of the position interval with a lower ranking; According to the sampling rate, clusters from the first cluster group corresponding to each position interval are extracted, and all samples in each extracted cluster are labeled.

2. The sample screening method according to claim 1, characterized in that: The step of performing cluster analysis on the unlabeled sample set and the labeled sample set to obtain a first cluster group and a second cluster group, and calculating the distance between the unlabeled sample set and the labeled sample set based on the first cluster group and the second cluster group comprises: Using the KM algorithm, cluster analysis is performed on the unlabeled sample set and the labeled sample set to obtain a first cluster group and a second cluster group; Selecting a preset number of clusters from each of the first cluster group and the second cluster group, and extracting the centroid of each cluster; The sum of the Euclidean distances between each centroid from the first cluster group and all centroids from the second cluster group is calculated to obtain the distance between the unlabeled sample set and the labeled sample set.

3. The sample screening method according to claim 2, characterized in that: The step of inputting the unlabeled sample set into the pre-trained language recognition model, extracting corresponding representation parameters through an activation function, and calculating the activity value of each representation parameter includes: Inputting each cluster selected from the first cluster group into a pre-trained language recognition model, and extracting corresponding representation parameters through an activation function; Calculate the average value of each characterization parameter and determine the corresponding activity value based on the average value; the larger the average value, the higher the corresponding activity value.

4. The sample screening method according to claim 1, characterized in that: The step of selecting some samples from the unlabeled sample set for labeling according to the activity value includes: From each cluster selected from the first cluster group, a cluster whose activity value is less than a preset activity value and whose sorting order is before the preset order is selected, and all samples in each selected cluster are labeled.

5. The sample screening method according to claim 1, wherein: Also includes: Inputting the unlabeled sample set into a pre-trained language recognition model to obtain corresponding classification probabilities; Calculate the entropy value of each classification probability, and eliminate unlabeled samples with entropy values ​​greater than a preset entropy value.

6. A method for training a language recognition model, characterized in that: include: Step 1: Using the sample screening method according to any one of claims 1 to 5, screen the samples in the second training set to obtain a new second training set; Step 2: Use the new second training set to train the initial language recognition model under the constraint of the cross entropy loss function; During the training process, it is determined whether the initial language recognition model has converged. If convergence occurs, stop training and obtain the trained language recognition model; Otherwise, return to step 1 and re-screen the samples in the second training set.

7. A sample screening system, characterized in that: include: A sample construction module, configured to construct a first training set, wherein the first training set includes an unlabeled sample set and a labeled sample set; a distance calculation module, configured to perform cluster analysis on the unlabeled sample set and the labeled sample set respectively to obtain a first cluster group and a second cluster group, calculate the distance between the unlabeled sample set and the labeled sample set based on the first cluster group and the second cluster group, and sort the unlabeled sample set and the labeled sample set in ascending order to obtain a sorting result; an activity value calculation module, configured to input the unlabeled sample set into a pre-trained language recognition model, extract corresponding representation parameters through an activation function, and calculate the activity value of each representation parameter; A processing module is configured to select some samples from the unlabeled sample set for labeling based on the sorting result and the activity value, and add the labeled samples to the labeled sample set to form a new labeled sample set; wherein the step of selecting some samples from the unlabeled sample set for labeling based on the sorting result includes: According to the sorting results, select multiple position intervals; Assigning a sampling rate to each position interval, wherein the sampling rate of the position interval with a higher ranking is greater than the sampling rate of the position interval with a lower ranking; According to the sampling rate, the clusters corresponding to each position interval from the first cluster group are extracted, and all samples in each extracted cluster are marked. The sample construction module is further configured to combine the remaining samples in the unlabeled sample set and the new labeled sample set to form a screened first training set.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the sample screening method described in any one of claims 1 to 5 is implemented, or when the processor executes the computer program, the language recognition model training method described in claim 6 is implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the sample screening method described in any one of claims 1 to 5, or when the computer program is executed by a processor, it implements the language recognition model training method described in claim 6.

Images