Data classification and speech recognition method, device, electronic device and storage medium

Abstract

The present invention provides a data classification and speech recognition method, device, electronic device and storage medium, wherein the method includes determining data to be classified, the data to be classified includes original speech, recognized text and modified text; determining feature values ​​of the data to be classified under preset features, the preset features are used to characterize the difference information between the recognized text and the modified text and the user information of the user; based on the feature values, determining the validity classification result of the data to be classified. The method, device, electronic device and storage medium provided by the present invention can obtain each feature value in the candidate features of the sample data, and select preset features from the candidate features according to the number of valid labeled samples and the number of invalid labeled samples corresponding to each feature value in the candidate features, thereby removing invalid features from the candidate features, and improving the execution efficiency of the judgment without affecting the accuracy of the validity judgment.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a data classification and speech recognition method, device, electronic device and storage medium. Background Art

[0002] At present, in terms of data usage of deep neural networks, supervised learning is still better than unsupervised learning in terms of results. However, in speech recognition, speech data is easy to obtain, while manual annotation of large amounts of speech data is difficult to obtain. Therefore, weakly supervised annotation is often used for learning in speech recognition.

[0003] However, the existing weakly supervised speech data selection scheme compares the phoneme edit distance between the recognized text and the modified text at the phoneme level, and uses the phoneme edit distance as the basis for judging the quality of weakly supervised annotation. The selection scheme is too single and cannot cover all situations, resulting in a high possibility of misjudgment, which in turn affects the training effect of the speech recognition model. Summary of the invention

[0004] The present invention provides a data classification and speech recognition method, device, electronic device and storage medium, which are used to solve the defect that the single judgment condition in the weak supervision speech data selection scheme in the prior art leads to judgment errors.

[0005] The present invention provides a data classification method, comprising:

[0006] Determine data to be classified, the data to be classified includes original speech, recognized text and modified text, the recognized text is obtained by performing speech recognition on the original speech, and the modified text is obtained by modifying the recognized text in response to a modification input of a user;

[0007] Determining a feature value of the data to be classified under a preset feature, wherein the preset feature is used to characterize difference information between the recognized text and the modified text and user information of the user;

[0008] Based on the feature value, a validity classification result of the data to be classified is determined.

[0009] According to a data classification method provided by the present invention, the preset features are determined based on the following steps:

[0010] Identify candidate features;

[0011] Based on the feature values ​​of the sample data under the candidate features and the validity annotations of the sample data, determine the number of valid annotated samples and the number of invalid annotated samples corresponding to each feature value of the candidate features;

[0012] Based on the number of valid labeled samples and the number of invalid labeled samples corresponding to each feature value of the candidate feature, it is determined whether to use the candidate feature as the preset feature.

[0013] According to a data classification method provided by the present invention, the determining whether to use the candidate feature as the preset feature based on the number of valid labeled samples and the number of invalid labeled samples corresponding to each feature value of the candidate feature includes:

[0014] Determine the correlation between the candidate feature and the validity annotation of the sample data based on the number of valid annotated samples and the number of invalid annotated samples corresponding to each feature value of the candidate feature;

[0015] If the correlation is greater than a preset threshold, the candidate feature is used as the preset feature.

[0016] According to a data classification method provided by the present invention, the number of valid labeled samples and the number of invalid labeled samples corresponding to each feature value of the candidate feature are respectively determined, and the correlation between the candidate feature and the validity label of the sample data is determined, including:

[0017] Based on the number of valid labeled samples and the number of invalid labeled samples corresponding to any feature value in the candidate features, determining the degree of dispersion of the number of valid labeled samples and the number of invalid labeled samples corresponding to any feature value;

[0018] Determine the correlation between any one of the eigenvalues ​​and the validity annotation of the sample data based on the total number of samples corresponding to any one of the eigenvalues, the total number of samples of the sample data, and the degree of dispersion of the number of valid annotated samples and the number of invalid annotated samples corresponding to any one of the eigenvalues;

[0019] Based on the correlation between each feature value of the candidate feature and the validity mark of the sample data, the correlation between the candidate feature and the validity mark of the sample data is determined.

[0020] According to a data classification method provided by the present invention, the candidate features include: the candidate features include: the text edit distance and phoneme edit distance between the sample recognition text and the sample modification text in the sample data, the syntax of the sample recognition text and the sample modification text, the part of speech of the modification point in the sample recognition text and the sample modification text, the position of the modification point in the sample recognition text and the sample modification text, the number of words of the modification point in the sample recognition text and the sample modification text, the user usage frequency and at least one of the user age group;

[0021] The modification point is a text portion having different contents between the sample recognition text and the sample modification text.

[0022] According to a data classification method provided by the present invention, the user age group is determined based on the sample voice in the sample data.

[0023] According to a data classification method provided by the present invention, determining the validity classification result of the data to be classified based on the feature value includes:

[0024] Inputting the feature value into a classification model to obtain a validity classification result output by the classification model;

[0025] The classification model is obtained through training based on the feature values ​​of the training data under the preset features and the validity annotation of the training data.

[0026] The present invention also provides a speech recognition method, comprising:

[0027] Determine the speech to be recognized;

[0028] Inputting the speech to be recognized into a speech recognition model to obtain a recognition result output by the speech recognition model;

[0029] The speech recognition model is trained based on the original speech and modified text in the valid data; the valid data is sample data whose classification result is that the modified text matches the original speech, and the classification result is obtained by classifying the sample data based on any of the data classification methods described above.

[0030] The present invention also provides a data classification device, comprising:

[0031] a determination module, used to determine data to be classified, wherein the data to be classified includes original speech, recognized text and modified text, wherein the recognized text is obtained by performing speech recognition on the original speech, and the modified text is obtained by modifying the recognized text in response to a modification input of a user;

[0032] A feature value module, used to determine a feature value of the data to be classified under a preset feature, wherein the preset feature is used to characterize the difference information between the recognized text and the modified text and the user information of the user;

[0033] A matching module is used to determine whether the modified text matches the original speech based on the feature value.

[0034] The present invention also provides a speech recognition device, comprising:

[0035] A determination module, used to determine the speech to be recognized;

[0036] A recognition module, used for inputting the speech to be recognized into a speech recognition model to obtain a recognition result output by the speech recognition model;

[0037] The speech recognition model is trained based on the original speech and modified text in the valid data; the valid data is sample data whose classification result is that the modified text matches the original speech, and the classification result is obtained by classifying the sample data based on any of the data classification methods described above.

[0038] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the processor implements any one of the data classification methods described above or implements the speech recognition method described above.

[0039] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the computer program implements any of the data classification methods described above or the speech recognition method described above.

[0040] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any one of the data classification methods described above or implements the speech recognition method described above.

[0041] The data classification and speech recognition method, device, electronic device and storage medium provided by the present invention determine whether the modified text and the original speech match by using the feature values ​​of preset features used to characterize multiple information such as the difference information between the recognized text and the modified text and the user information of the user, thereby realizing the use of the feature values ​​under multiple preset features as the judgment conditions for the validity classification of the data to be classified, judging and screening the effective weakly supervised speech data from the full amount of weakly supervised speech data, and improving the accuracy of judging the weakly supervised speech data as effective data. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0043] Figure 1 It is a flow chart of the data classification method provided by the present invention;

[0044] Figure 2 This is one of the flow charts of the preset feature acquisition method provided by the present invention;

[0045] Figure 3 This is the second flow chart of the preset feature acquisition method provided by the present invention;

[0046] Figure 4 It is a flow chart of the method for obtaining the relevance of validity annotation provided by the present invention;

[0047] Figure 5 It is a flowchart of the training set acquisition and classification model training method provided by the present invention;

[0048] Figure 6 It is a flow chart of the speech recognition method provided by the present invention;

[0049] Figure 7 It is a flowchart of the weakly supervised data selection method provided by the present invention;

[0050] Figure 8 It is a structural schematic diagram of the data classification device provided by the present invention;

[0051] Fig. 9 It is a structural schematic diagram of the speech recognition device provided by the present invention;

[0052] Fig.10 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] At present, when deep neural networks are applied to speech recognition, the acquisition of manual annotations of speech data requires a lot of manpower, high cost and long cycle, so it is difficult to obtain a large amount of supervised data required for speech recognition network training. In contrast, it is relatively easy to obtain weakly supervised annotations corresponding to speech data. The recognition text of the speech can be modified to obtain the modified text, forming the weakly supervised annotation text of the speech.

[0055] At the same time, the accuracy of weakly supervised annotated text is crucial for the speech recognition network. Weakly supervised annotated text is highly correlated with speech. Correct weakly supervised annotated text can correct the errors in the speech recognition network, which is very beneficial to improving the speech recognition network; incorrect weakly supervised data will have the opposite effect on the speech recognition network. To effectively use weakly supervised sample data, it is necessary to obtain effective high-quality weakly supervised sample data from a large amount of weakly supervised sample data.

[0056] The existing high-quality weakly supervised sample data mainly uses the phoneme editing distance between the recognized text and the modified text of the sample speech as the basis for judging whether the weakly supervised sample data is valid. However, judging based on this single condition alone cannot cover other situations, such as the syntax of the recognized text and the modified text, the part of speech at the modification point, etc. A single basis for judgment will lead to a very high judgment error rate.

[0057] Therefore, how to improve the accuracy of judging whether weakly supervised sample data is effective is a technical problem that needs to be solved urgently in this field.

[0058] In view of the above technical problems, an embodiment of the present invention provides a data classification method. Figure 1 Schematic diagram of the data classification method provided by the present invention. Figure 1 As shown, the method includes:

[0059] Step 110, determining the data to be classified, the data to be classified includes original speech, recognized text and modified text, the recognized text is obtained by performing speech recognition on the original speech, and the modified text is obtained by modifying the recognized text in response to the modification input of the user.

[0060] Specifically, the data to be classified is weakly supervised sample data, which includes: original speech, recognized text obtained by recognizing the original speech, and modified text obtained in response to the user modifying the recognized text based on the original speech. It should be noted that the modified text may be the same as the recognized text or may be different, and the embodiment of the present invention does not limit this.

[0061] Step 120, determining a feature value of the data to be classified under a preset feature, where the preset feature is used to characterize the difference information between the recognized text and the modified text and the user information of the user;

[0062] Step 130: Determine the validity classification result of the data to be classified based on the feature value.

[0063] In order to avoid the problem of errors caused by a single classification condition when classifying the validity of the data to be classified, in an embodiment of the present invention, the data to be classified is classified for validity according to the feature values ​​under multiple preset features to obtain the validity classification results of the data to be classified. Among them, the validity classification results indicate whether the data to be classified is valid or invalid, wherein if the original voice and the modified text in the data to be classified match, the data to be classified is valid, and vice versa, the data to be classified is invalid. Here, the preset features are used to characterize the difference information between the recognized text and the modified text in the data to be classified and the user information of the user, for example: the difference information can be the text editing distance or phoneme editing distance between the recognized text and the modified text, and can also be the grammatical layer information such as the part of speech of the recognized text and the modified text at the modification point, which can reflect the difference between the recognized text and the modified text; the user information can be the user's frequency of use or the user's age group, which can reflect the user's proficiency in using the original voice to modify the recognized text to obtain the modified text. By using the feature values ​​under multiple preset features as the judgment conditions for the validity classification of the data to be classified, the accuracy of the classification results can be improved.

[0064] It should be noted that in order to realize the use of feature values ​​under multiple preset features as judgment conditions for validity classification of data to be classified, it can be carried out by mapping the feature values ​​of preset features obtained by training the feature values ​​of sample data under preset features and the validity annotation of sample data with the validity classification results of sample data. It can also be carried out by training a model based on decision tree modeling to obtain a classification model through the feature values ​​of sample data under preset features and the validity annotation of sample data. The embodiments of the present invention are not limited to this.

[0065] In addition, the preset features can be all pre-set features, or can be based on the feature value of each feature and the number of valid samples and the number of invalid samples corresponding to the feature value. When training the above-mentioned mapping relationship or when training the classification model, valid features are selected from the full set of features as preset features. The embodiments of the present invention do not limit this.

[0066] The data classification method provided by the present invention determines whether the modified text and the original speech match by using the feature values ​​of preset features used to characterize multiple information such as the difference information between the identified text and the modified text and the user information of the user, thereby realizing the use of the feature values ​​under multiple preset features as the judgment conditions for the validity classification of the data to be classified, judging and screening the valid weakly supervised speech data from the full amount of weakly supervised speech data, and improving the accuracy of judging the weakly supervised speech data as valid data.

[0067] Based on the above embodiments, Figure 2 This is one of the flow charts of the preset feature acquisition method provided by the present invention. Figure 2As shown, the preset features in step 120 are determined based on the following steps:

[0068] Step 210, determining candidate features;

[0069] Specifically, the candidate features are pre-set full features, and the candidate features are used to represent the difference information between the recognition text and the modified text in the data to be classified and the user information of the user. The preset features are obtained based on the candidate features and are a subset of the candidate features.

[0070] Step 220, based on the feature values ​​of the sample data under the candidate features and the validity annotations of the sample data, determine the number of valid annotated samples and the number of invalid annotated samples corresponding to each feature value of the candidate features;

[0071] Step 230 , based on the number of valid labeled samples and the number of invalid labeled samples corresponding to each feature value of the candidate feature, it is determined whether to use the candidate feature as a preset feature.

[0072] Considering that the eigenvalues ​​of some features in the candidate features have no obvious effect on the validity classification of weakly supervised data, for example, the number of valid samples and the number of invalid samples corresponding to the eigenvalues ​​of a certain feature in the candidate features are close, when the eigenvalues ​​of the feature of the sample data and the validity annotation of the sample data are used for training, the relationship between the feature and the validity annotation of the sample data cannot be effectively obtained, that is, the above mapping relationship or classification model obtained after training cannot judge whether the data to be classified is valid based on the eigenvalues ​​of the feature. Therefore, if some features in the candidate features that have no obvious effect on the validity classification of weakly supervised data are removed, the execution efficiency during classification can be improved without affecting the accuracy of validity classification.

[0073] In order to select the preset feature from the candidate features, in the embodiment of the present invention, when the above-mentioned mapping relationship or classification model is trained, step 220 calculates the feature value of the sample data based on the candidate feature to obtain the feature value of the sample data under the candidate feature, and based on the respective feature values ​​of the candidate feature and the validity labeling of the sample data, the sample data is statistically analyzed to obtain the number of valid labeled samples and the number of invalid labeled samples corresponding to each feature value of the candidate feature. Step 230 determines whether the candidate feature can be used as the preset feature according to the number of valid labeled samples and the number of invalid labeled samples corresponding to each feature value of the candidate feature obtained in step 220.

[0074] It should be noted that, according to the number of valid labeled samples and the number of invalid labeled samples corresponding to each feature value of the candidate feature, whether the candidate feature can be used as a preset feature can be determined according to the ratio of the number of valid labeled samples corresponding to each feature value of the candidate feature to the total number of samples corresponding to the feature value, and the ratio of the number of invalid labeled samples to the total number of samples corresponding to the feature value, to determine whether the feature value is a valid feature value. If the total proportion of valid feature values ​​of the candidate feature is higher than the total proportion of invalid feature values, then it is considered that the candidate feature can be used as a preset feature, otherwise it cannot be used as a preset feature. The correlation between the candidate feature and the validity labeling of the sample data can also be determined according to the number of valid labeled samples and the number of invalid labeled samples corresponding to each feature value of the candidate feature. If the correlation is greater than a preset threshold, then it is considered that the candidate feature can be used as a preset feature, and the embodiments of the present invention do not limit this.

[0075] The data classification method provided by the present invention obtains each feature value in the candidate features of the sample data, and selects preset features from the candidate features according to the number of valid labeled samples and the number of invalid labeled samples corresponding to each feature value in the candidate features, thereby removing invalid features from the candidate features, and can improve the execution efficiency of the judgment without affecting the accuracy of the validity judgment.

[0076] Based on the above embodiments, Figure 3 This is the second flow chart of the preset feature acquisition method provided by the present invention. Figure 3 As shown, step 230 includes:

[0077] Step 231, based on the number of valid labeled samples and the number of invalid labeled samples corresponding to each feature value of the candidate feature, determine the correlation between the candidate feature and the validity label of the sample data;

[0078] Step 232: If the correlation is greater than a preset threshold, the candidate feature is used as a preset feature.

[0079] Considering that the closer the number of valid labeled samples and the number of invalid labeled samples corresponding to a certain feature value are, the lower the correlation between the feature value and the validity labeling of the sample data is. For example, if the number of valid labeled samples and the number of invalid labeled samples corresponding to a certain feature value are the same, then since the number of valid labeled samples and the number of invalid labeled samples are the same, it is impossible to determine whether the validity labeling is more biased towards valid or invalid based on the feature value. It can be seen that the feature value can be considered to be irrelevant to the validity labeling of the sample data. Therefore, in an embodiment of the present invention, the correlation between the candidate feature and the validity labeling of the sample data is determined based on the number of valid labeled samples and the number of invalid labeled samples corresponding to each feature value of the candidate feature, and the correlation is compared with a preset threshold. If the correlation is greater than the preset threshold, the candidate feature is used as a preset feature.

[0080] It should be noted that the correlation between the candidate feature and the validity annotation of the sample data can be determined according to the number of valid labeled samples and the number of invalid labeled samples corresponding to each eigenvalue of the candidate feature. The correlation between the candidate feature and the validity annotation of the sample data can be determined by the degree of discreteness between the number of valid labeled samples and the number of invalid labeled samples corresponding to each eigenvalue in the candidate feature. The correlation between the candidate feature and the validity annotation of the sample data can also be determined jointly by the degree of discreteness between the number of valid labeled samples and the number of invalid labeled samples corresponding to each eigenvalue in the candidate feature and the proportion of the total number of samples corresponding to the eigenvalue in the total number of samples of the sample data. The embodiments of the present invention are not limited to this.

[0081] Based on the above embodiments, Figure 4 FIG. 1 is a flow chart of the method for obtaining the validity annotation relevance provided by the present invention. Figure 4 As shown, step 231 includes:

[0082] Step 231-1, based on the number of valid labeled samples and the number of invalid labeled samples corresponding to any feature value in the candidate features, determine the degree of dispersion of the number of valid labeled samples and the number of invalid labeled samples corresponding to the feature value;

[0083] Step 231-2, determining the correlation between the feature value and the validity label of the sample data based on the total number of samples corresponding to the feature value, the total number of samples of the sample data, and the degree of dispersion of the number of valid labeled samples and the number of invalid labeled samples corresponding to the feature value;

[0084] Step 231 - 3 , based on the correlation between each feature value of the candidate feature and the validity annotation of the sample data, determine the correlation between the candidate feature and the validity annotation of the sample data.

[0085] Considering that when calculating the correlation between a certain eigenvalue and the validity annotation of sample data, on the premise of using the discrete degree of the number of valid annotated samples and the number of invalid annotated samples corresponding to the eigenvalue, adding the proportion of the total number of samples corresponding to the eigenvalue in the total number of samples of the sample data can reflect the weight of the eigenvalue in its corresponding candidate feature, and can better reflect the correlation between the candidate feature and the validity annotation of the sample data.

[0086] Therefore, in the embodiments of the present invention, by calculating the degree of dispersion between the number of valid annotation samples and the number of invalid annotation samples of each eigenvalue in the candidate features, the correlation between the eigenvalue and the valid annotation of the sample data is reflected, and by calculating the proportion of the total number of samples of the eigenvalue in the total number of samples of the sample data, the weight of the eigenvalue in the candidate features is reflected. After obtaining the correlation between each eigenvalue in the candidate features and the valid annotation of the sample data and the weight of each eigenvalue in the candidate features, according to the weight of each eigenvalue in the candidate features, the correlations corresponding to each eigenvalue are weighted and summed to obtain the correlation between the candidate feature and the valid annotation of the sample data.

[0087] The data classification method provided by the present invention determines the correlation between the candidate feature and the valid annotation of the sample data by calculating the degree of dispersion between the number of valid annotation samples and the number of invalid annotation samples corresponding to each eigenvalue in the candidate features, and the proportion of the total number of samples of each eigenvalue in the total number of samples of the sample data, realizes the association between each eigenvalue in the candidate feature and the valid annotation of the sample data, and can reflect the correlation between the candidate feature and the valid annotation of the sample data. Furthermore, the candidate features with weak correlation can be removed based on the correlation. When using the features with strong correlation to perform validity classification on the data to be classified in the later stage, the execution efficiency during classification can be further improved while ensuring the classification accuracy.

[0088] Based on the above embodiments, the present invention provides a preferred embodiment. Specifically, the candidate features include at least one of the text edit distance and the phoneme edit distance between the sample recognition text and the sample modification text in the sample data, the syntax of the sample recognition text and the sample modification text, the part of speech of the modification point in the sample recognition text and the sample modification text, the position of the modification point in the sample recognition text and the sample modification text, the number of characters of the modification point in the sample recognition text and the sample modification text, the user usage frequency, and the user age group; wherein, the modification point is the text part with different content between the sample recognition text and the sample modification text.

[0089] It should be noted that the text edit distance and the phoneme edit distance between the sample recognition text and the sample modification text in the sample data can be calculated based on the edit distance algorithm. For example, in the Levenshtein algorithm, for "老张" (lao3zhang1) and "老章" (lao3 zhang1), the text edit distance in the sample data is 1, and the phoneme edit distance in the sample data is 0.

[0090] The syntax of the sample recognition text and the sample modification text can be obtained through natural language processing tools. The syntax specifically includes: SBV (subject-predicate relationship), VOB (verb-object relationship), IOB (indirect object relationship), FOB (preposed object), DBL (causative), ATT (attributive-middle relationship), ADV (adverbial-middle structure), CMP (verb-complement structure), COO (coordinate relationship), POB (prepositional-object relationship), LAD (left adjunct relationship), RAD (right adjunct relationship), IS (independent structure), HED (core relationship).

[0091] The part-of-speech of the modification point in the sample recognition text and the sample modification text can be obtained through natural language processing tools. For example, if the sample recognition text is "I Zhifan" and the sample modification text is "I eat", then the recognition text at the modification point is "Zhifan", and its part-of-speech is a proper noun. The part-of-speech specifically includes: a (adjective), b (other modifiers of nouns), c (conjunction), d (adverb), e (interjection), g (morpheme word), h (prefix), i (idiom), j (abbreviation), k (suffix), m (number), n (common noun), nd (direction noun), nh (person name), ni (company name), nl (location noun), ns (geographical noun), nt (time noun), nz (other noun), o (onomatopoeia), p (preposition), q (measure word), r (pronoun), u (auxiliary word), v (verb), wp (punctuation), ws (foreign word), x (non-word).

[0092] The position of the modification point in the sample recognition text and the sample modification text is specifically divided into three types: the beginning of the sentence, the middle of the sentence, and the end of the sentence.

[0093] The number of characters of the modification point in the sample recognition text and the sample modification text can be determined by the length of the modification point string. For example, if the sample recognition text is "I Zhifan" and the sample modification text is "I eat", the number of characters of the recognition text and the modification text at the modification point is both 2.

[0094] The user usage frequency is the single-day word frequency of the user using the speech recognition service; high-frequency users may be more familiar with the recognition system and are more likely to make meaningful modifications; the usage frequency information of the original speech corresponding to the user in the candidate features can be obtained through the service background.

[0095] The user age group represents the age group of the user corresponding to the original speech in the candidate features, which can be obtained through audio analysis and is divided into three types: children, adults, and the elderly.

[0096] Based on the above embodiment, the present invention also provides a preset feature selector for executing the preset feature acquisition method provided in the above embodiment, and the preset feature selector is used to select features that are more useful for specific classification tasks from the above candidate features as preset features, which are used for subsequent training of classification models and for effectiveness classification of data to be classified after the classification model training is completed. The selection basis of the preset feature selector is that for each feature in the candidate features, the correlation between the feature and the sample feature effectiveness label is calculated, which can be obtained by training with the following formula:

[0097]

[0098] Where V i is the eigenvalue of the i-th feature among the candidate features, n(V i ) is the sample data, the eigenvalue is V i The number of samples, N is the total number of samples of sample data, var(V i ) are all eigenvalues ​​V i The standard deviation of the number of valid labeled samples and the number of invalid labeled samples (for example, there are 10 samples with a text editing distance of 2 between the recognized text and the modified text, and the classification results are that the number of "data valid" (valid labeling) and "data invalid" (invalid labeling) are 5 and 5 respectively, then its value is the standard deviation of the sample [5, 5], which is 0).

[0099] The preset feature selector selects preset features whose relevance exceeds a certain threshold value T (eg, T=0.1) from the candidate features for subsequent classification model training or for the classification model to perform validity classification on the data to be classified.

[0100] Based on the above embodiments, the present invention provides a preferred embodiment, in which the user age group is determined based on the sample speech in the sample data. It should be noted that the user age group can be determined based on the speech by extracting acoustic features, such as MFCC (Mel Frequency Cepstrum), from the sample speech, and then classifying the age group through a classification algorithm, such as a GMM model (Gaussian Fuzzy Model).

[0101] Based on the above embodiments, the present invention provides a preferred embodiment, step 130 includes:

[0102] Input the feature value into the classification model to obtain the validity classification result output by the classification model;

[0103] The classification model is obtained based on the feature values ​​of the training data under preset features and the validity annotation of the training data.

[0104] It should be noted that the feature value under the preset feature can be a fusion feature value based on each feature value under the preset feature. The fusion feature value can be obtained by specifying the feature order in the preset feature and spacing each feature value with a special symbol. For example, the preset features include feature A, feature B, and feature C. The specified feature order is A, B, C. The feature value of feature A of a certain training data is V A , the eigenvalue of feature B is V B , and the eigenvalue of the C feature is V c , and connected with special symbols |, the fusion feature is V A |V B |V C The classification model is trained based on the fusion features under the preset features corresponding to each sample in the sample data and the validity annotation of the sample. The training data and the sample data for obtaining the preset features can be the same data set or different data sets, which is not limited in the embodiment of the present invention.

[0105] Based on the above embodiments, the present invention provides a preferred embodiment. Figure 5 is a flow chart of the training set acquisition and classification model training method provided by the present invention, such as Figure 5 As shown, the training set acquisition and classification model training method specifically include:

[0106] The decision tree is used to build the initial classification model. The feature values ​​and validity labels under the preset features of the training data are combined into a training set, and the initial classification model is trained according to the training set until the training is completed and the classification model is obtained. Among them, the decision tree uses the gini criterion as the branch criterion, class_weight uses balance, and is trained using the python sklearn.tree tool; the preset features are selected from the candidate features according to the above preset feature selector; the feature values ​​are calculated based on the candidate features of the training data.

[0107] Figure 6 Schematic diagram of the flow of the speech recognition method provided by the present invention. Figure 6 As shown, an embodiment of the present invention provides a speech recognition method, comprising:

[0108] Step 610, determining the speech to be recognized;

[0109] Specifically, the speech to be recognized may be speech received by a microphone in real time, or may be speech in a pre-recorded audio file, which is not limited in the embodiment of the present invention.

[0110] Step 620, inputting the speech to be recognized into the speech recognition model to obtain the recognition result output by the speech recognition model;

[0111] The speech recognition model is trained based on the original speech and modified text in the valid data; the valid data is the sample data whose classification result is that the modified text matches the original speech, and the classification result is obtained by classifying the sample data based on any data classification method provided in the above embodiments.

[0112] Specifically, the speech to be recognized is input into a speech recognition model trained by using the original speech in the valid data and the modified text corresponding to the original speech as labels, and a recognition result corresponding to the speech to be recognized is obtained. The valid data is sample data whose classification result is that the modified text matches the original speech in the full amount of sample data, and the classification result is obtained by classifying the sample data according to any data classification method provided in the above embodiments.

[0113] The embodiment of the present invention provides a speech recognition method, which trains a speech recognition model by using valid data obtained by filtering sample data, thereby reducing the negative impact of incorrectly labeled data in weakly supervised data on the speech recognition model, thereby improving the recognition accuracy of the speech recognition model.

[0114] The present invention also provides a weakly supervised data selection method. Figure 7 Schematic diagram of the weak supervision data selection method provided by the present invention. Figure 7 As shown, the method includes:

[0115] Determine the full amount of weak supervision data, which includes: original speech, recognized text and modified text corresponding to the original speech;

[0116] The eigenvalues ​​of the full amount of weakly supervised data are calculated according to the preset features to obtain the eigenvalues ​​of the preset features corresponding to each piece of data in the full amount of weakly supervised data.

[0117] The feature values ​​of the preset features corresponding to each piece of data in the full amount of weakly supervised data are added to the classification model, and the classification model outputs the validity classification result of each piece of data in the full amount of weakly supervised data; the classification model is trained based on the training set acquisition and classification model training method provided in the above embodiment.

[0118] The classification output by the classification model, that is, the validity classification result, if the validity classification result is weakly supervised data of the valid category, is used as valid data, and the valid data is used as the training input for subsequent training of other models, such as: the speech recognition model provided in the above embodiment.

[0119] The data classification device and speech recognition device provided by the present invention are described below. The data classification device described below and the data classification method described above can be referenced to each other, and the speech recognition device described below and the speech recognition method described above can be referenced to each other.

[0120] Figure 8 Schematic diagram of the structure of the data classification device provided by the present invention. Figure 8 As shown, the device includes: a determination module 810, a feature value module 820 and a matching module 830.

[0121] in,

[0122] The determination module 810 is used to determine the data to be classified, where the data to be classified includes original speech, recognized text and modified text, where the recognized text is obtained by performing speech recognition on the original speech, and the modified text is obtained by modifying the recognized text in response to a modification input by the user;

[0123] A feature value module 820 is used to determine a feature value of the data to be classified under a preset feature, where the preset feature is used to characterize the difference information between the recognized text and the modified text and the user information of the user;

[0124] The matching module 830 is used to determine whether the modified text matches the original speech based on the feature value.

[0125] In an embodiment of the present invention, a determination module is used to determine the data to be classified, and the data to be classified includes original speech, recognized text and modified text. The recognized text is obtained by performing speech recognition on the original speech, and the modified text is obtained by modifying the recognized text in response to the modification input of the user; a feature value module is used to determine the feature value of the data to be classified under a preset feature, and the preset feature is used to characterize the difference information between the recognized text and the modified text, and the user information of the user; a matching module is used to determine whether the modified text matches the original speech based on the feature value, so that the feature values ​​under multiple preset features are used as the judgment conditions for the validity classification of the data to be classified, and the effective weakly supervised speech data is judged and screened from the full amount of weakly supervised speech data, thereby improving the accuracy of judging the weakly supervised speech data as valid data.

[0126] Based on any of the above embodiments, the feature value module 820 includes:

[0127] A candidate feature determination submodule is used to determine candidate features;

[0128] The sample number statistics submodule is used to determine the number of valid labeled samples and the number of invalid labeled samples corresponding to each characteristic value of the candidate feature based on the characteristic value of the sample data under the candidate feature and the validity labeling of the sample data;

[0129] The preset feature selection submodule is used to determine whether to use the candidate feature as the preset feature based on the number of valid labeled samples and the number of invalid labeled samples corresponding to each feature value of the candidate feature.

[0130] Based on any of the above embodiments, the preset feature selection submodule includes:

[0131] A feature relevance submodule is used to determine the relevance between the candidate feature and the validity annotation of the sample data based on the number of valid annotated samples and the number of invalid annotated samples corresponding to each feature value of the candidate feature;

[0132] The preset feature judgment submodule is used to use the candidate feature as the preset feature if the correlation is greater than a preset threshold.

[0133] Based on any of the above embodiments, the feature relevance submodule includes:

[0134] The discrete degree calculation submodule is used to determine the discrete degree of the number of valid labeled samples and the number of invalid labeled samples corresponding to any feature value in the candidate features based on the number of valid labeled samples and the number of invalid labeled samples corresponding to the feature value;

[0135] The eigenvalue correlation submodule is used to determine the correlation between the eigenvalue and the validity annotation of the sample data based on the total number of samples corresponding to the eigenvalue, the total number of samples of the sample data, and the degree of dispersion of the number of valid annotated samples and the number of invalid annotated samples corresponding to the eigenvalue;

[0136] The correlation calculation submodule is used to determine the correlation between the candidate feature and the validity annotation of the sample data based on the correlation between each feature value of the candidate feature and the validity annotation of the sample data.

[0137] Based on any of the above embodiments, the candidate features in the candidate feature determination submodule include: the text editing distance and phoneme editing distance between the sample recognition text and the sample modification text in the sample data, the syntax of the sample recognition text and the sample modification text, the part of speech of the modification point in the sample recognition text and the sample modification text, the position of the modification point in the sample recognition text and the sample modification text, the number of words in the modification point in the sample recognition text and the sample modification text, the user usage frequency and at least one of the user age groups; wherein the modification point is the text portion with different content between the sample recognition text and the sample modification text.

[0138] Based on any of the above embodiments, the candidate feature determination submodule includes:

[0139] The user age group acquisition submodule is used to determine the user age group based on the sample voice in the sample data.

[0140] Based on any of the above embodiments, the matching module 830 includes:

[0141] The model classification submodule is used to input the feature value into the classification model to obtain the validity classification result output by the classification model;

[0142] The model training submodule is used to train the classification model based on the feature values ​​of the training data under preset features and the validity annotation of the training data.

[0143] Fig. 9 Schematic diagram of the structure of the speech recognition device provided by the present invention. Fig. 9 As shown, the device includes: a determination module 910 and an identification module 920.

[0144] in,

[0145] A determination module 910 is used to determine a speech to be recognized;

[0146] The recognition module 920 is used to input the speech to be recognized into the speech recognition model and obtain the recognition result output by the speech recognition model;

[0147] The speech recognition model is trained based on the original speech and modified text in the valid data; the valid data is sample data whose classification result is that the modified text matches the original speech, and the classification result is obtained by classifying the sample data based on the data classification method provided in any of the above embodiments.

[0148] In an embodiment of the present invention, a determination module is used to determine a speech to be recognized; a recognition module is used to input the speech to be recognized into a speech recognition model to obtain a recognition result output by the speech recognition model; the speech recognition model is trained based on the original speech and modified text in the valid data; the valid data is sample data whose classification result is a match between the modified text and the original speech, and the classification result is obtained by classifying the sample data based on the data classification method provided in any of the above embodiments, thereby reducing the negative impact of incorrectly labeled data in weakly supervised data on the speech recognition model, thereby improving the recognition accuracy of the speech recognition model.

[0149] Fig.10 An example of a physical structure diagram of an electronic device is shown in FIG. Fig.10As shown, the electronic device may include: a processor 1010, a communication interface 1020, a memory 1030 and a communication bus 1040, wherein the processor 1010, the communication interface 1020 and the memory 1030 communicate with each other through the communication bus 1040. The processor 1010 may call the logic instructions in the memory 1030 to execute a data classification method or a speech recognition method, wherein the data classification method includes: determining the data to be classified, the data to be classified includes the original speech, the recognition text and the modified text, the recognition text is obtained by performing speech recognition on the original speech, and the modified text is obtained by modifying the recognition text in response to the modification input of the user; determining the feature value of the data to be classified under the preset feature, the preset feature is used to characterize the difference information between the recognition text and the modified text and the user information of the user; based on the feature value, determining the validity classification result of the data to be classified.

[0150] The speech recognition method includes: determining the speech to be recognized; inputting the speech to be recognized into a speech recognition model to obtain a recognition result output by the speech recognition model; the speech recognition model is trained based on the original speech and modified text in the valid data; the valid data is sample data whose classification result is that the modified text matches the original speech, and the classification result is obtained by classifying the sample data based on the data classification method provided in any of the above embodiments.

[0151] In addition, the logic instructions in the above-mentioned memory 1030 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk.

[0152] On the other hand, the present invention also provides a computer program product, which includes a computer program, and the computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the data classification method or speech recognition method provided by the above-mentioned methods, wherein the data classification method includes: determining the data to be classified, the data to be classified includes original speech, recognized text and modified text, the recognized text is obtained by performing speech recognition on the original speech, and the modified text is obtained by modifying the recognized text in response to the user's modification input; determining the feature value of the data to be classified under a preset feature, the preset feature is used to characterize the difference information between the recognized text and the modified text and the user information of the user; based on the feature value, determining the validity classification result of the data to be classified.

[0153] The speech recognition method includes: determining the speech to be recognized; inputting the speech to be recognized into a speech recognition model to obtain a recognition result output by the speech recognition model; the speech recognition model is trained based on the original speech and modified text in the valid data; the valid data is sample data whose classification result is that the modified text matches the original speech, and the classification result is obtained by classifying the sample data based on the data classification method provided in any of the above embodiments.

[0154] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the data classification method or speech recognition method provided by the above-mentioned methods, wherein the data classification method comprises: determining the data to be classified, the data to be classified comprises original speech, recognized text and modified text, the recognized text is obtained by performing speech recognition on the original speech, and the modified text is obtained by modifying the recognized text in response to the modification input of the user; determining the feature value of the data to be classified under a preset feature, the preset feature is used to characterize the difference information between the recognized text and the modified text and the user information of the user; based on the feature value, determining the validity classification result of the data to be classified.

[0155] The speech recognition method includes: determining the speech to be recognized; inputting the speech to be recognized into a speech recognition model to obtain a recognition result output by the speech recognition model; the speech recognition model is trained based on the original speech and modified text in the valid data; the valid data is sample data whose classification result is that the modified text matches the original speech, and the classification result is obtained by classifying the sample data based on the data classification method provided in any of the above embodiments.

[0156] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0157] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0158] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data classification method, characterized in that: include: Determine data to be classified, the data to be classified includes original speech, recognized text and modified text, the recognized text is obtained by performing speech recognition on the original speech, and the modified text is obtained by modifying the recognized text in response to a modification input of a user; Determining a feature value of the data to be classified under a preset feature, wherein the preset feature is used to characterize difference information between the recognized text and the modified text and user information of the user, wherein the user information reflects the user's proficiency in using the original voice to modify the recognized text to obtain the modified text; Based on the feature value, a validity classification result of the data to be classified is determined, the validity classification result being valid or invalid, wherein valid means that the original speech and the modified text in the data to be classified match.

2. The data classification method according to claim 1, characterized in that: The preset features are determined based on the following steps: Identify candidate features; Based on the feature values ​​of the sample data under the candidate features and the validity annotations of the sample data, determine the number of valid annotated samples and the number of invalid annotated samples corresponding to each feature value of the candidate features; Based on the number of valid labeled samples and the number of invalid labeled samples corresponding to each feature value of the candidate feature, it is determined whether to use the candidate feature as the preset feature.

3. The data classification method according to claim 2, characterized in that: The determining whether to use the candidate feature as the preset feature based on the number of valid labeled samples and the number of invalid labeled samples corresponding to each feature value of the candidate feature includes: Determine the correlation between the candidate feature and the validity annotation of the sample data based on the number of valid annotated samples and the number of invalid annotated samples corresponding to each feature value of the candidate feature; If the correlation is greater than a preset threshold, the candidate feature is used as the preset feature.

4. The data classification method according to claim 3, characterized in that: The determining, based on the number of valid labeled samples and the number of invalid labeled samples corresponding to each feature value of the candidate feature, the correlation between the candidate feature and the validity label of the sample data comprises: Based on the number of valid labeled samples and the number of invalid labeled samples corresponding to any feature value in the candidate features, determining the degree of dispersion of the number of valid labeled samples and the number of invalid labeled samples corresponding to any feature value; Determine the correlation between any one of the eigenvalues ​​and the validity annotation of the sample data based on the total number of samples corresponding to any one of the eigenvalues, the total number of samples of the sample data, and the degree of dispersion of the number of valid annotated samples and the number of invalid annotated samples corresponding to any one of the eigenvalues; Based on the correlation between each feature value of the candidate feature and the validity mark of the sample data, the correlation between the candidate feature and the validity mark of the sample data is determined.

5. The data classification method according to any one of claims 2 to 4, characterized in that: The candidate features include: at least one of the text edit distance and phoneme edit distance between the sample recognition text and the sample modified text in the sample data, the syntax of the sample recognition text and the sample modified text, the part of speech of the modified point in the sample recognition text and the sample modified text, the position of the modified point in the sample recognition text and the sample modified text, the number of words in the modified point in the sample recognition text and the sample modified text, the frequency of use by users, and the age group of users; The modification point is a text portion having different contents between the sample recognition text and the sample modification text.

6. The data classification method according to claim 5, characterized in that: The user age group is determined based on the sample voice in the sample data.

7. The data classification method according to any one of claims 1 to 4, characterized in that: The determining, based on the feature value, a validity classification result of the data to be classified includes: Inputting the feature value into a classification model to obtain a validity classification result output by the classification model; The classification model is obtained through training based on the feature values ​​of the training data under the preset features and the validity annotation of the training data.

8. A speech recognition method, characterized in that: include: Determine the speech to be recognized; Inputting the speech to be recognized into a speech recognition model to obtain a recognition result output by the speech recognition model; The speech recognition model is trained based on the original speech and modified text in the valid data; the valid data is sample data whose classification result is that the modified text matches the original speech, and the classification result is obtained by classifying the sample data based on the data classification method described in any one of claims 1 to 7.

9. A data classification device, characterized in that: include: a determination module, used to determine data to be classified, wherein the data to be classified includes original speech, recognized text and modified text, wherein the recognized text is obtained by performing speech recognition on the original speech, and the modified text is obtained by modifying the recognized text in response to a modification input of a user; A feature value module, used to determine a feature value of the data to be classified under a preset feature, wherein the preset feature is used to characterize the difference information between the recognized text and the modified text and the user information of the user, wherein the user information reflects the user's proficiency in using the original voice to modify the recognized text to obtain the modified text; A matching module is used to determine whether the modified text matches the original speech based on the feature value, and the validity classification result is valid or invalid, where valid means that the original speech and the modified text in the data to be classified match.

10. A speech recognition device, characterized in that: include: A determination module, used to determine the speech to be recognized; A recognition module, used for inputting the speech to be recognized into a speech recognition model to obtain a recognition result output by the speech recognition model; The speech recognition model is trained based on the original speech and modified text in the valid data; the valid data is sample data whose classification result is that the modified text matches the original speech, and the classification result is obtained by classifying the sample data based on the data classification method described in any one of claims 1 to 7.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the data classification method according to any one of claims 1 to 7 or the speech recognition method according to claim 8 is implemented.

12. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the data classification method according to any one of claims 1 to 7 or the speech recognition method according to claim 8 is implemented.

Images