Short voice language recognition method and related device
By performing feature processing and clustering on short speech data, and utilizing pre-trained multilingual speech recognition models and clustering algorithms, the problem of poor language recognition performance in short speech data has been solved, achieving higher recognition accuracy and more refined language differentiation.
Patent Information
- Application Number
- CN202610081376.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-01-21
- Publication Date
- 2026-03-27
AI Technical Summary
Existing deep learning-based language recognition models perform poorly in short speech scenarios, failing to effectively utilize the diversity and language information of short speech data, resulting in decreased recognition accuracy.
By performing feature processing on short speech data, feature representations are extracted using the encoder of a pre-trained multilingual speech recognition model and input into the short speech language recognition model. Combined with clustering algorithms, the same language is divided into more refined language subclasses, and a model capable of learning language subclass labels is trained.
It improves the accuracy of short speech language recognition, provides a finer recognition granularity, can better distinguish different languages, and reduces confusion in recognition.
Smart Images
Figure CN121747530A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of speech processing, and in particular to a short speech language recognition method and related device. BACKGROUND
[0002] Language recognition, also known as language identification, refers to a process of automatically determining a language category to which a speech segment belongs by a machine. A mainstream language recognition technology at present is a language recognition model based on deep learning. This method uses training corpus to train a language recognition model, and then uses the model to recognize a to-be-recognized speech into a group of scores, and determines the language classification attribute of the to-be-recognized speech according to the category corresponding to the maximum score.
[0003] However, the mainstream language recognition model has good recognition effect when dealing with long speech (such as speech effective duration not less than 30s) language recognition scene, but the recognition effect will be seriously reduced when directly using the technology to short speech (such as speech effective duration not more than 3s) language recognition scene.
[0004] Therefore, how to provide a short speech language recognition technology to improve the short speech language recognition effect has become a technical problem to be solved by those skilled in the art. SUMMARY
[0005] In view of the above problems, the present application provides a short speech language recognition method and related device to achieve the purpose of improving the short speech language recognition effect. The specific scheme is as follows:
[0006] The first aspect of the present application provides a short speech language recognition method, comprising:
[0007] obtaining short speech data to be recognized;
[0008] performing feature processing on the short speech data to obtain a feature representation of the short speech data;
[0009] inputting the feature representation of the short speech data into a short speech language recognition model to obtain a language sub-class recognition result output by the short speech language recognition model; the short speech language recognition model is trained using feature representations of short speech data samples and language sub-class labels thereof; and the language corresponding to the language sub-class recognition result is the language of the short speech data.
[0010] In a possible implementation, the feature processing on the short speech data to obtain the feature representation of the short speech data comprises:
[0011] performing acoustic feature processing on the short speech data to obtain acoustic features of the short speech data;
[0012] input the acoustic features of the short speech data into a preset encoder to obtain a feature representation output by the encoder as the feature representation of the short speech data, the encoder being an encoder in a pre-trained multilingual speech recognition model.
[0013] In a possible implementation, the feature representation of the short speech data sample is determined in the following manner:
[0014] determine a short speech data sample;
[0015] perform acoustic feature processing on the short speech data sample to obtain acoustic features of the short speech data sample;
[0016] input the acoustic features of the short speech data sample into a preset encoder to obtain a feature representation output by the encoder as the feature representation of the short speech data sample, the encoder being an encoder in a pre-trained multilingual speech recognition model.
[0017] In a possible implementation, the language sub-class label of the short speech data sample is determined in the following manner:
[0018] train a basic language recognition model by using the feature representation of the short speech data sample and the language label thereof;
[0019] input the feature representation of the short speech data sample into the basic language recognition model to obtain an embedding vector of the short speech data sample;
[0020] for each language, cluster the embedding vectors of the short speech data samples related to the language to obtain a language sub-class label of each short speech data sample.
[0021] In a possible implementation, the inputting of the feature representation of the short speech data sample into the basic language recognition model to obtain the embedding vector of the short speech data sample includes:
[0022] obtain a feature vector of a last layer before a classification layer of the basic language recognition model as the embedding vector of the short speech data sample.
[0023] In a possible implementation, the clustering of the embedding vectors of the short speech data samples related to the language to obtain the language sub-class label of each short speech data sample includes:
[0024] cluster the embedding vectors of the short speech data samples related to the language to obtain an initial clustering result, the initial clustering result including a sample quantity corresponding to each initial sub-class;
[0025] merge each initial sub-class whose sample number is less than the preset sub-class sample number threshold in the initial clustering result, to obtain a final sub-class clustering result; the final clustering result includes a sample number corresponding to each final sub-class;
[0026] perform language sub-class label naming on each final sub-class in the final clustering result, to obtain a language sub-class label of each short audio data sample.
[0027] In a possible implementation, the short audio language recognition model is trained in the following manner:
[0028] update the classification layer of the basic language model to obtain a short audio language recognition model to be trained, and the number of nodes in the last layer of the classification layer of the short audio language recognition model to be trained is consistent with the total number of language sub-class labels;
[0029] train the short audio language recognition model to be trained by using the feature representation of the short audio data sample and the language sub-class label thereof, to obtain a trained short audio language recognition model.
[0030] The second aspect of the application provides a short audio language recognition device, comprising:
[0031] an acquisition unit configured to acquire short audio data to be recognized;
[0032] a feature processing unit configured to perform feature processing on the short audio data, to obtain a feature representation of the short audio data;
[0033] a recognition unit configured to input the feature representation of the short audio data into a short audio language recognition model, to obtain a language sub-class recognition result output by the short audio language recognition model; the short audio language recognition model is trained by using a feature representation of a short audio data sample and a language sub-class label thereof; and the language corresponding to the language sub-class recognition result is the language of the short audio data.
[0034] In a possible implementation, the feature processing unit is specifically configured to:
[0035] perform acoustic feature processing on the short audio data, to obtain acoustic features of the short audio data;
[0036] input the acoustic features of the short audio data into a preset encoder, to obtain a feature representation output by the encoder as the feature representation of the short audio data, and the encoder is an encoder in a pre-trained multi-language speech recognition model.
[0037] In a possible implementation, the device further comprises:
[0038] The feature representation determination unit is specifically configured to:
[0039] determine a short speech data sample;
[0040] perform acoustic feature processing on the short speech data sample to obtain acoustic features of the short speech data sample;
[0041] input the acoustic features of the short speech data sample into a preset encoder to obtain a feature representation output by the encoder as a feature representation of the short speech data sample, the encoder being an encoder in a pre-trained multilingual speech recognition model.
[0042] In a possible implementation, the apparatus further includes:
[0043] The language sub-class label determination unit includes:
[0044] The base language recognition model training unit is configured to train a base language recognition model by using the feature representations of the short speech data samples and the language labels thereof;
[0045] The embedding vector determination unit is configured to input the feature representations of the short speech data samples into the base language recognition model to obtain embedding vectors of the short speech data samples.
[0046] The clustering unit is configured to, for each language, cluster the embedding vectors of the short speech data samples related to the language to obtain language sub-class labels of the short speech data samples.
[0047] In a possible implementation, the embedding vector determination unit is specifically configured to:
[0048] obtain feature vectors of a last layer before a classification layer of the base language recognition model as the embedding vectors of the short speech data samples.
[0049] In a possible implementation, the clustering unit is specifically configured to:
[0050] cluster the embedding vectors of the short speech data samples related to the language to obtain an initial clustering result, the initial clustering result including a sample quantity corresponding to each initial sub-class;
[0051] merge each initial sub-class in the initial clustering result whose sample quantity is less than a preset sub-class sample quantity threshold to obtain a final sub-class clustering result, the final clustering result including a sample quantity corresponding to each final sub-class;
[0052] name language sub-class labels for each final sub-class in the final clustering result to obtain the language sub-class labels of the short speech data samples.
[0053] In a possible implementation, the apparatus further includes:
[0054] The phrase speech language recognition model training unit is specifically configured to:
[0055] updating the classification layer of the base language model to obtain a to-be-trained phrase speech language recognition model, wherein the number of nodes of the last layer of the classification layer of the to-be-trained phrase speech language recognition model is consistent with the total number of the language sub-class labels;
[0056] training the to-be-trained phrase speech language recognition model by using the feature representation of the phrase speech data sample and the language sub-class label of the phrase speech data sample to obtain a trained phrase speech language recognition model.
[0057] The third aspect of the present application provides a computer program product, which includes computer readable instructions, when the computer readable instructions run on an electronic device, the electronic device implements the phrase speech language recognition method of the first aspect or any implementation manner of the first aspect.
[0058] The fourth aspect of the present application provides an electronic device, which includes at least one processor and a memory connected with the processor, wherein:
[0059] The memory is configured to store a computer program;
[0060] The processor is configured to execute the computer program, so that the electronic device can implement the phrase speech language recognition method of the first aspect or any implementation manner of the first aspect.
[0061] The fifth aspect of the present application provides a computer readable storage medium, which carries one or more computer programs, when the one or more computer programs are executed by an electronic device, the electronic device can implement the phrase speech language recognition method of the first aspect or any implementation manner of the first aspect.
[0062] By means of the above technical solution, the present application provides a phrase speech language recognition method and device, which pre-trains a phrase speech language recognition model by using the feature representation of the phrase speech data sample and the language sub-class label of the phrase speech data sample. The phrase speech language recognition model can learn the corresponding relationship between the feature representation of different phrase speech data of the same language and each language sub-class of the language. Based on this, the feature representation of the phrase speech data is input into the phrase speech language recognition model, so that the language sub-class recognition result output by the phrase speech language recognition model can be obtained. The language corresponding to the language sub-class recognition result is the language of the phrase speech data. The technical solution of the present application can provide more fine recognition granularity for the phrase speech data with limited information representing the language for each language, and thus improves the phrase speech language recognition accuracy. BRIEF DESCRIPTION OF DRAWINGS
[0063] The above and other features, aspects and advantages of the present disclosure will become more apparent with reference to the following detailed description when taken in conjunction with the accompanying drawings. Throughout the drawings, similar or same reference numerals are used to denote similar or same elements. It is to be understood that the drawings are schematic, and the proportions of the elements and the like are not necessarily to scale.
[0064] Figure 1 A flowchart of a short speech language recognition method provided by an embodiment of the present application;
[0065] Figure 2 A flowchart of a method for feature processing of short speech data to obtain a feature representation of the short speech data provided by an embodiment of the present application;
[0066] Figure 3 A flowchart of a determination manner of a feature representation of a short speech data sample provided by an embodiment of the present application;
[0067] Figure 4 A flowchart of a determination manner of a language sub-class label of a short speech data sample provided by an embodiment of the present application;
[0068] Figure 5 A flowchart of a method for clustering embedding vectors of short speech data samples related to a language to obtain language sub-class labels of the short speech data samples provided by an embodiment of the present application;
[0069] Figure 6 A flowchart of a training manner of a short speech language recognition model provided by an embodiment of the present application;
[0070] Figure 7 A structural diagram of a short speech language recognition device provided by an embodiment of the present application;
[0071] Figure 8 A structural diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION
[0072] The embodiments of the present application are described below in conjunction with the accompanying drawings. The terms used in the embodiment part of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.
[0073] The embodiments of the present application are described below in conjunction with the accompanying drawings. It is known to those skilled in the art that, as technology develops and new scenarios appear, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.
[0074] The terms "first", "second", and the like in the description and in the claims of the present application and above-described drawings are used to distinguish similar objects and are not necessarily used to describe a particular sequential or chronological order. It should be understood that terms so used are interchangeable under appropriate circumstances and that the embodiments of the present application are described by the description and drawings as a matter of convenience and the terms "first", "second", and the like are in no way intended to limit the scope of the present application or the application for which a patent is claimed. Furthermore, the terms "comprise", "comprising", "include", "including", and the like are intended to cover non-exclusive inclusion, such that a process, method, system, product, or apparatus that comprises a list of elements is not necessarily limited to those elements, but can include other elements not expressly listed or inherent to such process, method, system, product, or apparatus.
[0075] Language recognition, also known as language identification, refers to the process of automatically determining the language category to which a speech segment belongs by a machine. The current mainstream language recognition technology is a language recognition model based on deep learning. This method uses training corpus to train a language recognition model, and then uses the model to recognize the to-be-recognized speech into a group of scores, and determines the language classification attribute of the to-be-recognized speech according to the category corresponding to the maximum score.
[0076] For ease of understanding, an example of a language recognition model is provided in the present application, namely an Ecapa-Tdnn (Emphasized Channel Attention Propagation and Aggregation in TDNN, Chinese name: Enhanced Channel Attention Propagation and Aggregation Time Delay Neural Network) language recognition model. The Ecapa-Tdnn language recognition model is composed of a feature extraction module, a multi-scale feature fusion module, a global attention module, and a classification module.
[0077] In the feature extraction module, it extracts and converts the input 80-dimensional original FB feature. It integrates a one-dimensional Res2Net layer and a Squeeze-and-Excitation module to model the relationship between feature channels, and then obtains the key information of the feature representation from the FB feature.
[0078] In the multi-scale feature fusion module, it fuses multiple hierarchical features while utilizing the information of the shallow and deep layers of the network. The purpose of this is to complement the information and make full use of the low-level and high-level features of the feature representation to enhance the feature representation.
[0079] In the global attention module, it uses an attention mechanism-based pooling layer to generate a global attention-based feature representation. Through the attention mechanism, a weight (importance score) is assigned to each frame to generate a weighted average vector and a weighted standard deviation vector. This can highlight key frame information while suppressing redundant or noisy features, thereby improving the accuracy of language recognition.
[0080] Finally, the classification module projects the embedding of different languages output by the global attention module onto the corresponding language id node.
[0081] However, the current mainstream language recognition model has good recognition effect when dealing with long speech (such as speech effective duration not less than 30s) language recognition scene, but when directly used in short speech (such as speech effective duration not more than 3s) language recognition scene, the recognition effect will be seriously reduced.
[0082] The present inventors have found that one of the reasons for the poor short speech language recognition effect of the current mainstream language recognition model is that the recognition effect is heavily dependent on the number of training data sets, and the same language is only used as a category, without fully considering the diversity of the same language data in the short speech scene, such as effective duration distribution, accent difference, and noise difference. Moreover, the language recognition model itself has a crucial role in mining language information, and the current mainstream language recognition model has a relatively simple structure and relatively weak ability to mine language information from short speech. Although the recognition effect can be improved by increasing the labeled training data, it takes a lot of manpower and time to accumulate a considerable amount of labeled data, and when the training data reaches a certain amount, increasing the amount of training data has little effect on improving the recognition effect.
[0083] To solve the above problems, the present application provides a short speech language recognition method, which can accurately recognize the language of short speech data. The language recognition method of the present application will be described in detail below with reference to the accompanying drawings.
[0084] Reference Figure 1 , Figure 1 A flowchart of a short speech language recognition method provided by the present application is shown in FIG. 1, which can include the following steps, which will be described in detail below. Figure 1
[0085] S101: obtaining short speech data to be recognized;
[0086] In the present application, the short speech data can be speech data with a speech effective duration not exceeding a preset duration, and the preset duration can be set based on the scene requirements. For example, the preset duration can be set to 3s, and the present application does not make any limitation.
[0087] S102: performing feature processing on the short speech data to obtain a feature representation of the short speech data;
[0088] The current mainstream language recognition technology is to use the FB feature of the speech data as the feature representation of the speech data. In the present application, as a possible implementation manner, the acoustic feature of the short speech data, such as the FB (Filter Bank) feature, can be extracted as the feature representation of the short speech data. Considering that the effective time length of the short speech is short, the acoustic feature of the short speech data has weak ability to represent the language information. In the present application, as another possible implementation manner, the short speech data can be processed to obtain a feature with strong ability to represent the language information as the feature representation of the short speech data.
[0089] In order to improve the recognition effect, the short speech data can be preprocessed (such as filtering out invalid data such as silence and noise), and then the short speech data is processed.
[0090] S103: input the feature representation of the short speech data into a short speech language recognition model to obtain a language sub-class recognition result output by the short speech language recognition model; the short speech language recognition model is trained by using the feature representation of the short speech data sample and the language sub-class label thereof; the language corresponding to the language sub-class recognition result is the language of the short speech data.
[0091] The current mainstream language recognition model regards the same language as only one category. Considering that the effective time length of the short speech data is short, the information representing the language is limited, and factors such as accent difference and noise size are more likely to interfere with the recognition result and more likely to cause confusion between different languages. In the present application, the same language is no longer regarded as only one category, but the diversity of the short speech data of the same language is further considered, the same language is divided into more refined language sub-classes, and a short speech language recognition model is pre-trained by using the feature representation of the short speech data sample and the language sub-class label thereof. The short speech language recognition model can learn the correspondence between the feature representation of the different short speech data of the same language and the language sub-classes of the language, based on which the feature representation of the short speech data is input into the short speech language recognition model, and the language sub-class recognition result output by the short speech language recognition model is obtained. The language corresponding to the language sub-class recognition result is the language of the short speech data.
[0092] The technical solution of the present application can provide more refined recognition granularity for the short speech data with limited information representing the language for each language, thereby improving the short speech language recognition accuracy.
[0093] As mentioned in the foregoing, the acoustic feature of the short speech data has weak ability to represent language information due to the short effective duration of the short speech, in the present application, as another possible implementation, the short speech data can be processed to obtain a feature with strong ability to represent language information as the feature representation of the short speech data. Next, the specific implementation is described in detail.
[0094] In a possible implementation, with reference to Figure 2 , Figure 2 A flowchart of a method for processing short speech data to obtain a feature representation of the short speech data according to an embodiment of the present application is shown in FIG. 2, which includes the following steps: Figure 2
[0095] S201: Acoustic feature processing is performed on the short speech data to obtain acoustic features of the short speech data.
[0096] In the present application, the acoustic features of the short speech data can be filter bank (FB) features, mel-frequency cepstral coefficients (MFCC) features, etc., which are not limited in the present application.
[0097] S202: The acoustic features of the short speech data are input into a preset encoder to obtain a feature representation output by the encoder as the feature representation of the short speech data, the encoder being an encoder in a pre-trained multilingual speech recognition model.
[0098] In a possible implementation, the encoder can be an encoder layer in a multilingual speech recognition model. The multilingual speech recognition model is trained in advance using a large amount of multilingual data. For example, the multilingual speech recognition model can be a whisper-base model.
[0099] Of course, the multilingual speech recognition model can also be other models, which are not limited in the present application. However, the size of the encoder cannot be too large, otherwise even if the feature representation obtained by encoding has strong ability to represent language information, the speed of language recognition will be greatly reduced.
[0100] The FB feature only stays in the acoustic physical layer, and in the embodiment, the acoustic information bottleneck of the short speech FB feature can be fundamentally made up by migrating high-level semantic representation. Specifically, the pre-trained multilingual speech recognition model has strong context modeling capability through large-scale data training, and even if the input speech fragment is short, the extracted features have already integrated phoneme, syllable, and even word-level linguistic constraints, forming high-level representation that is resistant to noise and accent and is universal across languages, and the discrimination is significantly better than the FB feature that simply relies on the spectrum. In addition, the model implicitly learns shared pronunciation units and distinguishing boundaries across languages during the training process, so that the extracted features naturally have contrastive learning properties and can better distinguish systematic differences between different languages. These features, after multi-layer nonlinear transformation, encode the mapping relationship between pronunciation patterns and language symbols, are closer to "what is said" rather than "how to make sound", and can effectively suppress irrelevant variations such as speaker personality and channel difference, thereby greatly improving the discrimination between languages.
[0101] Therefore, the feature representation of the short speech data obtained by using the encoder in the pre-trained multilingual speech recognition model to process the short speech data has strong ability to represent language information, thereby laying a solid foundation for the accuracy of short speech language recognition.
[0102] In a possible implementation, with reference to Figure 3 , Figure 3 A flowchart of a determination manner of a feature representation of a short speech data sample provided by an embodiment of the present application is shown in Figure 3 , and includes the following steps:
[0103] S301: Determine a short speech data sample;
[0104] In the present application, a business expert can prepare short speech data of each language, and collect at least 1 hour of data for each language, and perform speed processing (such as speed 0.9 and speed 1.1 times speed processing) on part of the short speech data of each language prepared by the business expert to obtain a short speech data sample, and the short speech data sample has a language label.
[0105] S302: Perform acoustic feature processing on the short speech data sample to obtain acoustic features of the short speech data sample;
[0106] In the present application, the acoustic features of the short speech data sample can be FB (Filter Bank) features, MFCC features, etc., and the present application does not make any limitation thereto.
[0107] S303: input the acoustic features of the short voice data sample into a preset encoder to obtain a feature representation output by the encoder as the feature representation of the short voice data sample, the encoder being an encoder in a pre-trained multilingual speech recognition model.
[0108] In a possible implementation, the encoder can be an encoder layer in a multilingual speech recognition model. The multilingual speech recognition model is well trained in advance by using a large amount of multilingual data. For example, the multilingual speech recognition model can be a whisper-base model.
[0109] Of course, the multilingual speech recognition model can also be other models, and the present application does not make any limitation thereto. However, the size of the encoder cannot be too large, otherwise even if the feature representation obtained by encoding has strong ability to represent language information, the speed of language recognition will be greatly reduced.
[0110] As described in the foregoing embodiments, the FB feature only stays at the acoustic physical level, and in the present embodiment, the acoustic information bottleneck of the short voice FB feature can be fundamentally made up by migrating high-level semantic representation. Specifically, the pre-trained multilingual speech recognition model has strong context modeling ability through large-scale data training, and even if the input voice segment is short, the features extracted by the model have already integrated phoneme, syllable and even word-level linguistic constraints, forming a high-level representation that is resistant to noise and accent and is universal across languages, and the discrimination is significantly better than the FB feature that simply relies on the spectrum. In addition, the model implicitly learns shared pronunciation units and distinguishing boundaries across languages during the training process, so that the extracted features naturally have a contrast learning attribute and can better distinguish the systematic differences between different languages. These features are transformed by multiple nonlinear layers, and the mapping relationship between pronunciation patterns and language symbols is encoded, which is closer to "what is said" rather than "how to pronounce", and can effectively suppress irrelevant variations such as speaker personality and channel difference, thereby greatly improving the discrimination between languages.
[0111] Therefore, the feature representation of the short voice data sample obtained by using the encoder in the pre-trained multilingual speech recognition model to process the short voice data sample has strong ability to represent language information, which lays a solid foundation for the training effect of the short voice language recognition model.
[0112] In a possible implementation, with reference to Figure 4 , Figure 4 A flowchart of a manner provided by an embodiment of the present application for determining a language sub-class label of a short voice data sample is shown in Figure 4 , and includes the following steps:
[0113] S401: training a base language recognition model by using the feature representation of the short speech data sample and the language label thereof;
[0114] In the present application, when the base language recognition model is trained by using the feature representation of the short speech data sample and the language label thereof, the iteration number of the network can be set as iter max1, and the initial learning rate can be set as lr init1. The feature representation of the randomly sampled sample in one batch N1 is sent to the base language recognition model for training. The specific values of iter max1, lr init1 and batch N1 can be set based on the scene requirements, and the present application does not make any limitation thereto.
[0115] The loss function used by the base language recognition model can be a classification loss function, such as an AM-Softmax loss function. The trained base language recognition model can be obtained by repeatedly iterating until the loss function of the base language recognition model is stable or the maximum iteration number is reached.
[0116] S402: inputting the feature representation of the short speech data sample into the base language recognition model to obtain the embedding vector of the short speech data sample;
[0117] In the present application, after the feature representation of the short speech data sample is input into the base language recognition model, the feature vector of any layer between the input layer and the classification layer of the base language recognition model can be obtained as the embedding vector of the short speech data sample. Preferably, the feature vector of the last layer before the classification layer of the base language recognition model can be obtained as the embedding vector of the short speech data sample. Because the information contained in the embedding vector of this layer is the most complete.
[0118] S403: for each language, clustering the embedding vectors of the short speech data samples related to the language to obtain the language sub-class label of each short speech data sample.
[0119] In the present application, the clustering algorithm can be a Kmeans clustering algorithm, or other clustering algorithms, which can be set based on the scene requirements, and the present application does not make any limitation thereto.
[0120] In this embodiment, the short speech embedding vectors of each language are clustered to obtain language sub-class labels, and the core advantage is to realize fine modeling and adaptive enhancement by mining the structured variation patterns within the language. Specifically, clustering can unsupervisedly reveal the multi-cluster distribution formed by factors such as dialects, accents and channels within the same language, thereby converting the traditional fuzzy modeling of "overall language" to accurate representation of "language style subdivision". This process provides more rich discrimination clues for the back-end classifier, enabling it to simultaneously learn the difference boundaries between languages and between language sub-classes, thereby significantly improving the ability to distinguish between easily confused variants of languages. In addition, aggregating statistical information at the sub-class level can effectively alleviate the instability problem of single short speech sample representation caused by information sparseness, and improve the model estimation reliability. Finally, the acoustic variation originally belonging to interference factors is converted into learnable structured knowledge, and a more accurate and more interpretable language discrimination benchmark is constructed in the feature space.
[0121] In a possible implementation, with reference to Figure 5 , Figure 5 A flowchart of a method for clustering embedding vectors of short speech data samples related to a language to obtain language sub-class labels of each short speech data sample is provided in this embodiment, as shown in Figure 5 .
[0122] S501: Cluster the embedding vectors of the short speech data samples related to the language to obtain an initial clustering result, and the initial clustering result includes the number of samples corresponding to each initial sub-class;
[0123] For ease of understanding, assume that there are 5000 short speech data samples related to the Chinese language, and the embedding vectors of the 5000 short speech data samples are extracted. The initial clustering result obtained by clustering the embedding vectors of the 5000 short speech data samples can be as follows:
[0124] The embedding vectors of 2500 short speech data samples in the first initial sub-class;
[0125] The embedding vectors of 1200 short speech data samples in the second initial sub-class;
[0126] The embedding vectors of 800 short speech data samples in the third initial sub-class;
[0127] The embedding vectors of 400 short speech data samples in the fourth initial sub-class;
[0128] The embedding vectors of 100 short speech data samples in the fifth initial sub-class.
[0129] S502: merging each initial sub-class with a sample quantity less than the preset sub-class sample quantity threshold in the initial clustering result, to obtain a final sub-class clustering result; the final clustering result includes a sample quantity corresponding to each final sub-class;
[0130] For ease of understanding, still taking the above clustering example as an example, assuming that the preset sub-class sample quantity threshold is 1000, the embedding vectors of the 800 short voice data samples in the third initial sub-class, the embedding vectors of the 400 short voice data samples in the fourth initial sub-class, and the embedding vectors of the 100 short voice data samples in the fifth initial sub-class are merged, and the final sub-class clustering result obtained can be as follows:
[0131] The embedding vectors of 2500 short voice data samples in the first final sub-class;
[0132] The embedding vectors of 1200 short voice data samples in the second final sub-class;
[0133] The embedding vectors of 1300 short voice data samples in the third final sub-class.
[0134] In the present application, by merging each initial sub-class with a sample quantity less than the preset sub-class sample quantity threshold in the initial clustering result, to obtain a final sub-class clustering result, it can be prevented that the short voice language recognition model training fails due to too few sub-classes.
[0135] S503: naming a language sub-class label for each final sub-class in the final clustering result, to obtain a language sub-class label for each short voice data sample.
[0136] For ease of understanding, still taking the above clustering example as an example, assuming that the language sub-class labels of Chinese languages are represented as Chinese 1, Chinese 2, Chinese 3, …, the first final sub-class can be named as Chinese 1, the second final sub-class can be named as Chinese 2, and the third final sub-class can be named as Chinese 3, and the language sub-class label of the short voice data sample in the first final sub-class is Chinese 1, the language sub-class label of the short voice data sample in the second final sub-class is Chinese 2, and the language sub-class label of the short voice data sample in the third final sub-class is Chinese 3.
[0137] In a possible implementation, with reference to Figure 6 , Figure 6 A flowchart of a training mode of a short voice language recognition model provided by an embodiment of the present application is shown as follows: Figure 6
[0138] S601: update the classification layer of the base language model to obtain a short speech language recognition model to be trained, the number of nodes of the last layer of the classification layer of the short speech language recognition model to be trained is consistent with the total number of sub-class labels of languages;
[0139] In the present application, the backbone network parameters of the base language recognition model can be fixed, the classification layer of the base language recognition model is removed and the classification layer is reinitialized to obtain a short speech language recognition model to be trained; the number of nodes of the last layer of the classification layer of the short speech language recognition model to be trained is the total number of sub-class labels of languages.
[0140] S602: train the short speech language recognition model to be trained using the feature representation of the short speech data sample and the sub-class label of the language thereof to obtain a trained short speech language recognition model.
[0141] In the present application, when training the short speech language recognition model to be trained using the feature representation of the short speech data sample and the sub-class label of the language thereof, the iteration number of the network can be set as iter_max2, the initial learning rate can be set as lr_init2, and one batch_N2 of feature representations is sent to the short speech language recognition model to be trained at a time. The specific values of iter_max2, lr_init2 and batch_N2 can be set based on the scene requirements, and the present application does not make any limitation thereto.
[0142] Wherein, the rules of balanced sampling of batch_N2 are as follows: assuming that the total number of sub-class labels of languages is L, k1 classes are randomly selected from L classes each time, and k2 samples are randomly collected from each of the k1 classes. Keep k1 x k2 = M, and the value of M remains unchanged. The specific values of k1 and k2 can be set based on the scene requirements, and the present application does not make any limitation thereto.
[0143] The loss function used by the short speech language recognition model can also be a classification loss function, such as AM-Softmax loss function. The trained short speech language recognition model can be obtained by repeatedly iterating until the loss function of the short speech language recognition model is stable or the maximum iteration number is reached.
[0144] In this embodiment, the base language recognition model is first trained, and then the classification layer of the base language model is updated, so that the node number of the last layer of the classification layer of the short audio language recognition model to be trained is consistent with the total number of language sub-class labels, and the short audio language recognition model to be trained is obtained. Finally, the short audio language recognition model to be trained is trained to obtain the trained short audio language recognition model. This two-stage training method provides more abundant internal supervision signals for the model through the language sub-class labels obtained in the first stage, so that the classification layer can be retrained using these language sub-class labels in the second stage, so that the classification layer can learn to distinguish more subtle acoustic mode differences, thereby improving the short audio language recognition effect of the model.
[0145] The above introduces a short audio language recognition method provided by an embodiment of the application. The following introduces a device for executing the short audio language recognition method.
[0146] Please refer to Figure 7 , Figure 7 The structure of a short audio language recognition device provided by an embodiment of the application is shown in FIG. 1. As shown in FIG. 1, the short audio language recognition device includes: Figure 7
[0147] The acquisition unit 11 is configured to acquire short audio data to be recognized.
[0148] The feature processing unit 12 is configured to perform feature processing on the short audio data to obtain a feature representation of the short audio data.
[0149] The recognition unit 13 is configured to input the feature representation of the short audio data into a short audio language recognition model to obtain a language sub-class recognition result output by the short audio language recognition model. The short audio language recognition model is trained using feature representations of short audio data samples and language sub-class labels thereof. The language corresponding to the language sub-class recognition result is the language of the short audio data.
[0150] In a possible implementation, the feature processing unit is specifically configured to:
[0151] perform acoustic feature processing on the short audio data to obtain acoustic features of the short audio data.
[0152] input the acoustic features of the short audio data into a preset encoder to obtain a feature representation output by the encoder as the feature representation of the short audio data, the encoder being an encoder in a pre-trained multi-language speech recognition model.
[0153] In a possible implementation, the device further includes:
[0154] a feature representation determination unit of short audio data samples, specifically configured to:
[0155] determine a short speech data sample;
[0156] perform acoustic feature processing on the short speech data sample to obtain an acoustic feature of the short speech data sample;
[0157] input the acoustic feature of the short speech data sample into a preset encoder to obtain a feature representation output by the encoder as a feature representation of the short speech data sample, the encoder being an encoder in a pre-trained multilingual speech recognition model.
[0158] In a possible implementation, the apparatus further includes:
[0159] a language sub-class label determination unit for the short speech data sample, including:
[0160] a basic language identification model training unit for training a basic language identification model by using the feature representation of the short speech data sample and a language label thereof;
[0161] an embedding vector determination unit for inputting the feature representation of the short speech data sample into the basic language identification model to obtain an embedding vector of the short speech data sample;
[0162] a clustering unit for clustering, for each language, embedding vectors of short speech data samples related to the language to obtain a language sub-class label of each of the short speech data samples.
[0163] In a possible implementation, the embedding vector determination unit is specifically configured to:
[0164] obtain a feature vector of a last layer before a classification layer of the basic language identification model as the embedding vector of the short speech data sample.
[0165] In a possible implementation, the clustering unit is specifically configured to:
[0166] cluster the embedding vectors of the short speech data samples related to the language to obtain an initial clustering result, the initial clustering result including a sample quantity corresponding to each initial sub-class;
[0167] perform merging processing on each initial sub-class in the initial clustering result with a sample quantity less than a preset sub-class sample quantity threshold to obtain a final sub-class clustering result; the final clustering result including a sample quantity corresponding to each final sub-class;
[0168] perform language sub-class label naming on each final sub-class in the final clustering result to obtain the language sub-class label of each of the short speech data samples.
[0169] In a possible implementation, the apparatus further includes:
[0170] The short speech language recognition model training unit is specifically configured to:
[0171] updating the classification layer of the basic language model to obtain a short speech language recognition model to be trained, the number of nodes of the last layer of the classification layer of the short speech language recognition model to be trained being consistent with the total number of sub-class labels of languages;
[0172] training the short speech language recognition model to be trained by using the feature representation of the short speech data sample and the sub-class label of the language of the short speech data sample to obtain a trained short speech language recognition model.
[0173] The various units in the short speech language recognition apparatus can be all or partially implemented by software, hardware, or a combination thereof. The various units can be embedded in or independent of a processor in a computer device in a hardware form, or stored in a memory of the computer device in a software form, so as to be called and executed by a processor to perform the operations corresponding to the various units.
[0174] An electronic device is also provided in the embodiments of the present application. Referring to Figure 8 , a structural schematic diagram suitable for implementing the electronic device in the embodiments of the present application is shown. The electronic device in the embodiments of the present application can include, but is not limited to, a fixed terminal such as a mobile phone, a notebook computer, a PDA (Personal Digital Assistant), a PAD (Tablet Personal Computer), a desktop computer, and the like. Figure 8 The electronic device shown is only an example, and should not bring any limitation to the functions and use range of the embodiments of the present application.
[0175] As shown in Figure 8 , the electronic device can include a processing device (for example, a central processing unit, a graphics processing unit, or the like) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 to a random access memory (RAM) 603. In a state where the electronic device is powered on, the RAM 603 also stores various programs and data required for the operation of the electronic device. The processing device 601, the ROM 602, and the RAM 603 are connected to each other through a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.
[0176] In general, the following devices can be connected to the I / O interface 605: input devices 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, and the like; output devices 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, and the like; storage devices 608 including, for example, a memory card, a hard disk, and the like; and communication devices 609. The communication devices 609 can allow the electronic device to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 8 An electronic device having various devices is shown, but it is understood that all of the shown devices are not required to be implemented or present. More or less devices can alternatively be implemented or present.
[0177] The embodiment of the present application further provides a computer program product comprising computer readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the short speech language recognition methods provided by the embodiments of the present application.
[0178] The embodiment of the present application further provides a computer readable storage medium, which carries one or more computer programs, when the one or more computer programs are executed by an electronic device, can cause the electronic device to implement any of the short speech language recognition methods provided by the embodiments of the present application.
[0179] In addition, it should be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e. they can be located in one place, or distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments. In addition, the connection relationship between the modules in the device embodiment provided by the present application indicates that there is a communication connection between them, which can be implemented as one or more communication buses or signal lines.
[0180] Those skilled in the art can clearly understand that the application can be implemented by means of software plus necessary universal hardware, and of course can also be implemented by means of dedicated hardware including special integrated circuit, special CPU, special memory, special component, etc. Generally, any function completed by computer program can be easily implemented by corresponding hardware, and the specific hardware structure for implementing the same function can also be various, such as analog circuit, digital circuit or special circuit, etc. However, for the application, software program implementation is a better embodiment. Based on such understanding, the technical solution of the application or the part of the application which makes contribution to the prior art can be embodied in the form of software product, which is stored in readable storage medium, such as computer floppy disk, U disk, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc., and includes a plurality of instructions for making a computer device (which can be personal computer, training device or network device, etc.) execute the method described in various embodiments of the application.
[0181] In the above embodiments, the implementation can be achieved by software, hardware, firmware or any combination thereof, entirely or partially. When implemented by software, the implementation can be in the form of computer program product.
[0182] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on the computer, the flow or function described in the embodiments of the application is generated entirely or partially. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another, for example, the computer instructions can be transferred from one website, computer, training device or data center to another website, computer, training device or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that can be stored by the computer or a data storage device such as training device, data center, etc. integrated with one or more available media sets. The available medium can be magnetic medium (such as floppy disk, hard disk, magnetic tape), optical medium (such as DVD) or semiconductor medium (such as solid state disk (SSD)) etc.
Claims
1. A method for short speech language recognition, characterized in that, include: Acquire short speech data to be recognized; The short speech data is subjected to feature processing to obtain the feature representation of the short speech data; The feature representation of the short speech data is input into the short speech language identification model to obtain the language seed category identification result output by the short speech language identification model; the short speech language identification model is trained using the feature representation of the short speech data sample and its language seed category label; the language corresponding to the language seed category identification result is the language of the short speech data.
2. The method according to claim 1, characterized in that, The step of performing feature processing on the short speech data to obtain a feature representation of the short speech data includes: The short speech data is subjected to acoustic feature processing to obtain the acoustic features of the short speech data; The acoustic features of the short speech data are input into a preset encoder, and the feature representation output by the encoder is used as the feature representation of the short speech data. The encoder is the encoder in a pre-trained multilingual speech recognition model.
3. The method according to claim 1, characterized in that, The feature representation of the short speech data sample is determined as follows: Identify short speech data samples; The short speech data samples are subjected to acoustic feature processing to obtain the acoustic features of the short speech data samples; The acoustic features of the short speech data sample are input into a preset encoder, and the feature representation output by the encoder is used as the feature representation of the short speech data sample. The encoder is the encoder in a pre-trained multilingual speech recognition model.
4. The method according to claim 1, characterized in that, The method for determining the language seed class label of the short speech data sample includes: A basic language recognition model is obtained by training the feature representations and language labels of the short speech data samples. The feature representation of the short speech data sample is input into the basic language recognition model to obtain the embedding vector of the short speech data sample; For each language, the embedding vectors of short speech data samples related to that language are clustered to obtain language subclass labels for each short speech data sample.
5. The method according to claim 4, characterized in that, The step of inputting the feature representation of the short speech data sample into the basic language recognition model to obtain the embedding vector of the short speech data sample includes: The feature vector of the last layer before the classification layer of the basic language recognition model is obtained as the embedding vector of the short speech data sample.
6. The method according to claim 4, characterized in that, The process of clustering the embedding vectors of the language-related short speech data samples to obtain language subclass labels for each short speech data sample includes: Clustering is performed on the embedding vectors of the language-related short speech data samples to obtain an initial clustering result, which includes the number of samples corresponding to each initial subclass. The initial subclasses whose sample counts are less than a preset threshold for the number of subclass samples in the initial clustering results are merged to obtain the final subclass clustering results; the final clustering results include the sample counts corresponding to each final subclass. Each final subclass in the final clustering result is named with a language seed class label to obtain the language seed class label of each short speech data sample.
7. The method according to claim 4, characterized in that, The training method for the short speech language recognition model is as follows: The classification layer of the basic language model is updated to obtain the short speech language recognition model to be trained. The number of nodes in the last layer of the classification layer of the short speech language recognition model to be trained is consistent with the total number of language seed class labels. The short speech language recognition model to be trained is trained by using the feature representations and language seed class labels of short speech data samples to obtain a trained short speech language recognition model.
8. A computer program product, characterized in that, It includes computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the short speech language recognition method as described in any one of claims 1 to 7.
9. An electronic device, characterized in that, It includes at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program to enable the electronic device to implement the short speech language recognition method as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, The storage medium carries one or more computer programs that, when executed by an electronic device, enable the electronic device to implement the short speech language recognition method as described in any one of claims 1 to 7.