Language classification model training method and device, language recognition method and device and intelligent equipment

Abstract

The invention is suitable for the technical field of intelligent voice processing, and provides a language classification model training method, a language recognition method and device and intelligent equipment. The language classification model training method comprises the following steps: acquiring a first acoustic feature, wherein the first acoustic feature refers to an acoustic feature of an audio clip; the acoustic features of the audio clips are sequentially input into N layers of encoders of an initial language classification model, first encoding features output by the N layers of encoders respectively are obtained, and N is an integer larger than 1; inputting each first coding feature into a fusion module of the initial language classification model to obtain a first fusion feature; and based on the first fusion feature and a language recognition module of the initial language classification model, training the initial language classification model to obtain a target language classification model, the target language classification model being used for recognizing a language category of the audio data. According to the invention, the accuracy and robustness of language recognition can be improved.

Description

Technical Field

[0001] The present application belongs to the field of intelligent speech processing technology, and in particular relates to a language classification model training method, a language recognition method, an apparatus and an intelligent device. Background Art

[0002] In recent years, with the rapid development of smart car technology, in-vehicle voice interaction systems have become one of the core functions to enhance user experience. In the context of globalization, automobile manufacturers need to cope with multi-language and multi-scenario voice recognition needs in overseas markets. Reliable and robust spoken language identification (LID) is essential to achieve high performance and high accuracy in these systems. Existing language recognition technologies perform poorly in multi-language mixed and complex scenarios, which reduces the accuracy and robustness of language recognition. Summary of the invention

[0003] The embodiments of the present application provide a language classification model training method, a language recognition method, an apparatus and an intelligent device, which can improve the accuracy and robustness of language recognition.

[0004] In a first aspect, an embodiment of the present application provides a language classification model training method, comprising:

[0005] Acquiring a first acoustic feature, where the first acoustic feature refers to an acoustic feature of an audio segment, where the audio segment includes at least one character of a language category;

[0006] Inputting the acoustic features of the audio clip into N layers of encoders of the initial language classification model in sequence, to obtain first encoding features outputted by each of the N layers of encoders, where N is an integer greater than 1;

[0007] Inputting each of the first coding features into a fusion module of the initial language classification model to obtain a first fusion feature;

[0008] Based on the first fusion feature and the language recognition module of the initial language classification model, the initial language classification model is trained to obtain a target language classification model, and the target language classification model is used to identify the language category of the audio data.

[0009] In an embodiment of the present application, by acquiring the acoustic features of the audio clip and inputting the acoustic features into the N-layer encoder of the initial language classification model in sequence, the first coding features output by each of the N-layer encoders can be obtained, and each first coding feature is input into the fusion module of the initial language classification model to obtain the first fusion feature. Based on the first fusion feature and the language recognition module of the initial classification, the initial language classification model can be trained to obtain the target language classification model. This solution fuses the coding features output by each of the N-layer encoders to capture the global and local features of the audio clip. On this basis, the initial language classification model is trained based on the fused features (i.e., the first fusion features) and the language recognition module, which can improve the accuracy and robustness of the language recognition of the target language classification model.

[0010] In a second aspect, an embodiment of the present application provides a language recognition method, including:

[0011] Obtain audio data to be recognized;

[0012] Acquiring a second acoustic feature, where the second acoustic feature refers to an acoustic feature of the audio data to be recognized;

[0013] Determine the language category of the audio data to be recognized based on the second acoustic feature and a target language classification model; the target language classification model is obtained by training an initial language classification model based on the method described in the first aspect above;

[0014] The target language classification model includes an N-layer encoder, a fusion module and a language recognition module, where N is an integer greater than 1;

[0015] The N-layer encoder is used to: determine a corresponding second coding feature based on the second acoustic feature;

[0016] The fusion module is used to: fuse the second coding features to obtain a target fusion feature;

[0017] The language identification module is used to determine the language category to which the audio data to be identified belongs based on the target fusion feature.

[0018] In an embodiment of the present application, by obtaining the acoustic features of the audio data to be identified (i.e., the second acoustic features), and inputting the acoustic features into the target language classification model, the language category to which the audio data to be identified belongs can be obtained. Among them, the above-mentioned target language classification model includes an N-layer encoder, a fusion module and a language recognition module, the N-layer encoder is used to determine the corresponding second coding features based on the acoustic features of the audio data to be identified, the fusion module is used to fuse each of the second coding features to obtain a target fusion feature, and the language recognition module is used to determine the language category to which the audio data to be identified belongs based on the target fusion feature. When this solution performs language recognition on the audio data to be identified through the target language classification model, it can capture the global features and local features of the audio data to be identified, thereby improving the accuracy and robustness of language recognition.

[0019] In a third aspect, an embodiment of the present application provides a language classification model training device, which can be applied to a smart device or can be used as a smart device, and the language classification model training device includes:

[0020] A feature acquisition module, configured to acquire a first acoustic feature, wherein the first acoustic feature refers to an acoustic feature of an audio segment, wherein the audio segment includes at least one character of a language category;

[0021] A feature encoding module, used for sequentially inputting the first acoustic feature into N layers of encoders of an initial language classification model to obtain first encoding features outputted by each of the N layers of encoders, where N is an integer greater than 1;

[0022] A feature input module, used for inputting each of the first coding features into a fusion module of the initial language classification model to obtain a first fusion feature;

[0023] A model training module is used to train the initial language classification model based on the first fusion feature and the language recognition module of the initial language classification model to obtain a target language classification model, and the target language classification model is used to identify the language category of the audio data.

[0024] In a fourth aspect, an embodiment of the present application provides a language recognition device, which can be applied to a smart device or can be used as a smart device, and the language recognition device includes:

[0025] A data acquisition module, used to acquire audio data to be recognized;

[0026] An acoustic acquisition module, used for acquiring a second acoustic feature, where the second acoustic feature refers to an acoustic feature of the audio data to be recognized;

[0027] A language recognition module, configured to determine the language category to which the audio data to be recognized belongs based on the audio data to be recognized and a target language classification model; the target language classification model is obtained by training an initial language classification model based on the method described in the first aspect above;

[0028] The target language classification model includes an N-layer encoder, a fusion module and a language recognition module, where N is an integer greater than 1;

[0029] The N-layer encoders are used to: determine the second encoding features corresponding to the respective ones based on the second acoustic features;

[0030] The fusion module is used to: fuse the second coding features to obtain a target fusion feature;

[0031] The language identification module is used to determine the language category to which the audio data to be identified belongs based on the target fusion feature.

[0032] In a fifth aspect, an embodiment of the present application provides an intelligent device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the electronic device implements the language classification model training method described in the first aspect above, or implements the language recognition method described in the second aspect above.

[0033] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a computer, the language classification model training method as described in the first aspect above is implemented, or the language recognition method as described in the second aspect above is implemented.

[0034] In a seventh aspect, an embodiment of the present application provides a computer program product, comprising a computer program, which, when executed, enables the language classification model training method as described in the first aspect above to be executed, or enables the language recognition method as described in the second aspect above to be executed.

[0035] It can be understood that the beneficial effects of the third to seventh aspects mentioned above can be found in the relevant descriptions of the first and second aspects mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0037] Figure 1 It is a flowchart of the language classification model training method provided in the embodiment of the present application;

[0038] Figure 2 It is a flowchart of the language recognition method provided in the embodiment of the present application;

[0039] Figure 3 It is a structural diagram of a language classification model training device provided in an embodiment of the present application;

[0040] Figure 4 is a schematic diagram of the structure of a language recognition device provided in an embodiment of the present application;

[0041] Figure 5 It is a schematic diagram of the structure of the smart device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0042] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0043] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0044] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0045] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0046] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0047] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0048] See also Figure 1 , Figure 1 A flow chart of the language classification model training method provided in the embodiment of the present application is shown. As an example but not limitation, the method is applied to a smart device (for example, a smart device may include a driving device, a smart car, a robot, etc.) or an electronic device. For example, the smart device is a smart car. The electronic device may be a server, a mobile phone, a tablet computer, a computer, or other related electronic device with a model training function. For example, the server may also be a cloud server that communicates with the smart device.

[0049] Specifically, Figure 1 The method provided in the embodiment of the present application is described by taking the method being applicable to a smart device as an example. Specifically, the method comprises the following steps:

[0050] Step 101: Obtain a first acoustic feature, where the first acoustic feature refers to an acoustic feature of an audio clip.

[0051] The audio clip includes at least one character of the language category. On this basis, the audio clip can contain language information, so that it can be used to train a language classification model.

[0052] Optionally, in order to realize language recognition of short audio, the audio length of the above audio segment can be limited to a preset length range, and the maximum value of the preset length range can be the length threshold described below. For example, the preset length range is 640 milliseconds to 2 seconds.

[0053] In order to improve the recognition accuracy and robustness of the language classification model, more audio clips can be used to train the language classification model, that is, the number of audio clips is multiple. Of course, it is understandable that the number of the above audio clips can be one, and this application does not limit this.

[0054] In order to facilitate the distinction between an untrained language classification model and a trained language classification model, the untrained language classification model can be referred to as an initial language classification model, and the trained language classification model can be referred to as a target language classification model.

[0055] The first acoustic feature and the second acoustic feature described below are acoustic features of the same type, and the present application does not limit the specific type of the acoustic feature. For example, the acoustic feature may be a filter bank (FilterBank, FBank) feature, a Mel-Frequency Cepstrum Coefficients (Mel-Frequency Cepstrum Coefficients, MFCC), a Linear Predictive Cepstral Coefficient (Linear Predictive Cepstral Coefficient, LPCC) or a Perceptual Linear Predictive (Perceptual Linear Predictive, PLP) coefficient.

[0056] As an example but not a limitation, the first acoustic feature and the second acoustic feature in the present application are FBank features, which can process audio signals in a manner similar to the human ear, thereby improving the performance of speech recognition.

[0057] In one embodiment, the smart device can obtain at least one audio training sample, and can directly use each audio training sample as an audio segment, or can randomly crop each audio training sample to obtain an audio segment in order to increase the diversity of the audio segment. This application is not limited to this.

[0058] The at least one audio training sample constitutes a training data set of the initial language classification model, which is used to train the initial language classification model. When training the initial language classification model based on the training data set, audio segments can be randomly cut out based on each audio training sample in the training data set, and then the language classification model is trained based on the audio segments. That is to say, audio segments need to be randomly cut out based on the audio training samples in each training, which can ensure the diversity of audio segments.

[0059] For the cropped audio clips, data enhancement techniques such as time domain perturbation, noise addition, frequency domain transformation, speed change, time domain and frequency domain masking can be applied to enhance the audio clips to improve the robustness of the target language classification model.

[0060] In one embodiment, the smart device can collect audio data of relevant languages ​​on the Internet and user voice data actually collected in the vehicle, and use these data as original audio data. The original audio data can be directly used as audio training samples, or the original audio data can be pre-processed through a voice activity detection (VAD) module to remove silent segments in the original audio data and crop out audio training samples containing valid language information, thereby eliminating the influence of non-human voice data on the training process of the initial language classification model.

[0061] The smart device may crop each audio training sample to obtain an audio segment, which may include: obtaining the timestamp of each character in each audio training sample; for any audio training sample, randomly cropping the audio training sample based on the timestamp of each character in the audio training sample during each training to obtain multiple audio segments.

[0062] The timestamp of a character records the time information of the character in the corresponding audio training sample.

[0063] Optionally, the audio training samples can be processed by the Whisper model to obtain the timestamp of each character in the audio training samples, that is, the timestamp is a word-level timestamp, and the audio training samples can be more accurately randomly cropped based on the word-level timestamp, thereby ensuring that the audio segment obtained after random cropping includes at least one character. Among them, for audio training samples that are not labeled with language categories, or audio training samples labeled with multiple language categories, the above-mentioned Whisper model can be used to perform language recognition on the audio training samples to obtain the true language value (i.e., expression label) of the audio training samples, and the initial language classification model can be trained based on the language true value.

[0064] This embodiment performs random cropping based on the timestamp of each character, which can ensure that each audio segment includes at least one character, so that the audio segment contains language information and can be used to train the initial language classification model. And the random cropping of audio training samples can simulate the randomness of the audio length and the diversified audio characteristics of voice commands in real scenes (such as car scenes), so as to better train the initial language classification model.

[0065] It usually takes more rounds of training from the beginning of training the initial language classification model to the convergence of the initial language classification model. During each training, the audio training samples are randomly cropped again. Due to the random cropping, the audio segments obtained from the same audio training sample at different times of cropping may also be different, which can make the training data (i.e., the audio segments) more in line with the actual vehicle scene and improve the robustness of the target language classification model.

[0066] In one embodiment, a length threshold may be pre-set, and an audio segment suitable for short audio recognition may be obtained from the audio training sample through the length threshold, that is, the audio length of the audio segment is shorter. By training the initial language classification model with the audio segment, the accuracy and robustness of the target language classification model for short audio language recognition can be improved.

[0067] Among them, the scheme for obtaining an audio segment suitable for short audio recognition from an audio training sample through a length threshold includes: if the audio length of the audio training sample exceeds the length threshold, the audio training sample is randomly cropped based on the timestamp of each character in the audio training sample during each training; if the audio duration of the audio training sample does not exceed the length threshold, the audio training sample is determined to be an audio segment.

[0068] Optionally, the length threshold can be set according to the scenario requirements, and the specific value of the length threshold of the present application is not limited. As an example but not a limitation, the length threshold is 2 seconds.

[0069] In real car scenarios, user voice commands usually appear in the form of short audio clips (for example, the audio length is less than 1 to 2 seconds). In streaming recognition scenarios, it is necessary to identify the language category of the audio within a few hundred milliseconds to facilitate the subsequent speech recognition and natural language understanding (NLU) related decision logic. This embodiment uses the short audio enhancement method of VAD cropping and random cropping to make the audio clips used to train the initial language classification model more consistent with real car scenarios, and can also more accurately distinguish the features of languages ​​with higher similarity, thereby improving the accuracy and robustness of the target language classification model for language recognition of short audio, and achieving efficient recognition of short audio in multiple languages, which is suitable for real car scenarios and streaming recognition scenarios.

[0070] As an example but not a limitation, in a real vehicle scenario, the smart device collects audio data of relevant languages ​​on the Internet and real vehicle data, and first uses the VAD module to crop these audio data (which may include audio data of one language or multiple languages) to crop audio training samples that conform to the real scenario, and then uses the Whisper model to further identify the language of audio training samples that are not labeled with language categories or have low language category accuracy, and outputs the timestamp of each character in the audio training samples to obtain a high-quality multi-language training data set. In order to further improve the short audio classification capability, the audio training samples can be randomly cropped based on the timestamp to obtain an audio clip containing at least one character of the corresponding language category. During random cropping, audio training samples with an audio length greater than 2 seconds are cropped, and audio clips with a length of 640 milliseconds to 2 seconds containing real word pronunciations are extracted. Data enhancement techniques such as speed change, time domain perturbation, noise addition and frequency change are applied to the cropped audio clips. The FBank feature extraction method is then used to extract 80-dimensional FBank features from the enhanced audio clips, which are then masked in the time and frequency domains and used as input for subsequent model training to enhance the model's robustness to noise and interference.

[0071] Step 102, input the first acoustic feature into the N-layer encoders of the initial language classification model in sequence to obtain the first encoding features output by each of the N-layer encoders.

[0072] Wherein, N is an integer greater than 1.

[0073] Optionally, the above-mentioned N-layer encoders may be some encoders in the speech recognition model, or may be all encoders, which is not limited in this application.

[0074] Each layer of the encoder may include, in sequence: a first position-wise feedforward neural network (FFN), a multi-head self-attention (MHA) layer module, a convolution module, and a second FFN.

[0075] As an example but not a limitation, taking the i-th layer encoder as an example, the i-th layer encoder is any encoder in the N-layer encoder except the first layer encoder, and the encoding features output by the i-th layer encoder can be expressed as follows:

[0076] encoder_out i =FFN i (Conv i (MHA i (FFN_Macaron i (encoder_outi-1 ))))

[0077] Among them, encoder_out i Represents the encoding features output by the i-th layer encoder, FFN_Macaron i represents the first FFN in the i-th layer encoder, FFN i represents the second FFN in the i-th layer encoder, Conv i represents the convolutional module in the i-th layer encoder, MHA i represents the MHA in the i-th layer encoder, encoder_out i-1 Represents the encoded features output by the i-1th layer encoder.

[0078] Optionally, the above-mentioned speech recognition model can be any type of speech recognition model, which is not limited in this application.

[0079] As an example but not limitation, the above-mentioned speech recognition model is a Wav2Vec2.0-BERT model including a 12-layer encoder, and the above-mentioned N-layer encoder may refer to the 12-layer encoder.

[0080] Step 103: input each first coding feature into a fusion module of an initial language classification model to obtain a first fusion feature.

[0081] Optionally, after inputting the first coding features into the fusion module, the fusion module can splice the first coding features to integrate more semantic information in the audio clip, thereby capturing the global characteristics and fine-grained differences of the language through the synergy of hierarchical information, solving the problem of single-layer feature instability, and is suitable for language classification tasks in multi-language mixtures and complex scenarios.

[0082] Since this embodiment selects N layers of encoders in the speech recognition model for feature extraction instead of selecting the entire speech recognition model for feature extraction, the above-mentioned fusion module can also be called a lightweight layer fusion module, which can avoid the computational complexity problem caused by direct splicing of the entire layer, while also taking into account the real-time requirements of voice interaction scenarios (such as in-vehicle voice interaction scenarios).

[0083] The lightweight layer fusion module in this embodiment is isolated from the language recognition module, which speeds up the training process of the initial language classification model, can increase the diversity of language recognition, and can perform targeted optimization of language recognition, which not only improves the accuracy and robustness of language recognition of the target language classification model, but also meets the performance requirements of real-time voice interaction in the vehicle.

[0084] Step 104: Based on the first fusion feature and the language recognition module of the initial language classification model, the initial language classification model is trained to obtain a target language classification model.

[0085] The target language classification model is used to identify the language category of the audio data.

[0086] In one possible implementation, after the first fusion feature is input into the language recognition module, the N-layer encoder, the fusion module and the language recognition module can be trained based on the output of the language recognition module. Alternatively, a trained speech recognition model (i.e., the N-layer encoder has been pre-trained) can be used for feature encoding in step 102, and a trained fusion module can be used for feature fusion in step 103. On this basis, the language recognition module is trained based on the output of the language recognition module. This method can avoid additional training costs.

[0087] In a possible implementation, the language recognition module includes a first multi-head attention layer, a first attention statistical pooling layer, a second multi-head attention layer, and a second attention statistical pooling layer in sequence; the language recognition module based on the first fusion feature and the initial language classification model trains the initial language classification model to obtain a target language classification model, including:

[0088] Input the first fused feature into the first multi-head attention layer to perform attention weighting on the time dimension of the first fused feature to obtain a second fused feature;

[0089] Input the second fused feature into the first attention statistical pooling layer to fuse the features of different time steps in the second fused feature to obtain a third fused feature;

[0090] Input the third fused feature into the second multi-head attention layer to perform attention weighting on the encoder dimension of the third fused feature to obtain a fourth fused feature;

[0091] Inputting the fourth fused feature into the second attention statistical pooling layer to fuse the encoded features output by the N-layer encoder contained in the fourth fused feature to obtain a fifth fused feature;

[0092] Based on the fifth fusion feature, the initial language classification model is trained to obtain the target language classification model.

[0093] The first multi-head attention layer performs weighted attention on the time dimension of the first fusion feature, and can extract the relationship between features at different time steps in the first fusion feature. The second multi-head attention layer performs weighted attention on the encoder dimension of the third fusion feature, and can extract the relationship between the encoded features output by the N-layer encoder contained in the third fusion feature.

[0094] The first multi-head attention layer focuses on the feature relationship at different time steps in the audio clip, and can capture the temporal correlation and dynamic change patterns in the audio clip. The second multi-head attention layer focuses on the information interaction and fusion between the outputs of different encoders, extracting rich contextual semantics from multiple levels. By combining the first multi-head attention layer and the second multi-head attention layer, the language classification model can achieve a full range of feature expression in the two key dimensions of time and layer, providing a more discriminative feature representation for language classification tasks.

[0095] The first attention statistical pooling layer can dynamically adjust the attention distribution according to the actual characteristics of the audio clip, so as to effectively focus on the key information related to the language and ignore noise or redundancy. The second attention statistical pooling layer highlights the contribution of different encoders through weight distribution and realizes the dynamic aggregation of layer features. By combining the first attention statistical pooling layer and the second attention statistical pooling layer, not only the adaptability of the language classification model to complex audio data is improved, but also the robustness and accuracy of language classification are significantly enhanced. After the first coding features are fused, the dimension of the first fused feature obtained is relatively high. In order to avoid the interference of redundant information on downstream classification tasks and improve the computational efficiency, the above-mentioned initial language recognition module can also include a downsampling layer. Before the first fused feature is input into the first multi-head attention layer, the first fused feature is first input into the downsampling layer to perform dimensionality reduction processing on the first fused feature. The dimensionality reduction processing can effectively reduce the computational complexity while retaining the important information in the first fused feature.

[0096] In a possible implementation, on the basis of using the trained speech recognition model and the fusion module to train the initial language classification model (that is, the N-layer encoder and the fusion module of the initial language classification model are pre-trained), the training of the initial language classification model based on the fifth fusion feature includes:

[0097] When the parameters of the N-layer encoder are the first parameters and the parameters of the fusion module are the second parameters, based on the fifth fusion feature, updating the parameters of the language recognition module to be the third parameters;

[0098] An initial language classification model obtained when the parameters of the N-layer encoder are the first parameters, the parameters of the fusion module are the second parameters, and the parameters of the language recognition module are the third parameters is used as the target language classification model.

[0099] Since the N-layer encoder of the initial language classification model is pre-trained, the parameters of the N-layer encoder are fixed, and the first parameter is the parameter of the trained N-layer encoder. Since the fusion module of the initial language classification model is also pre-trained, the parameters of the fusion module are also fixed, and the second parameter is the parameter of the trained fusion module.

[0100] After the language recognition module is trained based on the fifth fusion feature, the parameters of the language recognition module are also fixed, so the third parameter is the parameter of the trained language recognition module.

[0101] In this embodiment, when the parameters of the N-layer encoder are the first parameters and the parameters of the fusion module are the second parameters, the parameters of the language recognition module are updated, that is, when the parameters of the N-layer encoder and the parameters of the fusion module are frozen, only the parameters of the language recognition module are updated, that is, in the training of the language classification model, the N-layer encoder and the fusion module only generate features in the forward reasoning process and do not participate in parameter updating. This method can fully utilize the powerful feature extraction capability of the encoder and the feature fusion capability of the fusion module, and avoid additional training costs, thereby improving the efficiency and stability of model training.

[0102] This embodiment can maintain feature stability by freezing the parameters of the N-layer encoder and the parameters of the fusion module, and can fully mine the information of the time dimension and layer dimension by combining the multi-head attention layer and the attention statistical pooling layer in the time dimension and layer dimension.

[0103] Optionally, the above-mentioned language recognition module can also include a fully connected layer and an activation function, that is, the fifth fusion feature is input into the fully connected layer and the activation function in sequence to obtain the language prediction value of the corresponding audio clip, the language prediction value and the true language value of the audio clip, and update the parameters of the downsampling layer, the first multi-head attention layer, the first attention statistical pooling layer, the second multi-head attention layer, the second attention statistical pooling layer and the fully connected layer in the language recognition module.

[0104] During the training process, the language classification model can be optimized through the standard cross entropy loss function, so that it can efficiently perform multi-language recognition on short-term audio. After the training is completed, the target language classification model is used for language discrimination.

[0105] After obtaining the target language classification model, a threshold mechanism can be constructed based on positive and negative sample data. As an example but not a limitation, by analyzing a large number of positive samples (target language audio data) and negative samples (non-target language audio data) of vehicle-mounted scenes, the classification score distribution of the target language classification model for different languages ​​is calculated. According to the score distribution of positive and negative samples, the score threshold for language discrimination is set. After setting the score threshold, it can be applied to language recognition scenarios. For example: when the score of the target language is greater than the score threshold, it is considered that the audio data belongs to the language; otherwise, it is considered that the language classification result is not clear enough, and the general multilingual resource package is recorded. After the classification score of the target language is greater than the score threshold, the pronunciation dictionary and weighted finite state transition machine (Weighted Finite-State Transducer, WFST) of the target language are loaded to optimize the accuracy of the speech decoding process. Call the hot word list related to the target language to ensure that important words in a specific field are recognized first in speech recognition. Call the NLU module corresponding to the target language to further improve the ability to parse the intent of speech input. Use the target language classification model for language discrimination.

[0106] The embodiment of the present application fuses the coding features output by each of the N layers of encoders, which can capture the global and local features of the audio clip. On this basis, the initial language classification model is trained based on the fused features and the language recognition module, which can improve the accuracy and robustness of language recognition of the target language classification model, and provide reliable technical support for intelligent voice interaction and multi-language support.

[0107] See also Figure 2 , Figure 2 The flow chart of the language recognition method provided in the embodiment of the present application is shown. As an example but not a limitation, the method is applied to smart devices (for example, smart devices may include driving devices, smart cars, robots, etc.) or electronic devices such as mobile phones. For example, the smart device is a smart car. Specifically, Figure 2 The method is described by taking the application of the method to a smart device as an example. Specifically, the smart device has a target language classification model. Specifically, the method includes the following steps:

[0108] Step 201: Acquire audio data to be recognized.

[0109] The audio data to be identified may refer to audio data for which language identification is required.

[0110] As an example but not limitation, the audio data to be recognized may be audio data collected by a voice collection unit in the smart device, or audio data to be recognized input by a user into the smart device via an interactive interface of the smart device.

[0111] Taking a smart device as a vehicle as an example, when a user is riding in the vehicle, the voice collection unit in the vehicle can collect the language of one or more users communicating in the vehicle in real time, and then use the collected language as audio data to be recognized.

[0112] After obtaining the audio data to be recognized, the audio data to be recognized can be preprocessed through the VAD module to remove silent segments in the audio data to be recognized, crop out the audio segments to be recognized containing valid language information, and perform language recognition based on the audio segments to be recognized, thereby eliminating the influence of non-human voice data on the language recognition of the audio data to be recognized.

[0113] Step 202: Acquire a second acoustic feature, where the second acoustic feature refers to an acoustic feature of the audio data to be identified.

[0114] When the audio data to be recognized is preprocessed by the VAD module, the second acoustic feature can be extracted from the audio segment to be recognized.

[0115] Step 203: Determine the language category to which the audio data to be recognized belongs based on the second acoustic feature and the target language classification model.

[0116] The target language classification model includes an N-layer encoder, a fusion module, and a language recognition module, where N is an integer greater than 1;

[0117] The N-layer encoder is used to: determine a corresponding second coding feature based on the second acoustic feature;

[0118] The fusion module is used to: fuse the second coding features to obtain a target fusion feature;

[0119] The language recognition module is used to determine the language category of the audio data to be recognized based on the target fusion features.

[0120] As an example, the target language classification model can be generated by a smart device or server through Figure 1 The method shown is obtained by training, or obtained by the smart device or server from device A, and device A is executed Figure 1 The method shown is used to train a device to obtain the target language classification model.

[0121] Based on the second acoustic feature and the target language classification model, determining the language category of the audio data to be recognized, including:

[0122] In one case, a target language classification model is deployed in the smart device or server, and the second acoustic feature is input into the target language classification model to obtain the language category to which the audio data to be identified belongs.

[0123] In another case, if the target language classification model is not deployed in the smart device or server, the smart device sends the second acoustic feature or the audio data to be recognized to the device deployed with the target language classification model. Then, the device deployed with the target language classification model obtains the language category to which the audio data to be recognized belongs based on the second acoustic feature or the audio data to be recognized, and sends it to the smart device or server.

[0124] The language classification model in the embodiment of the present application combines short audio enhancement and model layer fusion technology. When the language is recognized by the language classification model, the accuracy and robustness of language recognition can be improved. It is particularly suitable for multilingual speech recognition systems and related application scenarios.

[0125] As an example but not a limitation, in actual applications, the target language classification model can be used to perform real-time language recognition on voice commands collected in real time. The language recognition results are used to optimize the subsequent voice recognition process (such as selecting language-specific voice recognition language models and corresponding language dictionaries, as well as the language that the downstream NLU needs to switch to). Combining real-time requirements with vehicle-mounted scene characteristics, a seamless integration of multi-language recognition and voice interaction is achieved.

[0126] When performing language recognition on audio data to be recognized through a target language classification model, the embodiment of the present application can capture the global features and local features of the audio data to be recognized, thereby improving the accuracy and robustness of language recognition.

[0127] It is worth noting that the above Figure 1 The embodiments shown and Figure 2 When the embodiment shown is executed by the same smart device (server), the smart device (server) may first execute Figure 1 In the embodiment shown, if there is a need for audio category recognition, the Figure 2 The embodiment shown.

[0128] It should be noted that the solutions not described in detail in this embodiment can refer to the description of the relevant solutions in the aforementioned embodiments, and will not be repeated here.

[0129] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0130] It is worth noting that the smart device or server that executes the language classification model training method and the smart device or server that executes the language recognition method can be the same device or different devices.

[0131] For example, the server executes the language classification model training method, and the smart device performs the language recognition method based on the target language classification model obtained by the server executing the language classification model training method.

[0132] For example, after the server obtains the target language classification model through training in the above manner, the target language classification model is sent to the smart device, or the smart device obtains the target language classification model from the server, and then the smart device executes the above language recognition method based on the target language classification model.

[0133] Alternatively, the devices for executing the language classification model training method and the devices for executing the language recognition method are different servers. For example, server 1 executes the language classification model training method, and then sends the target language classification model to server 2, and then server 2 executes the above-mentioned language recognition method according to the target language classification model.

[0134] Alternatively, the devices that execute the language classification model training method and the devices that execute the language recognition method are different intelligent devices. For example, intelligent device 1 executes the language classification model training method, and then sends the target language classification model to intelligent device 2, and then intelligent device 2 executes the above-mentioned language recognition method according to the target language classification model.

[0135] Alternatively, after server 1 or smart device 1 obtains the target language classification model by executing the above-mentioned language classification model training method, the target language classification model is stored in server 1 or smart device 1. When server 2 or smart device 2 needs to perform language recognition on the audio data to be recognized, server 2 or smart device 2 sends the audio data to be recognized to server 1 or smart device 1, and then server 1 or smart device 1 performs language category recognition on the audio data to be recognized according to the target language classification model, and then sends the recognition result to server 2 or smart device 2.

[0136] Corresponding to the language classification model training method described in the above embodiment, Figure 3 The structural diagram of the language classification model training device provided in the embodiment of the present application is shown, and the language classification model training device can be applied to a smart device or the language classification model training device can be used as a smart device. For ease of description, only the parts related to the embodiment of the present application are shown.

[0137] Reference Figure 3 , the device comprises:

[0138] A feature acquisition module 301 is used to acquire a first acoustic feature, where the first acoustic feature refers to an acoustic feature of an audio segment, and the audio segment includes at least one character;

[0139] A feature encoding module 302 is used to sequentially input the first acoustic feature into N layers of encoders of an initial language classification model to obtain first encoding features output by each of the N layers of encoders, where N is an integer greater than 1;

[0140] A feature input module 303, used for inputting each of the first coding features into a fusion module of the initial language classification model to obtain a first fusion feature;

[0141] The model training module 304 is used to train the initial language classification model based on the first fusion feature and the language recognition module of the initial language classification model to obtain a target language classification model, and the target language classification model is used to identify the language category of the audio data.

[0142] Optionally, the above device further comprises:

[0143] A sample acquisition module, used to acquire at least one audio training sample;

[0144] A timestamp acquisition module, used to acquire the timestamp of each character in each audio training sample;

[0145] The random cropping module is used to randomly crop any of the audio training samples based on the timestamps of the characters in the audio training sample during each training to obtain a plurality of the audio segments.

[0146] Optionally, the random cropping module is specifically used for:

[0147] If the audio length of the audio training sample exceeds a length threshold, the audio training sample is randomly cropped based on the timestamp of each character in the audio training sample during each training.

[0148] Optionally, the language recognition module includes a first multi-head attention layer, a first attention statistical pooling layer, a second multi-head attention layer, and a second attention statistical pooling layer in sequence; the above-mentioned model training module 304 includes:

[0149] A first input unit, configured to input the first fused feature into the first multi-head attention layer to perform attention weighting on the time dimension of the first fused feature to obtain a second fused feature;

[0150] A second input unit is used to input the second fused feature into the first attention statistical pooling layer to fuse features at different time steps in the second fused feature to obtain a third fused feature;

[0151] A third input unit, used for inputting the third fused feature into the second multi-head attention layer to perform attention weighting on the encoder dimension of the third fused feature to obtain a fourth fused feature;

[0152] a fourth input unit, configured to input the fourth fused feature into the second attention statistical pooling layer, so as to fuse the encoded features output by the N layers of encoders contained in the fourth fused feature, so as to obtain a fifth fused feature;

[0153] A training unit is used to train the initial language classification model based on the fifth fusion feature to obtain the target language classification model.

[0154] Optionally, the training unit is specifically used for:

[0155] When the parameter of the encoder of the N-layer is the first parameter and the parameter of the fusion module is the second parameter, based on the fifth fusion feature, updating the parameter of the language recognition module to the third parameter;

[0156] The initial language classification model obtained when the parameters of the N-layer encoder are the first parameters, the parameters of the fusion module are the second parameters, and the parameters of the language recognition module are the third parameters is used as the target language classification model.

[0157] Corresponding to the language identification method described in the above embodiment, Figure 4 The structure diagram of the language recognition device provided by the embodiment of the present application is shown, and the language recognition device can be applied to a smart device or the language recognition device can be used as a smart device. For the convenience of description, only the parts related to the embodiment of the present application are shown.

[0158] Reference Figure 4 , the device comprises:

[0159] The data acquisition module 401 is used to acquire the audio data to be recognized;

[0160] An acoustic acquisition module 402, configured to acquire a second acoustic feature, where the second acoustic feature refers to an acoustic feature of the audio data to be recognized;

[0161] A language recognition module 403 is used to obtain the language category of the audio data to be recognized based on the second acoustic feature and a target language classification model; the target language classification model is obtained by training an initial language classification model based on the above-mentioned language classification model training method;

[0162] The language classification model includes an N-layer encoder, a fusion module and a language recognition module, where N is an integer greater than 1;

[0163] The N-layer encoder is used to: determine a corresponding second coding feature based on the second acoustic feature;

[0164] The fusion module is used to: fuse the second coding features to obtain a target fusion feature;

[0165] The language identification module is used to determine the language category to which the audio data to be identified belongs based on the target fusion feature.

[0166] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0167] Figure 5 This is a schematic diagram of the structure of the smart device provided in the embodiment of the present application. Figure 5 As shown, the smart device 5 of this embodiment includes: at least one processor 50 ( Figure 5 Only one is shown in the figure), a memory 51, and a computer program 52 stored in the memory 51 and executable on the at least one processor 50, wherein the processor 50 implements the steps of any of the above-mentioned method embodiments when executing the computer program 52.

[0168] The smart device may include, but is not limited to, a processor 50 and a memory 51. Those skilled in the art will appreciate that Figure 5 It is only an example of the smart device 5 and does not constitute a limitation on the smart device 5. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include input and output devices, network access devices, etc.

[0169] The processor 50 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0170] In some embodiments, the memory 51 may be an internal storage unit of the smart device 5, such as a hard disk or memory of the smart device 5. In other embodiments, the memory 51 may also be an external storage device of the smart device 5, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the smart device 5. Further, the memory 51 may also include both an internal storage unit and an external storage device of the smart device 5. The memory 51 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program. The memory 51 may also be used to temporarily store data that has been output or is to be output.

[0171] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0172] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the device / intelligent device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electric carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0173] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0174] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0175] In the embodiments provided in the present application, it should be understood that the disclosed devices / intelligent devices and methods can be implemented in other ways. For example, the device / intelligent device embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0176] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0177] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A language classification model training method, characterized in that: include: Acquiring a first acoustic feature, where the first acoustic feature refers to an acoustic feature of an audio segment, where the audio segment includes at least one character of a language category; Inputting the first acoustic feature into N layers of encoders of the initial language classification model in sequence to obtain first encoding features output by each of the N layers of encoders, where N is an integer greater than 1; Inputting each of the first coding features into a fusion module of the initial language classification model to obtain a first fusion feature; Based on the first fusion feature and the language recognition module of the initial language classification model, the initial language classification model is trained to obtain a target language classification model, and the target language classification model is used to identify the language category of the audio data.

2. The language classification model training method according to claim 1, characterized in that: Before acquiring the first acoustic feature, the method further includes: Obtain at least one audio training sample; Obtaining a timestamp of each of the characters in each of the audio training samples; For any of the audio training samples, during each training, the audio training sample is randomly cropped based on the timestamps of the characters in the audio training sample to obtain a plurality of the audio segments.

3. The language classification model training method according to claim 2, characterized in that: The randomly cutting the audio training samples based on the timestamps of the characters in the audio training samples during each training comprises: If the audio length of the audio training sample exceeds a length threshold, the audio training sample is randomly cropped based on the timestamp of each character in the audio training sample during each training.

4. The language classification model training method according to any one of claims 1 to 3, characterized in that: The language recognition module includes a first multi-head attention layer, a first attention statistical pooling layer, a second multi-head attention layer, and a second attention statistical pooling layer in sequence; the language recognition module based on the first fusion feature and the initial language classification model trains the initial language classification model to obtain a target language classification model, including: Inputting the first fused feature into the first multi-head attention layer to perform attention weighting on the time dimension of the first fused feature to obtain a second fused feature; Inputting the second fused feature into the first attention statistical pooling layer to fuse features at different time steps in the second fused feature to obtain a third fused feature; Inputting the third fused feature into the second multi-head attention layer to perform attention weighting on the encoder dimension of the third fused feature to obtain a fourth fused feature; Inputting the fourth fused feature into the second attention statistical pooling layer to fuse the encoded features output by the N layers of encoders contained in the fourth fused feature to obtain a fifth fused feature; Based on the fifth fusion feature, the initial language classification model is trained to obtain the target language classification model.

5. The language classification model training method according to claim 4, characterized in that: The step of training the initial language classification model based on the fifth fusion feature to obtain the target language classification model includes: When the parameters of the encoder of the N-layer are the first parameters and the parameters of the fusion module are the second parameters, based on the fifth fusion feature, updating the parameters of the language recognition module to be the third parameters; The initial language classification model obtained when the parameters of the N-layer encoder are the first parameters, the parameters of the fusion module are the second parameters, and the parameters of the language recognition module are the third parameters is used as the target language classification model.

6. A language recognition method, characterized in that: include: Obtain audio data to be recognized; Acquiring a second acoustic feature, where the second acoustic feature refers to an acoustic feature of the audio data to be recognized; Based on the second acoustic feature and the target language classification model, determine the language category to which the audio data to be recognized belongs; the target language classification model is obtained by training the initial language classification model based on the method described in any one of claims 1 to 5; wherein the target language classification model includes an N-layer encoder, a fusion module and a language recognition module, where N is an integer greater than 1; The N-layer encoder is used to: determine a corresponding second coding feature based on the second acoustic feature; The fusion module is used to: fuse the second coding features to obtain a target fusion feature; The language identification module is used to determine the language category to which the audio data to be identified belongs based on the target fusion feature.

7. A language classification model training device, characterized in that: include: A feature acquisition module, configured to acquire a first acoustic feature, wherein the first acoustic feature refers to an acoustic feature of an audio segment, wherein the audio segment includes at least one character of a language category; A feature encoding module, used for sequentially inputting the first acoustic feature into N layers of encoders of an initial language classification model to obtain first encoding features outputted by each of the N layers of encoders, where N is an integer greater than 1; A feature input module, used for inputting each of the first coding features into a fusion module of the initial language classification model to obtain a first fusion feature; A model training module is used to train the initial language classification model based on the first fusion feature and the language recognition module of the initial language classification model to obtain a target language classification model, and the target language classification model is used to identify the language category of the audio data.

8. A language recognition device, characterized in that: include: A data acquisition module, used to acquire audio data to be recognized; An acoustic acquisition module, used for acquiring a second acoustic feature, where the second acoustic feature refers to an acoustic feature of the audio data to be recognized; A language recognition module, configured to determine the language category to which the audio data to be recognized belongs based on the second acoustic feature and a target language classification model; the target language classification model is obtained by training an initial language classification model based on the method described in any one of claims 1 to 5; The target language classification model includes an N-layer encoder, a fusion module and a language recognition module, where N is an integer greater than 1; The N-layer encoder is used to: determine a corresponding second coding feature based on the second acoustic feature; The fusion module is used to: fuse the second coding features to obtain a target fusion feature; The language identification module is used to determine the language category to which the audio data to be identified belongs based on the target fusion feature.

9. An intelligent device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the smart device implements the language classification model training method as described in any one of claims 1 to 6, or implements the language recognition method as described in claim 7.

10. A computer program product, characterized in that It comprises a computer program, which, when executed, enables the language classification model training method as described in any one of claims 1 to 6 to be executed, or enables the language recognition method as described in claim 7 to be executed.

Images