Multilingual recognition method and system based on asr information

Abstract

The multilingual recognition method and system based on ASR information, the method carries out silence detection on test data, removes the silence segment, and obtains non-silence data; the Fbank feature and the pitch feature are extracted from the obtained non-silence data, the extracted Fbank feature and the pitch feature are sent to a multilingual mixed ASR model, and the Encoder output information is obtained through the multilingual mixed ASR model; the obtained Encoder output information is sent to a multilingual classification model, and the language type prediction result is obtained through the multilingual classification model. The training process carries out speed disturbance, volume disturbance, Gaussian noise adding and spectral enhancement data enhancement processing. The model language recognition performance is improved; in the resource scarce and high confusion task, high performance can also be achieved, and the recognition accuracy under the existing general technology is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of language recognition, and in particular to a multi-language recognition method and system based on ASR information. BACKGROUND

[0002] In recent years, small language speech recognition has attracted more and more attention. According to statistics, small languages used by less than one million people account for 80% of all languages in the world. However, among the mature speech recognition products at present, only 4% of all languages in the world are covered. As a pre-processing module of multi-language speech recognition, the performance of language recognition directly affects the speech recognition result, and small language recognition plays a key role in the downstream speech recognition task. Therefore, the research on small languages is particularly important.

[0003] In the traditional language classification method, MFCC, Fbank and other features are usually used to represent the information of the input audio, or the original audio is directly convolved to obtain the features. These methods have limited classification information of the audio, which limits the upper limit of the performance of the classification model. In actual production, we can usually obtain the ASR (Automatic Speech Recognition Technology, which can convert human speech into text) text labeling information of multi-language audio. How to use these effective information to improve the language classification accuracy is of practical significance. SUMMARY

[0004] Therefore, the present application provides a multi-language recognition method and system based on ASR information, which solves the problem of low language classification accuracy caused by data scarcity, easy confusion, limited audio classification information, and limited classification model performance in actual application scenarios.

[0005] In order to achieve the above purpose, the present application provides the following technical scheme: a multi-language recognition method based on ASR information, comprising:

[0006] Performing silence detection on the test data, and obtaining non-silence data after removing the silence segments;

[0007] Extracting Fbank features and Pitch features from the obtained non-silence data, and inputting the extracted Fbank features and Pitch features into a multi-language mixed ASR model to obtain Encoder output information through the multi-language mixed ASR model;

[0008] Inputting the obtained Encoder output information into a multi-language classification model to obtain a language prediction result through the multi-language classification model.

[0009] As a preferred scheme of the multi-language recognition method based on ASR information, the training steps of the multi-language mixed ASR model comprise:

[0010] The ASR multi-lingual data is processed, and the audio files of each language are mixed at equal proportions to obtain training data.

[0011] Character granularity is used to construct pronunciation dictionaries of each language, and then the multi-lingual pronunciation dictionaries are combined to construct a unified pronunciation dictionary.

[0012] Fbank features and pitch features are extracted from the training data, and an end-to-end multi-lingual mixed ASR model is constructed and fed with data for training.

[0013] As an optimal scheme of the multi-lingual recognition method based on ASR information, the training step of the multi-lingual classification model comprises:

[0014] The original multi-lingual training data is subjected to silence detection, and the silence segments are removed.

[0015] Data enhancement processing is performed to expand the training data, Fbank features and pitch features are extracted, and the multi-lingual mixed ASR model is fed with the Encoder output information.

[0016] The Encoder output information is taken as features and fed into the multi-lingual classification model for training.

[0017] As an optimal scheme of the multi-lingual recognition method based on ASR information, the data enhancement processing comprises speed disturbance, volume disturbance, Gaussian noise addition and spectrum enhancement.

[0018] The Fbank features contain correlation information between audio spectrum features, and the pitch features contain the fundamental frequency features of the speaker.

[0019] As an optimal scheme of the multi-lingual recognition method based on ASR information, the multi-lingual mixed ASR model adopts a conformer structure, the multi-lingual mixed ASR model comprises 12 layers of Encoder and 6 layers of Decoder, and a CTC+Attention Decoder loss joint training method is used during training.

[0020] The multi-lingual classification model adopts a 7-layer tdnn structure, the first 5 layers are frame-level calculation, and the last 2 layers are segment-level calculation, and the predicted language category results are obtained through softmax output.

[0021] The application further provides a multi-lingual recognition system based on ASR information, comprising a recognition unit, wherein the recognition unit comprises:

[0022] The test data silence detection module is used for performing silence detection on the test data, and obtaining non-silence data after removing the silence segments.

[0023] a feature extraction module configured to extract Fbank features and Pitch features from the obtained non-silence data;

[0024] an ASR module configured to transmit the extracted Fbank features and Pitch features to a multi-lingual mixed ASR model to obtain Encoder output information through the multi-lingual mixed ASR model;

[0025] a language recognition module configured to transmit the obtained Encoder output information to a multi-lingual classification model to obtain a language prediction result through the multi-lingual classification model.

[0026] As a preferred solution of the multi-lingual recognition system based on ASR information, the system further comprises an ASR model training unit configured to train the multi-lingual mixed ASR model, including:

[0027] a training data processing module configured to process ASR multi-lingual data, and mix audio files of each language at a same proportion to obtain training data;

[0028] a pronunciation dictionary construction module configured to construct pronunciation dictionaries of each language in a character granularity, and then merge the multi-lingual pronunciation dictionaries to construct a unified pronunciation dictionary;

[0029] a first training module configured to extract Fbank features and Pitch features from the training data, construct an end-to-end multi-lingual mixed ASR model, and input data for training.

[0030] As a preferred solution of the multi-lingual recognition system based on ASR information, the system further comprises a classification model training unit configured to train the multi-lingual classification model, including:

[0031] a training data silence detection module configured to perform silence detection on original multi-lingual training data, and remove silence segments;

[0032] a data enhancement module configured to perform data enhancement processing and expand training data;

[0033] an Encoder processing module configured to extract Fbank features and Pitch features, input the features into the trained multi-lingual mixed ASR model, and obtain Encoder output information;

[0034] a second training module configured to input the Encoder output information as features into the multi-lingual classification model for training.

[0035] As a preferred solution of the multi-lingual recognition system based on ASR information, the data enhancement processing of the data enhancement module includes speed disturbance, volume disturbance, Gaussian noise addition, and spectrum enhancement;

[0036] Among the feature extraction module, the first training module and the second training module, the Fbank feature contains correlation information between audio spectrum features; and the Pitch feature contains a fundamental frequency feature of a speaker.

[0037] As a preferred solution of the multilingual recognition system based on ASR information, the multilingual mixed ASR model adopts a conformer structure, the multilingual mixed ASR model includes 12 layers of Encoder and 6 layers of Decoder, and a CTC+Attention Decoder loss joint training method is adopted during training.

[0038] The multilingual classification model adopts a 7-layer tdnn structure, the first 5 layers are frame-level calculation, and the last 2 layers are segment-level calculation, and a predicted language category result is obtained through softmax output.

[0039] The application has the following advantages: the test data is subjected to silence detection, and after the silence segments are removed, non-silence data is obtained; the obtained non-silence data is subjected to Fbank feature and Pitch feature extraction, the extracted Fbank feature and Pitch feature are fed into a multilingual mixed ASR model, Encoder output information is obtained through the multilingual mixed ASR model; the obtained Encoder output information is fed into a multilingual classification model, and a language prediction result is obtained through the multilingual classification model. The application improves the language recognition performance of the model; in a resource-scarce and high-confusion task, high performance can also be achieved, and the recognition accuracy under the existing general technology is improved. BRIEF DESCRIPTION OF DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the application or the technical solutions in the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are only exemplary, and for those skilled in the art, other drawings can also be derived from the provided drawings without creative labor.

[0041] The structures, proportions, sizes, etc. shown in the specification are only used to cooperate with the content disclosed in the specification, to be understood and read by those skilled in the art, and do not define the limiting conditions for the implementation of the application, so they do not have technical significance. Any modification of structure, change of proportion relationship or adjustment of size, without affecting the effect and purpose that the application can produce, should still fall within the scope of the technical content disclosed by the application.

[0042] Figure 1 The multilingual recognition method based on ASR information provided for embodiment 1 of the application is shown in the flowchart.

[0043] Figure 2 A multi-language mixed ASR model training flowchart in a multi-language recognition method based on ASR information provided for Embodiment 1 of the present application;

[0044] Figure 3 A multi-language classification model training flowchart in a multi-language recognition method based on ASR information provided for Embodiment 1 of the present application;

[0045] Figure 4 A multi-language recognition system based on ASR information provided in Embodiment 2 of the present application. DETAILED DESCRIPTION

[0046] The embodiments of the present application will be described in detail by specific embodiments, and those skilled in the art can easily understand other advantages and effects of the present application from the content disclosed in the specification. Obviously, the described embodiments are part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0047] Language recognition is a technology for automatically recognizing the language category of input speech signal, which plays an irreplaceable role in practical application. Language recognition can be used as a pre-processing module of multi-language speech recognition, and directly affects the final effect of speech recognition.

[0048] Language recognition belongs to a technology of pattern recognition in the field of artificial intelligence, mainly including model training and model decision. In the actual application scene of language recognition, there is a problem that the similarity between several languages is high. The features extracted by the traditional recognition method do not consider the difference between different languages, and perform poorly in high confusion tasks, which cannot achieve satisfactory results.

[0049] Therefore, the present application provides a multi-language recognition technical scheme based on ASR information, which fully utilizes ASR information, refines the original features, further excavates the difference between different languages, and improves the discrimination ability of the model.

[0050] Embodiment 1

[0051] Referring to Figure 1 , Embodiment 1 of the present application provides a multi-language recognition method based on ASR information, comprising:

[0052] S11, performing silence detection on test data, and obtaining non-silence data after removing the silence segment;

[0053] S12, extracting Fbank features and Pitch features from the obtained non-silence data;

[0054] S13, deliver the extracted Fbank features and Pitch features to a multi-lingual mixed ASR model, and obtain Encoder output information through the multi-lingual mixed ASR model;

[0055] S14, deliver the obtained Encoder output information to a multi-lingual classification model, and obtain a language prediction result through the multi-lingual classification model.

[0056] In this embodiment, the silence detection adopts the method of NN-VAD, the training data is based on the frame-level annotation information obtained by forced alignment of the speech recognition model, and a 4-layer TDNN structure is used to train the human voice and silence classification model. The multi-lingual training and test data are detected by VAD to exclude the influence of non-human voice data on the system.

[0057] In this embodiment, the multi-lingual mixed ASR model adopts a conformer structure, and the multi-lingual mixed ASR model includes 12 layers of Encoder and 6 layers of Decoder. During training, a CTC+Attention Decoder loss joint training method is used.

[0058] In this embodiment, the feature extraction adopts a combination method of 80-dimensional Fbank+3-dimensional Pitch. Experiments prove that the Pitch feature plays an important role in multi-lingual speech recognition. If the Pitch feature is removed, the model training is not easy to converge.

[0059] In this embodiment, the multi-lingual classification model adopts a 7-layer tdnn structure, the first 5 layers are frame-level calculation, and the last 2 layers are segment-level calculation. The predicted language category results are obtained through softmax output. Among them, the slice size is set to 300, that is, 3s audio is classified as a segment.

[0060] Referring to Figure 2 In this embodiment, the training steps of the multi-lingual mixed ASR model include:

[0061] S21, processing ASR multi-lingual data, and mixing each language audio file at a same proportion to obtain training data;

[0062] S22, constructing a pronunciation dictionary of each language in character granularity, and then merging the multi-lingual pronunciation dictionaries to construct a unified pronunciation dictionary;

[0063] S23, extracting Fbank features and Pitch features from the training data;

[0064] S24, constructing an end-to-end multi-lingual mixed ASR model, and delivering the data for training.

[0065] Specifically, the training process of the multi-language mixed ASR model first extracts features from the training data, specifically including Fbank features and Pitch features, wherein the Fbank features contain the correlation information between audio spectrum features, and the Pitch features contain the fundamental frequency features of the speaker; then an end-to-end framework with a conformer structure is built, including an Encoder and a Decoder, the features are input for training, and a CTC+Attention Decoder loss joint training method is adopted.

[0066] Referring to Figure 3 In this embodiment, the training step of the multi-language classification model includes:

[0067] S31, performing silence detection on the original multi-language training data to remove the silent segments;

[0068] S32, performing data enhancement processing to expand the training data;

[0069] S33, extracting Fbank features and Pitch features, and inputting them into the trained multi-language mixed ASR model to obtain Encoder output information;

[0070] S34, inputting the Encoder output information as features into the multi-language classification model for training.

[0071] Specifically, in the training process of the multi-language classification model, first, the data is subjected to VAD silence detection to filter out pure silent segments and retain segments with speech sounds; considering the problem of data scarcity in model training, data enhancement processing is performed, including speed variation, amplitude modulation, noise addition, and spectral enhancement, which alleviates the problem of insufficient training data; then feature extraction is performed, which is the same as the training of the multi-language mixed ASR model; then the extracted Fbank features and Pitch features are input into the multi-language mixed ASR model to obtain the Encoder output result; finally, the Encoder output of the multi-language mixed ASR model is taken as features (which plays a role in feature refinement) and input into the multi-language classification model for classification training.

[0072] Specifically, in the data enhancement processing process, first, speed disturbance is performed, and the disturbance parameter is randomly selected between 0.9 and 1.1, that is, the original data is processed at 0.9 or 1.1 times the speed. Secondly, the volume disturbance is disturbed, and the disturbance parameter is randomly selected from -12db to 12db. Then add Gaussian noise, and the signal-to-noise ratio is randomly selected from 10 to 30. Finally, the spectrum is enhanced. First, the time domain signal is processed by fft to obtain the time-frequency domain signal. Then, the time domain and frequency domain are randomly masked, and finally the ifft processing is performed to obtain the time domain signal. Through the above enhancement method, 4 times of enhanced data can be expanded. Experiments prove that the performance of the combination method of randomly selecting 2 times of data amount of the original data from the enhanced data and the original data is optimal, and the performance will decrease with more disturbance data.

[0073] In summary, the application detects the test data for silence, obtains non-silence data after removing the silence segment, extracts Fbank features and Pitch features from the obtained non-silence data, inputs the extracted Fbank features and Pitch features into a multi-language mixed ASR model, obtains Encoder output information through the multi-language mixed ASR model, inputs the obtained Encoder output information into a multi-language classification model, and obtains a language prediction result through the multi-language classification model. In the aspect of silence detection, the application adopts a VAD method based on NN to solve the problem of inaccurate silence detection in a noisy environment. In the aspect of data processing, the application adopts methods such as noise addition, speed variation, amplitude modulation, and spectrum enhancement to effectively alleviate the problem of scarce training data in actual tasks. The application uses a multi-language mixed ASR model as a feature extractor to further refine the original features into more information conducive to language classification, further improving the model recognition performance. In the resource-scarce and high-confusion task, the application can also achieve high performance, improving the recognition accuracy under the existing general technology.

[0074] It should be noted that the method of the embodiment of the present disclosure can be executed by a single device, such as a computer or a server. The method of the embodiment can also be applied in a distributed scenario, and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only execute one or more steps in the method of the embodiment of the present disclosure, and the multiple devices will interact with each other to complete the method.

[0075] It is to be understood that the foregoing description is directed to some embodiments of the disclosure. Other embodiments fall within the scope of the following claims. In some cases, the actions or steps recited in the claims can be performed in a different order and still accomplish the desired result. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order to achieve the desired results. In certain implementations, multitasking and parallel processing can be advantageous.

[0076] Embodiment 2

[0077] Referring to Figure 4 Embodiment 2 of the present disclosure provides a multi-lingual recognition system based on ASR information, comprising a recognition unit 1, the recognition unit 1 comprising:

[0078] A test data silence detection module 11 is configured to perform silence detection on the test data, and obtain non-silence data after removing the silence segments.

[0079] A feature extraction module 12 is configured to perform Fbank feature and pitch feature extraction on the obtained non-silence data.

[0080] An ASR module 13 is configured to deliver the extracted Fbank feature and pitch feature to a multi-lingual mixed ASR model, and obtain Encoder output information through the multi-lingual mixed ASR model.

[0081] A language recognition module 14 is configured to deliver the obtained Encoder output information to a multi-lingual classification model, and obtain a language prediction result through the multi-lingual classification model.

[0082] In this embodiment, an ASR model training unit 2 is further included, which is configured to train the multi-lingual mixed ASR model, comprising:

[0083] A training data processing module 21 is configured to process ASR multi-lingual data, and obtain training data by mixing audio files of each language at a same proportion.

[0084] A pronunciation dictionary construction module 22 is configured to construct pronunciation dictionaries of each language in character granularity, and then merge the multi-lingual pronunciation dictionaries to construct a unified pronunciation dictionary.

[0085] A first training module 23 is configured to extract Fbank feature and pitch feature from the training data, construct an end-to-end multi-lingual mixed ASR model, and deliver the data for training.

[0086] In this embodiment, a classification model training unit 3 is further included, which is configured to train the multi-lingual classification model, comprising:

[0087] The training data silence detection module 31 is configured to perform silence detection on the original multi-lingual training data, and remove the silence segments.

[0088] The data enhancement module 32 is configured to perform data enhancement processing to expand the training data.

[0089] The encoder processing module 33 is configured to extract Fbank features and pitch features, and input the Fbank features and the pitch features into the trained multi-lingual mixed ASR model to obtain encoder output information.

[0090] The second training module 34 is configured to input the encoder output information as features into a multi-lingual classification model for training.

[0091] In the embodiment, the data enhancement processing performed by the data enhancement module 32 includes speed perturbation, volume perturbation, Gaussian noise addition, and spectrum enhancement.

[0092] In the feature extraction module 12, the first training module 23, and the second training module 34, the Fbank features include correlation information between audio spectrum features, and the pitch features include speaker fundamental frequency features.

[0093] In the embodiment, the multi-lingual mixed ASR model adopts a conformer structure, the multi-lingual mixed ASR model includes 12 layers of encoders and 6 layers of decoders, and a CTC+Attention Decoder loss joint training method is used during training.

[0094] The multi-lingual classification model adopts a 7-layer tdnn structure, the first 5 layers are frame-level calculations, the last 2 layers are segment-level calculations, and a softmax output is used to obtain predicted results of each language category.

[0095] It should be noted that the information interaction and execution process between the modules / units of the system described above are based on the same concept as the method embodiments in Embodiment 1 of the present application, and the technical effects brought by the method embodiments are the same as those of the method embodiments of the present application. For specific content, refer to the description of the method embodiments described above.

[0096] Embodiment 3

[0097] Embodiment 3 of the present application provides a non-transitory computer readable storage medium, the computer readable storage medium stores a program code of a multi-lingual recognition method based on ASR information, and the program code includes instructions for executing the multi-lingual recognition method based on ASR information in Embodiment 1 or any possible implementation manner thereof.

[0098] The computer-readable storage medium can be any available media or a set of one or more available media that is accessible by a computer, a server, a data center, etc. data storage device integrated. The available media can be a magnetic medium, (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk (SSD)), etc.

[0099] Embodiment 4

[0100] Embodiment 4 of the present application provides an electronic device, comprising: a memory and a processor;

[0101] The processor and the memory complete the communication between each other through the bus; the memory stores program instructions executable by the processor, and the processor calling the program instructions can execute the multilingual recognition method based on ASR information of embodiment 1 or any possible implementation manner thereof.

[0102] Specifically, the processor can be implemented by hardware or software. When implemented by hardware, the processor can be a logic circuit, an integrated circuit, etc. When implemented by software, the processor can be a general-purpose processor, which realizes by reading software codes stored in the memory. The memory can be integrated in the processor or exist independently outside the processor.

[0103] In the above embodiments, all or part of them can be realized by software, hardware, firmware or any combination thereof. When realized by software, all or part of them can be realized in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another, for example, the computer instructions can be transferred from one website, computer, server or data center to another through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode.

[0104] It should be apparent to those skilled in the art that the modules or steps of the application described above can be implemented with a general purpose computing device, which can be centralized on a single computing device or distributed across a network of multiple computing devices, and optionally implemented with program code executable by a computing device, which can be stored in a storage device and executed by a computing device, and in some cases, the steps shown or described can be performed in an order different from that shown, or made into individual integrated circuit modules, or multiple modules or steps made into a single integrated circuit module. Thus, the application is not limited to any particular combination of hardware and software.

[0105] While the application has been described in detail and with reference to specific embodiments thereof, it will be apparent to those skilled in the art that various modifications and changes can be made therein without departing from the spirit of the application. Accordingly, it is intended that all such modifications and changes be included within the scope of the application as claimed.

Claims

1. A multilingual recognition method based on ASR information, characterized in that: include: Perform silence detection on the test data and remove the silent segments to obtain the non-silent data; Extracting Fbank features and Pitch features from the obtained non-silent data, transmitting the extracted Fbank features and Pitch features to a multilingual hybrid ASR model, and obtaining encoder output information through the multilingual hybrid ASR model; Transmitting the obtained encoder output information to a multilingual classification model, and obtaining a language prediction result through the multilingual classification model; The training steps of the multilingual classification model include: Perform silence detection on the original multilingual training data and remove silent segments; Perform data augmentation processing to expand the training data; extract Fbank features and Pitch features, and feed them into the trained multilingual hybrid ASR model to obtain Encoder output information; The encoder output information is used as features and fed into the multilingual classification model for training; Data enhancement processing includes speed perturbation, volume perturbation, Gaussian noise and spectral enhancement; The Fbank feature contains the correlation information between audio spectrum features; the Pitch feature contains the speaker's fundamental frequency feature; The multilingual hybrid ASR model adopts a conformer structure, including a 12-layer encoder and a 6-layer decoder, and is trained using a CTC+Attention Decoderloss joint training method. The multilingual classification model adopts a 7-layer TDNN structure. The first 5 layers are frame-level calculations, and the last 2 layers are segment-level calculations. The predicted results of each language category are obtained through softmax output. Feature extraction uses a combination of 80-dimensional Fbank + 3-dimensional Pitch methods; during the data enhancement process, speed perturbation is first performed, and the perturbation parameter is randomly selected between 0.9 and 1.1, that is, the original data is speed-changed by 0.9 or 1.1 times; secondly, volume perturbation is performed, and the perturbation parameter is randomly selected from -12dB to 12dB; then Gaussian noise is added, and the signal-to-noise ratio is randomly selected from 10 to 30; finally, spectrum enhancement is performed, first the time domain signal is FFT processed to obtain the time-frequency domain signal; then the time domain and frequency domain are randomly masked respectively, and finally ifft processing is performed to obtain the time domain signal.

2. The multilingual recognition method based on ASR information according to claim 1, characterized in that: The training steps of the multilingual hybrid ASR model include: Process ASR multilingual data and mix audio files of different languages ​​in equal proportions to obtain training data; Use character granularity to build pronunciation dictionaries for each language, and then merge multiple language pronunciation dictionaries to build a unified pronunciation dictionary; Extract Fbank features and Pitch features from the training data; build an end-to-end multilingual hybrid ASR model and input data for training.

3. A multilingual recognition system based on ASR information, characterized by: The invention comprises an identification unit, wherein the identification unit comprises: The test data silence detection module is used to perform silence detection on the test data and obtain the non-silent data after removing the silent segments; Feature extraction module, used to extract Fbank features and Pitch features from the obtained non-silent data; An ASR module, configured to transmit the extracted Fbank features and Pitch features to a multilingual hybrid ASR model, and obtain encoder output information through the multilingual hybrid ASR model; A language identification module is used to transmit the obtained encoder output information to a multilingual classification model and obtain a language prediction result through the multilingual classification model; The system further includes a classification model training unit, which is used for training a multilingual classification model and includes: The training data silence detection module is used to detect silence in the original multilingual training data and remove silent segments; Data enhancement module, used to perform data enhancement processing and expand training data; The encoder processing module is used to extract Fbank features and Pitch features, and input them into the trained multilingual hybrid ASR model to obtain encoder output information; The second training module is used to feed the encoder output information as features into the multilingual classification model for training; The data enhancement module performs data enhancement processing including speed disturbance, volume disturbance, Gaussian noise addition and spectrum enhancement; In the feature extraction module and the second training module, the Fbank feature includes correlation information between audio spectrum features; the Pitch feature includes the fundamental frequency feature of the speaker; The multilingual hybrid ASR model adopts a conformer structure, including a 12-layer encoder and a 6-layer decoder, and is trained using a CTC+Attention Decoderloss joint training method. The multilingual classification model adopts a 7-layer TDNN structure. The first 5 layers are frame-level calculations, and the last 2 layers are segment-level calculations. The predicted results of each language category are obtained through softmax output. Feature extraction uses a combination of 80-dimensional Fbank + 3-dimensional Pitch methods; during the data enhancement process, speed perturbation is first performed, and the perturbation parameter is randomly selected between 0.9 and 1.1, that is, the original data is speed-changed by 0.9 or 1.1 times; secondly, volume perturbation is performed, and the perturbation parameter is randomly selected from -12dB to 12dB; then Gaussian noise is added, and the signal-to-noise ratio is randomly selected from 10 to 30; finally, spectrum enhancement is performed, first the time domain signal is FFT processed to obtain the time-frequency domain signal; then the time domain and frequency domain are randomly masked respectively, and finally ifft processing is performed to obtain the time domain signal.

4. The multilingual recognition system based on ASR information according to claim 3, characterized in that: The system further includes an ASR model training unit, which is used for training the multilingual mixed ASR model and includes: The training data processing module is used to process ASR multilingual data and mix audio files of different languages ​​in equal proportions to obtain training data; A pronunciation dictionary construction module is used to construct pronunciation dictionaries for each language using character granularity, and then merge the pronunciation dictionaries of multiple languages ​​to construct a unified pronunciation dictionary; The first training module is used to extract Fbank features and Pitch features from training data; build an end-to-end multilingual hybrid ASR model, and input data for training.

Images