Voice recognition engine switching method, language recognition model training method and device, computer equipment, storage medium and computer program product

Abstract

The invention relates to a voice recognition engine switching method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: recognizing a to-be-recognized audio stream by adopting a current voice recognition engine, and inputting the audio features of each audio frame in the to-be-recognized audio stream into a language recognition model to obtain the probability that each audio window belongs to a preset language; determining the number of target audio windows in a first preset number of continuous audio windows; when the number of the target audio windows is greater than or equal to a second preset number, determining that the to-be-recognized audio stream belongs to a preset language, and determining a voice recognition engine corresponding to the preset language as a target voice recognition engine; and when the target voice recognition engine is inconsistent with the current voice recognition engine, switching the current voice recognition engine to the target voice recognition engine, and recognizing the to-be-recognized audio stream through the target voice recognition engine. By adopting the method, the speech recognition effect of the multilingual speech interaction scene can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of audio processing, and particularly to a method for switching a speech recognition engine, a method for training a language recognition model, an apparatus, a computer device, a storage medium, and a computer program product. Background Art

[0002] With the development of deep learning technology, great progress has been made in language recognition technology, which has begun to be applied in various fields, such as smart homes, vehicle-mounted systems, and medical diagnosis.

[0003] In the scenario of large-screen voice interaction, different users may come from different regions, and the language types used by different users are diverse, and there may be voice interaction requirements in multiple languages. However, the current single speech recognition engine cannot provide multi-language voice interaction support, resulting in unsatisfactory speech recognition effects.

[0004] Therefore, there is a problem of poor speech recognition effect in the multi-language voice interaction scenario in the traditional technology. Summary of the Invention

[0005] Based on this, it is necessary to provide a method for switching a speech recognition engine, a method for training a language recognition model, an apparatus, a computer device, a storage medium, and a computer program product that can improve the speech recognition effect in the multi-language voice interaction scenario for the above technical problems.

[0006] A method for switching a speech recognition engine, the method includes:

[0007] Using the current speech recognition engine to recognize the audio stream to be recognized, and inputting the audio features of each audio frame in the audio stream to be recognized into a pre-trained language recognition model to obtain the probability that each audio window belongs to a preset language; each audio window includes a plurality of audio frames determined according to preset sliding window sampling parameters;

[0008] Determining the number of target audio windows in a continuous first preset number of audio windows; the target audio window is an audio window whose probability of belonging to a preset language is greater than a preset threshold;

[0009] In the case where the number of target audio windows is greater than or equal to a second preset number, determining that the audio stream to be recognized belongs to a preset language, and determining the speech recognition engine corresponding to the preset language as the target speech recognition engine;

[0010] In the case where the target speech recognition engine is inconsistent with the current speech recognition engine, switching the current speech recognition engine to the target speech recognition engine, and recognizing the audio stream to be recognized through the target speech recognition engine.

[0011] In one embodiment, there are multiple preset languages, and the method further includes:

[0012] When the number of target audio windows is less than the second preset number, input the audio stream to be recognized into the speech recognition engines corresponding to the preset languages, and obtain the recognition results output by the speech recognition engines corresponding to the preset languages;

[0013] Input the recognition result output by the speech recognition engine corresponding to any one of the preset languages into the language model corresponding to the preset language, and obtain the score information indicating that the audio stream to be recognized belongs to the preset language;

[0014] According to the score information indicating that the audio stream to be recognized belongs to each preset language, determine the target language among the preset languages, and determine the speech recognition engine corresponding to the target language as the target speech recognition engine.

[0015] In one embodiment, the pre-trained language recognition model includes a feature dimensionality reduction network and a feature classification network. The feature dimensionality reduction network includes N feature dimensionality reduction units and a flattening layer, where N is a positive integer greater than or equal to 2. Each feature dimensionality reduction unit includes a convolutional layer and a pooling layer. The feature classification network includes a gated recurrent unit, a fully connected layer, and an activation layer. Inputting the audio features of each audio frame in the audio stream to be recognized into the pre-trained language recognition model to obtain the probabilities that each audio window belongs to the preset languages includes:

[0016] Input the audio features of each audio frame into the feature dimensionality reduction network to obtain the dimensionality-reduced features;

[0017] Input the dimensionality-reduced features into the feature classification network to obtain the probabilities that each audio window belongs to the preset languages.

[0018] A method for training a language recognition model, the method includes:

[0019] Obtain a training sample set; any training sample in the training sample set includes the audio features of each audio frame in the audio sample, the language appearance situation label, and the language occurrence boundary label; the language appearance situation label represents each preset language that appears in the audio sample; the language occurrence boundary label includes the start and end points of the audio segments of each preset language in the audio sample, and the start and end points of the blank audio segments in the audio sample;

[0020] Input the audio features of each audio frame in the audio sample into the language recognition model to be trained, and obtain the probabilities that each audio window in the audio sample belongs to each preset language; each audio window includes multiple audio frames determined according to the preset sliding window sampling parameters;

[0021] Based on the probabilities of each audio window belonging to each preset language, a first prediction probability that the audio sample is a language occurrence boundary label is determined using a voice boundary operator, and, based on the probabilities of each audio window belonging to each preset language, a second prediction probability that the audio sample is a language occurrence situation label is aggregated using a linear aggregation model;

[0022] Based on the first prediction probability and the second prediction probability, loss information is determined;

[0023] Based on the loss information, the language recognition model to be trained is trained.

[0024] In one embodiment, a training sample set is obtained, including:

[0025] An original audio set is obtained; the original audio set includes original audios of each preset language;

[0026] Based on the original audio set, audio splicing is performed to obtain a plurality of spliced audios;

[0027] Noise addition processing is performed on each spliced audio to obtain a spliced audio with added noise corresponding to each spliced audio;

[0028] Audio amplitude smoothing processing is performed on the audio splicing part of each spliced audio with added noise to obtain each processed spliced audio;

[0029] Based on each processed spliced audio, a training sample set is generated.

[0030] In one embodiment, the language recognition model includes a feature dimensionality reduction network and a feature classification network. The feature dimensionality reduction network includes N feature dimensionality reduction units and a flattening layer, where N is a positive integer greater than or equal to 2. Each feature dimensionality reduction unit includes a convolutional layer and a pooling layer. The feature classification network includes a gated recurrent unit, a fully connected layer, and an activation layer. Inputting the audio features of each audio frame in the audio sample into the language recognition model to be trained to obtain the probabilities of each audio window belonging to each preset language includes:

[0031] Inputting the audio features of each audio frame in the audio sample into the feature dimensionality reduction network to obtain reduced-dimensional features;

[0032] Inputting the reduced-dimensional features into the feature classification network to obtain the probabilities of each audio window belonging to each preset language.

[0033] A switching device for a speech recognition engine, the device includes:

[0034] A feature input module, configured to use the current speech recognition engine to recognize the audio stream to be recognized, and input the audio features of each audio frame in the audio stream to be recognized into a pre-trained language recognition model to obtain the probabilities of each audio window belonging to a preset language; each audio window includes a plurality of audio frames determined according to preset sliding window sampling parameters;

[0035] A quantity determination module, configured to determine the quantity of target audio windows in a consecutive first preset quantity of audio windows; a target audio window is an audio window whose probability of belonging to a preset language is greater than a preset threshold.

[0036] A language determination module, configured to determine that the audio stream to be recognized belongs to a preset language and determine the speech recognition engine corresponding to the preset language as the target speech recognition engine when the quantity of target audio windows is greater than or equal to a second preset quantity.

[0037] An engine switching module, when the target speech recognition engine is inconsistent with the current speech recognition engine, switches the current speech recognition engine to the target speech recognition engine and recognizes the audio stream to be recognized through the target speech recognition engine.

[0038] A computer device, including a memory and a processor, where the memory stores a computer program, and when the processor executes the computer program, the steps of the above method are implemented.

[0039] A computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0040] A computer program product, including a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0041] The above method, device, computer device, storage medium, and computer program product for switching a speech recognition engine recognize an audio stream to be recognized by using a current speech recognition engine, input the audio features of each audio frame in the audio stream to be recognized into a pre-trained language recognition model, and obtain the probability that each audio window belongs to a preset language; each audio window includes a plurality of audio frames determined according to preset sliding window sampling parameters; in a continuous first preset number of audio windows, determine the number of target audio windows; a target audio window is an audio window whose probability of belonging to a preset language is greater than a preset threshold; in the case where the number of target audio windows is greater than or equal to a second preset number, determine that the audio stream to be recognized belongs to the preset language, and determine the speech recognition engine corresponding to the preset language as the target speech recognition engine; in the case where the target speech recognition engine is inconsistent with the current speech recognition engine, switch the current speech recognition engine to the target speech recognition engine, and recognize the audio stream to be recognized by using the target speech recognition engine; thus, it is possible to receive an audio stream in real time, quickly recognize the language to which the audio stream belongs, and in the case where the current speech recognition engine is not the speech recognition engine corresponding to the language, switch the current speech recognition engine to the speech recognition engine corresponding to the language, and quickly realize seamless language recognition and automatic switching of the speech recognition engine without complex operations, improving the speech recognition accuracy and the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following will briefly introduce the drawings required for use in the description of the embodiments or related technologies. Obviously, the drawings in the following description are only some embodiments of the present application, and those of ordinary skill in the art can obtain other drawings without creative efforts based on these drawings.

[0043] Figure 1 It is an application environment diagram of a method for switching a speech recognition engine in an embodiment;

[0044] Figure 2 It is a flowchart of a method for switching a speech recognition engine in an embodiment;

[0045] Figure 3 It is a flowchart of a method for switching a speech recognition engine in another embodiment;

[0046] Figure 4 It is a flowchart of a method for training a language recognition model in an embodiment;

[0047] Figure 5 It is a flowchart of a method for training a language recognition model in another embodiment;

[0048] Figure 6 Flow chart of a language switching system in one embodiment;

[0049] Figure 7 Schematic diagram of language localization performance of different tags under CTC loss in one embodiment;

[0050] Figure 8 Schematic diagram of the network structure of a language recognition model in one embodiment;

[0051] Figure 9 Schematic diagram of a first decision logic in one embodiment;

[0052] Figure 10 Block diagram of the structure of a switching device for a speech recognition engine in one embodiment;

[0053] Figure 11 Block diagram of the structure of a training device for a language recognition model in one embodiment;

[0054] Figure 12 Internal structure diagram of a computer device in one embodiment. Detailed implementation manners

[0055] In order to make the objectives, technical solutions and advantages of the present application clearer and more understandable, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0056] Speech recognition technology is widely used in various scenarios, such as smart homes, intelligent home appliances, intelligent cockpits, various voice assistants, subtitle generation, etc. However, there are some deficiencies in the existing speech recognition technology, mainly including: the single ability of the speech recognition engine, the cumbersome manual switching of the speech recognition engine, and the scarcity of language data.

[0057] In view of the problem of the single ability of the speech recognition engine, in the large-screen scenario, since the user groups come from all over the world and have diverse language needs, in order to meet the needs of different user groups, it is usually necessary to introduce multiple speech recognition engines. However, different speech recognition engines often only show better recognition effects in specific speech scenarios, lacking universality and flexibility, and there are often obvious limitations in complex and changeable actual application environments. For the application scenario of the home large screen, there are often requirements for multiple languages, and a single speech recognition engine is difficult to provide sufficient support, resulting in unsatisfactory recognition effects. Therefore, a single terminal device is often difficult to meet the speech recognition needs in a multi-language environment, and the capabilities of multiple speech recognition engines cannot be fully utilized.

[0058] Regarding the problem of the cumbersome manual switching of speech recognition engines, the current voice assistant APK based on the large TV screen has options for configuring multiple speech recognition engines. However, the method of manually switching speech recognition engines is cumbersome, inefficient, and the path is hidden, affecting the user experience and having a high operation complexity.

[0059] Regarding the problem of less language data, when recognizing speech in some languages, there are often challenges such as insufficient data volume, high difficulty in language annotation, long annotation cycle, and high manual annotation cost.

[0060] To solve the above problems, the embodiments of the present application provide a method for switching speech recognition engines, which solves the problem of poor speech recognition effect in multi-language demand scenarios caused by the single ability of speech recognition engines, the cumbersome manual switching of speech recognition engines, and the scarcity of language data.

[0061] The method for switching speech recognition engines provided by the embodiments of the present application can be applied to an application environment such as Figure 1 shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or placed in the cloud or other network servers. The server 104 uses the current speech recognition engine to recognize the audio stream to be recognized, and inputs the audio features of each audio frame in the audio stream to be recognized into a pre-trained language recognition model to obtain the probability that each audio window belongs to a preset language; each audio window includes a plurality of audio frames determined according to preset sliding window sampling parameters; the server 104 determines the number of target audio windows in a continuous first preset number of audio windows; the target audio window is an audio window whose probability of belonging to a preset language is greater than a preset threshold; the server 104 determines that the audio stream to be recognized belongs to the preset language when the number of target audio windows is greater than or equal to a second preset number, and determines the speech recognition engine corresponding to the preset language as the target speech recognition engine; when the target speech recognition engine is inconsistent with the current speech recognition engine, the server 104 switches the current speech recognition engine to the target speech recognition engine, and recognizes the audio stream to be recognized through the target speech recognition engine. Among them, the terminal 102 can be, but is not limited to, various personal computers, laptop computers, smart phones, tablet computers, Internet of Things devices, and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. The portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server 104 can be implemented by an independent server or a server cluster composed of multiple servers.

[0062] In an exemplary embodiment, as Figure 2 shown, a method for switching speech recognition engines is provided, and this method is applied toFigure 1 Taking the server 104 in [reference] as an example, the following steps S202 to S208 are included. Among them:

[0063] Step S202: Use the current speech recognition engine to recognize the audio stream to be recognized, and input the audio features of each audio frame in the audio stream to be recognized into the pre-trained language recognition model to obtain the probability that each audio window belongs to a preset language; each audio window includes a plurality of audio frames determined according to the preset sliding window sampling parameters.

[0064] Among them, the current speech recognition engine can be the default speech recognition engine. In practical applications, for people mainly using the first language, the default speech recognition engine can be set to the speech recognition engine corresponding to the first language, and for people mainly using the second language, the default speech recognition engine can be set to the speech recognition engine corresponding to the second language.

[0065] Among them, the audio stream to be recognized can refer to the real-time input streaming audio.

[0066] Among them, the audio features of each audio frame in the audio stream to be recognized can refer to the audio features obtained after feature extraction of the audio data of each audio sampling frame in the audio stream to be recognized.

[0067] Among them, the pre-trained language recognition model can refer to the language recognition model obtained by pre-training the language recognition model with multi-language training data, which can accurately identify the language to which the currently input streaming audio belongs.

[0068] Among them, the preset language can be any language, such as the first language, the second language, the third language, the fourth language, etc., and can be determined according to the multi-language requirements in the actual application scenario.

[0069] Among them, the audio window is a plurality of audio frames determined according to the preset sliding window sampling parameters.

[0070] Among them, the preset sliding window sampling parameters can include the size parameter of the audio window and the step length parameter of the audio window movement. For example, the size parameter of the audio window can be set to 28 frames, and the movement step length parameter of the audio window can be set to 4 frames. In this application, the receptive field in the network structure of the language recognition model can be 28 frames, and the frame shift can be 4 frames. Then, after inputting the audio features of 28 frames, the language recognition model aggregates a probability value, and this probability value is used as the probability value corresponding to the audio window formed by these 28 frames. In practical applications, according to the speech rate of four characters per second, 28 frames approximately correspond to the duration of one character. Therefore, the probability value of each audio window roughly represents the probability that a character belongs to the preset language. In the streaming audio detection, the language recognition model will perform a covering detection according to 24 frames.

[0071] Among them, the probability that an audio window belongs to a preset language can be the probability that the audio window belongs to the first language, the second language, the third language, or the fourth language.

[0072] Optionally, the current speech recognition engine (such as the speech recognition engine corresponding to the first language) is used to recognize the audio stream to be recognized. Among them, the audio features of each audio frame of the audio stream to be recognized will be cached in the data cache pool. After caching 28 frames, they will start to be input into the pre-trained language recognition model to obtain the probabilities that each audio window belongs to the first language, the second language, the third language, and the fourth language.

[0073] Step S204: Determine the number of target audio windows among a continuous first preset number of audio windows; the target audio window is an audio window whose probability of belonging to a preset language is greater than a preset threshold.

[0074] Among them, the first preset number can be set to 5, or can be set to other values according to actual needs.

[0075] Among them, the target audio window can refer to an audio window whose probability of belonging to the first language is greater than the preset threshold, or can refer to an audio window whose probability of belonging to the second language is greater than the preset threshold, or can refer to an audio window whose probability of belonging to the third language is greater than the preset threshold, or can refer to an audio window whose probability of belonging to the fourth language is greater than the preset threshold.

[0076] Among them, the preset threshold can be set to 0.7, or can be set to other values according to actual needs.

[0077] Optionally, taking the first preset number as 5, the preset threshold as 0.7, and the preset language as the third language as an example, among 5 consecutive audio windows, determine the number of these 5 audio windows whose probability of belonging to the first language is greater than 0.7.

[0078] Step S206: In the case where the number of target audio windows is greater than or equal to a second preset number, determine that the audio stream to be recognized belongs to the preset language, and determine the speech recognition engine corresponding to the preset language as the target speech recognition engine.

[0079] Among them, the second preset number can be set to 3, or can be set to other values according to actual needs.

[0080] Among them, the speech recognition engine corresponding to the preset language can refer to the speech recognition engine corresponding to the first language, the speech recognition engine corresponding to the second language, the speech recognition engine corresponding to the third language, the speech recognition engine corresponding to the fourth language, or the speech recognition engine corresponding to other languages.

[0081] Among them, the target speech recognition engine can refer to the speech recognition engine that matches the currently input audio stream to be recognized. For example, if it is detected that the audio stream to be recognized belongs to the third language, the target speech recognition engine is the speech recognition engine corresponding to the third language.

[0082] Optionally, taking the first preset quantity as 5, the second preset quantity as 3, the preset threshold as 0.7, and the preset language as the third language as an example, if the probability that each of the 5 consecutive audio windows is the third language is greater than 0.7, it is determined that the third language is the language of the audio stream to be recognized. Each 5 frames correspond to approximately 2.5 characters in the actual audio. In 5 consecutive frames, up to two probabilities lower than 0.7 (false negative cases) are allowed. That is, as long as it is satisfied that the probability that most of the audio windows belong to the third language exceeds the threshold of 0.7, it can be determined that the audio stream to be recognized belongs to the third language, and the speech recognition engine corresponding to the third language is determined as the target speech recognition engine.

[0083] Step S208, in the case where the target speech recognition engine is inconsistent with the current speech recognition engine, switch the current speech recognition engine to the target speech recognition engine, and recognize the audio stream to be recognized through the target speech recognition engine.

[0084] Optionally, in the case where the target speech recognition engine is the speech recognition engine corresponding to the third language and the current speech recognition engine is the speech recognition engine corresponding to the second language, it is determined that the target speech recognition engine is inconsistent with the current speech recognition engine. Then, it is necessary to switch the speech recognition engine corresponding to the second language to the speech recognition engine corresponding to the third language, and use the speech recognition engine corresponding to the third language to recognize the audio stream to be recognized.

[0085] In practical applications, in the case where the target speech recognition engine is consistent with the current speech recognition engine, there is no need to switch the speech recognition engine.

[0086] In the above method for switching the speech recognition engine, the to-be-recognized audio stream is recognized by using the current speech recognition engine, and the audio features of each audio frame in the to-be-recognized audio stream are input into a pre-trained language recognition model to obtain the probability that each audio window belongs to a preset language; each audio window includes a plurality of audio frames determined according to preset sliding window sampling parameters; in a continuous first preset number of audio windows, the number of target audio windows is determined; a target audio window is an audio window whose probability of belonging to the preset language is greater than a preset threshold; in the case where the number of target audio windows is greater than or equal to a second preset number, it is determined that the to-be-recognized audio stream belongs to the preset language, and the speech recognition engine corresponding to the preset language is determined as the target speech recognition engine; in the case where the target speech recognition engine is inconsistent with the current speech recognition engine, the current speech recognition engine is switched to the target speech recognition engine, and the to-be-recognized audio stream is recognized by the target speech recognition engine; thus, the audio stream can be received in real time, the language to which the audio stream belongs can be quickly recognized, and in the case where the current speech recognition engine is not the speech recognition engine corresponding to this language, the current speech recognition engine is switched to the speech recognition engine corresponding to this language, and the language recognition and the automatic switching of the speech recognition engine can be quickly realized without perception without complex operations, improving the speech recognition accuracy, and improving the user experience.

[0087] In an exemplary embodiment, there are multiple preset languages, and the method further includes: in the case where the number of target audio windows is less than the second preset number, inputting the to-be-recognized audio stream into the speech recognition engines corresponding to each preset language to obtain the recognition results output by the speech recognition engines corresponding to each preset language; inputting the recognition result output by any speech recognition engine corresponding to a preset language into the language model corresponding to the preset language to obtain the score information indicating that the to-be-recognized audio stream belongs to the preset language; according to the score information indicating that the to-be-recognized audio stream belongs to each preset language, determining a target language among the preset languages, and determining the speech recognition engine corresponding to the target language as the target speech recognition engine.

[0088] Wherein, the recognition result output by the speech recognition engine may be the speech content recognized by the speech recognition engine.

[0089] Wherein, the language model corresponding to the preset language may refer to the language models corresponding to the first language, the second language, the third language, and the fourth language, and the language model is used to score the speech content recognized by the speech recognition engine.

[0090] Wherein, the score information indicating that the to-be-recognized audio stream belongs to the preset language may refer to the score values indicating that the to-be-recognized audio belongs to the first language, the second language, the third language, and the fourth language.

[0091] Wherein, the target language may refer to any preset language.

[0092] Optionally, the audio data of the audio stream to be recognized is respectively input into the speech recognition engines of each preset language for recognition, and the recognition results output by the speech recognition engines of each preset language are obtained. Then, the recognition results output by the speech recognition engines of each preset language are scored by the N-gram language models of each preset language. By comparing the scoring values of the N-gram language models of each preset language, it is determined whether the preset language with the highest score meets the lowest score threshold of its corresponding N-gram language model. If it meets, it is determined that this language is the target language, and the speech recognition engine corresponding to this target language is used as the target speech recognition engine.

[0093] In this embodiment, when the number of target audio windows is less than the second preset number, the audio stream to be recognized is input into the speech recognition engines corresponding to each preset language, and the recognition results output by the speech recognition engines corresponding to each preset language are obtained; the recognition result output by the speech recognition engine corresponding to any preset language is input into the language model corresponding to the preset language to obtain the score information indicating that the audio stream to be recognized belongs to the preset language; according to the score information indicating that the audio stream to be recognized belongs to each preset language, the target language is determined among each preset language, and the speech recognition engine corresponding to the target language is determined as the target speech recognition engine. In this way, when the number of target audio windows is less than the second preset number, the recognition results output by the speech recognition engines corresponding to each preset language can be scored by combining the language model, and the speech recognition engine with the highest score is used as the target speech recognition engine. Through two determination rules, the false detection rate can be reduced without incurring additional costs.

[0094] In an exemplary embodiment, the pre-trained language recognition model includes a feature dimensionality reduction network and a feature classification network. The feature dimensionality reduction network includes N feature dimensionality reduction units and a flattening layer, where N is a positive integer greater than or equal to 2. Each feature dimensionality reduction unit includes a convolutional layer and a pooling layer. The feature classification network includes a gated recurrent unit, a fully connected layer, and an activation layer. Inputting the audio features of each audio frame in the audio stream to be recognized into the pre-trained language recognition model to obtain the probability that each audio window belongs to a preset language includes: inputting the audio features of each audio frame into the feature dimensionality reduction network to obtain the dimensionality-reduced features; inputting the dimensionality-reduced features into the feature classification network to obtain the probability that each audio window belongs to a preset language.

[0095] Among them, the feature dimensionality reduction network may refer to a network structure used to reduce the dimensionality of the audio features of each audio frame in the audio stream to be recognized.

[0096] Among them, the feature dimensionality reduction unit may be composed of a convolutional layer and a pooling layer. In the feature dimensionality reduction network, there may be multiple feature dimensionality reduction units.

[0097] Among them, the flattening layer can be a network structure used to flatten multi-dimensional features into a one-dimensional vector for input into a fully connected layer or other classification networks.

[0098] Among them, the gated recurrent unit can refer to the GRU unit.

[0099] Among them, the fully connected layer can refer to the fullconnect unit.

[0100] Among them, the activation layer can refer to the sigmood function.

[0101] Among them, the feature after dimensionality reduction can refer to the one-dimensional vector output by the flattening layer.

[0102] Optionally, the server inputs the audio features of each audio frame into the feature dimensionality reduction network of the language recognition model, performs dimensionality reduction processing through each feature dimensionality reduction unit in the feature dimensionality reduction network. The convolutional layer of the feature dimensionality reduction unit is used for convolutional processing, and the pooling layer is used for pooling the processing result output by the convolutional layer. The flattening layer in the feature dimensionality reduction network flattens the pooling result output by the pooling layer of the last feature dimensionality reduction unit to obtain the feature after dimensionality reduction. The server then inputs the feature after dimensionality reduction into the feature classification network of the language recognition model, captures the features in the feature after dimensionality reduction through the gated recurrent unit in the feature classification network, maps the features captured by the gated recurrent unit in the feature after dimensionality reduction to the category space through the fully connected layer in the feature classification network, and converts the output of the fully connected layer into a probability value through the activation layer in the feature classification network, so as to obtain the probability that each audio window belongs to a preset language.

[0103] In this embodiment, by inputting the audio features of each audio frame into the feature dimensionality reduction network, the feature after dimensionality reduction is obtained; by inputting the feature after dimensionality reduction into the feature classification network, the probability that each audio window belongs to a preset language is obtained. In this way, the probability that each audio window of the audio stream to be recognized belongs to a preset language can be accurately determined.

[0104] In another embodiment, as Figure 3 shown, a method for switching a speech recognition engine is provided. Taking the server 104 in Figure 1 as an example for illustration, the method includes the following steps:

[0105] Step S302, using the current speech recognition engine to recognize the audio stream to be recognized, and inputting the audio features of each audio frame in the audio stream to be recognized into a pre-trained language recognition model to obtain the probability that each audio window belongs to a preset language; each of the audio windows includes a plurality of audio frames determined according to preset sliding window sampling parameters.

[0106] Step S304, determine the number of target audio windows among a consecutive first preset number of the audio windows; the target audio windows are audio windows with a probability of belonging to the preset language greater than a preset threshold.

[0107] Step S306, when the number of the target audio windows is greater than or equal to a second preset number, determine that the audio stream to be recognized belongs to the preset language, and determine the speech recognition engine corresponding to the preset language as the target speech recognition engine.

[0108] Step S308, when the target speech recognition engine is inconsistent with the current speech recognition engine, switch the current speech recognition engine to the target speech recognition engine, and recognize the audio stream to be recognized through the target speech recognition engine.

[0109] Step S310, when the number of the target audio windows is less than the second preset number, input the audio stream to be recognized into the speech recognition engines corresponding to each of the preset languages, and obtain the recognition results output by the speech recognition engines corresponding to each of the preset languages.

[0110] Step S312, input the recognition result output by the speech recognition engine corresponding to any one of the preset languages into the language model corresponding to the preset language, and obtain the score information indicating that the audio stream to be recognized belongs to the preset language.

[0111] Step S314, according to the score information indicating that the audio stream to be recognized belongs to each of the preset languages, determine the target language among each of the preset languages, and determine the speech recognition engine corresponding to the target language as the target speech recognition engine.

[0112] It should be noted that the specific limitations of the above steps can refer to the specific limitations of a method for switching a speech recognition engine described above.

[0113] In an exemplary embodiment, as Figure 4 shown, a training method for a language recognition model is provided. Taking the method being applied to Figure 1 the server 104 as an example, it includes the following steps S402 to S408. Among them:

[0114] Step S402, obtain a training sample set; any training sample in the training sample set includes the audio features of each audio frame in the audio sample, the language appearance situation label, and the language occurrence boundary label; the language appearance situation label represents each preset language that has appeared in the audio sample; the language occurrence boundary label includes the start point and end point of the audio segment of each preset language in the audio sample, and the start point and end point of the blank audio segment in the audio sample.

[0115] Among them, the training sample set includes a large number of training samples. Each training sample includes an audio sample, the audio features of each audio frame of the audio sample, the language occurrence situation label of the audio sample, and the language occurrence boundary label of the audio sample.

[0116] Among them, the language occurrence situation label of the audio sample can be a label marked with the preset languages that appear in the audio sample, and the language occurrence situation label is an unordered label.

[0117] Among them, the language occurrence boundary label of the audio sample can be a label marked with the starting and ending points of the audio segments of each preset language in the audio sample, and the language occurrence boundary label is an ordered label. For example, in an audio, the speaker first speaks the first language and then the second language, then the language occurrence boundary label is defined as U s U e M s M e , where, U s is the starting point of the first language, U e is the ending point of the first language, M s is the starting point of the second language, M e is the ending point of the second language.

[0118] Optionally, the server obtains a training sample set including multiple preset languages.

[0119] Step S404: Input the audio features of each audio frame in the audio sample into the language recognition model to be trained, and obtain the probabilities that each audio window in the audio sample belongs to each preset language; each audio window includes multiple audio frames determined according to the preset sliding window sampling parameters.

[0120] Optionally, the server inputs the audio features of each audio frame in the audio sample into the language recognition model to be trained, and predicts the probabilities that each audio window in the audio sample belongs to each preset language through the language recognition model to be trained.

[0121] Step S406: Use the speech boundary operator to determine the first prediction probability that the audio sample is the language occurrence boundary label based on the probabilities that each audio window belongs to each preset language, and use the linear aggregation model to aggregate the second prediction probability that the audio sample is the language occurrence situation label based on the probabilities that each audio window belongs to each preset language.

[0122] Among them, the speech boundary operator can be an operator used to calculate and derive the probabilities of the occurrence boundaries of each audio window in the audio sample for each preset language.

[0123] Among them, the first prediction probability can be the probability that the audio sample is the language occurrence boundary label calculated and derived by the speech boundary operator.

[0124] Among them, the linear aggregation model can be a probability prediction model based on a linear pooling function, which is used to fuse the probabilities of each audio window to form an overall decision.

[0125] Among them, the second predicted probability can be the probability that the audio sample aggregated by the linear aggregation model is the language occurrence situation label.

[0126] Optionally, the server uses a speech boundary operator to determine the first predicted probability that the audio sample is the language occurrence boundary label based on the probabilities of each audio window belonging to each preset language, and, based on a linear pooling function, aggregates the probabilities of each audio window belonging to each preset language to obtain the second predicted probability that the audio sample is the language occurrence situation label.

[0127] Step S408, determine loss information according to the first predicted probability and the second predicted probability.

[0128] Among them, the loss information can be a loss value determined based on the first predicted probability and the second predicted probability.

[0129] Optionally, after determining the first predicted probability and the second predicted probability, the server determines the loss information loss(CTP) and the loss information loss(MIL) based on the first predicted probability, and determines the final loss information loss = loss(MIL) + loss(CTP) based on loss(CTP) and loss(MIL).

[0130] Step S410, train the language recognition model to be trained based on the loss information.

[0131] Optionally, the server trains the language recognition model to be trained based on the loss information loss.

[0132] The training method of the above language recognition model includes obtaining a training sample set; any training sample in the training sample set includes the audio features of each audio frame in the audio sample, the language occurrence situation label, and the language occurrence boundary label; the language occurrence situation label represents each preset language that appears in the audio sample; the language occurrence boundary label includes the start and end points of the audio segments of each preset language in the audio sample, and the start and end points of the blank audio segments in the audio sample; inputting the audio features of each audio frame in the audio sample into the language recognition model to be trained to obtain the probability that each audio window in the audio sample belongs to each preset language; each audio window includes a plurality of audio frames determined according to the preset sliding window sampling parameters; using a speech boundary operator to determine the first prediction probability that the audio sample is the language occurrence boundary label based on the probability that each audio window belongs to each preset language, and using a linear aggregation model to aggregate the second prediction probability that the audio sample is the language occurrence situation label based on the probability that each audio window belongs to each preset language; determining loss information according to the first prediction probability and the second prediction probability; training the language recognition model to be trained based on the loss information; in this way, the language recognition model can be jointly trained by combining the speech boundary operator and the linear aggregation model, thereby increasing the coverage of the training sample set and improving the accuracy of the language recognition model.

[0133] For the convenience of understanding by those skilled in the art, the following provides a training process of a language recognition model, which specifically includes:

[0134] Step 1: Perform various data augmentations on the training audio and extract MFCC features;

[0135] Step 2: Perform deep feature extraction through the talnet network structure;

[0136] Step 3: For the data with sequential labels, calculate loss(CTP) through the CTP loss function and the cross-entropy loss function loss(MIL) of multi-instance learning, and determine the final loss function based on loss(CTP) and loss(MIL); for the data without labels, only calculate the cross-entropy loss function loss(MIL) as the final loss function; this can reduce the label threshold of the data.

[0137] Step 4: Backpropagation and gradient optimization training.

[0138] In an exemplary embodiment, obtaining a training sample set includes: obtaining an original audio set; the original audio set includes the original audio of each preset language; performing audio splicing on the original audio set to obtain a plurality of spliced audios; performing noise addition processing on each spliced audio to obtain the spliced audio after noise addition corresponding to each spliced audio; performing audio amplitude smoothing processing on the audio splicing part of each spliced audio after noise addition to obtain each processed spliced audio; generating a training sample set based on each processed spliced audio.

[0139] Among them, the original audio set may refer to a data set formed by a large number of audio segments without any processing.

[0140] The original audio of this application comes from open-source data, and almost no additional manpower is required for data annotation, which can effectively save labor costs and maximize the use of existing resources. Compared with training a hybrid speech recognition model, the cost is significantly reduced.

[0141] Among them, the spliced audio can be an audio with multiple languages obtained by splicing audio segments of different preset languages.

[0142] Among them, the spliced audio after adding noise can be an audio obtained by adding random noise to an audio with multiple languages.

[0143] Among them, the processed spliced audio can be an audio obtained by smoothing an audio with multiple languages and added with random noise.

[0144] Optionally, the server obtains a certain number of audio segments of multiple preset languages, and then splices them according to random numbers and random orders. During the splicing process, random noise needs to be added at the front, middle, and back positions of the audio, and at the same time, smoothing processing of the audio amplitude is performed at the splicing positions to enhance the authenticity of the audio data.

[0145] In this embodiment, by obtaining the original audio set; the original audio set includes the original audio of each preset language; based on the original audio set, audio splicing is performed to obtain multiple spliced audios; noise addition processing is performed on each spliced audio to obtain the spliced audio after adding noise corresponding to each spliced audio; smoothing processing of the audio amplitude is performed at the audio splicing positions of each spliced audio after adding noise to obtain each processed spliced audio; based on each processed spliced audio, a training sample set is generated; in this way, the authenticity of each audio sample in the training sample set can be enhanced, thereby reducing the problem of model overfitting that may be brought by too many obvious silent segments, and thus improving the generalization ability of the model.

[0146] In an exemplary embodiment, the language recognition model includes a feature dimensionality reduction network and a feature classification network. The feature dimensionality reduction network includes N feature dimensionality reduction units and a flattening layer, where N is a positive integer greater than or equal to 2. Each feature dimensionality reduction unit includes a convolutional layer and a pooling layer. The feature classification network includes a gated recurrent unit, a fully connected layer, and an activation layer. Inputting the audio features of each audio frame in the audio sample into the language recognition model to be trained, obtaining the probabilities that each audio window in the audio sample belongs to each preset language, including: inputting the audio features of each audio frame in the audio sample into the feature dimensionality reduction network to obtain the dimensionality-reduced features; inputting the dimensionality-reduced features into the feature classification network to obtain the probabilities that each audio window belongs to each preset language.

[0147] Optionally, the server inputs the audio features of each audio frame in the audio sample into the feature dimensionality reduction network of the language recognition model to be trained, and performs dimensionality reduction processing through each feature dimensionality reduction unit in the feature dimensionality reduction network. The convolutional layer of the feature dimensionality reduction unit is used for convolutional processing, and the pooling layer is used for pooling the processing result output by the convolutional layer. The flattening layer in the feature dimensionality reduction network flattens the pooling result output by the pooling layer of the last feature dimensionality reduction unit to obtain the dimensionality-reduced features. The server then inputs the dimensionality-reduced features into the feature classification network of the language recognition model to be trained, captures the features in the dimensionality-reduced features through the gated recurrent unit in the feature classification network, maps the features captured by the gated recurrent unit in the dimensionality-reduced features to the category space through the fully connected layer in the feature classification network, and converts the output of the fully connected layer into a probability value through the activation layer in the feature classification network, so as to obtain the probability that each audio window in the audio sample belongs to a preset language.

[0148] In this embodiment, by inputting the audio features of each audio frame in the audio sample into the feature dimensionality reduction network, dimensionality-reduced features are obtained; by inputting the dimensionality-reduced features into the feature classification network, the probability that each audio window belongs to each preset language is obtained; in this way, the probability that each audio window of the audio sample belongs to the preset language can be accurately determined.

[0149] In another embodiment, as Figure 5 shown, a training method for a language recognition model is provided. Taking the server 104 in Figure 1 as an example, the method includes the following steps:

[0150] Step S502, obtaining an original audio set; the original audio set includes the original audio of each preset language.

[0151] Step S504, performing audio splicing based on the original audio set to obtain a plurality of spliced audios.

[0152] Step S506, performing noise addition processing on each spliced audio to obtain a spliced audio with added noise corresponding to each spliced audio.

[0153] Step S508, performing audio amplitude smoothing processing on the audio splicing part of each spliced audio with added noise to obtain each processed spliced audio.

[0154] Step S510, generating a training sample set based on each processed spliced audio; any training sample in the training sample set includes the audio features of each audio frame in the audio sample, a language appearance situation label, and a language occurrence boundary label; the language appearance situation label represents each preset language that has appeared in the audio sample; the language occurrence boundary label includes the start point and end point of the audio segment of each preset language in the audio sample, and the start point and end point of the blank audio segment in the audio sample.

[0155] Step S512: Input the audio features of each audio frame in the audio sample into the language recognition model to be trained, and obtain the probabilities that each audio window in the audio sample belongs to each preset language; each audio window includes multiple audio frames determined according to preset sliding window sampling parameters.

[0156] Step S514: Use a voice boundary operator to determine the first prediction probability that the audio sample is a language occurrence boundary label based on the probabilities that each audio window belongs to each preset language, and use a linear aggregation model to aggregate the second prediction probability that the audio sample is a language occurrence situation label based on the probabilities that each audio window belongs to each preset language.

[0157] Step S516: Determine the loss information according to the first prediction probability and the second prediction probability.

[0158] Step S518: Train the language recognition model to be trained based on the loss information.

[0159] It should be noted that the specific limitations of the above steps can refer to the specific limitations of a method for training a language recognition model described above.

[0160] This application aims to solve the problem of multi-language speech recognition in the large screen scenario, and proposes a method for switching a speech recognition engine, which can flexibly switch the language recognition engine in the large screen scenario. By designing a new architecture, this method can dynamically select the most suitable speech recognition engine according to different speech features of the input audio, thereby improving the accuracy of speech recognition and enhancing the user experience.

[0161] This application optimizes the commonly used loss function CTC (Connectionist Temporal Classification) loss function in speech recognition to make it better adapt to the application scenario of multi-language recognition. The advantage of the CTC loss function is that it can automatically align the input data with the labels without explicit label alignment information, solving the problem of label alignment in traditional speech recognition. By minimizing the total path cost and peak clustering characteristics, the CTC function can calculate the most likely alignment position of the labels. However, the CTC loss function mainly focuses on the order of the labels rather than accurately locating the starting point of the speech event, which makes it have limitations in dealing with multi-language recognition tasks that require accurate positioning of the starting point of the language.

[0162] To address this issue, this application improves the CTC loss function in combination with the special requirements of multi-language recognition, enabling it to accurately locate the starting point of a language solely through ordered tags. This optimization method not only retains the advantages of the CTC loss function in label alignment but also overcomes its deficiencies in language starting point localization, thereby achieving more efficient and accurate language recognition, which is named CTP (Connectionist Temporal Position).

[0163] This application performs language detection for language recognition in streaming speech. To reduce model misdetection and prevent frequent incorrect replacement of the speech engine, based on the above improved loss function, two decision-making logics are used to assist the final decision for each frame prediction result of the model. The first decision-making logic is rule-based. The prediction value of each frame must be greater than a certain threshold and last for a certain length to give the final decision. If the first decision-making logic cannot give a result, the second decision-making logic is triggered. The corresponding speech recognition engine is used in combination with the corresponding ngram language model to score and determine the final language category, and the corresponding recognition result is updated to the final recognition result on the terminal side.

[0164] For the convenience of understanding by those skilled in the art, the following uses four languages, namely the first language, the second language, the third language, and the fourth language, as examples to illustrate the overall method architecture of this application. The specific implementation process structure is as Figure 6 shown, mainly consisting of three modules: Module 1 is the data and label processing module; Module 2 is the joint training based on the CTP loss function and the multi-instance learning method; Module 3 is the inference and decision-making process of language recognition. Specifically:

[0165] (1) Module 1, the data and label processing module, includes three main steps: audio labeling, audio construction, and audio preprocessing.

[0166] The purpose of audio sample labeling is to obtain the ordered labels (language occurrence order sequence labels) and unordered labels (language appearance situation labels) of the audio samples. The ordered labels are sequential annotations. During the labeling process, it is necessary to annotate the boundary order of different languages in an audio sample to obtain the sequential labels of the audio sample. For example, if an audio consists of the first language spoken first and then the second language, the boundary labels of language occurrence are defined as the start point U of the first language s 、the end point U of the first language e 、the start point M of the second language s 、the end point M of the second language e , that is, U s U e M s M e。Unordered tags mean with or without annotation. During the tagging process, only the languages that appear in an audio sample need to be shown, without annotating the order in which different languages appear. Table 1 below provides the specific tagging differences between ordered and unordered tags.

[0167] Table 1

[0168]

[0169]

[0170] The purpose of audio construction is to obtain a certain number of audio samples. During the construction process, only audio segments of four languages in a certain quantity need to be obtained, and then they are spliced randomly in terms of quantity and order. During the splicing process, noise needs to be randomly added at the front, middle, and end positions of the audio, and at the same time, the audio amplitude at the splicing points is smoothed to enhance the authenticity of the audio data. This method can effectively reduce the problem of model overfitting that may be caused by too many obvious silent segments, thereby improving the generalization ability of the model.

[0171] The purpose of audio preprocessing is to extract and amplify audio features. During the preprocessing process, the audio needs to be pre-emphasized first. Framing is performed with 25 ms as one frame and 10 ms as the frame shift to maintain a partial overlap. The Hamming window is used to smooth the input speech signal and the fast Fourier transform (FFT) is used to obtain frequency-domain features. The signal is filtered using the Mel filter bank (Mel Banks) to obtain FBank features. After obtaining the FBank features of each frame, in order to obtain the up and down correlations of the time series signal, framing with a unit of 5 is performed. In order to reduce the computational complexity, after every 5 frames are framed, 1 frame can be skipped for the next segment of framing.

[0172] (2) Module 2 is jointly trained based on the CTP loss function and the multi-instance learning method, including the training process based on CTP and the training process based on multi-instance learning.

[0173] It is found from the statistical data that more than 80% of the start point tags and end point tags in the training data are adjacent. Therefore, the CTC loss function will misinterpret this pair of tags as the same language tag and cannot recognize them as the start and end nodes of a language. Secondly, the CTC loss function only stipulates the order of the predicted tags without time interval restrictions. The network will be more inclined to simply output the start and end tags closely together without considering whether there should be a certain interval between them in actual time. The specific training performance is as Figure 7As shown, whether there is a language label or not, only one peak will be predicted for the target language. The sequential start and end labels tend to predict the start and end labels in close positions. Ideal label prediction needs to accurately locate the start and end points of the language audio in order to accurately identify the start and end points of different languages. This application optimizes the CTC loss to make it more suitable for the corresponding scenario and makes certain adjustments to the forward algorithm. Generally speaking, in the part trained based on the CTP loss function, the frame-level probability of each language is output by using the TALNET network structure, and a fixed algorithm is added in the next layer to derive the language boundary probability. Then, the total probability of the true label sequence is calculated according to these frame-level boundary probabilities through the forward algorithm.

[0174] In the training process based on CTP, the TALNET network structure is used, as Figure 8 shown. After the GRU layer, the fully connected layer outputs the frame-level probability of each language (i.e., the probability of each audio frame belonging to each preset language). Then, a boundary operator is added in the next layer of the fully connected layer to derive the frame-level boundary probability of each language (i.e., the probability of the language boundary of each audio frame being different preset languages). Finally, the forward algorithm calculates the total probability of the true label sequence according to the frame-level boundary probability of each language. Among them, the boundary operator derives the probability of the language boundary of each audio frame being a preset language from the probability of each audio frame being a preset language. For example, y t (l) represents the probability that the preset language l appears in the t-th frame in the range of 1 < t < T, where T represents the total number of frames of an audio sample. The start point probability z t (l s ) and the end point probability z t (l e ) can be calculated by the following formula:

[0175] z t (l s ) = max[0, y t (l) - y t-1 (l)];

[0176] z t (l e ) = max[0, y t-1 (l) - y t (l)];

[0177] Among them, the value range of t is [1, T + 1] to accommodate the language audio starting from the first frame or ending at the last frame. y0(l) and y T+1 (l) are assumed to be 0. For the prediction value of a certain frame, the language boundary probabilities z t(l) may sum to more than 1. The probabilities of language boundaries for each frame being of different preset languages can be considered independent. Based on this assumption, the probability that each frame does not belong to the language boundary of any preset language can be determined (analogous to the probability that each frame is the blank label "blank" in CTC), expressed as: Among them, l represents the start boundary or end boundary of all preset languages, and the probability of the boundary occurrence of a single preset language l is expressed as: To simplify the formula, the following definitions are made: Then the probability of the boundary occurrence of a single preset language l in the formula can be converted to: p t (l) = ∈ t ·δ t (l).

[0178] The independence assumption of the boundary occurrence probabilities of different preset languages in the same frame above no longer requires the blank label probability of CTC, but the original CTC blank label has two functions: one is to allow no language to occur in a certain frame; the other is to distinguish continuously repeated identical labels. Under the premise of the independence assumption, the first function has been achieved. For the second point, the frame-level labels can be mapped to coarse-grained language order labels, and continuously repeated labels are no longer merged into a single label. Through this simplification, the blank label can be completely removed. The independence assumption can help evaluate the probability of emitting multiple labels in the same frame, which cannot be done in standard Connectionist Temporal Classification (CTC). The probability of emitting multiple labels simultaneously at each frame is:

[0179]

[0180] The optimized loss function CTP of this application aims to find the total probability of the true label sequence L = l1,..., l k , regardless of which frames can output which labels. Given the probability y t (l) of the language label, through the above formula, the probability p t (l1,..., l k ) of outputting zero, one, or multiple labels for each frame can be deduced. Define a t (i) as the probability of exactly outputting the first i labels in the sequence L after the t-th frame. Then the recurrence formula for the derivative of a t (i) is:

[0181]

[0182] In the summation operation, the index j represents the number of language labels output at the t-th frame, and the initial value is: The negative logarithm of the final a T+1 (L) is the value of the loss function for optimization. The final loss function is as follows:

[0183] loss(CTP) = -log(a T+1 (L)).

[0184] It should be noted that this application has significant advantages in training cost. The core technology lies in optimizing the CTC loss function. On the basis of the automatic alignment of the CTC loss, the present invention further solves the peak clustering problem. By combining the CTP loss with the boundary operator, low-cost model training is achieved.

[0185] Based on the training process of multi-instance learning, since the above CTP method depends on ordered labels, in order to make full use of training data from various sources and increase the fitting ability of the language identification model to complex data, the present invention combines the multi-instance learning method with lower requirements for data annotation to jointly train the language identification model, reduce the model training threshold, increase the coverage of training data, and improve the accuracy of the final identification. The multi-instance learning (MIL) framework deals with classification tasks with unclear sample annotations. In sound event detection, an entire recording can be regarded as a "bag", and each frame in it is regarded as a sample, thus transforming it into an MIL problem. The model first predicts the probability that each sample is positive, and then converts these probabilities into the probability that the entire "bag" is positive through an aggregation function, so as to minimize the difference between the prediction result and the unordered label of the bag.

[0186] Similar to the training process based on CTP, the TALNET network structure is also used. After the GRU layer, the fully connected layer outputs the frame-level probability of each language (that is, the probability that each audio frame belongs to each preset language), and its matrix size is [T*L], where T is the total number of currently output frames, and L is the total number of languages for model training. The predicted probability y of each frame t , represents the predicted value of the t-th frame signal. Since the labels used here are with or without labels, the model needs to aggregate the probability values of all frames to generate the overall label corresponding to the audio segment. This aggregation process uses a linear pooling function to fuse the probabilities of each frame and finally form an overall decision. The specific calculation formula is The corresponding cross-entropy loss function is loss(MIL) = -llogy - (1 - l)log(1 - y), where l is the true label and y is the output of the aggregation function. Finally, this application uses the following joint loss to train the language identification model: loss = loss(MIL) + loss(CTP).

[0187] (3) The inference and determination process of module three language identification involves the first determination logic and the second determination logic.

[0188] The network structure of the language identification model of the present invention is as Figure 5 shown. Its receptive field is 28 frames, and the frame shift is 4 frames. After inputting 28 frames, the language identification model can aggregate the probability of one frame. According to the speaking speed of four characters per second, 28 frames approximately correspond to the duration of one character. Therefore, the probability prediction of each frame roughly represents the probability that a character belongs to a certain preset language. And in the streaming detection, the language identification model will perform overlapping detection based on 24 frames.

[0189] In the first decision logic, the probability threshold of each frame is set to 0.7. If a certain preset language meets the requirement that the probability is greater than 0.7 in 5 consecutive frames, then it is determined that this language is the language of the current audio. Every 5 frames correspond to approximately 2.5 characters in the actual audio. If in 5 consecutive frames, at most two probabilities lower than 0.7 (false negative cases) are allowed, that is, as long as the probabilities of most frames exceed the threshold, the final decision will be given. If the decision condition is still not met based on 5 consecutive frames, the counter is cleared and the subsequent frames are continued to be detected. If the language decision is still not given finally, the second decision logic is entered.

[0190] In the second decision logic, as Figure 6 shown, when the first decision logic cannot make a decision, a speech recognition engine is further combined with an N-gram language model for secondary scoring to determine the most suitable language. The specific process is as Figure 6 shown: First, the audio data is input into all speech recognition engines for recognition respectively, and the recognition results of each engine are input into the corresponding N-gram language model for scoring. To optimize the execution speed, a 2-gram language model can be used for language model scoring. Then, by comparing the language model scores of each language, it is determined whether the language with the highest score meets the lowest threshold of the corresponding language model. If it meets, it is determined that this language is the final language, and this result is used as the basis for language switching. Throughout the process, the default speech engine is always used on the terminal side for streaming recognition and result display. For the languages of non-default engines, the final decision result will overwrite the preliminary recognition result on the terminal side, thus realizing the automatic recognition function of multiple languages.

[0191] This application optimizes the model prediction accuracy through a multi-level decision logic, avoiding the impact of frequent false detections on the user experience. Among them, the first decision logic focuses on reducing the misjudgment of the acoustic model and can effectively screen out the correct language; the second decision logic can provide a fallback guarantee when the first decision logic cannot make a decision, focusing on preventing false negatives. The test results show that this application has an extremely low false detection rate and high recognition accuracy, demonstrating strong practicability.

[0192] It should be understood that although the steps in the flowcharts involved in the above-described embodiments are sequentially shown according to the arrows, these steps are not necessarily executed sequentially in the order indicated by the arrows. Unless there is a clear indication in this article, the execution of these steps has no strict order limit, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-described embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be executed alternately or in turn with at least a part of other steps or steps or stages in other steps.

[0193] Based on the same inventive concept, an embodiment of the present application further provides a voice recognition engine switching device for implementing the above-mentioned voice recognition engine switching method. The solution provided by this device to solve the problem is similar to the solution described in the above method. Therefore, the specific limitations in one or more embodiments of the voice recognition engine switching device provided below can refer to the limitations on the voice recognition engine switching method in the above text, and will not be repeated here.

[0194] In an exemplary embodiment, as Figure 10 shown, a voice recognition engine switching device is provided, including: a feature input module 1002, a quantity determination module 1004, a language determination module 1006, and an engine switching module 1008, where:

[0195] The feature input module 1002 is configured to recognize the audio stream to be recognized using the current voice recognition engine, and input the audio features of each audio frame in the audio stream to be recognized into a pre-trained language recognition model to obtain the probability that each audio window belongs to a preset language; each audio window includes a plurality of audio frames determined according to preset sliding window sampling parameters;

[0196] The quantity determination module 1004 is configured to determine the quantity of target audio windows in a continuous first preset quantity of audio windows; the target audio window is an audio window whose probability of belonging to a preset language is greater than a preset threshold;

[0197] The language determination module 1006 is configured to determine that the audio stream to be recognized belongs to a preset language when the quantity of target audio windows is greater than or equal to a second preset quantity, and determine the voice recognition engine corresponding to the preset language as the target voice recognition engine;

[0198] The engine switching module 1008, when the target voice recognition engine is inconsistent with the current voice recognition engine, switches the current voice recognition engine to the target voice recognition engine, and recognizes the audio stream to be recognized using the target voice recognition engine.

[0199] In one embodiment, there are multiple preset languages. The language determination module 1006 is further configured to, when the number of target audio windows is less than a second preset number, input the audio stream to be recognized into the speech recognition engines corresponding to the preset languages, and obtain the recognition results output by the speech recognition engines corresponding to the preset languages; input the recognition result output by the speech recognition engine corresponding to any one of the preset languages into the language model corresponding to the preset language, and obtain the score information indicating that the audio stream to be recognized belongs to the preset language; determine the target language from the preset languages according to the score information indicating that the audio stream to be recognized belongs to the preset languages, and determine the speech recognition engine corresponding to the target language as the target speech recognition engine.

[0200] In one embodiment, the pre-trained language recognition model includes a feature dimensionality reduction network and a feature classification network. The feature dimensionality reduction network includes N feature dimensionality reduction units and a flattening layer, where N is a positive integer greater than or equal to 2. Each feature dimensionality reduction unit includes a convolutional layer and a pooling layer. The feature classification network includes a gated recurrent unit, a fully connected layer, and an activation layer. The feature input module 1002 is specifically configured to input the audio features of each audio frame into the feature dimensionality reduction network to obtain the dimensionality-reduced features; input the dimensionality-reduced features into the feature classification network to obtain the probabilities that each audio window belongs to the preset languages.

[0201] Based on the same inventive concept, an embodiment of the present application further provides a training device for a language recognition model for implementing the training method of the language recognition model involved above. The implementation solutions provided by this device for solving problems are similar to the implementation solutions described in the above method. Therefore, the specific limitations in one or more embodiments of the training device for the language recognition model provided below can refer to the limitations on the training method of the language recognition model in the above text, and will not be repeated here.

[0202] In an exemplary embodiment, as Figure 11 shown, a training device for a language recognition model is provided, including: an acquisition module 1102, an input module 1104, a prediction module 1106, a determination module 1108, and a training module 1110, where:

[0203] The acquisition module 1102 is configured to acquire a training sample set; any training sample in the training sample set includes the audio features of each audio frame in the audio sample, a language appearance situation label, and a language occurrence boundary label; the language appearance situation label represents each preset language that appears in the audio sample; the language occurrence boundary label includes the start and end points of the audio segments of each preset language in the audio sample, and the start and end points of the blank audio segments in the audio sample;

[0204] An input module 1104 for inputting the audio features of each audio frame in an audio sample into a language recognition model to be trained, and obtaining the probabilities that each audio window in the audio sample belongs to each preset language; each audio window includes a plurality of audio frames determined according to preset sliding window sampling parameters;

[0205] A prediction module 1106 for determining a first prediction probability that the audio sample is a language occurrence boundary label based on the probabilities that each audio window belongs to each preset language by using a voice boundary operator, and aggregating a second prediction probability that the audio sample is a language occurrence situation label based on the probabilities that each audio window belongs to each preset language by using a linear aggregation model;

[0206] A determination module 1108 for determining loss information according to the first prediction probability and the second prediction probability;

[0207] A training module 1110 for training the language recognition model to be trained based on the loss information.

[0208] In one embodiment, an acquisition module 1102 is specifically configured to acquire an original audio set; the original audio set includes original audios of each preset language; perform audio splicing on the original audio set to obtain a plurality of spliced audios; perform noise addition processing on each spliced audio to obtain a spliced audio with added noise corresponding to each spliced audio; perform audio amplitude smoothing processing on the audio splicing part of each spliced audio with added noise to obtain each processed spliced audio; generate a training sample set based on each processed spliced audio.

[0209] In one embodiment, the language recognition model includes a feature dimensionality reduction network and a feature classification network. The feature dimensionality reduction network includes N feature dimensionality reduction units and a flattening layer, where N is a positive integer greater than or equal to 2. Each feature dimensionality reduction unit includes a convolutional layer and a pooling layer. The feature classification network includes a gated recurrent unit, a fully connected layer, and an activation layer. The input module 1104 is specifically configured to input the audio features of each audio frame in the audio sample into the feature dimensionality reduction network to obtain reduced-dimensional features; input the reduced-dimensional features into the feature classification network to obtain the probabilities that each audio window belongs to each preset language.

[0210] Each module in the above voice recognition engine switching device and language recognition model training device can be implemented in whole or in part by software, hardware, and their combination. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0211] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as Figure 12As shown in the figure. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O), and a communication interface. Among them, the processor, the memory, and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store the switching of the speech recognition engine and the training data of the language recognition model. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals through a network connection. When the computer program is executed by the processor, it is used to implement a method for switching a speech recognition engine and a method for training a language recognition model.

[0212] Those skilled in the art can understand that Figure 12 the structure shown in the figure is only a block diagram of some structures related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0213] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program. When the computer program is executed by the processor, the processor is caused to execute the steps of the above-mentioned method for switching a speech recognition engine and the method for training a language recognition model. Here, the steps of the method for switching a speech recognition engine and the method for training a language recognition model may be the steps in the method for switching a speech recognition engine and the method for training a language recognition model in the above-mentioned various embodiments.

[0214] In one embodiment, a computer-readable storage medium is provided, storing a computer program. When the computer program is executed by the processor, the processor is caused to execute the steps of the above-mentioned method for switching a speech recognition engine and the method for training a language recognition model. Here, the steps of the method for switching a speech recognition engine and the method for training a language recognition model may be the steps in the method for switching a speech recognition engine and the method for training a language recognition model in the above-mentioned various embodiments.

[0215] In one embodiment, a computer program product is provided, including a computer program, which, when executed by a processor, causes the processor to execute the steps of the above-mentioned method for switching a speech recognition engine and the method for training a language recognition model. Here, the steps of the method for switching a speech recognition engine and the method for training a language recognition model may be the steps in the method for switching a speech recognition engine and the method for training a language recognition model in each of the above embodiments.

[0216] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it may include the processes of the embodiments of the above-mentioned methods. Among them, any reference to a memory, database, or other medium used in the various embodiments provided in the present application may include at least one of non-volatile and volatile memories. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The databases involved in the various embodiments provided in the present application may include at least one of relational databases and non-relational databases. Non-relational databases may include distributed databases based on blockchain, etc., without limitation. The processors involved in the various embodiments provided in the present application may be general-purpose processors, central processing units, graphics processors, digital signal processors, programmable logic devices, data processing logics based on quantum computing, etc., without limitation.

[0217] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered to be within the scope described in this specification.

[0218] The above-described embodiments merely represent several implementation manners of the present application. The description thereof is relatively specific and detailed, but it should not be construed as a limitation on the patent scope of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several variations and improvements can still be made, and these all fall within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the appended claims.

Claims

1. A method for switching a speech recognition engine, characterized in that, The method includes: Using a current speech recognition engine to recognize an audio stream to be recognized, and inputting the audio features of each audio frame in the audio stream to be recognized into a pre-trained language recognition model to obtain the probability that each audio window belongs to a preset language; each of the audio windows includes a plurality of audio frames determined according to preset sliding window sampling parameters; Determining the number of target audio windows in a continuous first preset number of the audio windows; the target audio window is an audio window whose probability of belonging to the preset language is greater than a preset threshold; When the number of the target audio windows is greater than or equal to a second preset number, determining that the audio stream to be recognized belongs to the preset language, and determining the speech recognition engine corresponding to the preset language as the target speech recognition engine; When the target speech recognition engine is inconsistent with the current speech recognition engine, switching the current speech recognition engine to the target speech recognition engine, and recognizing the audio stream to be recognized through the target speech recognition engine.

2. The method according to claim 1, wherein There are multiple preset languages, and the method further includes: When the number of the target audio windows is less than the second preset number, inputting the audio stream to be recognized into the speech recognition engines corresponding to each of the preset languages to obtain the recognition results output by the speech recognition engines corresponding to each of the preset languages; Inputting the recognition result output by any one of the speech recognition engines corresponding to the preset languages into the language model corresponding to the preset language to obtain the score information indicating that the audio stream to be recognized belongs to the preset language; According to the score information indicating that the audio stream to be recognized belongs to each of the preset languages, determining a target language among each of the preset languages, and determining the speech recognition engine corresponding to the target language as the target speech recognition engine.

3. The method according to claim 1, wherein The pre-trained language recognition model includes a feature dimensionality reduction network and a feature classification network. The feature dimensionality reduction network includes N feature dimensionality reduction units and a flattening layer, where N is a positive integer greater than or equal to 2. Each feature dimensionality reduction unit includes a convolutional layer and a pooling layer. The feature classification network includes a gated recurrent unit, a fully connected layer, and an activation layer. The step of inputting the audio features of each audio frame in the audio stream to be recognized into the pre-trained language recognition model to obtain the probability that each audio window belongs to a preset language includes: Inputting the audio features of each of the audio frames into the feature dimensionality reduction network to obtain the dimensionality-reduced features; Inputting the dimensionality-reduced features into the feature classification network to obtain the probability that each audio window belongs to the preset language.

4. A training method for a language identification model, characterized in that, The method includes: Obtaining a training sample set; any training sample in the training sample set includes the audio features of each audio frame in the audio sample, a language appearance situation label, and a language occurrence boundary label; the language appearance situation label represents each preset language that has appeared in the audio sample; the language occurrence boundary label includes the start point and end point of the audio segment of each preset language in the audio sample, and the start point and end point of the blank audio segment in the audio sample; Input the audio features of each audio frame in the audio sample into the language recognition model to be trained, and obtain the probabilities that each audio window in the audio sample belongs to each of the preset languages; each of the audio windows includes multiple audio frames determined according to preset sliding window sampling parameters. Use a speech boundary operator to determine the first prediction probability that the audio sample is a boundary label of the language occurrence based on the probabilities that each audio window belongs to each of the preset languages, and use a linear aggregation model to aggregate the second prediction probability that the audio sample is a label of the language occurrence situation based on the probabilities that each audio window belongs to each of the preset languages. Determine loss information according to the first prediction probability and the second prediction probability. Train the language recognition model to be trained based on the loss information.

5. The method according to claim 4, wherein The obtaining of the training sample set includes: Obtain an original audio set; the original audio set includes original audios of each of the preset languages. Perform audio splicing on the original audio set to obtain a plurality of spliced audios. Perform noise addition processing on each of the spliced audios to obtain a spliced audio with added noise corresponding to each of the spliced audios. Perform audio amplitude smoothing processing on the audio splicing part of each of the spliced audios with added noise to obtain each processed spliced audio. Generate the training sample set based on each of the processed spliced audios.

6. The method according to claim 4, wherein The language recognition model includes a feature dimensionality reduction network and a feature classification network. The feature dimensionality reduction network includes N feature dimensionality reduction units and a flattening layer, where N is a positive integer greater than or equal to 2. Each feature dimensionality reduction unit includes a convolutional layer and a pooling layer. The feature classification network includes a gated recurrent unit, a fully connected layer, and an activation layer. The step of inputting the audio features of each audio frame in the audio sample into the language recognition model to be trained and obtaining the probabilities that each audio window in the audio sample belongs to each of the preset languages includes: Input the audio features of each of the audio frames in the audio sample into the feature dimensionality reduction network to obtain reduced-dimensional features. Input the reduced-dimensional features into the feature classification network to obtain the probabilities that each audio window belongs to each of the preset languages.

7. A switching device for a speech recognition engine, characterized in that, The device includes: A feature input module, configured to use a current speech recognition engine to recognize an audio stream to be recognized, and input the audio features of each audio frame in the audio stream to be recognized into a pre-trained language recognition model to obtain the probabilities that each audio window belongs to a preset language; each of the audio windows includes multiple audio frames determined according to preset sliding window sampling parameters. A quantity determination module, configured to determine the quantity of target audio windows in a continuous first preset quantity of the audio windows; the target audio window is an audio window whose probability of belonging to the preset language is greater than a preset threshold. A language determination module, configured to determine that the audio stream to be recognized belongs to the preset language when the quantity of the target audio windows is greater than or equal to a second preset quantity, and determine the speech recognition engine corresponding to the preset language as the target speech recognition engine. The engine switching module, in the case where the target speech recognition engine is inconsistent with the current speech recognition engine, switches the current speech recognition engine to the target speech recognition engine, and recognizes the audio stream to be recognized through the target speech recognition engine.

8. A computer device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method described in any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that, When this computer program is executed by a processor, it implements the steps of the method described in any one of claims 1 to 6.

Images