Speech recognition method, device and system, electronic equipment and storage medium

Abstract

The invention provides a voice recognition method, device and system, electronic equipment and a storage medium, and the method is applied to terminal equipment, and comprises the steps: carrying out the acoustic feature extraction of a voice signal based on the language information of the voice signal, obtaining an acoustic feature, decoding the acoustic feature, and obtaining a plurality of initial recognition results of the voice signal; determining a voice recognition result of the voice signal; the speech recognition result is obtained by applying a speech recognition model to perform semantic error correction on the basis of each initial recognition result, the acoustic feature and the language information; a speech recognition model is constructed on the basis of a large-scale language model, the defects that current multi-language speech recognition is low in accuracy and prone to misjudgment are overcome, through a two-step recognition process, a plurality of initial recognition results are generated locally and rapidly, and then the speech recognition accuracy is improved through the strong semantic understanding ability of the large-scale model. And the acoustic features and the language information are fused to carry out multi-modal deep error correction, so that the accuracy of multi-language speech recognition is greatly improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of natural language processing, and in particular to a speech recognition method, device, system, electronic device and storage medium. BACKGROUND

[0002] With the acceleration of globalization, cross-lingual academic exchanges and travel are becoming more frequent, and the demand for cross-lingual communication is increasing. In this case, multilingual speech recognition technology is attracting attention, which is expected to break down language barriers and achieve efficient and convenient communication. However, the current multilingual speech recognition technology still has obvious bottlenecks and cannot meet the diversified needs of practical applications.

[0003] At present, in order to improve the performance of multilingual speech recognition, the common way is to build an end-to-end deep learning model, such as a sequence-to-sequence model based on attention mechanism. This model can directly map the speech signal to the text sequence, without the need for explicit training of acoustic models and language models, simplifying the training process. However, different languages have great differences in speech features, grammatical structures, vocabulary semantics, etc. For example, there are a large number of similar phonemes in some languages or unique grammatical structures, which bring great difficulties to the optimization of the model, making it difficult to accurately process multiple languages at the same time, especially some small languages and languages with complex speech characteristics. The recognition accuracy of the model is low, and misjudgment is prone to occur. SUMMARY

[0004] The present application provides a speech recognition method, device, system, electronic device and storage medium to solve the problem that the prior art cannot accurately process multiple languages at the same time, the accuracy of multilingual speech recognition is low, and misjudgment is prone to occur, thereby improving the accuracy of multilingual speech recognition.

[0005] The present application provides a speech recognition method, comprising: determining a speech signal to be recognized and language information of the speech signal; based on the language information, performing acoustic feature extraction on the speech signal to obtain acoustic features of the speech signal, and decoding the acoustic features to obtain a plurality of initial recognition results of the speech signal; determining a speech recognition result of the speech signal; the speech recognition result is obtained by applying a speech recognition model to perform semantic correction on the basis of each initial recognition result, the acoustic features and the language information; The speech recognition model is constructed on the basis of a large language model.

[0006] According to the speech recognition method provided by the application, the speech signal is subjected to acoustic feature extraction based on the language information, and acoustic features of the speech signal are obtained, which comprises: The speech signal and the language information are input into a first encoding layer of an encoder to obtain initial acoustic features output by the first encoding layer; The initial acoustic features are input into a second encoding layer in the encoder to obtain the acoustic features output by the second encoding layer; The second encoding layer comprises a plurality of expert models corresponding to the language information and a feature fusion layer, each expert model is used for modeling acoustic features based on the initial acoustic features to obtain expert acoustic features, and the feature fusion layer is used for feature fusion of each expert acoustic feature to obtain the acoustic features.

[0007] According to the speech recognition method provided by the application, the initial acoustic features are input into a second encoding layer in the encoder to obtain the acoustic features output by the second encoding layer, which comprises: The initial acoustic features are input into a convolutional self-attention block in the second encoding layer to obtain intermediate acoustic features output by the convolutional self-attention block; The intermediate acoustic features are input into a plurality of expert models in the second encoding layer to obtain the expert acoustic features output by the expert models; Each expert acoustic feature is input into a feature fusion layer in the second encoding layer to obtain the acoustic features output by the feature fusion layer; The feature fusion layer is used for determining the weights of the expert models based on the intermediate acoustic features and the expert acoustic features, and performing feature fusion on the expert acoustic features based on the weights of the expert models to obtain the acoustic features.

[0008] According to the speech recognition method provided by the application, the feature fusion layer is used for splicing the intermediate acoustic features and the expert acoustic features output by any expert model, and projecting the spliced features to obtain the context features of the any expert model; The feature fusion layer is also used for projecting the context features of the any expert model to obtain target context features of the any expert model, and determining the weights of the expert models based on the target context features of the expert models.

[0009] According to the speech recognition method provided by the application, the encoder is trained based on the following steps: determining a training data set, and a current training mode; the training data set containing a plurality of groups of training data, the training data containing training audio, training text corresponding to the training audio, and language information corresponding to the training audio; the current training mode being obtained by probabilistic sampling of an accurate prior mode and a zero prior mode; based on an initial encoder, determining initial acoustic features of the training audio in each group of training data, and in a case where the current training mode is the accurate prior mode, determining training acoustic features of each training audio based on the initial acoustic features and the language information of the training audio, and in a case where the current training mode is the zero prior mode, determining training acoustic features of each training audio based on the initial acoustic features of the training audio; decoding each training acoustic feature to obtain predicted text corresponding to the training audio in each group of training data; based on the predicted text and the training text corresponding to the training audio in each group of training data, performing parameter iteration on the initial encoder to obtain the encoder.

[0010] According to the speech recognition method provided by the application, the training acoustic features of any training audio are determined based on the following steps: based on the expert models, performing acoustic feature modeling on the initial acoustic features of any training audio to obtain expert acoustic features of the training audio output by the expert models; in a case where the current training mode is the accurate prior mode, taking the expert model corresponding to the language information of the training audio as a target expert model, and performing feature fusion on the expert acoustic features of the training audio output by the target expert model to obtain the training acoustic features of the training audio; in a case where the current training mode is the zero prior mode, performing feature fusion on the expert acoustic features of the training audio output by the expert models to obtain the training acoustic features of the training audio.

[0011] According to the speech recognition method provided by the application, the determination of the speech recognition result of the speech signal comprises: in a case where the confidence of each initial recognition result is less than a confidence threshold, sending the initial recognition results, the acoustic features, and the language information to a cloud server, so that the cloud server applies the speech recognition model to perform semantic correction based on the initial recognition results, the acoustic features, and the language information, and obtains and returns the speech recognition result of the speech signal; In a case where a confidence corresponding to any initial recognition result is greater than or equal to the confidence threshold, a speech recognition result of the speech signal is determined based on the initial recognition results.

[0012] According to the speech recognition method provided by the application, the cloud server is configured to align the acoustic features to a semantic space of the speech recognition model through an adapter to obtain semantic features, and input the semantic features, the initial recognition results and the language information into the speech recognition model to obtain a speech recognition result output by the speech recognition model. The speech recognition model is configured to determine semantic information of the initial recognition results and matching information between the initial recognition results and the semantic features, and perform semantic correction on the initial recognition results based on the semantic information, the matching information and the language information to obtain a speech recognition result of the speech signal.

[0013] According to the speech recognition method provided by the application, the adapter is trained based on the following steps: A sample data set is determined, the sample data set contains sample audio text pairs of multiple languages, the sample audio text pairs contain sample audio and sample text corresponding to the sample audio; Sample acoustic features of the sample audio in each sample audio text pair and sample semantic features of the sample text are determined; Based on an initial adapter, predicted semantic features corresponding to each sample acoustic feature are determined, and the initial adapter is constructed based on a learnable linear transformation layer and a nonlinear activation function; Based on feature similarities between the predicted semantic features corresponding to each sample acoustic feature and sample semantic features of the sample text in the sample audio text pair in which the sample audio corresponding to the sample acoustic feature is located, and feature similarities between the predicted semantic features corresponding to each sample acoustic feature and sample semantic features of the sample text in other sample audio text pairs, a contrastive loss is determined; Based on the contrastive loss, parameter iteration is performed on the initial adapter to obtain the adapter.

[0014] The application further provides a speech recognition method applied to a cloud server, comprising: In a case where confidences corresponding to multiple initial recognition results of a speech signal to be recognized by a terminal device are all less than a confidence threshold, the speech signal is received, acoustic features, language information and the initial recognition results of the speech signal are sent, the acoustic features are obtained by performing acoustic feature extraction on the speech signal based on the language information, and the initial recognition results are obtained by decoding the acoustic features; application of the speech recognition model is based on the each initial recognition result, the acoustic feature and the language information. The speech recognition model is constructed on the basis of a large language model.

[0015] The application further provides a speech recognition device applied to a terminal device, comprising: A speech information determination unit is configured to determine a speech signal to be recognized and language information of the speech signal. An initial result determination unit is configured to extract an acoustic feature of the speech signal based on the language information, to obtain the acoustic feature of the speech signal, and to decode the acoustic feature to obtain a plurality of initial recognition results of the speech signal. A recognition result determination unit is configured to determine a speech recognition result of the speech signal; the speech recognition result is obtained by application of a speech recognition model to semantic correction based on each initial recognition result, the acoustic feature and the language information. The speech recognition model is constructed on the basis of a large language model.

[0016] The application further provides a speech recognition device applied to a cloud server, comprising: A receiving unit is configured to receive the acoustic feature, the language information and each initial recognition result of the speech signal sent by a terminal device in a case where confidence degrees corresponding to a plurality of initial recognition results of the speech signal to be recognized are all less than a confidence threshold; the acoustic feature is obtained by extraction of an acoustic feature of the speech signal based on the language information, and each initial recognition result is obtained by decoding the acoustic feature. A recognition unit is configured to obtain a speech recognition result of the speech signal by application of a speech recognition model to semantic correction based on each initial recognition result, the acoustic feature and the language information, and to return the speech recognition result to the terminal device. The speech recognition model is constructed on the basis of a large language model.

[0017] The application further provides a speech recognition system comprising a terminal device and a cloud server. The terminal device is configured to extract an acoustic feature of a speech signal to be recognized based on language information of the speech signal, to obtain the acoustic feature of the speech signal, and to decode the acoustic feature to obtain a plurality of initial recognition results of the speech signal; in a case where confidence degrees corresponding to each initial recognition result are all less than a confidence threshold, the terminal device is configured to send each initial recognition result, the acoustic feature and the language information to the cloud server. The cloud server is configured to receive the initial recognition results, the acoustic features and the language information sent by the terminal device, and based on the initial recognition results, the acoustic features and the language information, apply a speech recognition model to perform semantic correction to obtain a speech recognition result of the speech signal, and return the speech recognition result to the terminal device. The speech recognition model is constructed based on a large language model.

[0018] The application further provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, and the processor implements the speech recognition method according to any one of the preceding embodiments when executing the computer program.

[0019] The application further provides a non-transitory computer-readable storage medium having a computer program stored thereon, and the computer program is executed by a processor to implement the speech recognition method according to any one of the preceding embodiments.

[0020] The speech recognition method, device, system, electronic device and storage medium provided by the application are based on language information, acoustic feature extraction of a speech signal, acoustic feature decoding of the speech signal, and determination of a speech recognition result of the speech signal. The speech recognition result is obtained by applying a speech recognition model to perform semantic correction based on the initial recognition results, acoustic features and language information. The speech recognition model is constructed based on a large language model, which overcomes the low multi-language speech recognition accuracy and easy misjudgment defects in the prior art. Through a two-step recognition process, multiple initial recognition results are quickly generated locally, and the powerful semantic understanding ability of the large model is utilized to perform multi-modal deep correction by combining acoustic features and language information, thereby greatly improving the accuracy of multi-language speech recognition. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the application or the prior art, the following will briefly introduce the drawings needed in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0022] Figure 1 is one of the flowcharts of the speech recognition method provided by the application; Figure 2 is a schematic diagram of the acoustic feature extraction process provided by the application; Figure 3 is a general framework diagram of the speech recognition method provided by the application; is a general framework diagram of the speech recognition method provided by the application;Figure 4 is a flowchart of a voice recognition method provided by the present application; Figure 5 is a structural schematic diagram of a voice recognition device provided by the present application; Figure 6 is a structural schematic diagram of a voice recognition device provided by the present application; Figure 7 is a structural schematic diagram of a voice recognition system provided by the present application; Figure 8 is a structural schematic diagram of an electronic device provided by the present application. DETAILED DESCRIPTION

[0023] To make the objectives, technical solutions, and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below with reference to the drawings in the present application. Obviously, the described embodiments are some, but not all, of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0024] In view of the fact that current multilingual speech recognition cannot adapt to the differences in speech characteristics, grammatical structure, vocabulary semantics, etc. of different languages, leading to difficulties in improving model performance, and thus making it difficult to accurately process multiple languages at the same time, especially for some small languages or languages with complex speech characteristics, the recognition accuracy is low, and misjudgment is prone to occur. The present application provides a voice recognition method, which aims to introduce a large model, utilize the powerful semantic understanding ability of the large model, and fuse acoustic features, language information, and multiple initial recognition results for voice recognition to obtain accurate and reliable voice recognition results, overcoming the defects of low accuracy and easy misjudgment in current multilingual speech recognition, and greatly improving the accuracy of multilingual speech recognition.

[0025] Figure 1 is a flowchart of a voice recognition method provided by the present application, as shown in Figure 1 the method is applied to a terminal device, which can be a smart phone, a tablet computer, a vehicle-mounted voice assistant, etc. The method comprises: Step 110, determining a voice signal to be recognized, and language information of the voice signal; Step 120, based on the language information, performing acoustic feature extraction on the voice signal to obtain acoustic features of the voice signal, and decoding the acoustic features to obtain multiple initial recognition results of the voice signal; In step 130, a speech recognition result of the speech signal is determined; the speech recognition result is obtained by applying a speech recognition model to perform semantic correction on the basis of the initial recognition results, the acoustic features and the language information; and the speech recognition model is constructed on the basis of a large language model.

[0026] Specifically, before speech recognition is performed, the recognition object, i.e., the speech signal to be recognized, needs to be acquired first. Here, the speech signal to be recognized can be acquired in real time by a terminal device through a built-in microphone array, can be obtained by directly reading a corresponding audio file stored in the terminal device when recognition is needed, or can be acquired through other manners, which are not limited in the embodiments of the present application.

[0027] The speech signal to be recognized can be a speech signal of a single language, such as a pure Chinese or pure English speech signal, or can be a mixed speech signal containing multiple languages, such as a speech signal containing Chinese, English, Korean, etc., which are not limited in the embodiments of the present application. In the embodiments of the present application, in order to solve the problems existing in current multi-language speech recognition, as a preferred option, the speech signal here can be a mixed speech signal containing multiple languages.

[0028] After the speech signal to be recognized is determined, in order to improve the accuracy of multi-language speech recognition and ensure the accuracy of the final recognition result, the language information of the speech signal also needs to be determined. The language information is used to indicate the language types contained in the speech signal.

[0029] The language information can be provided by a user, such as being input through an interactive interface on the terminal device, or being selected by framing the language types corresponding to the speech signal, and the terminal device can determine the language information of the speech signal after receiving the instruction of the user. The language information can also be obtained by recognition through algorithms, models, etc., such as a lightweight language recognition model, such as Language Identification, can be built in the terminal device, which can quickly analyze the speech signal and output the language types contained therein, thereby obtaining the language information of the speech signal. The language information can also be obtained by context inference, such as in continuous dialogue of a vehicle-mounted voice assistant or two-party conversation in a conference scenario, it can be assumed that the language of the subsequent speech signal is consistent with that of the previous dialogue.

[0030] It should be noted that the purpose of providing the language information of the speech signal to be recognized in the process of multi-language speech recognition is to further improve the recognition accuracy and maximize the recognition accuracy. If the language information is not provided or cannot be obtained, multi-language speech recognition can still be performed. In the subsequent recognition process, the terminal device can automatically perceive the language information, and the recognition effect is guaranteed. Compared with the current multi-language speech recognition technology, the recognition accuracy is greatly improved.

[0031] Further, after determining the speech signal to be recognized and the language information of the speech signal, the speech recognition can be performed based on the same in the embodiment of the application. The speech recognition process is the preliminary recognition performed by the terminal device, and the obtained result corresponds to the initial speech recognition result.

[0032] Specifically, the acoustic feature of the speech signal can be extracted using the language information first, so as to obtain the acoustic feature of the speech signal. That is, the original one-dimensional time-domain speech signal waveform is converted into a multi-dimensional feature vector sequence that is easier for the terminal device to understand and process. Specifically, an acoustic model and a traditional acoustic feature extraction algorithm can be used to extract the acoustic feature of the speech signal, such as extracting the mel frequency cepstrum coefficient feature or the Fbank feature. The acoustic feature of the speech signal can also be extracted through a cross-language language model, or the acoustic feature extraction can be realized through a coding and decoding architecture, which is not limited in the embodiment of the application.

[0033] Preferably, the acoustic feature extraction is completed by an encoder in the coding and decoding architecture in the embodiment of the application. That is, the speech signal to be recognized and the language information can be taken as inputs, and the encoder can process the speech signal according to the language information, so as to output a high-dimensional vector sequence that can represent the content of the speech signal, that is, the acoustic feature. The acoustic feature fuses the acoustic information and the language information of the speech signal, and provides a high-quality input for subsequent decoding.

[0034] Subsequently, after obtaining the acoustic feature of the speech signal, the acoustic feature can be decoded to obtain a plurality of initial recognition results. This process is essentially mapping the high-dimensional vector sequence to the corresponding text sequence, which can be realized through a decoder. Specifically, the acoustic feature of the speech signal extracted by the acoustic feature extraction can be input into the decoder, and the decoder can perform decoding processing thereon, and output a probability distribution through a search algorithm such as bundle search. The probability distribution contains a plurality of recognition texts and the confidence degrees of the recognition texts. Since the recognition texts obtained at this time are the preliminary recognition results generated by the terminal device locally, they can also be called initial recognition results.

[0035] The decoder can be a CTC (Connectionist Temporal Classification) decoder, an RNN-T (Recurrent Neural Network Transducer) decoder, or another decoder, and the embodiments of the present application do not make specific limitations.

[0036] Further, considering that the initial recognition result obtained through preliminary identification may have errors and is not necessarily very accurate, in order to further improve the accuracy of multi-lingual speech recognition, a semantic error correction mechanism based on a large language model is introduced in the embodiments of the present application to correct the plurality of initial recognition results obtained through initial recognition and obtain accurate speech recognition results.

[0037] Specifically, a model for correcting the initial recognition result, i.e., a speech recognition model, can be determined first. The model is constructed based on a large language model (referred to as a large model). The large language model is a pre-trained language model based on a Transformer architecture and having a parameter quantity of hundreds of millions or more, which has strong natural language understanding, analysis, and processing (such as semantic reasoning and text generation) capabilities. Therefore, a speech recognition model can be constructed based on this to be used for semantic error correction. Then, the plurality of initial recognition results, the acoustic features, and the language information obtained in the foregoing steps can be collectively provided as input to the speech recognition model, and the speech recognition model can comprehensively utilize the information in the three dimensions for deep fusion analysis and output the final recognition result, i.e., the speech recognition result of the speech signal.

[0038] Here, it should be noted that the process of calling the speech recognition model for recognition analysis can be performed locally on the terminal device, or can be performed in the cloud, i.e., the terminal device can send the initial recognition results, the acoustic features, and the language information to the cloud server, and the speech recognition model deployed on the cloud server can process and output the final speech recognition result; the embodiments of the present application do not make specific limitations.

[0039] In the process of semantic error correction, the speech recognition model can analyze the differences between the initial recognition results, evaluate the grammatical structure, word collocation, and semantic rationality of each initial recognition result using its strong language knowledge, analyze the matching degree between the initial recognition results and the acoustic features, such as which initial recognition result is more matched with the real pronunciation, analyze the adaptation degree of the intermediate result to the language during the error correction process to ensure that the final recognition result conforms to the language habits of the corresponding language, and finally obtain the speech recognition result of the speech signal.

[0040] In the embodiment of the present application, through comprehensive consideration of multi-modal information, the speech recognition model can accurately correct errors (such as spelling errors, homonym errors, syntax errors, etc.) in the initial recognition result, and finally output a recognition text with the most coherent semantics and the most matched acoustic features, that is, the speech recognition result, thereby realizing multi-language speech recognition and obtaining accurate and reliable speech recognition results.

[0041] The speech recognition method provided by the present application extracts acoustic features of a speech signal based on language information, obtains the acoustic features of the speech signal, decodes the acoustic features, and obtains multiple initial recognition results of the speech signal; determines a speech recognition result of the speech signal; the speech recognition result is obtained by applying a speech recognition model for semantic error correction based on each initial recognition result, acoustic features, and language information; the speech recognition model is constructed based on a large language model, which overcomes the defects of low multi-language speech recognition accuracy and easy misjudgment in traditional schemes. Through a two-step recognition process, multiple initial recognition results are quickly generated locally, and the powerful semantic understanding ability of the large model is utilized to perform multi-modal deep error correction by combining acoustic features and language information, thereby greatly improving the accuracy of multi-language speech recognition, especially for languages with a large number of similar phonemes or unique grammar structures, and for some complex or ambiguous sentences, the recognition accuracy is greatly improved.

[0042] Based on the above embodiment, in step 120, acoustic features of a speech signal are extracted based on language information to obtain acoustic features of the speech signal, including: inputting the speech signal and the language information into a first encoding layer of an encoder to obtain initial acoustic features output by the first encoding layer; inputting the initial acoustic features into a second encoding layer in the encoder to obtain acoustic features output by the second encoding layer; The second encoding layer includes multiple expert models corresponding to the language information and a feature fusion layer. Each expert model is used to model acoustic features based on the initial acoustic features to obtain expert acoustic features. The feature fusion layer is used to fuse each expert acoustic feature to obtain the acoustic features.

[0043] Specifically, considering the current end-to-end deep learning model, when it processes multiple languages, a mixture of experts (MoE) module is introduced into the model structure, and through a data-driven dynamic routing mechanism, the model learns the commonalities and differentiated characteristics of different languages. Although this approach can improve the model capacity and efficiency, the data-driven nature of the MoE router, which relies on data distribution decisions, on the one hand, can lead to insufficient training of low-resource language experts, with the router favoring high-frequency languages, forming a "Matthew effect", which further reduces the recognition effect of low-resource languages; on the other hand, the expert switching is unstable, and the time sequence continuity of the speech signal is easily destroyed by the discrete expert switching of the MoE, resulting in a decline in recognition effect.

[0044] In view of this, in the embodiments of the present application, the encoder for acoustic feature extraction can be improved to dynamically aggregate the outputs of each expert branch according to different language prior scenarios, thereby alleviating the problem of the MoE encoder router favoring high-frequency languages leading to a decline in the recognition effect of low-resource languages, and the instability of expert switching leading to a decline in the recognition effect. By improving the accuracy and robustness of acoustic feature extraction in a multi-language speech recognition scenario, the recognition effect of multi-language speech recognition is improved.

[0045] In detail, Figure 2 is a schematic diagram of the acoustic feature extraction process provided by the present application, as Figure 2 indicated, the acoustic feature extraction process can be completed by an encoder. The encoder adopts a three-level cascading architecture to build a complete processing link from the original speech signal to the acoustic feature. It includes three layers, the first layer is a convolution module, the middle layer is a first encoding layer, and the third layer is a second encoding layer. Among them, the convolution module can be composed of multiple two-dimensional convolution networks, and its core function is to capture the local time-frequency features in the speech signal. Through the sliding window operation of the multi-layer two-dimensional convolution kernel, the edge features and texture information in the mel spectrum graph can be effectively extracted, thereby providing high-quality input for subsequent modeling. This feature extraction method based on local receptive field not only preserves the physical characteristics of the speech signal, but also lays a foundation for subsequent global modeling.

[0046] The first encoding layer that follows is stacked by multiple Conformer Blocks (convolutional self-attention blocks), which is responsible for modeling the local time-frequency features output by the convolution module to a deeper level, capturing long-range dependencies and temporal dynamics in the speech signal. Here, the cascading design of multiple Conformer Blocks in the first encoding layer forms a deep feature extraction network that can abstract high-level semantic features in the speech signal layer by layer. This progressive feature extraction mechanism can gradually strip noise interference and focus on language essential features.

[0047] The second encoding layer is the core of the encoder to realize language perception. It is composed of three parts: Conformer Block, multi-expert model, and feature fusion layer based on prior perception attention mechanism. Among them, the expert model corresponds to the language information, and each expert model focuses on the feature space learning of a specific language. This customized design enables each language to have a dedicated processing channel that matches its characteristics, effectively solving the feature dilution problem of traditional unified models in a multi-lingual scenario.

[0048] Specifically, when performing acoustic feature extraction, the speech signal and language information can be input into the convolution module of the encoder first. The convolution module extracts the edge features and texture information in the extracted mel-spectrogram of the speech signal and outputs them. Then, the features output by the convolution module can be input into the first encoding layer for preliminary acoustic feature extraction. The feature vector output by the first encoding layer is the initial acoustic feature. This initial acoustic feature is a more general intermediate representation of the speech signal that contains high-level semantic information and serves as the basis for subsequent language-specific processing.

[0049] After that, the initial acoustic feature can be input into the second encoding layer of the encoder for processing, and the acoustic feature output by the second encoding layer can be obtained finally. The second encoding layer is used to receive the initial acoustic feature output by the first encoding layer and perform targeted and refined processing on it, finally outputting high-quality acoustic features for decoding.

[0050] Specifically, the second encoding layer contains multiple expert models and a feature fusion layer. The expert models here correspond to the language information, i.e., the language information points to how many language categories, and the corresponding language category expert model is enabled here. For example, there are 3 language categories, such as Chinese, English, and Korean, and the expert model corresponds to include Chinese expert model, English expert model, and Korean expert model. Here, each expert model is equipped with an independent parameter matrix, which can model the pronunciation characteristics, prosodic patterns, etc. of the corresponding language in detail.

[0051] In a specific implementation, after the initial acoustic features are input into the second encoding layer, each expert model can model the acoustic features based on the initial acoustic features, thereby obtaining the expert acoustic features output by each expert model. Here, specifically, the initial acoustic features input into the second encoding layer can be distributed to each enabled expert model, and each expert model uses its own knowledge that can be directed to a specific language to individually process the initial acoustic features, thereby outputting expert acoustic features with unique marks of the language. For example, for Chinese with rich tones, the corresponding expert model (Chinese expert model) will focus on enhancing prosodic features such as the fundamental frequency track; and for English with more consonant clusters, the corresponding expert model (English expert model) will focus on enhancing features such as plosive sounds. This differentiated processing strategy can enable the features of each language to be fully retained and strengthened.

[0052] After that, the expert acoustic features output by each expert model can be input into the feature fusion layer for fusion through the feature fusion layer, and finally the acoustic features of the speech signal can be obtained. Here, the feature fusion layer receives the expert acoustic features output by each expert model, and uses an intelligent aggregation strategy to fuse these parallel expert acoustic features with different language characteristics into a single unified feature vector, which is the acoustic feature finally output by the encoder and sent to the decoder for decoding. In this process, the encoder can select which expert models to enable according to the language information, and use the expert acoustic features output by these expert models in the feature fusion process. Through adaptive fusion of the expert acoustic features, the optimal acoustic features can be generated, thereby providing a data basis for subsequent speech recognition.

[0053] In the embodiment of the present application, multiple expert models corresponding to languages and a feature fusion layer are arranged in the second encoding layer, which can model the acoustic characteristics of each language in depth, effectively solving the problem of feature dilution and poor recognition performance for low-resource languages caused by the model parameters needing to consider all languages when the traditional model processes multi-language speech signals, greatly improving the overall performance and robustness of multi-language speech recognition.

[0054] Based on the above embodiment, the initial acoustic features are input into the second encoding layer of the encoder to obtain the acoustic features output by the second encoding layer, including: The initial acoustic features are input into the convolutional self-attention block in the second encoding layer to obtain the intermediate acoustic features output by the convolutional self-attention block; The intermediate acoustic features are input into the multiple expert models in the second encoding layer to obtain the expert acoustic features output by each expert model; The expert acoustic features are input into a feature fusion layer in the second encoding layer to obtain acoustic features output by the feature fusion layer; the feature fusion layer is configured to determine weights of the expert models based on the intermediate acoustic features and the expert acoustic features, and perform feature fusion on the expert acoustic features based on the weights of the expert models to obtain the acoustic features.

[0055] Specifically, the process of obtaining the acoustic features of the speech signal through the second encoding layer can specifically include: Referring to Figure 2 It can be seen that after the initial acoustic features output by the first encoding layer are input into the second encoding layer, the initial acoustic features will not be directly sent to the expert models for processing. In order to further enhance the context awareness ability of the features, the initial acoustic features will first pass through the Conformer Block (convolutional self-attention block) in the second encoding layer and be processed by the convolutional self-attention block, that is, the initial acoustic features are “deeply processed” to capture long-distance global context information and fine-grained local acoustic patterns at the same time, thereby obtaining intermediate acoustic features with higher quality and richer information.

[0056] After that, the intermediate acoustic features output by the convolutional self-attention block can be input into the expert models, and acoustic feature modeling is performed by the expert models to obtain expert acoustic features output by the expert models. Since the intermediate acoustic features already contain more abundant context information, each expert model can perform specific modeling of the respective language on the basis of a higher-quality feature representation, thereby outputting more accurate expert acoustic features that can reflect the acoustic details of the specific language.

[0057] After that, the expert acoustic features output by the expert models also need to be input into the feature fusion layer for feature fusion, and finally acoustic features output by the feature fusion layer can be obtained.

[0058] Here, the feature fusion layer adopts a priori perception attention mechanism. Instead of simply averaging or concatenating the outputs of all experts, this mechanism aggregates the expert acoustic features output by all expert models in a dynamic way. Specifically, the feature fusion layer first determines the weights of the expert models according to the intermediate acoustic features and the expert acoustic features output by the expert models; the weights quantify the “speaking rights” or “importance” of each expert model in the context of the input speech signal. Then, the expert acoustic features output by the expert models can be fused according to the weights, and finally the acoustic features can be obtained.

[0059] wherein the determination of the weight is dynamic; that is, the feature fusion layer simultaneously refers to two aspects of information, one of which is the expert acoustic feature output by each expert model, and the other of which is the intermediate acoustic feature input to the expert model. Through joint analysis of these two parts of information, the feature fusion layer can intelligently determine which analysis results of the expert models are more reliable and more relevant to the input speech signal, and assign higher weights to these expert models; conversely, assign lower weights to expert models with lower relevance.

[0060] After determining the weight of each expert model, the feature fusion layer can perform a feature fusion operation. Here, the feature fusion operation can be weighted summation; that is, multiply the expert acoustic feature output by each expert model by its corresponding weight, and then add all the weighted results to obtain a single feature vector that is the final output of the encoder and is sent to the decoder for decoding.

[0061] In the embodiments of the present application, a dynamic weighted fusion strategy based on an attention mechanism is adopted to adaptively adjust and aggregate the outputs of different expert models based on the acoustic content of the speech signal, thereby effectively alleviating the "Matthew effect" problem caused by pure data-driven routers in traditional MoE, avoiding the problem of timing instability that may be caused by hard switching of expert models, and enabling the finally generated acoustic feature to accurately capture subtle differences of a specific language while maintaining the continuity and integrity of the speech signal.

[0062] Based on the above embodiments, the feature fusion layer is used to splice the intermediate acoustic feature and the expert acoustic feature output by any expert model, and project the spliced feature to obtain the context feature of the expert model; The feature fusion layer is also used to project the context feature of the expert model to obtain the target context feature of the expert model, and determine the weight of each expert model based on the target context features of the expert models.

[0063] Specifically, the feature fusion layer based on the prior perception attention mechanism determines the weight of each expert model through the following steps when performing feature fusion: For any expert model, the feature fusion layer first splices the input and output of the expert model, that is, splices the intermediate acoustic feature and the expert acoustic feature output by the expert model, and then projects the spliced feature to obtain the context feature of the expert model. After that, the context feature can be further projected to obtain the target context feature of the expert model, and the weight of each expert model can be determined based on the target context features of the expert models.

[0064] In detail, to evaluate the importance of the output of the expert model in the context, the feature fusion layer performs a context-aware scoring mechanism. The first step of the mechanism is to construct a context-rich representation, i.e., concatenating the intermediate acoustic features as the common input of the expert model with the expert acoustic features output by the expert model itself in the feature dimension, and then splicing to form a wider and more informative feature. Then, the spliced feature can be projected to map it to a new feature space that may have different dimensions; then, a nonlinear activation function (such as the hyperbolic tangent function tanh) is used for processing, and the context feature of the expert model is obtained. The context feature here can be understood as a preliminary quantitative score of the output quality of the expert model.

[0065] After that, the context feature of the expert model can be further processed to obtain the final "decision score" used to calculate the weight. The process is as follows: first, the context feature of the expert model is projected to map it to a scalar value, which can be used to decide the weight of the expert model.

[0066] Then, a mask function can be applied to obtain the target context feature of the expert model. The mask function is the key to achieving prior perception. Given the language information, i.e., in the accurate prior mode, the mask function retains the target context features of the enabled expert models (i.e., the target context features of the expert model corresponding to the language information), and sets the target context features of all expert models outside the language category contained in the language information to a minimum value (such as negative infinity). Here, it should be noted that when the expert model is used in the second encoding layer, the enabled expert model can be only the expert model corresponding to the language information, or all expert models can be enabled, and then masked.

[0067] Correspondingly, when the language information is not given, i.e., in the zero prior mode, the mask function does not work and retains the target context features of all expert models. It can also be understood that in this case, all expert models in the second encoding layer are enabled, and all target context features of the expert models are retained when the features are fused.

[0068] Then, the weights of each expert model can be determined based on its target context features. Repeating this process yields the target context features for each expert model. Once these features are determined, they can be normalized by inputting them into the Softmax function. The Softmax function normalizes these scalar values, transforming them into a distribution with a sum of 1. Each value in this distribution represents the weight of the corresponding expert model. Expert models with higher "decision scores" receive larger weights, and vice versa.

[0069] In this embodiment of the invention, by splicing and multi-step projection of intermediate acoustic features and acoustic features of each expert, a target context feature capable of fully evaluating the output quality of each expert model is constructed. Based on this, weight evaluation enables the feature fusion layer to accurately quantify the "contribution" of each expert model in a context-aware manner, thereby achieving dynamic weighted fusion of acoustic features of each expert. This allows the final output acoustic features to absorb the advantageous information output by relevant expert models to the greatest extent and discard the noise interference from irrelevant expert models, thus greatly improving the encoder's feature extraction capability in complex multilingual environments.

[0070] Based on the above embodiments, the process of determining the weights of each expert model can be represented by the following formula: In the formula, Indicates intermediate acoustic features. Indicates the first The expert acoustic features output by an expert model The number of expert models enabled. and These are the encoder's hyperparameters. Indicates to and Perform splicing (to obtain dimension ) (characteristics) Indicates to and The spliced ​​features are then projected. It is a non-linear activation function. Indicates the first The contextual features output by an expert model and These are trainable parameters. This indicates that the context features are projected. For masking functions, express The processed first The target context features of an expert model This represents the target context features of each expert model. This is a distribution with a sum of 1, where each value in the distribution is the weight of the corresponding expert model.

[0071] Different masking strategies will be used depending on the scenario. In the exact prior mode, all expert model target context features other than the language type included in the language information will be set to... If it is in zero-prior mode, it will not have any effect, and all target context features of the expert model will be preserved.

[0072] Based on the above embodiments, the encoder is trained using the following steps: The training dataset and the current training mode are determined. The training dataset contains multiple sets of training data, including training audio, training text corresponding to the training audio, and language information corresponding to the training audio. The current training mode is obtained by probability sampling of the exact prior mode and the zero prior mode. Based on the initial encoder, the initial acoustic features of the training audio in each group of training data are determined. When the current training mode is the exact prior mode, the training acoustic features of each training audio are determined based on the initial acoustic features and language information. When the current training mode is the zero prior mode, the training acoustic features of each training audio are determined based on the initial acoustic features. Decode each training acoustic feature to obtain the predicted text corresponding to the training audio in each group of training data; Based on the predicted text and training text corresponding to the training audio in each group of training data, the parameters of the initial encoder are iterated to obtain the encoder.

[0073] Specifically, the encoder training process may include the following steps: First, the dataset used for training, i.e., the training dataset, is determined. Considering that in practical applications, the encoder faces multilingual scenarios, a large-scale training dataset is needed in this embodiment of the invention to ensure that the trained encoder can accurately process multiple languages. This training dataset contains multiple sets of training data, each set of training data being a "triple," specifically including a training audio segment (such as an English or Chinese recording), training text that is completely identical to the content of the training audio segment (i.e., the correctly recognized text of the recording), and the language information of the training audio segment.

[0074] Before starting each round of training, it is also necessary to determine the current training mode. The training mode here includes two, namely the accurate prior mode and the zero prior mode. In the specific training process, a certain mode is not fixedly used, but the training mode of the current training round is determined by probability sampling. For example, a sampling probability can be set, such as 70%, at the beginning of each round of training, with a probability of 70% to select the accurate prior mode, and a probability of 30% to select the zero prior mode. The purpose of this mixed training strategy is to improve the generalization ability and robustness of the encoder, so that it can not only handle scenarios with clear language information, but also cope with complex scenarios with unknown or ambiguous language information, and has strong scene adaptability.

[0075] After determining the current training mode by probability sampling, the training of the current round can be formally started. Specifically, the initial acoustic features of the training audio in each group of training data are first determined by the initial encoder, and in the case that the current training mode is the accurate prior mode, the training acoustic features of each training audio are determined based on the initial acoustic features of each training audio and the language information, and in the case that the current training mode is the zero prior mode, the training acoustic features of each training audio are determined based on the initial acoustic features of each training audio.

[0076] Among them, the initial encoder can adopt a three-level cascade architecture, and its specific network architecture is consistent with the encoder used for actual acoustic feature extraction, or it can be understood that the encoder used for actual acoustic feature extraction is a trained encoder obtained by training the initial encoder; the difference is that the internal parameters of the initial encoder here are randomly initialized or obtained by pre-training other tasks, while the parameters of the actual encoder are trained.

[0077] For any group of training data in the training data set, the training audio first passes through the first encoding layer of the initial encoder to obtain the initial acoustic feature. Subsequently, the initial acoustic feature of the training audio is sent to the second encoding layer in the initial encoder for processing to obtain the final training acoustic feature. The processing method here strictly depends on the current training mode determined in the previous step: that is, the initial acoustic feature can be processed by all expert models in the initial encoder to obtain the expert acoustic features of the training audio output by all expert models; then, in the case that the current training mode is the accurate prior mode, the second encoding layer will use the language information corresponding to the training audio to guide the feature fusion process. That is, through mechanisms such as masks, the weight of the expert model corresponding to the language information is greatly improved, while the expert models of other language information are suppressed, so as to obtain a highly language-related training acoustic feature.

[0078] Correspondingly, in the case that the current training mode is the zero prior mode, the second encoding layer ignores the language information of the training audio, and lets the internal feature fusion layer (such as the prior perceptual attention mechanism) completely adaptively determine and aggregate the outputs of the expert models based on the expert acoustic features of the training audio output by the expert models, so as to obtain the training acoustic features.

[0079] Then, the training acoustic features of each training audio can be decoded, that is, the training acoustic features of each training audio are input into the initial decoder, and the initial decoder decodes the training acoustic features to generate the predicted text that it considers to be the most likely prediction. Here, the architecture of the initial decoder is consistent with that of the decoder in actual application, and the difference may be different parameters, or the parameters may be consistent, and the decoder does not need to be trained.

[0080] Then, the initial encoder can be trained according to the predicted text and the training text corresponding to the training audio in each group of training data, and finally a trained encoder can be obtained. Specifically, by comparing the difference between the predicted text and the training text provided in the training data set, that is, the training text, and quantifying the difference through a loss function (such as CTC Loss), the loss value quantified reflects the "error degree" of the initial encoder in this recognition task in the current training round. Based on this loss value, the parameters of the initial encoder can be adjusted, that is, parameter iteration, and the goal of iteration is to update the parameters of the initial encoder in the direction that can reduce the difference between the predicted text and the training text in the next iteration. Through multiple rounds of training and multiple parameter iterations, a trained encoder that can be used for actual recognition tasks can be finally obtained.

[0081] In the embodiments of the present application, through the mixed training strategy of probability sampling of the accurate prior mode and the zero prior mode, the finally trained encoder can not only perform efficient and accurate feature extraction when obtaining clear language prior, but also perform robust adaptive recognition when lacking prior information, thereby effectively avoiding overfitting of the model to a specific training mode, and greatly enhancing the generalization and reliability of the encoder in real and variable application multilingual scenarios.

[0082] Based on the above embodiments, the training acoustic features of any training audio are determined based on the following steps: Based on the expert models, the initial acoustic features of the training audio are modeled to obtain expert acoustic features of the training audio output by the expert models; In the case that the current training mode is the accurate prior mode, the expert model corresponding to the language information of the training audio is taken as the target expert model, and the expert acoustic features of the training audio output by the target expert model are fused to obtain the training acoustic features of the training audio; In the case that the current training mode is the zero-prior mode, the expert acoustic features of the training audio output by each expert model are fused to obtain the training acoustic feature of the training audio.

[0083] Specifically, in the training process of the encoder, for any training audio, its final training acoustic feature is dynamically generated based on the current training mode.

[0084] In detail, for any training audio, its training acoustic feature can be determined by the following steps: First, when the initial acoustic feature of the training audio is sent into the second encoding layer of the initial encoder, it will be simultaneously distributed to all expert models contained in the layer. Each expert model will independently model the acoustic feature of the initial acoustic feature, i.e., forward calculation using its own unique network parameters, to obtain the expert acoustic features of the training audio output by all expert models.

[0085] Then, different feature fusion strategies can be adopted according to the current training mode: First, in the case that the current training mode is the accurate-prior mode, i.e., when the current training mode is selected as the accurate-prior mode through probability sampling, at this time, the language information provided in the training data can be used to identify the expert model corresponding to the language information of the training audio from all expert models, and determine it as the target expert model of the training audio. For example, if the language information of the training audio contains Chinese and English, the Chinese expert model and the English expert model are the target expert models of the training audio.

[0086] In this mode, the feature fusion process is strongly guided. That is, the feature fusion layer of the initial encoder will assign almost all weights (e.g., weight values close to 1.0) to the target expert model through masking and other methods, and set or tend to 0 the weights of all other non-target expert models. Therefore, feature fusion at this time is actually equivalent to a selection operation, and the final output training acoustic feature is almost completely determined by the expert acoustic feature output by the target expert model, and the output of other experts is effectively ignored.

[0087] Second, in the case that the current training mode is the zero-prior mode, i.e., when the current training mode is selected as the zero-prior mode through probability sampling, at this time, the language information in the training data will be ignored to simulate the scenario in the real world where the language is unknown or ambiguous.

[0088] In this mode, the process of feature fusion is data-driven and adaptive. The feature fusion layer processes the expert acoustic features output by all expert models. That is, a weight is dynamically calculated for each expert model according to the relevance of the expert acoustic features output by all expert models to the training audio. Generally, the expert model that matches the acoustic characteristics of the training audio better will obtain a higher weight. For example, for an English training audio, even in the zero-prior mode, a well-trained English expert model can output more relevant expert acoustic features, thereby obtaining a higher weight in the feature fusion process of the initial encoder. The final training acoustic features are obtained by weighted fusion of the expert acoustic features output by all expert models.

[0089] In the embodiments of the present application, by explicitly distinguishing the specific strategies of feature fusion in the two different training modes, the core mechanism of the encoder learning language perception ability is disclosed in detail. In the accurate prior mode, the model is forced to learn to activate the corresponding expert using explicit language labels, achieving accurate modeling in a "point strike" manner; in the zero-prior mode, the model is trained without external guidance, relying only on the acoustic content itself to make "independent decisions". This differentiated training method enables the final obtained encoder to flexibly cope with various prior conditions, ensuring the modeling depth on high-frequency, resource-rich languages, and improving the recognition robustness in low-resource or unknown language scenarios, thereby optimizing the performance of the multi-language speech recognition system as a whole.

[0090] Based on the above embodiments, step 130 comprises: In the case where the confidence of each initial recognition result is less than the confidence threshold, the initial recognition results, acoustic features and language information are sent to a cloud server, so that the cloud server applies a speech recognition model to perform semantic correction based on the initial recognition results, acoustic features and language information, to obtain and return a speech recognition result of the speech signal; In the case where the confidence of any initial recognition result is greater than or equal to the confidence threshold, a speech recognition result of the speech signal is determined based on the initial recognition results.

[0091] Specifically, the determination process of the speech recognition result of the speech signal specifically comprises: Considering the computing power of the terminal device, that is, the computing power of the terminal device is limited compared to the cloud server, and the speech recognition model usually requires a large amount of computing resources for running, if it is run locally on the terminal device, it may cause the terminal device to be difficult to load, thereby seriously affecting the normal operation of the terminal device.

[0092] In view of this, in the embodiments of the present application, an intelligent decision-making mechanism of terminal-cloud collaboration is introduced. This mechanism aims to balance the real-time and accuracy of speech recognition, while achieving privacy protection.

[0093] In detail, Figure 3 is the overall framework of the speech recognition method provided by the present application, as shown in Figure 3 The acoustic feature extraction and decoding process of the speech signal are performed on the terminal device, and the process of calling the speech recognition model for semantic error correction is performed on the cloud server. The specific execution process includes the following steps: First, after the terminal device obtains multiple initial recognition results through decoding, it can perform conditional judgment, i.e., checking the confidence of each initial recognition result. The confidence is a quantitative score, which reflects the "degree of certainty" of the terminal device that the recognition result is correct. The higher the confidence, the higher the possibility that the recognition result is correct, and vice versa. If the confidence of each initial recognition result is less than the confidence threshold, it can be determined that the terminal device does not have enough "confidence" at this time, and needs to request the cloud server for assistance. That is, the cloud collaborative mechanism is triggered, and the terminal device will send the local generated multiple initial recognition results, the acoustic features output by the encoder, and the language information to the cloud server for further error correction processing.

[0094] The confidence threshold is a value that can be dynamically adjusted through experiments or according to the requirements of the application scenario, such as 0.75 or 0.8.

[0095] Here, it is worth noting that the terminal device does not transmit the original speech signal in the process of transmitting information, but only transmits the acoustic features processed by the encoder. This not only greatly reduces the data transmission amount and reduces the network bandwidth consumption, but more importantly, it effectively protects the privacy and security of the user, avoiding the risk of privacy leakage of the original speech signal in the transmission process. Here, in order to further ensure data security during data transmission, the transmitted information can also be encrypted.

[0096] Subsequently, the cloud server can receive the initial recognition results, acoustic features, and language information transmitted by the terminal device. Then, the cloud server can call the speech recognition model to process the received information to achieve semantic error correction, and finally obtain the speech recognition result, which can be returned to the terminal device. Since the cloud server has powerful computing resources, it can deploy and run a speech recognition model with a large number of parameters, thereby performing more accurate deep analysis and error correction to obtain a high-quality speech recognition result and return it to the terminal device.

[0097] Correspondingly, if the confidence corresponding to any one or at least one initial recognition result is greater than or equal to the confidence threshold, it can be determined that the terminal device has high "confidence" for this initial recognition result at this time, and the cloud server does not need to be requested to assist in processing at this time.

[0098] That is, the speech recognition result of the speech signal can be directly determined based on each initial recognition result. Here, specifically, the initial recognition result with the highest confidence can be directly selected as the final speech recognition result. In this case, the speech recognition result is directly determined locally, which not only avoids network interaction with the cloud, greatly reduces recognition delay, and guarantees high real-time performance in the simple and clear speech input scenario.

[0099] In the embodiments of the application, the recognition path can be dynamically selected according to the degree of uncertainty of local recognition: for a high-confidence scenario, the result is directly output locally to ensure extremely fast response; for a low-confidence complex scenario, multi-modal information is uploaded to the cloud server, and the powerful capabilities of the speech recognition model are used for semantic correction to ensure high accuracy, thereby balancing real-time performance and accuracy, and at the same time, by transmitting only acoustic features rather than original speech signals, network efficiency and user privacy protection are taken into account, making the entire speech recognition process more efficient, secure and reliable.

[0100] Based on the above embodiments, the cloud server is configured to align the acoustic features to the semantic space of the speech recognition model through the adapter to obtain semantic features, and input the semantic features, each initial recognition result and language information to the speech recognition model to obtain a speech recognition result output by the speech recognition model; The speech recognition model is configured to determine semantic information of each initial recognition result and matching information between each initial recognition result and the semantic features, and perform semantic correction on each initial recognition result based on the semantic information, the matching information and the language information to obtain a speech recognition result of the speech signal.

[0101] Specifically, referring to Figure 3It can be seen that after the cloud server receives the acoustic features, instead of directly applying them, the acoustic features are converted through an adapter. This is because there is a modal gap between the acoustic features and the semantic space of the speech recognition model. The acoustic features are vectors that encode the physical properties of sound, while the speech recognition model mainly operates in a semantic space composed of lexical and sentence meanings. In order to bridge this gap, an adapter is configured in the cloud server. The adapter can be a lightweight neural network, such as a neural network constructed from a learnable linear transformation layer and a nonlinear activation function, which is used to align the acoustic features to the semantic space of the speech recognition model. It can also be understood that the adapter receives acoustic features from the terminal device as input, and through its internal transformation, outputs a new feature vector. This new feature vector is in the same or highly similar feature space as the semantic feature vector understood by the speech recognition model, i.e., the semantic feature. The semantic feature can be regarded as the projection of the speech signal in the semantic space.

[0102] After obtaining the semantic features, the cloud server can input the semantic features, the initial recognition results, and the language information into the speech recognition model. After receiving these multi-modal inputs, the speech recognition model will first analyze and compare the semantic reasonableness, contextual coherence, and matching degree of the different initial recognition results, i.e., it can analyze whether the vocabulary of each initial recognition result is correct, whether the grammar is smooth, and whether it is reasonable in a specific language (such as English) context, to obtain the semantic information of each initial recognition result. At the same time, it can compare the text representation of each initial recognition result with the semantic features converted by the adapter, which represent the original sound meaning, to judge the matching degree of each initial recognition result in terms of "listening" with the original pronunciation, to obtain the matching information between each initial recognition result and the semantic features.

[0103] After that, the speech recognition model can perform semantic error correction on each initial recognition result based on the semantic information, matching information, and language information, and finally output an error-corrected optimal recognition text, i.e., the speech recognition result. This process is actually a comprehensive decision-making process. The speech recognition model will combine all "clues" to reason, such as combining "which initial recognition result reads the most smoothly?", "which initial recognition result sounds the most like the original pronunciation?", and "which language environment is it in now?" to comprehensively score, sort, filter, or even reorganize all initial recognition results, and finally generate a final text that is most semantically reasonable, most acoustically matched, and conforms to the specified language specifications, i.e., the speech recognition result of the speech signal.

[0104] In the embodiment of the present application, the adapter is introduced to successfully map the information of the acoustic mode to the semantic space which is good at the speech recognition model, solving the problem of multi-modal information fusion; on this basis, deep error correction is carried out from two dimensions of semantic rationality of text and matching degree of text and sound, compared with traditional pure text semantic error correction, there is additional discrimination basis from acoustic characteristics, which can more accurately capture the subtle changes in the voice, handle homophones, near homophones and other difficult problems, thereby greatly improving the accuracy of multi-lingual speech recognition.

[0105] Based on the above embodiment, the adapter is trained based on the following steps: A sample data set is determined, the sample data set containing a plurality of sample audio text pairs of different languages, each sample audio text pair containing a sample audio and a sample text corresponding to the sample audio; The sample acoustic features of the sample audio and the sample semantic features of the sample text in each sample audio text pair are determined; Based on the initial adapter, the predicted semantic features corresponding to each sample acoustic feature are determined; the initial adapter is constructed based on a learnable linear transformation layer and a nonlinear activation function; Based on the feature similarity between the predicted semantic features corresponding to each sample acoustic feature and the sample semantic features of the sample text in the sample audio text pair in which the sample audio is located, and the feature similarity between the predicted semantic features corresponding to each sample acoustic feature and the sample semantic features of the sample text in other sample audio text pairs, a contrastive loss is determined; Based on the contrastive loss, the parameters of the initial adapter are iterated to obtain the adapter.

[0106] Specifically, the training process of the adapter can include the following steps: Firstly, the sample data set required for training needs to be determined. The sample data set contains a plurality of sample audio text pairs of different languages. Each sample audio text pair is a paired data unit, which includes a sample audio and a sample text which is completely consistent with the content of the sample audio. The sample data set needs to be as large as possible in scale and widely cover the supported languages to ensure that the adapter can learn the acoustic semantic mapping relationship across languages.

[0107] Then, the acoustic features of the sample audio in each sample audio text pair are extracted to obtain the sample acoustic features of the sample audio; at the same time, the semantic features of the sample text corresponding to each sample audio are extracted to obtain the sample semantic features of each sample text.

[0108] Here, the process of acoustic feature extraction can be achieved by a conventional feature extraction algorithm, or by a pre-trained acoustic encoder with stable performance, and the embodiments of the present application do not make specific limitations. Semantic feature extraction can be achieved by a conventional feature extraction algorithm, or by a pre-trained powerful text encoder, and the embodiments of the present application also do not make specific limitations.

[0109] In this way, each sample audio text pair is associated with two features: one from the acoustic modality and one from the semantic modality.

[0110] After that, the initial adapter can be used to determine the sample semantic features corresponding to the sample acoustic features. Here, specifically, an initial adapter can be determined first, which can be constructed on the basis of a learnable linear transformation layer and a nonlinear activation function. Then, for the sample acoustic features of any sample audio, the initial adapter can perform forward calculation (i.e., linear transformation and nonlinear activation) on them, outputting the converted features, i.e., the predicted semantic features corresponding to the sample acoustic features of the sample audio.

[0111] And since the training target of the initial adapter is to update the parameters so that the predicted semantic features and their paired sample semantic features are as close as possible in the feature space. Therefore, after obtaining the sample semantic features corresponding to each sample audio, in the embodiments of the present application, the contrastive loss can be determined according to the feature similarity between the predicted semantic features corresponding to each sample acoustic feature and the sample semantic features of the sample text in the sample audio text pair to which the sample audio belongs, and the feature similarity between the predicted semantic features corresponding to each sample acoustic feature and the sample semantic features of the sample text in other sample audio text pairs.

[0112] Here, the process of loss measurement adopts the idea of contrastive learning. That is, for the sample acoustic features of any sample audio, the predicted semantic features obtained after conversion by the initial adapter have two relationships with the sample semantic features of all sample texts in the sample data set: Positive sample relationship: the sample semantic features of the sample text from the same sample audio text pair as the sample audio are the only positive samples. The two are completely corresponding in semantics.

[0113] Negative sample relationship: the sample semantic features of other sample texts different from the sample audio, i.e., the sample semantic features of the sample texts in other sample audio text pairs except the sample audio text pair to which the sample audio belongs, are all negative samples. The two are not related in semantics.

[0114] Based on the two relationships, the loss can be measured to determine the contrast loss, that is, the feature similarity (such as cosine similarity) between the predicted semantic feature of each sample audio and its corresponding positive sample is calculated, and the feature similarity (such as cosine similarity) between the predicted semantic feature of each sample audio and all negative samples is calculated, and the contrast loss can be obtained based on the two. Since the training target is to maximize the feature similarity between the predicted semantic feature of each sample audio and its corresponding positive sample, that is, to bring them closer in the feature space, and to minimize the feature similarity between the predicted semantic feature of each sample audio and its corresponding negative sample, that is, to push them away in the feature space, therefore, if the feature similarity between the positive sample is high and the feature similarity between the negative sample is low, it can be determined that the contrast loss is small, and vice versa if the feature similarity between the positive sample is low and / or the feature similarity between the negative sample is high, it can be determined that the contrast loss is large.

[0115] After that, the initial adapter can be iterated according to the contrast loss, that is, the parameters of the initial adapter that can be learned are adjusted according to the calculated contrast loss, that is, parameter iteration. The direction of iteration is towards the direction that can reduce the contrast loss. This process will be repeated on the entire sample data set until the loss converges or reaches the preset training number. After training, the obtained model is the final adapter that can achieve accurate acoustic semantic alignment.

[0116] In the embodiment of the application, the contrast learning strategy utilizes the inherent pairing relationship of the data itself to construct the learning target, without additional labeling, and can efficiently learn the complex mapping of acoustic features to semantic space, forcing the initial adapter to not only learn to identify the correct acoustic semantic pairing, but also to learn to distinguish the incorrect pairing, so that the final obtained adapter has strong discrimination ability and can generate semantic features highly aligned with the text semantics, thereby providing favorable data support for subsequent speech recognition models to perform accurate multi-modal semantic error correction.

[0117] The application further provides a speech recognition method, Figure 4 is a second flowchart of the speech recognition method provided by the application, as Figure 4 shown, the method is applied to a cloud server, and the method comprises the following steps. Step 410: receiving the acoustic features, language information and initial recognition results of the speech signal sent by the terminal device in the case where the confidence degrees corresponding to the initial recognition results of the speech signal to be recognized are all less than a confidence threshold; the acoustic features are obtained by performing acoustic feature extraction on the speech signal based on the language information, and the initial recognition results are obtained by decoding the acoustic features; At step 420, based on the initial recognition result, the acoustic feature and the language information, a speech recognition model is applied for semantic error correction to obtain a speech recognition result of the speech signal, and the speech recognition result is returned to the terminal device; the speech recognition model is constructed on the basis of a large language model.

[0118] Specifically, the cloud server, as a platform with powerful computing and storage capabilities, can handle complex error correction tasks that the terminal device cannot complete with high quality. The cloud server and the terminal device communicate through a network.

[0119] In detail, before speech recognition, the recognition object, i.e., the speech signal to be recognized, needs to be obtained first. Here, the speech signal to be recognized can be real-time collected by the terminal device through the built-in microphone array, can be obtained by directly reading the corresponding audio file stored in the terminal device when recognition is needed, or can be obtained by other means, which is not limited in the embodiment of the application.

[0120] The speech signal to be recognized can be a single language speech signal, such as a pure Chinese or pure English speech signal; or can be a mixed speech signal containing multiple languages, such as a speech signal containing Chinese, English, Korean, etc.; the embodiment of the application does not make specific limitation. In the embodiment of the application, in order to solve the problem of current multi-language speech recognition, as a preferred embodiment, the speech signal here can be a mixed speech signal containing multiple languages.

[0121] After determining the speech signal to be recognized, in order to improve the accuracy of multi-language speech recognition and ensure the accuracy of the final recognition result, the language information of the speech signal also needs to be determined. The language information is used to indicate the language types contained in the speech signal.

[0122] The language information can be provided by the user, such as input through the interactive interface on the terminal device, frame selection of the language types corresponding to the speech signal, and the terminal device can determine the language information of the speech signal after receiving the user's instruction; or can be obtained by algorithm, model, etc., such as a lightweight language recognition model, such as Language Identification, can be built in the terminal device, which can quickly analyze the speech signal and output the language types contained therein, thereby obtaining the language information of the speech signal; or can be inferred by the context, such as in the continuous dialogue of the car-mounted voice assistant or the two-way conversation in the conference scene, the language of the subsequent speech signal can be assumed to be consistent with that of the previous dialogue.

[0123] It should be noted that the purpose of providing the language information of the speech signal to be recognized in the process of multi-language speech recognition is to further improve the recognition accuracy and maximize the recognition accuracy. If the language information is not provided or the language information of the speech signal to be recognized cannot be obtained, multi-language speech recognition can still be performed. In the subsequent recognition process, the terminal device can automatically perceive the language information, and the recognition effect is guaranteed. Compared with the current multi-language speech recognition technology, the recognition accuracy is greatly improved.

[0124] Further, after determining the speech signal to be recognized and the language information of the speech signal, in the embodiment of the present application, the speech signal can be processed on the terminal device side first, that is, speech recognition is performed based on this. The speech recognition process is the preliminary recognition of the terminal device, and the result obtained corresponds to the initial speech recognition result.

[0125] Specifically, the terminal device can first extract acoustic features of the speech signal using the language information, thereby obtaining acoustic features of the speech signal. That is, the original one-dimensional time-domain speech signal waveform is converted into a multi-dimensional feature vector sequence that is easier for the terminal device to understand and process. Specifically, an acoustic model, a traditional acoustic feature extraction algorithm can be used to extract acoustic features of the speech signal, such as extracting mel frequency cepstral coefficient features or Fbank features. The acoustic features of the speech signal can also be extracted through a cross-language language model, or the acoustic feature extraction can be realized through a coding and decoding architecture, which is not limited in the embodiment of the present application.

[0126] As a preferred embodiment, the acoustic feature extraction is completed by an encoder in a coding and decoding architecture. That is, the speech signal to be recognized and the language information can be input, and the encoder can process the speech signal according to the language information, thereby outputting a high-dimensional vector sequence that can represent the content of the speech signal, that is, the acoustic features. The acoustic features fuse the acoustic information and the language information of the speech signal, and provide high-quality input for subsequent decoding.

[0127] Subsequently, after obtaining the acoustic features of the speech signal, the terminal device can decode the acoustic features to obtain a plurality of initial recognition results. This process is essentially mapping a high-dimensional vector sequence to a corresponding text sequence, which can be realized through a decoder. Specifically, the acoustic features of the speech signal extracted by the acoustic feature extraction can be input into the decoder, and the decoder can perform decoding processing thereon, and output a probability distribution through a search algorithm such as bundle search. The probability distribution contains a plurality of recognition texts and the confidence of each recognition text. Since the recognition texts obtained at this time are the preliminary recognition results generated by the terminal device locally, they can also be called initial recognition results.

[0128] The decoder can be a Connectionist Temporal Classification (CTC) decoder, an RNN-T (Recurrent Neural Network Transducer) decoder, or another decoder, and embodiments of the present application do not make specific limitations thereto.

[0129] After that, the confidence of the initial recognition result can be determined, i.e., the terminal device checks the confidence corresponding to each initial recognition result. The confidence is a quantitative score, which reflects the "degree of certainty" of the terminal device that the recognition result is correct. The higher the confidence, the higher the possibility that the recognition result is correct, and vice versa.

[0130] Further, if the confidence corresponding to each initial recognition result is less than a confidence threshold, it can be determined that the terminal device does not have enough "confidence" and needs to request the cloud server for assistance. That is, the terminal device sends the local generated multiple initial recognition results, the acoustic features output by the encoder, and the language information to the cloud server. The confidence threshold is a value that can be dynamically adjusted through experiments or according to the requirements of the application scenario, such as 0.75 or 0.8.

[0131] It is worth noting that the terminal device does not transmit the original voice signal in the process of transmitting information, but only transmits the acoustic features processed by the encoder. This not only greatly reduces the data transmission amount and reduces the network bandwidth consumption, but more importantly, it effectively protects the privacy and security of the user and avoids the risk of privacy leakage of the original voice signal in the transmission process. In order to further ensure data security during data transmission, the transmitted information can also be encrypted.

[0132] The cloud server can receive the initial recognition results, acoustic features, and language information transmitted by the terminal device, and can call the speech recognition model to process the received information to achieve semantic error correction. Finally, the speech recognition result can be obtained and returned to the terminal device. Since the cloud server has powerful computing resources, it can deploy and run a speech recognition model with a large number of parameters to perform more accurate deep analysis and error correction, obtain a high-quality speech recognition result, and return it to the terminal device.

[0133] Here, the speech recognition model is constructed on the basis of a large language model (referred to as a large model). The large language model is a pre-trained language model based on a Transformer architecture and having a parameter quantity of hundreds of millions or more, and has strong natural language understanding, analysis and processing (such as semantic reasoning and text generation) capabilities. Therefore, a speech recognition model can be constructed based on this to be used for semantic error correction.

[0134] In the speech recognition model, when performing semantic error correction, the speech recognition model analyzes each initial recognition result in combination with acoustic features and language information, such as analyzing the differences between multiple initial recognition results and using the strong language knowledge to evaluate the grammatical structure, word collocation and semantic rationality of each initial recognition result; for example, analyzing the matching degree between each initial recognition result and acoustic features, such as which initial recognition result is more matched with the real pronunciation; for example, analyzing the adaptation degree of the intermediate result to the language during the error correction process to ensure that the final recognition result conforms to the language habits of the corresponding language; and finally the speech recognition result of the speech signal can be obtained.

[0135] Correspondingly, if the confidence of any one or at least one initial recognition result is greater than or equal to the confidence threshold, it can be determined that the terminal device has a very high "confidence" for this initial recognition result at this time, and the cloud server does not need to be requested to assist in processing at this time. That is, the initial recognition result with the highest confidence can be directly selected as the final speech recognition result. In this case, the speech recognition result is determined locally on the terminal device, which not only avoids network interaction with the cloud, but also greatly reduces the recognition delay and ensures high real-time performance in the simple and clear speech input scenario.

[0136] In the embodiment of the application, through comprehensive consideration of multi-modal information, the speech recognition model can accurately correct errors (such as spelling errors, homonym errors, syntax errors, etc.) in the initial recognition result, and finally output an identification text that is most coherent in semantics and most matched with acoustic features, i.e., a speech recognition result, thereby realizing multi-language speech recognition and obtaining an accurate and reliable speech recognition result.

[0137] The speech recognition method provided by the application, in the case that the confidence degrees corresponding to multiple initial recognition results of a speech signal to be recognized of a terminal device are all less than a confidence degree threshold, the speech signal acoustic features, language information and each initial recognition result are sent; the acoustic features are obtained by performing acoustic feature extraction on the speech signal based on the language information, and each initial recognition result is obtained by decoding the acoustic features; based on each initial recognition result, the acoustic features and the language information, a speech recognition model is applied to perform semantic error correction to obtain and return a speech recognition result, overcoming the defects of low multi-language speech recognition accuracy, easy misjudgment and privacy leakage risk, through an end-cloud collaborative mechanism, on-demand allocation of computing resources is realized, the cloud only intervenes when the terminal device encounters recognition difficulty, while ensuring low delay and high real-time of local recognition of the terminal device in a simple scenario, the powerful ability of the cloud large model is used to ensure high recognition accuracy in a complex scenario. In addition, only acoustic features and initial recognition results are transmitted during data transmission, while realizing high-performance recognition, privacy protection and network bandwidth efficiency are taken into account.

[0138] The speech recognition device provided by the application is described below, and the speech recognition device described below can be mutually corresponding with the speech recognition method described above.

[0139] Figure 5 is one of the structure schematic diagrams of the speech recognition device provided by the application, as Figure 5 indicated, the device is applied to a terminal device, and the device comprises: A speech information determination unit 510 is configured to determine a speech signal to be recognized and language information of the speech signal. An initial result determination unit 520 is configured to perform acoustic feature extraction on the speech signal based on the language information to obtain acoustic features of the speech signal, and decode the acoustic features to obtain multiple initial recognition results of the speech signal. A recognition result determination unit 530 is configured to determine a speech recognition result of the speech signal; the speech recognition result is obtained by applying a speech recognition model to perform semantic error correction based on each initial recognition result, the acoustic features and the language information. The speech recognition model is constructed based on a large language model.

[0140] The speech recognition device provided by the application extracts acoustic features of a speech signal based on language information, obtains acoustic features of the speech signal, decodes the acoustic features, and obtains multiple initial recognition results of the speech signal; determines a speech recognition result of the speech signal; the speech recognition result is obtained by applying a speech recognition model to semantic error correction on the basis of each initial recognition result, acoustic features, and language information; the speech recognition model is constructed on the basis of a large language model, and overcomes the defects of low multi-language speech recognition accuracy and easy misjudgment in the traditional scheme. Through a two-step recognition process, multiple initial recognition results are quickly generated locally, and the powerful semantic understanding capability of the large model is utilized to perform multi-modal deep error correction by fusing acoustic features and language information, thereby greatly improving the accuracy of multi-language speech recognition, especially for languages with a large number of similar phonemes or unique grammar structures, and for some complex or ambiguous sentences, the recognition accuracy is greatly improved.

[0141] Based on the above embodiment, the initial result determination unit 520 is configured to: input the speech signal and the language information into a first encoding layer of an encoder to obtain initial acoustic features output by the first encoding layer; input the initial acoustic features into a second encoding layer in the encoder to obtain the acoustic features output by the second encoding layer; The second encoding layer includes a plurality of expert models corresponding to the language information and a feature fusion layer. Each expert model is configured to model acoustic features based on the initial acoustic features to obtain expert acoustic features. The feature fusion layer is configured to fuse the expert acoustic features to obtain the acoustic features.

[0142] Based on the above embodiment, the initial result determination unit 520 is configured to: input the initial acoustic features into a convolution self-attention block in the second encoding layer to obtain intermediate acoustic features output by the convolution self-attention block; input the intermediate acoustic features into a plurality of expert models in the second encoding layer to obtain the expert acoustic features output by the expert models; input the expert acoustic features into a feature fusion layer in the second encoding layer to obtain the acoustic features output by the feature fusion layer; The feature fusion layer is configured to determine weights of the expert models based on the intermediate acoustic features and the expert acoustic features, and fuse the expert acoustic features based on the weights of the expert models to obtain the acoustic features.

[0143] Based on the above embodiment, the feature fusion layer is configured to concatenate the intermediate acoustic feature and the expert acoustic feature output by any expert model, and project the concatenated feature to obtain the context feature of the any expert model. The feature fusion layer is further configured to project the context feature of the any expert model to obtain a target context feature of the any expert model, and determine the weight of the expert model based on the target context feature of the expert model.

[0144] Based on the above embodiment, the apparatus further includes a training unit configured to: determine a training data set and a current training mode; the training data set includes a plurality of groups of training data, and each group of training data includes a training audio, a training text corresponding to the training audio, and language information corresponding to the training audio; the current training mode is obtained by probabilistic sampling of an accurate prior mode and a zero prior mode; based on the initial encoder, determine an initial acoustic feature of the training audio in each group of training data, and based on the initial acoustic feature and the language information of each training audio, determine a training acoustic feature of the training audio in the case that the current training mode is the accurate prior mode, and based on the initial acoustic feature of each training audio, determine a training acoustic feature of the training audio in the case that the current training mode is the zero prior mode; decode each training acoustic feature to obtain a predicted text corresponding to the training audio in each group of training data; based on the predicted text and the training text corresponding to the training audio in each group of training data, perform parameter iteration on the initial encoder to obtain the encoder.

[0145] Based on the above embodiment, the training unit is configured to: based on the expert model, perform acoustic feature modeling on the initial acoustic feature of the any training audio to obtain an expert acoustic feature of the any training audio output by the expert model; in the case that the current training mode is the accurate prior mode, take the expert model corresponding to the language information of the any training audio as a target expert model, and perform feature fusion on the expert acoustic feature of the any training audio output by the target expert model to obtain a training acoustic feature of the any training audio; in the case that the current training mode is the zero prior mode, perform feature fusion on the expert acoustic feature of the any training audio output by the expert model to obtain a training acoustic feature of the any training audio.

[0146] Based on the above embodiment, the recognition result determination unit 530 is configured to: In a case where the confidence degree corresponding to each initial recognition result is less than the confidence degree threshold, the initial recognition result, the acoustic feature, and the language information are sent to a cloud server, so that the cloud server applies the speech recognition model to perform semantic correction based on the initial recognition result, the acoustic feature, and the language information, and returns a speech recognition result of the speech signal; In a case where the confidence degree corresponding to any initial recognition result is greater than or equal to the confidence degree threshold, a speech recognition result of the speech signal is determined based on the initial recognition result.

[0147] Based on the above embodiment, the cloud server is configured to align the acoustic feature to a semantic space of the speech recognition model through an adapter to obtain a semantic feature, and input the semantic feature, the initial recognition result, and the language information into the speech recognition model to obtain a speech recognition result output by the speech recognition model; The speech recognition model is configured to determine semantic information of the initial recognition result, and matching information between the initial recognition result and the semantic feature, and perform semantic correction on the initial recognition result based on the semantic information, the matching information, and the language information to obtain a speech recognition result of the speech signal.

[0148] Based on the above embodiment, the training unit is further configured to: determine a sample data set, the sample data set containing a plurality of sample audio text pairs of different languages, the sample audio text pair containing a sample audio and a sample text corresponding to the sample audio; determine a sample acoustic feature of the sample audio and a sample semantic feature of the sample text in each sample audio text pair; based on an initial adapter, determine a predicted semantic feature corresponding to each sample acoustic feature; the initial adapter is constructed based on a learnable linear transformation layer and a nonlinear activation function; based on a feature similarity between the predicted semantic feature corresponding to each sample acoustic feature and a sample semantic feature of a sample text in a sample audio text pair in which the sample audio corresponding to the sample acoustic feature is located, and a feature similarity between the predicted semantic feature corresponding to each sample acoustic feature and a sample semantic feature of a sample text in other sample audio text pairs, determine a contrastive loss; based on the contrastive loss, perform parameter iteration on the initial adapter to obtain the adapter.

[0149] The speech recognition device provided by the application is described below, and the speech recognition device described below can be mutually corresponding and referred to with the speech recognition method described above.

[0150] Figure 6This is a second structural schematic diagram of the voice recognition device provided by the present invention, as shown below. Figure 6 As shown, this device is used in a cloud server and includes: The receiving unit 610 is configured to receive the acoustic features, language information, and initial recognition results of the speech signal transmitted by the terminal device when the confidence levels corresponding to multiple initial recognition results of the speech signal to be recognized are all less than a confidence threshold; the acoustic features are obtained by extracting acoustic features from the speech signal based on the language information, and the initial recognition results are obtained by decoding the acoustic features. The recognition unit 620 is used to perform semantic error correction by applying a speech recognition model based on the initial recognition results, the acoustic features and the language information, to obtain the speech recognition result of the speech signal, and to return the speech recognition result to the terminal device; The speech recognition model is built on top of a large-scale language model.

[0151] The speech recognition device provided by this invention receives the acoustic features, language information, and initial recognition results of a speech signal sent by a terminal device when the confidence levels corresponding to multiple initial recognition results of the speech signal to be recognized are all less than a confidence threshold. The acoustic features are obtained by extracting acoustic features from the speech signal based on the language information, and each initial recognition result is obtained by decoding the acoustic features. Based on each initial recognition result, acoustic features, and language information, a speech recognition model is applied for semantic error correction to obtain and return the speech recognition result. This overcomes the shortcomings of current multilingual speech recognition, such as low accuracy, susceptibility to misjudgment, and privacy leakage risks. Through an end-to-cloud collaborative mechanism, computing resources are allocated on demand, with the cloud only intervening when the terminal device encounters recognition difficulties. While ensuring low latency and high real-time performance for local recognition on the terminal device in simple scenarios, the powerful capabilities of the cloud's large model ensure high recognition accuracy in complex scenarios. Furthermore, only acoustic features and initial recognition results are transmitted during data transmission, achieving high-performance recognition while also considering privacy protection and network bandwidth efficiency.

[0152] The present invention also provides a speech recognition system. Figure 7 This is a schematic diagram of the speech recognition system provided by the present invention, as shown below. Figure 7 As shown, the system includes terminal devices and a cloud server; The terminal device is configured to extract acoustic features from the speech signal based on the language information of the speech signal to be identified, obtain the acoustic features of the speech signal, and decode the acoustic features to obtain multiple initial recognition results of the speech signal; if the confidence level corresponding to each initial recognition result is less than the confidence level threshold, the initial recognition results, the acoustic features, and the language information are sent to the cloud server. The cloud server is configured to receive the initial recognition results, the acoustic features and the language information sent by the terminal device, and based on the initial recognition results, the acoustic features and the language information, apply a speech recognition model to perform semantic error correction to obtain a speech recognition result of the speech signal, and return the speech recognition result to the terminal device; the speech recognition model is constructed based on a large language model.

[0153] The speech recognition system provided by the application comprises a terminal device and a cloud server; the terminal device is configured to extract acoustic features from a speech signal by using language information to obtain acoustic features, and decode the acoustic features to obtain a plurality of initial recognition results; when the confidence levels of the initial recognition results are all less than a confidence threshold, the terminal device sends the initial recognition results, the acoustic features and the language information to the cloud server; the cloud server is configured to receive the initial recognition results, the acoustic features and the language information, and based on the initial recognition results, the acoustic features and the language information, apply a speech recognition model to perform semantic error correction to obtain and return a speech recognition result, thereby overcoming the defects of low multi-language speech recognition accuracy, easy misjudgment and privacy leakage risk, and realizing on-demand allocation of computing resources through an end-cloud collaborative mechanism; the cloud server only intervenes when the terminal device encounters recognition difficulties, thereby ensuring low delay and high real-time performance of local recognition of the terminal device in a simple scenario, and using the powerful capability of the cloud server to ensure high recognition accuracy in a complex scenario. In addition, only acoustic features and initial recognition results are transmitted during data transmission, thereby balancing privacy protection and network bandwidth efficiency.

[0154] Figure 8 An example of an entity structure diagram of an electronic device is shown in FIG. 1. Figure 8As shown, the electronic device can include a processor 810, a communications interface 820, a memory 830, and a communications bus 840, wherein the processor 810, the communications interface 820, and the memory 830 complete mutual communication through the communications bus 840. The processor 810 can invoke a logical instruction in the memory 830 to execute a speech recognition method, which includes determining a speech signal to be recognized and language information of the speech signal; performing acoustic feature extraction on the speech signal based on the language information to obtain acoustic features of the speech signal, and decoding the acoustic features to obtain a plurality of initial recognition results of the speech signal; determining a speech recognition result of the speech signal; the speech recognition result is obtained by applying a speech recognition model for semantic error correction on the basis of each initial recognition result, the acoustic features, and the language information; or the speech recognition model is constructed on the basis of a large language model; or includes receiving acoustic features, language information, and each initial recognition result of the speech signal sent by a terminal device in a case where confidence degrees corresponding to the plurality of initial recognition results of the speech signal to be recognized are all less than a confidence threshold; the acoustic features are obtained by performing acoustic feature extraction on the speech signal based on the language information, and the each initial recognition result is obtained by decoding the acoustic features; applying a speech recognition model for semantic error correction on the basis of the each initial recognition result, the acoustic features, and the language information to obtain a speech recognition result of the speech signal, and returning the speech recognition result to the terminal device; and the speech recognition model is constructed on the basis of a large language model.

[0155] In addition, the logical instructions in the memory 830 described above can be implemented in the form of a software function unit and sold or used as an independent product, and can be stored in a computer-readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in various embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0156] In another aspect, the present application also provides a computer program product, which comprises a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions which, when executed by a computer, enable the computer to perform the speech recognition method provided by any of the above methods, the method comprising: determining a speech signal to be recognized, and language information of the speech signal; performing acoustic feature extraction on the speech signal based on the language information to obtain acoustic features of the speech signal, and decoding the acoustic features to obtain a plurality of initial recognition results of the speech signal; determining a speech recognition result of the speech signal; the speech recognition result being obtained by applying a speech recognition model to perform semantic error correction on the basis of each initial recognition result, the acoustic features and the language information; the speech recognition model being constructed on the basis of a large language model; or comprising: receiving acoustic features, language information and each initial recognition result of a speech signal sent by a terminal device in a case where confidence degrees corresponding to a plurality of initial recognition results of the speech signal to be recognized are all less than a confidence threshold; the acoustic features being obtained by performing acoustic feature extraction on the speech signal on the basis of the language information, and the each initial recognition result being obtained by decoding the acoustic features; applying a speech recognition model to perform semantic error correction on the basis of the each initial recognition result, the acoustic features and the language information to obtain a speech recognition result of the speech signal, and returning the speech recognition result to the terminal device; the speech recognition model being constructed on the basis of a large language model.

[0157] In yet another aspect, the present application also provides a non-transitory computer-readable storage medium having stored thereon a computer program, which, when executed by a processor, implements the speech recognition method provided by any of the above methods, and the method comprises: determining a speech signal to be recognized, and language information of the speech signal; performing acoustic feature extraction on the speech signal based on the language information to obtain acoustic features of the speech signal, and decoding the acoustic features to obtain a plurality of initial recognition results of the speech signal; determining a speech recognition result of the speech signal; the speech recognition result is obtained by applying a speech recognition model to perform semantic correction on the basis of each initial recognition result, the acoustic features and the language information; or comprising: receiving acoustic features, language information and each initial recognition result of the speech signal sent by a terminal device in a case where confidence degrees corresponding to the plurality of initial recognition results of the speech signal to be recognized are all less than a confidence threshold; the acoustic features are obtained by performing acoustic feature extraction on the speech signal based on the language information, and the each initial recognition result is obtained by decoding the acoustic features; applying a speech recognition model to perform semantic correction based on the each initial recognition result, the acoustic features and the language information to obtain a speech recognition result of the speech signal, and returning the speech recognition result to the terminal device; and the speech recognition model is constructed on the basis of a large language model.

[0158] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment scheme according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0159] From the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be realized by means of software plus a necessary general hardware platform, and of course can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0160] It should be pointed out finally that the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit the same; and although the present application has been described in detail with reference to the foregoing embodiments, it should be appreciated by those skilled in the art that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features thereof can be replaced equivalently; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A voice recognition method, characterized by, The application is applied to a terminal device, comprising: determining a voice signal to be recognized and language information of the voice signal; based on the language information, performing acoustic feature extraction on the voice signal to obtain acoustic features of the voice signal, and decoding the acoustic features to obtain a plurality of initial recognition results of the voice signal; determining a voice recognition result of the voice signal; the voice recognition result is obtained by applying a voice recognition model to perform semantic correction on the basis of each initial recognition result, the acoustic features and the language information; the voice recognition model is constructed on the basis of a large language model.

2. The voice recognition method of claim 1, wherein, the acoustic feature extraction based on the language information comprises: inputting the voice signal and the language information into a first encoding layer of an encoder to obtain initial acoustic features output by the first encoding layer; inputting the initial acoustic features into a second encoding layer in the encoder to obtain the acoustic features output by the second encoding layer; wherein the second encoding layer comprises a plurality of expert models corresponding to the language information and a feature fusion layer; each expert model is used to model acoustic features based on the initial acoustic features to obtain expert acoustic features; and the feature fusion layer is used to fuse each expert acoustic feature to obtain the acoustic features.

3. The voice recognition method of claim 2, wherein, the inputting of the initial acoustic features into the second encoding layer in the encoder to obtain the acoustic features output by the second encoding layer comprises: inputting the initial acoustic features into a convolutional self-attention block in the second encoding layer to obtain intermediate acoustic features output by the convolutional self-attention block; inputting the intermediate acoustic features into a plurality of expert models in the second encoding layer to obtain the expert acoustic features output by the expert models; inputting each expert acoustic feature into a feature fusion layer in the second encoding layer to obtain the acoustic features output by the feature fusion layer; the feature fusion layer is used to determine the weights of the expert models based on the intermediate acoustic features and the expert acoustic features, and fuse the expert acoustic features based on the weights of the expert models to obtain the acoustic features.

4. The voice recognition method of claim 3, wherein: the feature fusion layer is used to splice the intermediate acoustic features and the expert acoustic features output by any expert model, and project the spliced features to obtain context features of the any expert model; the feature fusion layer is further used to project the context features of the any expert model to obtain target context features of the any expert model, and determine the weights of the expert models based on the target context features of the expert models.

5. The speech recognition method of any one of claims 2 to 4, characterized in that, the encoder is trained based on the following steps: determining a training data set and a current training mode; the training data set comprises a plurality of groups of training data, and each group of training data comprises training audio, training text corresponding to the training audio, and language information corresponding to the training audio; The current training mode is obtained by probabilistic sampling of the accurate prior mode and the zero prior mode; Based on the initial encoder, the initial acoustic features of the training audios in each group of training data are determined, and in the case that the current training mode is the accurate prior mode, the training acoustic features of each training audio are determined based on the initial acoustic features of each training audio and the language information, and in the case that the current training mode is the zero prior mode, the training acoustic features of each training audio are determined based on the initial acoustic features of each training audio; Each training acoustic feature is decoded to obtain the predicted text corresponding to the training audio in each group of training data; Based on the predicted text and the training text corresponding to the training audio in each group of training data, the parameters of the initial encoder are iterated to obtain the encoder.

6. The voice recognition method of claim 5, wherein, The training acoustic feature of any training audio is determined based on the following steps: Based on the expert models, the initial acoustic features of any training audio are modeled to obtain the expert acoustic features of any training audio output by the expert models; In the case that the current training mode is the accurate prior mode, the expert model corresponding to the language information of any training audio is taken as a target expert model, and the expert acoustic features of any training audio output by the target expert model are fused to obtain the training acoustic features of any training audio; In the case that the current training mode is the zero prior mode, the expert acoustic features of any training audio output by the expert models are fused to obtain the training acoustic features of any training audio.

7. The speech recognition method of any one of claims 1 to 4, characterized in that, The determination of the speech recognition result of the voice signal comprises: In the case that the confidence of each initial recognition result is less than a confidence threshold, the initial recognition results, the acoustic features and the language information are sent to a cloud server, so that the cloud server applies the speech recognition model to perform semantic correction based on the initial recognition results, the acoustic features and the language information, to obtain and return the speech recognition result of the voice signal; In the case that the confidence of any initial recognition result is greater than or equal to the confidence threshold, the speech recognition result of the voice signal is determined based on the initial recognition results.

8. The speech recognition method of claim 7, wherein The cloud server is configured to align the acoustic features to a semantic space of the speech recognition model through an adapter to obtain semantic features, and input the semantic features, the initial recognition results and the language information into the speech recognition model to obtain the speech recognition result output by the speech recognition model; The speech recognition model is configured to determine semantic information of the initial recognition results, and matching information between the initial recognition results and the semantic features, and perform semantic correction on the initial recognition results based on the semantic information, the matching information and the language information, to obtain the speech recognition result of the voice signal.

9. The voice recognition method of claim 8, wherein, The adapter is trained based on the following steps: determining a sample data set containing a plurality of sample audio-text pairs in different languages, each sample audio-text pair containing a sample audio and a sample text corresponding to the sample audio; determining sample acoustic features of the sample audio and sample semantic features of the sample text in each sample audio-text pair; determining predicted semantic features corresponding to each sample acoustic feature based on an initial adapter; the initial adapter is constructed based on a learnable linear transformation layer and a nonlinear activation function; determining a contrastive loss based on the feature similarity between the predicted semantic features corresponding to each sample acoustic feature and the sample semantic features of the sample text in the sample audio-text pair in which the sample audio corresponding to the sample acoustic feature is located, and the feature similarity between the predicted semantic features corresponding to each sample acoustic feature and the sample semantic features of the sample text in other sample audio-text pairs; iterating parameters of the initial adapter based on the contrastive loss to obtain the adapter.

10. A voice recognition method, characterized by, Applied to a cloud server, comprising: receiving acoustic features, language information and each initial recognition result of a voice signal sent by a terminal device in a case where confidence degrees corresponding to a plurality of initial recognition results of the voice signal to be recognized are all less than a confidence threshold; the acoustic features are obtained by performing acoustic feature extraction on the voice signal based on the language information, and each initial recognition result is obtained by decoding the acoustic features; applying a speech recognition model to perform semantic correction based on the initial recognition results, the acoustic features and the language information to obtain a speech recognition result of the voice signal, and returning the speech recognition result to the terminal device; the speech recognition model is constructed based on a large language model.

11. A speech recognition apparatus, characterized by comprising: Applied to a terminal device, comprising: a voice information determination unit configured to determine a voice signal to be recognized and language information of the voice signal; an initial result determination unit configured to perform acoustic feature extraction on the voice signal based on the language information to obtain acoustic features of the voice signal, and decode the acoustic features to obtain a plurality of initial recognition results of the voice signal; a recognition result determination unit configured to determine a speech recognition result of the voice signal; the speech recognition result is obtained by applying a speech recognition model to perform semantic correction based on each initial recognition result, the acoustic features and the language information; the speech recognition model is constructed based on a large language model.

12. A speech recognition apparatus, characterized by comprising: Applied to a cloud server, comprising: a receiving unit configured to receive acoustic features, language information and each initial recognition result of a voice signal sent by a terminal device in a case where confidence degrees corresponding to a plurality of initial recognition results of the voice signal to be recognized are all less than a confidence threshold; the acoustic features are obtained by performing acoustic feature extraction on the voice signal based on the language information, and each initial recognition result is obtained by decoding the acoustic features; The recognition unit is configured to apply a speech recognition model to perform semantic correction based on the initial recognition results, the acoustic features, and the language information, to obtain a speech recognition result of the speech signal, and to return the speech recognition result to the terminal device. The speech recognition model is constructed based on a large language model.

13. A speech recognition system characterized by The system comprises a terminal device and a cloud server. The terminal device is configured to perform acoustic feature extraction on the speech signal based on language information of the speech signal to be recognized, to obtain acoustic features of the speech signal, and to decode the acoustic features to obtain multiple initial recognition results of the speech signal. In a case where the confidence levels corresponding to the initial recognition results are all less than a confidence threshold, the initial recognition results, the acoustic features, and the language information are sent to the cloud server. The cloud server is configured to receive the initial recognition results, the acoustic features, and the language information sent by the terminal device, and to apply a speech recognition model to perform semantic correction based on the initial recognition results, the acoustic features, and the language information, to obtain a speech recognition result of the speech signal, and to return the speech recognition result to the terminal device. The speech recognition model is constructed based on a large language model.

14. An electronic device comprising a memory, a processor, and a computer program stored on the memory and running on the processor, characterized in that, The processor executes the computer program to implement the speech recognition method according to any one of claims 1 to 10. 15.A non-transitory computer-readable storage medium having stored thereon a computer program. The computer program is executed by the processor to implement the speech recognition method according to any one of claims 1 to 10.

Images