Speech recognition method and device, computer device and storage medium

Abstract

The present disclosure relates to the technical field of speech recognition, and discloses a speech recognition method and device, computer equipment and a storage medium. The method provided by the present disclosure comprises: obtaining a speech to be recognized; performing encoding processing based on the speech to be recognized to obtain acoustic coding information; performing information weight prediction corresponding to each language on the acoustic coding information to obtain information weights corresponding to the languages one by one, wherein the information weights comprise acoustic information and boundaries of the languages; fusing the information weights of the languages to obtain fused information weights; and performing decoding processing based on the fused information weights to obtain a speech recognition result. The method can distinguish the acoustic boundaries of each language in the speech recognition process by predicting the acoustic boundaries of each language independently, thereby improving the accuracy of the speech recognition result.

Description

Technical Field

[0001] This disclosure relates to the field of speech recognition, specifically to speech recognition methods, devices, computer equipment, and storage media. Background Technology

[0002] Code-switching (CS) refers to the alternation of two or more languages ​​in speech output. However, most existing speech recognition technologies focus on monolingual speech recognition, meaning they can only process one language at a time. Therefore, this speech recognition method is difficult to apply to scenarios involving two or more languages, leading to speech recognition errors. Summary of the Invention

[0003] In view of this, the present disclosure provides a speech recognition method, apparatus, computer device, and storage medium to solve the problem of low accuracy in speech recognition.

[0004] In a first aspect, this disclosure provides a speech recognition method, the method comprising:

[0005] Acquire the speech to be recognized;

[0006] The speech to be recognized is encoded to obtain acoustic coding information.

[0007] Information weight prediction corresponding to each language is performed on the acoustic coding information to obtain information weights that correspond one-to-one with the language. The information weights include the acoustic information and boundaries of the language.

[0008] The information weights of each language are combined to obtain the combined information weight;

[0009] The speech recognition result is obtained by decoding based on the fused information weights.

[0010] Secondly, this disclosure provides a speech recognition device, the device comprising:

[0011] The speech acquisition module is used to acquire the speech to be recognized;

[0012] The encoding module is used to perform encoding processing based on the speech to be recognized to obtain acoustic coding information;

[0013] The weight prediction module is used to predict the information weights corresponding to each language for the acoustic coding information, so as to obtain information weights that correspond one-to-one with the language. The information weights include the acoustic information and boundaries of the language.

[0014] The fusion module is used to fuse the information weights of the various languages ​​to obtain fused information weights.

[0015] The decoding module is used to perform decoding processing based on the fused information weights to obtain the speech recognition result.

[0016] Thirdly, this disclosure provides a computer device, including: a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to perform the speech recognition method of the first aspect or any corresponding embodiment described above.

[0017] Fourthly, this disclosure provides a computer-readable storage medium storing computer instructions for causing a computer to execute the speech recognition method of the first aspect or any corresponding embodiment described above.

[0018] The speech recognition method provided in this disclosure performs information weight prediction for each language separately to obtain the acoustic information and boundaries of each language. That is, by predicting the acoustic boundaries of each language independently, the acoustic boundaries of each language can be distinguished during the speech recognition process. After obtaining the information weights of each language separately, they are then fused to obtain fused information weights. Decoding is then performed on this basis, which can ensure that the speech recognition result is obtained without losing the fusion of each language, thereby improving the accuracy of the speech recognition result. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the specific embodiments of this disclosure or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0020] Figure 1 This is a schematic flowchart of a speech recognition method according to an embodiment of the present disclosure;

[0021] Figure 2 This is a flowchart illustrating a training method for a target speech recognition model according to an embodiment of the present disclosure;

[0022] Figure 3 This is a schematic diagram of a target speech recognition model according to an embodiment of the present disclosure;

[0023] Figure 4 This is a schematic diagram of an information weight prediction unit according to an embodiment of the present disclosure;

[0024] Figures 5a-5b These are schematic diagrams of a first decoder and a second decoder according to embodiments of this disclosure;

[0025] Figure 6 This is a flowchart illustrating a training method for a speech recognition model according to an embodiment of the present disclosure;

[0026] Figure 7 This is a schematic diagram illustrating the training of a speech recognition model according to an embodiment of the present disclosure;

[0027] Figure 8 This is a schematic diagram of language change detection according to an embodiment of the present disclosure;

[0028] Figure 9 This is a structural block diagram of a speech recognition device according to an embodiment of the present disclosure;

[0029] Figure 10 This is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present disclosure. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0031] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0032] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0033] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0034] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0035] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0036] In related technologies, code-switching speech recognition (CSSR) is performed on a hybrid speech recognition framework. However, with the widespread use of end-to-end models, some technologies employ end-to-end (E2E) strategies to solve CSSR tasks. Among these, attention-based E2E models were first applied to CSSR tasks, with language identification used as an auxiliary task to improve speech recognition accuracy. Furthermore, language-aware encoder structures and language-aware training strategies were applied to connection-temporal classification (CTC) models to decipher language-related information during encoding and generate frame-level language-aware representations. For the decoder, language-related attention mechanisms, non-autoregressive structures, and language models based on internal language model estimation were all used to mitigate the confusion caused by code-switching.

[0037] For example, in mixed Chinese and English speech, Mandarin characters typically represent individual syllables, and their acoustic boundaries are well-defined. English words, however, are obtained without any reference to acoustic knowledge, and their acoustic boundaries may be ambiguous. Therefore, language-specific acoustic boundary learning is necessary to obtain good acoustic boundaries for both Chinese and English in CSSR.

[0038] Based on this, the present disclosure provides a speech recognition method, which predicts the information weights of each language in the speech to be recognized to obtain information weights that correspond one-to-one with each language, fuses the information weights of each language to obtain fused information weights, and then decodes them to obtain the speech recognition result.

[0039] According to an embodiment of this disclosure, a speech recognition method embodiment is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0040] This embodiment provides a speech recognition method that can be used in the aforementioned computer devices, such as mobile terminals and computers. Figure 1 This is a flowchart of a speech recognition method according to an embodiment of the present disclosure, such as... Figure 1 As shown, the process includes the following steps:

[0041] Step S101: Obtain the speech to be recognized.

[0042] The speech to be recognized can be speech acquired in real time by the computer device, speech stored in the computer device, or speech obtained from a third-party device through communication between the computer device and the third-party device, etc. There are no restrictions on the source of the speech to be recognized; the specific settings are based on actual needs.

[0043] The speech to be recognized includes at least two languages, such as Chinese and English. Similarly, there is no limitation on the types of languages ​​it contains. Furthermore, the speech to be recognized may also include dialects from different regions; in this embodiment, dialects from different regions are considered different languages. For example, if the speech to be recognized includes Mandarin and a dialect from region A, then Mandarin can be considered language 1, and the dialect from region A can be considered language 2.

[0044] Step S102: Encode the speech to be recognized to obtain acoustic coding information.

[0045] After acquiring the speech to be recognized, it needs to undergo speech feature processing to convert it into a form that can be processed by computer equipment. This includes, but is not limited to, Mel-ray feature extraction of the speech to be recognized, followed by encoding of the extracted features to obtain acoustic coding information. The encoder structure is set according to actual needs; for example, it can be implemented using N stacked conformer modules. Of course, other model structures can also be used, and so on.

[0046] Step S103: Predict the information weights corresponding to each language for the acoustic coding information to obtain information weights that correspond one-to-one with each language.

[0047] The information weights include the acoustic information of the language and the boundaries.

[0048] After obtaining the acoustic coding information, information weight prediction is performed on this information corresponding to each language. Since information weight prediction is language-specific, the method for predicting information weights for each language needs to be determined before speech recognition. That is, in speech recognition methods, the method for information weight prediction is one-to-one with each language.

[0049] Information weight prediction is used to determine the acoustic information and boundaries of each language in the speech to be recognized. For example, for Chinese, the boundary is the acoustic boundary of a single syllable; for English, the boundary is the boundary of English words. The information weight prediction method is to extract features from the acoustic coding information corresponding to each language. That is, feature extraction is performed separately for each language to obtain the acoustic information and boundaries of each language.

[0050] Step S104: Integrate the information weights of each language to obtain the integrated information weights.

[0051] For acoustic encoders, information weights are predicted for each language. The resulting information weights correspond one-to-one with each language, while the speech to be recognized includes various languages. Therefore, after obtaining the information weights for each language, they are fused to obtain fused information weights. These fused information weights are then used for subsequent processing, ensuring that the speech recognition result is applicable to all languages ​​in the speech to be recognized.

[0052] The fusion methods include, but are not limited to, splicing, simple addition, etc. The specific method is set according to actual needs, and no restrictions are imposed on it here.

[0053] Step S105: Decode the speech based on the weights of the fused information to obtain the speech recognition result.

[0054] After obtaining the fused information weights, they are decoded to obtain the speech recognition result of the speech to be recognized. The decoding methods include, but are not limited to, autoregressive decoding, non-autoregressive decoding, etc. The specific settings are determined according to actual needs, and no limitations are imposed here.

[0055] The speech recognition method provided in this embodiment predicts information weights for each language separately to obtain the acoustic information and boundaries of each language. That is, by predicting the acoustic boundaries of each language independently, the acoustic boundaries of each language can be distinguished during the speech recognition process. After obtaining the information weights of each language separately, they are then fused to obtain fused information weights. Decoding is then performed on this basis, which can ensure that the speech recognition result is obtained without losing the fusion of each language, thereby improving the accuracy of the speech recognition result.

[0056] The speech recognition method in this embodiment is obtained by training a target speech recognition model. The input to the target speech recognition model can be the speech to be recognized or its acoustic features, and the output of the target speech recognition model is the speech recognition result. Based on this, this embodiment provides a method for training a speech recognition model, which can be used in computer devices, such as computers. Figure 2 This is a flowchart of a training method for a target speech recognition model according to an embodiment of the present disclosure, such as... Figure 2 As shown, the process includes the following steps:

[0057] Step S201: Obtain the sample speech and its label.

[0058] The tags include those used to characterize the target text.

[0059] Sample speech can be data from a speech dataset or speech collected with the user's consent. The labels for sample speech can be manually assigned or obtained by recording the user reading known text; the known text serves as the label for the sample speech. The languages ​​included in the sample speech determine the languages ​​the target speech recognition model can recognize. For example, if the sample speech includes Chinese and English, the target speech model can handle mixed Chinese and English speech; if the sample speech includes Mandarin and dialect 1, the target speech model can handle mixed Mandarin and dialect 1 speech. Therefore, it is necessary to obtain the corresponding sample speech according to the requirements of the target speech recognition model.

[0060] Step S202: Input the sample speech into the preset speech recognition model to obtain the predicted text.

[0061] The preset speech recognition model is used to predict the prediction information weights of each language, and decodes the fused prediction information weights to obtain the predicted text. The prediction information weights include the acoustic information and boundaries of each language in the sample speech.

[0062] The pre-set speech recognition model and the trained target speech recognition model can have the same or different model structures, but their model parameters will differ. The target speech recognition model is obtained after training the pre-set model. To obtain a higher-performing target speech recognition model, the pre-set model may include additional branches that are not present in the target speech recognition model. Therefore, the model structures of the pre-set and target speech recognition models can be different. Training the speech recognition model involves iteratively updating the parameters of the pre-set model to obtain a target speech recognition model with higher recognition performance. The model parameters of the pre-set speech recognition model can be randomly set, based on empirical values, etc.

[0063] The implementation principle of the preset speech recognition model is to predict the weights of the predicted information for each language, fuse these weights, and then decode the predicted text. Since the predicted information weights are language-specific, the preset speech model includes a language-specific information weight prediction unit. If mixed Chinese and English speech recognition is required, the preset speech model includes two information weight prediction units, one for Chinese and one for English.

[0064] The fusion of prediction information weights from different languages ​​can be achieved through concatenation, addition, or other methods. Decoding the fused prediction information weights is implemented using a decoder, such as an autoregressive decoder.

[0065] In some optional implementations, the preset speech recognition model includes an encoder, an information weight prediction unit corresponding to each language, a fusion unit, and a decoding processing unit. The information weight prediction unit is used to predict the information weight of each language, the fusion unit is used to fuse the predicted information weights of each language to obtain the predicted fused information weight, and the decoding processing unit is used to perform decoding processing based on the predicted fused information weight to obtain the predicted text.

[0066] The information weight prediction unit is language-specific; the fusion unit merges the outputs of each information weight prediction unit to obtain the predicted fused information weight; the decoding processing unit performs decoding processing based on the predicted fused information weight to obtain the predicted text.

[0067] As mentioned above, the model structure of the preset speech recognition model and the target speech recognition model can be the same. For example, Figure 3 This paper illustrates an optional structure for a target speech recognition model, designed to handle both Chinese and English languages. Based on this, the target speech recognition model includes an encoder, an English information weight prediction unit, a Chinese information weight prediction unit, and a fusion unit. Figure 3 (Not explicitly shown) includes the structure between the outputs of two information weight prediction units and the continuous distribution unit.

[0068] In some alternative implementations, Figure 4 The structure of the English information weight prediction unit is shown, consisting of three convolutional (Conv) layers, a fully connected (FC) layer, and an activation layer (Sigmoid). It should be noted that... Figure 4 This is merely an optional example and does not limit the scope of protection of this disclosure. For example, the number of convolutional layers and the type of activation layers included in the information weight prediction unit can be set according to actual needs.

[0069] Combination Figure 4 As shown, the input to the English information weight prediction unit is the encoder output h, and the output is the predicted information weight α corresponding to the English text. en .

[0070] The structure of the Chinese information weight prediction unit can be the same as or different from that of the English information weight prediction unit. With the same structure, the model parameters are different. Taking the case where both structures are the same, the input of the Chinese information weight prediction unit is the encoder output h, and the output is the predicted information weight α corresponding to the Chinese characters. ma .

[0071] By using a language-specific information weight prediction unit, the information weights of each language are predicted, enabling differentiated learning of information weights and improving the accuracy of the learned acoustic boundaries for each language. Furthermore, the predicted information weights of each language are fused by a fusion unit, resulting in a frame-level fusion of the predicted information weights from individual languages.

[0072] In some optional implementations, the decoding processing unit includes a continuous integration and distribution unit and a first decoder. Based on this, the method for determining the predicted text includes:

[0073] Step a1: The predicted fusion information weights are processed using the continuous integration and distribution unit.

[0074] Step a2: Use the first decoder to decode the processing result of the continuous integration and distribution unit to determine the predicted text.

[0075] The decoding processing unit includes a continuous integrate-and-fire (CIF) unit and a first decoder. Before decoding the predicted fusion information weights, the data first passes through the CIF unit. The CIF is a low-complexity, monotonically consistent sequence transformation mechanism that sequentially integrates arriving acoustic information. When the amount of integrated information reaches a recognition threshold, the integrated information is distributed for subsequent recognition. CIF is applied to the encoding / decoding framework. At each encoding time, CIF receives the predicted fusion information weights, continuously accumulates the weights, and integrates them with the acoustic encoded representation. When the accumulated weights reach a threshold, an acoustic boundary is defined. CIF divides the acquired predicted fusion information weights into two parts: one part is used to integrate the acoustic information of the current label, and the other part is used for integrating the acoustic information of the next label. CIF then distributes the integrated result to the first decoder for decoding, outputting the predicted text.

[0076] In some alternative implementations, Figure 5a The structure of the first decoder is shown. In this embodiment, the first decoder is an autoregressive decoder. The input to the first decoder is the result of processing the prediction fusion information weights from the CIF output at the current time. And the first decoder at the previous moment t-1 The output of the first decoder is the predicted text y at the current time t. mix .

[0077] like Figure 5a As shown, since the input of the first decoder includes two parameters, they need to be concatenated before being processed by N. aThe stacked processing units process the data and output fully connected and activation layers (FC & Softmax). Each processing unit includes a stacked attention network (SAN) and a feedforward neural network (FFN).

[0078] It should be noted that, Figure 5a This is merely one example of a first decoder and does not limit the scope of protection of this disclosure. The specific settings can be configured according to actual needs.

[0079] CIF continuously integrates the acoustic information arriving sequentially. When the amount of integrated information reaches the recognition threshold, the integrated information is distributed for subsequent recognition. Before decoding, the predictive fusion information weights are processed using a continuous integration and distribution unit to convert the frame-level acoustic representation into a tag-level acoustic embedding.

[0080] In some optional implementations, the determination methods for the aforementioned predictive fusion information weights include:

[0081] Step b1: Obtain the preset information loss rate.

[0082] Step b2: Based on the preset information loss rate, the predicted information weights for each language are randomly deactivated to obtain the processing results for each language.

[0083] Step b3: The processing results for each language are fused to obtain the predicted fusion information weights.

[0084] As mentioned above, the predicted fusion information weights are obtained by fusing the predicted information weights corresponding to different languages. Before fusing the predicted information weights of each language, random deactivation processing is performed on each language based on a preset information loss rate. This can prevent the speech recognition model from becoming overly dependent on one language and encourage it to learn from all languages.

[0085] The preset information loss rate is set according to actual needs, and its value ranges from (0, 1), for example, 0.1 or 0.2, etc. Figure 3 As shown, the outputs of the information weight prediction units for each language are first randomly deactivated before being fused to obtain the predicted fused information weights. For example, the predicted fused information weights can be expressed using the following formula:

[0086] α mix =Dropout(α) ma ,p)+Dropout(α en Formula (1)

[0087] In the formula, α mix To predict the weights of the fused information, α ma α represents the weight of the predicted information corresponding to the Chinese text. en is the weight of the predicted information corresponding to the English, p is the preset information loss rate, and Dropout is the random deactivation process.

[0088] Step S203: Calculate the loss based on the predicted text and the target text to obtain the loss calculation result.

[0089] After obtaining the predicted text, a loss calculation is performed on the predicted text and the target text based on the corresponding loss function to obtain the loss calculation result. The specific form of the chosen loss function is set according to actual needs and is not limited here.

[0090] Step S204: Iteratively update the parameters of the preset speech recognition model based on the loss calculation results to determine the target speech recognition model.

[0091] After obtaining the loss calculation result, the parameters of the preset speech recognition model are iteratively updated based on the set learning rate and other parameters. The stopping conditions for the iterative update include, but are not limited to, reaching the maximum number of iterations, the loss calculation result being less than the preset loss value, etc.

[0092] After the stopping condition for iterative updates is met, the parameters of the preset speech recognition model are fixed, and the target speech recognition model can be determined accordingly.

[0093] The training method for the target speech recognition model provided in this embodiment learns the acoustic boundaries of each language by predicting the acoustic boundaries of each language during the training process of the preset speech recognition model, so as to distinguish the acoustic boundaries of different languages. On this basis, the recognition effect of the trained target speech recognition model can be guaranteed.

[0094] The speech recognition method in this embodiment is obtained by training a target speech recognition model. This embodiment provides a method for training a speech recognition model, which can be used in computer devices, such as computers. Figure 6 This is a flowchart of a training method for a target speech recognition model according to an embodiment of the present disclosure, such as... Figure 6 As shown, the process includes the following steps:

[0095] Step S601: Obtain the sample speech and the tag of the sample speech.

[0096] The tags include those used to characterize the target text. See details below. Figure 2 Step S201 of the illustrated embodiment will not be described again here.

[0097] Step S602: Input the sample speech into the preset speech recognition model to obtain the predicted text.

[0098] The preset speech recognition model is used to predict the prediction information weights for each language, and decodes the fused prediction information weights to obtain the predicted text. The prediction information weights include the acoustic information and boundaries of each language in the sample speech. For details, please refer to [link to details]. Figure 2 Step S202 of the illustrated embodiment will not be described again here.

[0099] In some alternative implementations, Figure 7 A schematic diagram of the training process of the preset speech recognition model is shown.

[0100] Step S603: Calculate the loss based on the predicted text and the target text to obtain the loss calculation result.

[0101] Specifically, step S603 includes:

[0102] Step S6031: Obtain the target number of characters in the label.

[0103] As mentioned above, tags are used to represent target text, and once the target text is determined, the target number of characters in the target text is also determined accordingly. Consequently, the number of characters in each language can also be determined.

[0104] Step S6032: Calculate the loss based on the target quantity and the weight of the predicted fusion information to obtain the first loss calculation result.

[0105] For example, the first loss calculation result is obtained using the following formula:

[0106]

[0107] In the formula, As the result of the first loss calculation, U mix For the target quantity, To predict the fused information weight α mix The t-th value in the equation, where T′ is the length of the encoder's output h.

[0108] In some alternative implementations, such as Figure 7 As shown, step S6032 above includes:

[0109] Step c1: Using the continuous integration and distribution unit in the preset speech recognition model, the predicted information weights of each language are processed to obtain the processing results for each language.

[0110] Step c2: Use the second decoder, which corresponds one-to-one with the language, to decode the processing results for the corresponding language and obtain the predicted characters for each language.

[0111] Step c3: Calculate the loss based on the predicted characters for each language and the target characters for each language in the labels to obtain the single-language character loss for each language.

[0112] Step c4: Calculate the loss based on the target number and the predicted fusion information weights to obtain the fusion character loss.

[0113] Step c5: Based on the monolingual character loss and the fused character loss, the first loss calculation result is obtained.

[0114] Equation (2) above calculates the first loss result only for the prediction fusion information weights. Based on this, the single-language character loss can also be calculated by combining the number of predicted characters corresponding to each language. Specifically, the prediction information weights for each language are input into the CIF for processing to obtain the processing results for each language. The processing results for each language are then input into the second decoder for the corresponding language for decoding to obtain the predicted characters corresponding to each language.

[0115] As mentioned above, while obtaining the target number of characters in the target text, we can also obtain the number of characters in each language. Using the predicted characters and target characters corresponding to each language, we calculate the loss to obtain the single-language character loss for each language. Combining this with the fused character loss yields the first loss calculation result. The fusion method here includes, but is not limited to, weighted summation, direct addition, etc. Therefore, the first loss calculation result is obtained using the following formula:

[0116]

[0117] In the formula, As the result of the first loss calculation, U ma U represents the number of Chinese characters in the target text. en The number of English characters in the target text. This represents the t-th value in the predicted information weights corresponding to the Chinese text. t is the t-th value in the prediction information weights corresponding to the English text.

[0118] In some alternative implementations, Figure 5b An optional structure for the second decoder is shown. The structure of the second decoder is similar to... Figure 5a The structure of the first decoder shown is similar, but the differences lie in the model parameters, the number of stacked processing units, and the absence of concatenation units in the second decoder. Figure 5b In the second decoder stack, the number of processing units is N. nFor a detailed description of the structure of the second decoder, please refer to the structural details of the first decoder above; it will not be repeated here.

[0119] The first loss calculation result includes both monolingual character loss and fused character loss. Model training based on this can ensure the accuracy of character recognition results, thereby improving the accuracy of subsequent speech recognition.

[0120] In some alternative implementations, step c1 above includes:

[0121] Step c11: Obtain the number of characters corresponding to each language in the sample speech.

[0122] Step c12: Scaling the prediction information weights for each language based on the number of characters to obtain the processed prediction information weights.

[0123] Step c13: For each language, the weights of the processed prediction information are processed using the continuous integration and distribution unit to obtain the processing results.

[0124] During training, before processing the outputs of the fusion unit and the information weight prediction units for each language, CIF first scales the outputs to make them distinguishable between different languages. Since the number of characters in each language differs in the sample speech, scaling the predicted information weights for each language is applied based on the varying number of characters, making the processed predicted information weights more capable of distinguishing between languages. After scaling, the result is input into the entire distribution unit for further processing to obtain the final result.

[0125] For each language, the weights of the predicted information are scaled to better distinguish between languages ​​and improve the accuracy of language boundary learning.

[0126] In some optional implementations, step c12 above includes:

[0127] Step c121: Obtain the length of the encoded sequence of the sample speech. The length of the encoded sequence is the length of the encoded sequence output by the encoder.

[0128] Step c122: For each language, determine the scaling ratio based on the character length, the length of the encoded sequence, and the weight of the predicted information.

[0129] Step c123: Based on the scaling ratio of each language, the prediction information weights are scaled to obtain the processed prediction information weights.

[0130] The prediction information weights for each language are scaled using the following formula:

[0131]

[0132] Where τ is ma, en, or mix, α τ′ For the weights of the processed prediction information or the weights of the processed prediction fusion information, α τ To predict information weights or predict fused information weights.

[0133] The scaling ratio is calculated by combining the length of the encoded sequence and the length of the character, so that the calculated scaling ratio is closely related to the sample speech, rather than being arbitrarily set. This allows for the processing of the weights of the predicted information while ensuring accuracy.

[0134] Step S6033: Calculate the loss based on the predicted text and the target text to obtain the second loss calculation result.

[0135] The method for calculating the loss between the predicted text and the target text can be set according to actual needs, and no limitations are imposed here. After the loss calculation, a second loss result is obtained.

[0136] Step S6034: Determine the loss calculation result based on the fusion of the first loss calculation result and the second loss calculation result.

[0137] The fusion method between the first loss calculation result and the second loss calculation result includes, but is not limited to, direct addition or weighted addition, etc. The loss calculation result is determined after fusion processing.

[0138] In some optional implementations, the tag also includes language switching annotations for the sample speech. Based on this, step S6034 above includes:

[0139] Step d1: Detect language changes in the predicted fusion information weights and determine the language change detection results.

[0140] Step d2 involves calculating the loss based on the language change detection results and language switching annotations to obtain the third loss calculation result.

[0141] Step d3: Based on the fusion of the first loss calculation result, the second loss calculation result, and the third loss calculation result, determine the loss calculation result.

[0142] Language Change Detection (LCD) and the second encoders for each language mentioned above both serve to help the pre-defined speech recognition model learn and predict information weights, i.e., α. ma and α en These two parameters are strongly correlated with the acoustic boundary.

[0143] In some alternative implementations, Figure 8 A schematic diagram of the language change detection structure is shown. The language change detection model is similar to... Figure 5a The model structure of the first decoder shown is similar, the difference being the number of stacked processing units and the model parameters. Figure 8 The stack shown contains N1 processing units. For a detailed structural description, please refer to [link to relevant documentation]. Figure 5a The detailed information shown will not be repeated here.

[0144] Language switching detection is used to detect whether a language switch has occurred at each location; a value of 1 indicates a language switch, and a value of 0 indicates no language switch. This is merely an example and does not limit the scope of this disclosure.

[0145] After language switching detection, loss calculation is performed. The language switching detection results include both Chinese and English language switching detection results. Losses are calculated separately for each language, and then the two results are added together to obtain a third loss calculation result. The method of loss calculation is not restricted in this study; it can be set according to actual needs.

[0146] By detecting language changes in the predicted fusion information weights and calculating the loss accordingly, the trained model's ability to detect language changes within sentences is improved, thereby enhancing its language boundary learning ability.

[0147] Step S604: Based on the loss calculation results, iteratively update the parameters of the preset speech recognition model to determine the target speech recognition model. For details, please refer to [link to relevant documentation]. Figure 2 Step S204 of the illustrated embodiment will not be described again here.

[0148] As an example of the training process of a pre-defined speech recognition model, such as Figure 7 As shown, the preset speech recognition model includes an encoder, a language-discriminative information weight prediction unit, a CIF unit, an autoregressive (AR) decoder, a non-autoregressive (NAR) decoder, and a language change detection (LCD). The acoustic features x of the sample speech are converted into acoustic encoded information h by the encoder. The encoded output h is then passed through two language-discriminative information weight prediction units, which predict the acoustic information and boundaries in each frame. The predicted information weights for Chinese and English are denoted as α. ma and α en Predicting the fusion information weight α mix It is α ma and α en At the frame level, summation is performed. CIF units utilize language-distinguishing information weights α. ma α en and αmix Convert the frame-level acoustic representation h into a tag-level acoustic embedding c. ma c en and c mix Autoregressive decoder input acoustic embedding c mix To generate target prediction y mix Non-autoregressive decoder input monolingual acoustic embedding c ma and c en To predict the corresponding monolingual label sequence y ma and y en Finally, the LCD module inputs c. mix Based on the prediction results of the previous decoding step of the autoregressive decoder, the probability of predicting language change is l. mix The non-autoregressive decoder processes only the acoustic representation of a single language and utilizes a single-language cross-entropy loss to assist in learning acoustic boundary alignment. The descent of the non-autoregressive decoder's loss guides the model to better predict the single-language acoustic representation c. ma and c en The acoustic boundary.

[0149] Combination Figure 7 As shown, a joint training strategy is used to optimize the entire model. The training loss is obtained by fusing multiple losses. For example, the total loss can be expressed by the following formula.

[0150]

[0151] In the formula, The specific calculation method for the encoding loss (λ) is set according to actual needs. ctc , λ qua , λ nar and λ lcd These are adjustable hyperparameters; the values ​​can be set according to actual needs.

[0152] When using the target speech recognition module for inference, the non-autoregressive decoder and LCD module are not used; monolingual acoustic embedding is employed. ma and c en The computation is also omitted in the decoding stage. The main role of the non-autoregressive decoder and the LCD module is to help learn α. ma and α en These two parameters are strongly correlated with the acoustic boundary.

[0153] The training method for the target speech recognition model provided in this embodiment calculates the loss for the predicted fusion information weights and the predicted text separately during loss calculation, and then fuses the two calculated losses. Using multiple loss fusion methods can ensure the accuracy of the training results.

[0154] The speech recognition method, as an embodiment of this disclosure, is applied to a mobile terminal. Before speech recognition, the mobile terminal displays a pop-up window requesting user consent; only after user consent is given is the speech to be recognized collected. A target speech recognition model is used to perform speech recognition on the speech to be recognized, obtaining the target recognition result. Finally, the target recognition result is displayed on the mobile terminal, or it can be exported as a document for later user use.

[0155] In other implementations, speech recognition methods can also be applied to code-switching translation scenarios. With the user's consent, the speech to be recognized is collected, processed by a target recognition model to obtain recognized text, and then translated into the target language for output.

[0156] This embodiment also provides a voice recognition device for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0157] This embodiment provides a voice recognition device, such as... Figure 9 As shown, it includes:

[0158] The voice acquisition module 901 is used to acquire the voice to be recognized.

[0159] The encoding module 902 is used to perform encoding processing based on the speech to be recognized to obtain acoustic encoding information.

[0160] The weight prediction module 903 is used to predict the information weights corresponding to each language for the acoustic coding information, so as to obtain information weights that correspond one-to-one with the language. The information weights include the acoustic information and boundaries of the language.

[0161] The fusion module 904 is used to fuse the information weights of the various languages ​​to obtain fused information weights.

[0162] The decoding module 905 is used to perform decoding processing based on the fused information weights to obtain the speech recognition result.

[0163] In some optional implementations, the speech recognition method is obtained by training a target speech recognition model, the training device for the target speech recognition model comprising:

[0164] A sample acquisition module is used to acquire sample speech and tags for the sample speech, the tags including those used to characterize target text;

[0165] The prediction module is used to input the sample speech into a preset speech recognition model to obtain predicted text. The preset speech recognition model is used to predict the prediction information weights of each language and decode the fused prediction information weights to obtain the predicted text. The prediction information weights include the acoustic information and boundaries of each language in the sample speech.

[0166] The loss calculation module is used to perform loss calculation based on the predicted text and the target text to obtain the loss calculation result.

[0167] The update module is used to iteratively update the parameters of the preset speech recognition model based on the loss calculation results, so as to determine the target speech recognition model.

[0168] In some optional implementations, the preset speech recognition model includes an encoder, an information weight prediction unit corresponding to each language, a fusion unit, and a decoding processing unit. The information weight prediction unit is used to predict the information weight of the language. The fusion unit is used to fuse the predicted information weights of each language to obtain a predicted fused information weight. The decoding processing unit is used to perform decoding processing based on the predicted fused information weight to obtain the predicted text.

[0169] In some optional implementations, the decoding processing unit includes a continuous integration and distribution unit and a first decoder, and the predicted text determination module includes:

[0170] The CIF processing unit is used to process the predicted fusion information weights using the continuous integration and distribution unit.

[0171] A decoding unit is used to decode the processing result of the continuous integration and distribution unit using the first decoder to determine the predicted text.

[0172] In some optional implementations, the module for determining the predicted fusion information weights includes:

[0173] The loss rate acquisition unit is used to acquire the preset information loss rate.

[0174] The random deactivation processing unit is used to perform random deactivation processing on the prediction information weights of each language based on the preset information loss rate, so as to obtain the processing results corresponding to each language.

[0175] The fusion unit is used to fuse the processing results corresponding to each language to obtain the predicted fusion information weights.

[0176] In some optional implementations, the loss calculation module includes:

[0177] The quantity acquisition unit is used to acquire the target number of characters in the label.

[0178] The first loss unit is used to perform loss calculation based on the target quantity and the prediction fusion information weights to obtain the first loss calculation result.

[0179] The second loss unit is used to perform loss calculation based on the predicted text and the target text to obtain a second loss calculation result.

[0180] The loss fusion unit is used to determine the loss calculation result based on the fusion of the first loss calculation result and the second loss calculation result.

[0181] In some optional implementations, the label further includes language switching annotations for the sample speech, and the loss fusion unit includes:

[0182] The detection subunit is used to perform language change detection on the predicted fusion information weights and determine the language change detection result.

[0183] The loss calculation subunit is used to perform loss calculation based on the language change detection result and the language switching annotation to obtain a third loss calculation result.

[0184] The fusion subunit is used to determine the loss calculation result based on the fusion of the first loss calculation result, the second loss calculation result, and the third loss calculation result.

[0185] In some alternative implementations, the first loss unit includes:

[0186] The CIF subunit is used to process the prediction information weights of each language using the continuous integration and distribution unit in the preset speech recognition model, so as to obtain the processing results of each language.

[0187] The decoding subunit is used to decode the processing results of the corresponding language using a second decoder that corresponds one-to-one with the language, so as to obtain the predicted characters corresponding to each language.

[0188] The monolingual loss subunit is used to calculate the loss based on the predicted characters corresponding to each language and the target characters corresponding to each language in the label, so as to obtain the monolingual character loss corresponding to each language.

[0189] The fusion loss subunit is used to calculate the loss based on the target quantity and the predicted fusion information weights to obtain the fusion character loss.

[0190] The loss subunit is used to obtain the first loss calculation result based on the monolingual character loss and the fused character loss.

[0191] In some alternative implementations, the CIF subunit includes:

[0192] The character count acquisition subunit is used to acquire the character count corresponding to each language in the sample speech;

[0193] The scaling processing subunit is used to scale the prediction information weights of each language based on the number of characters to obtain the processed prediction information weights.

[0194] The CIF processing subunit is used to process the weights of the processed prediction information using the continuous integration and distribution unit for each language to obtain the processing result.

[0195] In some optional implementations, the scaling processing subunit includes:

[0196] The length acquisition subunit is used to acquire the length of the encoded sequence of the sample speech, wherein the length of the encoded sequence is the length of the encoded sequence output by the encoder.

[0197] The scaling ratio determination subunit is used to determine the scaling ratio for each language based on the character length, the encoded sequence length, and the prediction information weight.

[0198] The weight scaling processing subunit is used to scale the prediction information weights based on the scaling ratio of each language to obtain the processed prediction information weights.

[0199] In this embodiment, the voice recognition device is presented in the form of a functional unit. Here, a unit refers to an ASIC circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0200] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0201] This disclosure also provides a computer device having the above-described features. Figure 9 The voice recognition device shown.

[0202] Please see Figure 10 , Figure 10 This is a schematic diagram of the structure of a computer device provided in an optional embodiment of this disclosure, such as... Figure 10As shown, the computer device includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 10 Take a processor 10 as an example.

[0203] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GPA), or any combination thereof.

[0204] The memory 20 stores instructions executable by at least one processor 10 to cause the at least one processor 10 to perform the method shown in the above embodiments.

[0205] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the computer device. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, which can be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0206] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0207] The computer device also includes an input device 30 and an output device 40. The processor 10, memory 20, input device 30, and output device 40 can be connected via a bus or other means. Figure X Taking the example of a connection between China and Israel via a bus.

[0208] Input device 30 can receive input numerical or character information, and generate key signal inputs related to user settings and function control of the computer device, such as a touchscreen, keypad, mouse, trackpad, touchpad, joystick, one or more mouse buttons, trackball, joystick, etc. Output device 40 may include display devices, auxiliary lighting devices (e.g., LEDs), and haptic feedback devices (e.g., vibration motors). The aforementioned display devices include, but are not limited to, liquid crystal displays, light-emitting diodes, displays, and plasma displays. In some alternative embodiments, the display device may be a touchscreen.

[0209] This disclosure also provides a computer-readable storage medium in which the methods described in this disclosure can be implemented in hardware or firmware, or implemented as recordable on a storage medium, or implemented as computer code originally stored on a remote storage medium or a non-transitory machine-readable storage medium and subsequently stored on a local storage medium after being downloaded over a network. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium may be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium may also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code that, when accessed and executed by the computer, processor, or hardware, implements the methods shown in the above embodiments.

[0210] Although embodiments of the present disclosure have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A voice recognition method, characterized by, The method comprises: acquiring a voice to be recognized; performing encoding processing based on the voice to be recognized to obtain acoustic encoding information; performing information weight prediction corresponding to each language on the acoustic encoding information to obtain information weights corresponding to the languages one by one, the information weights comprising acoustic information and boundaries of the languages; fusing the information weights of the languages to obtain fused information weights; performing decoding processing based on the fused information weights to obtain a voice recognition result.

2. The method of claim 1, wherein, The voice recognition method is obtained by a target voice recognition model trained by the following steps: acquiring sample voice and labels of the sample voice, the labels comprising target text; inputting the sample voice into a preset voice recognition model to obtain predicted text, the preset voice recognition model being used to predict predicted information weights of the languages and to perform decoding on the fused predicted information weights to obtain the predicted text, the predicted information weights comprising acoustic information and boundaries of each language in the sample voice, wherein the fused predicted information weights are predicted fused information weights; performing loss calculation based on the predicted text and the target text to obtain a loss calculation result; iteratively updating parameters of the preset voice recognition model based on the loss calculation result to determine the target voice recognition model.

3. The method of claim 2, wherein, The preset voice recognition model comprises an encoder, information weight prediction units corresponding to the languages one by one, a fusion unit and a decoding processing unit, the information weight prediction units being used to predict information weights of the languages, the fusion unit being used to fuse predicted information weights of the languages to obtain the predicted fused information weights, and the decoding processing unit being used to perform decoding processing based on the predicted fused information weights to obtain the predicted text.

4. The method of claim 3, wherein, The decoding processing unit comprises a continuous integration and distribution unit and a first decoder, and the determination manner of the predicted text comprises: processing the predicted fused information weights by using the continuous integration and distribution unit; decoding the processing result of the continuous integration and distribution unit by using the first decoder to determine the predicted text.

5. The method of claim 2, wherein, The determination manner of the predicted fused information weights comprises: acquiring a preset information loss rate; performing random inactivation processing on predicted information weights of each language based on the preset information loss rate to obtain processing results corresponding to the languages; fusing the processing results corresponding to the languages to obtain the predicted fused information weights.

6. The method of claim 2, wherein, The loss calculation based on the predicted text and the target text to obtain a loss calculation result comprises: acquiring a target number of characters in the labels; performing loss calculation based on the target number and the predicted fused information weights to obtain a first loss calculation result; performing loss calculation based on the predicted text and the target text to obtain a second loss calculation result; determining the loss calculation result based on fusion of the first loss calculation result and the second loss calculation result.

7. The method of claim 6, wherein, The label further comprises a language switching label of the sample voice, and the loss calculation result is determined based on fusion of the first loss calculation result and the second loss calculation result. The predicted fusion information weight is subjected to language change detection to determine a language change detection result. Loss calculation is performed based on the language change detection result and the language switching label to obtain a third loss calculation result. The loss calculation result is determined based on fusion of the first loss calculation result, the second loss calculation result and the third loss calculation result.

8. The method of claim 6, wherein, The first loss calculation result is obtained by performing loss calculation based on the target quantity and the predicted fusion information weight. The predicted information weight of each language is processed by using a continuous integration and distribution unit in the preset voice recognition model to obtain a processing result of each language. The processing result of the corresponding language is decoded by using a second decoder corresponding to the language to obtain a predicted character corresponding to each language. Loss calculation is performed based on the predicted character corresponding to each language and the target character corresponding to each language in the label to obtain a single-language character loss corresponding to each language. Loss calculation is performed based on the target quantity and the predicted fusion information weight to obtain a fusion character loss. The first loss calculation result is obtained based on the single-language character loss and the fusion character loss.

9. The method of claim 8, wherein, The processing result of each language is obtained by processing the predicted information weight of each language by using a continuous integration and distribution unit in the preset voice recognition model. The number of characters corresponding to each language in the sample voice is obtained. The predicted information weight of each language is scaled based on the number of characters to obtain a processed predicted information weight. For each language, the processed predicted information weight is processed by using the continuous integration and distribution unit to obtain the processing result.

10. The method of claim 9, wherein, The number of characters corresponding to each language in the sample voice is obtained. For each language, a scaling ratio is determined based on the character length, the encoding sequence length and the predicted information weight. The predicted information weight is scaled based on the scaling ratio of each language to obtain the processed predicted information weight. The device comprises:

11. A speech recognition apparatus, characterized by comprising: A voice acquisition module for acquiring a voice to be recognized. An encoding module for performing encoding processing based on the voice to be recognized to obtain acoustic encoding information. A weight prediction module for predicting information weight corresponding to each language for the acoustic encoding information to obtain information weight corresponding to each language, wherein the information weight comprises acoustic information and boundaries of the language. A fusion module for fusing the information weight of each language to obtain fusion information weight. A decoding module for performing decoding processing based on the fusion information weight to obtain a voice recognition result. ​ 12. A computer device, comprising: ​ A memory and a processor, which are connected in communication with each other, the memory storing computer instructions, and the processor executing the computer instructions to perform the voice recognition method of any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions for causing a computer to perform the voice recognition method of any one of claims 1 to 10.

Images