Speech recognition method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a voice recognition method and device, electronic equipment and a storage medium, and relates to the fields of voice processing, artificial intelligence, cloud technology and the like. The method comprises the steps of obtaining to-be-recognized voice, recognizing the to-be-recognized voice through a trained voice recognition model to obtain a first voice recognition result, performing feature extraction on the to-be-recognized voice through a trained large audio recognition model to obtain corresponding text features, and recognizing the to-be-recognized voice based on the text features and the first voice recognition result. And performing autoregression decoding, sequentially decoding to obtain each target character, if the currently decoded target character is a first character, obtaining a target speech recognition result based on each currently decoded target character, and if the number of the currently decoded target characters is greater than the maximum character number, obtaining a target speech recognition result based on each currently decoded target character. And if yes, obtaining a target speech recognition result in a non-autoregression decoding mode. Based on the method, the accuracy and robustness of speech recognition are improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computers, and can relate to the fields of voice processing, artificial intelligence, and cloud technology. Specifically, the present application relates to a voice recognition method and device, an electronic device, and a storage medium. BACKGROUND

[0002] With the development of artificial intelligence technology, large language models (LLMs) are widely used in various fields as an important development direction of artificial intelligence technology. In order to bring a better voice interaction experience, LLMs are also applied to the field of voice processing, such as voice recognition, voice question and answer, etc.

[0003] However, the recognition result error rate of LLM in the voice recognition task in the related art is high, and the voice recognition effect still needs to be improved. SUMMARY

[0004] The purpose of the embodiments of the present application is to provide a voice recognition method, device, electronic device, and storage medium that can effectively improve the accuracy of voice recognition. To achieve this purpose, the technical solutions provided by the embodiments of the present application are as follows: On the one hand, the embodiments of the present application provide a voice recognition method, comprising: obtaining a to-be-recognized voice; obtaining a first voice recognition result of the to-be-recognized voice through a trained voice recognition model; based on the to-be-recognized voice and the first voice recognition result, performing the following operations through a trained audio recognition large model to obtain a target voice recognition result of the to-be-recognized voice: wherein the audio recognition large model comprises a voice preprocessing model and a large language model, and the target voice recognition result is obtained by the following way: extracting features of the to-be-recognized voice through the voice preprocessing model to obtain text features corresponding to the to-be-recognized voice; based on the text features and the first voice recognition result, performing autoregressive decoding through the large language model to sequentially decode each target character until a first condition is met; the first condition includes that the target character obtained by the current decoding is a first character, or the number of target characters that have been decoded is greater than the maximum number of characters of the to-be-recognized voice; if the first condition met is that the target character obtained by the current decoding is a first character, determining the target voice recognition result according to each target character that has been decoded; If the first condition met is that the number of currently decoded target characters is greater than the maximum number of characters of the speech to be recognized, then based on the text feature and the first speech recognition result, non-autoregressive decoding is performed through the large language model to obtain the target speech recognition result; wherein the maximum number of characters of the speech to be recognized is determined based on the number of characters of the first speech recognition result.

[0005] In another aspect, the embodiments of the present application also provide a speech recognition device, the device comprising: a speech acquisition module, configured to acquire speech to be recognized; a first speech recognition module, configured to obtain a first speech recognition result of the speech to be recognized through a trained speech recognition model; a second speech recognition module, configured to obtain a target speech recognition result of the speech to be recognized through a trained audio recognition large model based on the speech to be recognized and the first speech recognition result: wherein the audio recognition large model comprises a speech preprocessing model and a large language model, and the target speech recognition result is obtained through the following manner: extracting features of the speech to be recognized through the speech preprocessing model to obtain text features corresponding to the speech to be recognized; based on the text features and the first speech recognition result, autoregressive decoding is performed through the large language model to sequentially decode each target character until a first condition is met; the first condition includes that the target character currently decoded is a first character, or the number of currently decoded target characters is greater than the maximum number of characters of the speech to be recognized; if the first condition met is that the target character currently decoded is the first character, then the target speech recognition result is determined according to the target characters currently decoded; if the first condition met is that the number of currently decoded target characters is greater than the maximum number of characters of the speech to be recognized, then based on the text feature and the first speech recognition result, non-autoregressive decoding is performed through the large language model to obtain the target speech recognition result; wherein the maximum number of characters of the speech to be recognized is determined based on the number of characters of the first speech recognition result.

[0006] Optionally, the target speech recognition result is obtained through the following manner: based on the text feature and the first speech recognition result, a first target character is decoded; For each non-first character bit, a target character of the character bit is decoded according to a first character sequence corresponding to the character bit, the text feature, and the first speech recognition result; the first character sequence comprises target characters that have been decoded before the character bit.

[0007] Optionally, the speech preprocessing model comprises an encoding layer and a feature conversion layer. The second speech recognition module can be configured to: encode the to-be-recognized speech through the encoding layer to obtain speech features of the to-be-recognized speech; map the speech features of the to-be-recognized speech to a text feature space through the feature conversion layer to obtain text features corresponding to the to-be-recognized speech.

[0008] Optionally, the trained audio recognition large model comprises a trained speech preprocessing model and a trained large language model. The speech preprocessing model is trained in the following manner: a plurality of first training samples are obtained; each first training sample comprises a first sample speech and a labeled recognition text corresponding to the first sample speech; training operations are continuously performed on a to-be-trained first neural network model based on the plurality of first training samples until a first training end condition is met, and a trained speech preprocessing model is obtained; The training operations comprise: input the first sample speech of each first training sample into the first neural network model to obtain predicted text features of each first sample speech through the first neural network model; obtain predicted recognition texts corresponding to each first sample speech through a trained large language model according to the predicted text features of each first sample speech and the labeled recognition text corresponding to each first sample speech; determine a first training loss according to the predicted recognition texts corresponding to each first sample speech and the labeled recognition texts corresponding to the predicted recognition texts; adjust model parameters in the first neural network model based on the first training loss.

[0009] Optionally, for each first sample speech, the predicted recognition text corresponding to the first sample speech is obtained in the following manner: determine a first parameter corresponding to the first sample speech; If the first parameter corresponding to the first sample speech is less than or equal to the first threshold value, a first recognition text corresponding to the first sample speech is obtained according to the predicted text feature of the first sample speech and the labeled recognition text corresponding to the first sample speech by means of the trained large language model, and the first recognition text is taken as the predicted recognition text corresponding to the first sample speech. If the first parameter corresponding to the first sample speech is greater than the first threshold value, a second recognition text corresponding to the first sample speech is obtained according to the predicted text feature of the first sample speech by means of the trained large language model, and the second recognition text is taken as the predicted recognition text corresponding to the first sample speech.

[0010] Optionally, for each first sample speech, the predicted recognition text corresponding to the first sample speech is obtained by the following means: Based on the predicted text feature of the first sample speech and the corresponding labeled recognition text, a target character in a first position in the predicted recognition text is obtained. For each character position other than the first position in the predicted recognition text, a first character sequence corresponding to the character position is determined according to the labeled recognition text corresponding to the first sample speech, and a target character of the character position is decoded according to the first character sequence corresponding to the character position and the text feature; wherein the first character sequence includes each target character that has been decoded before the character position.

[0011] Optionally, the trained audio recognition large model includes a trained speech preprocessing model and a trained large language model. The audio recognition large model is obtained by the following means: A plurality of second training samples are obtained; wherein each second training sample includes a second sample speech and a labeled recognition text corresponding to the second sample speech. Based on the plurality of second training samples, a training operation is continuously performed on a second neural network model to be trained and a pre-trained large language model until a second training end condition is met, so as to obtain a trained audio recognition large model. The training operation includes: The second sample speech of each second training sample is input into the second neural network model, and the predicted text feature of each second sample speech is obtained by means of the second neural network model. According to the predicted text feature of each second sample speech and the labeled recognition text corresponding to each second sample speech, a predicted recognition text corresponding to each second sample speech is obtained by means of the pre-trained large language model. According to the predicted recognition text corresponding to each second sample speech and the corresponding labeled recognition text, a second training loss is determined. adjust model parameters in the second neural network model and the pre-trained large language model based on the second training loss.

[0012] The embodiments of the present application also provide an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor executes the computer program to implement the method provided in any optional embodiment of the present application.

[0013] In another aspect, the embodiments of the present application also provide a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the method provided in any optional embodiment of the present application.

[0014] In another aspect, the embodiments of the present application also provide a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the method provided in any optional embodiment of the present application.

[0015] The technical scheme provided by the embodiments of the present application has the following beneficial effects: The speech recognition method provided by the embodiments of the present application comprises the following steps: a speech recognition model is used to recognize the to-be-recognized speech to obtain a first speech recognition result; a trained audio recognition large model is used to extract features of the to-be-recognized speech to obtain corresponding text features; based on the text features and the first speech recognition result, each target character is sequentially decoded through autoregressive decoding; if a first character is obtained through the current decoding, a target speech recognition result is determined based on the target characters that have been decoded; if the number of the target characters that have been decoded is greater than a maximum number of characters, a target speech recognition result is obtained through non-autoregressive decoding, wherein the maximum number of characters of the to-be-recognized speech is determined based on the number of characters of the first speech recognition result. The method limits the number of characters of the target character obtained through autoregressive decoding of the audio recognition large model based on the first speech recognition result obtained through the speech recognition model, so that the target speech recognition result obtained through decoding is more accurate, the problem of fragment cycle in the output result of the audio recognition large model is avoided, the robustness of the large language model in the speech recognition task is improved, and the actual application requirements are better met. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the description of the embodiments of the present application will be briefly introduced.

[0017] Figure 1 A flowchart of a speech recognition method provided by the embodiments of the present application is shown in the figure. Figure 2 A structure diagram of a speech recognition system provided by the embodiments of the present application is shown in the figure. Figure 3 A structural schematic diagram of a large language model provided for an embodiment of the present application; Figure 4 A schematic diagram of a large language model constrained based on a transcription prompt word provided for an embodiment of the present application; Figure 5 An algorithm flowchart of a hybrid decoding provided for an embodiment of the present application; Figure 6 A schematic diagram of a speech recognition based on a hybrid decoding method provided for an embodiment of the present application; Figure 7 A structural schematic diagram of a speech recognition system provided for an embodiment of the present application; Figure 8 A structural schematic diagram of a speech recognition device provided for an embodiment of the present application; Figure 9 A structural schematic diagram of an electronic device provided for an embodiment of the present application. DETAILED DESCRIPTION

[0018] Embodiments of the present application will be described below with reference to the accompanying drawings. It should be understood that the embodiments described below in conjunction with the drawings are exemplary descriptions of the technical solutions of the embodiments of the present application, and do not limit the technical solutions of the embodiments of the present application.

[0019] Those skilled in the art can understand that, unless specifically stated otherwise, the singular forms "a", "an" and "the" used herein can also include the plural forms. It should be further understood that the terms "include" and "contain" used in the embodiments of the present application mean that the corresponding features can be implemented as the presented features, information, data, steps, operations, elements and / or components, but do not exclude other features, information, data, steps, operations, elements, components and / or their combinations supported by the present technology. It should be understood that when we say that an element is "connected" or "coupled" to another element, the element can be directly connected or coupled to the other element, or it can mean that the element and the other element are connected through an intermediate element. In addition, "connected" or "coupled" used herein can include wireless connection or wireless coupling. The term "and / or" used herein indicates that at least one of the items defined by the term, for example, "A and / or B" can be implemented as "A", or as "B", or as "A and B". When describing a plurality of (two or more) items, if the relationship between the plurality of items is not explicitly limited, the plurality of items can mean one, more or all of the plurality of items, for example, for the description of "parameters A including A1, A2, A3", the parameters A can include A1 or A2 or A3, or the parameters A can include at least two of the three parameters A1, A2, A3.

[0020] Currently, large language models may continuously output the same sentence or phrase when generating text (produce a fragment cycle), i.e., produce a "parrot phenomenon", and the reasons for this phenomenon are multifaceted, mainly including: (1) Bias in training data: There is a large amount of repeated text or some sentences or phrases with a high frequency in the data relied on by the model training, which leads the model to fail to learn enough diversity and tend to copy these data when generating information.

[0021] (2) Limitation of training target: Large language model training is based on a self-supervised learning method, which learns the language model by predicting the next word, leading the model to tend to generate text similar to the input.

[0022] In order to solve the above-mentioned "parrot problem" and avoid the continuous repetition of sentences or phrases in the output speech recognition result when the large language model is applied to the speech recognition task, the embodiment of the present application provides a speech recognition method which can be applied to scenarios such as speech recognition, machine question answering, machine translation, etc., and obtains a speech recognition result based on a large language model. The embodiment of the present application introduces a speech recognition model to recognize the speech to be recognized, and based on the first speech recognition result recognized by the speech recognition model, the number of characters of the target character obtained by the autoregressive decoding of the audio recognition large model is constrained, so that the target speech recognition result obtained by decoding is more accurate, avoiding the problem of fragment cycle in the output result of the audio recognition large model, improving the robustness of the large language model in the speech recognition task, and better meeting the actual application requirements.

[0023] Moreover, since the large language model has more abundant text information, speech recognition through the large language model improves the text modeling capability of the speech recognition system, and the speech recognition result is more accurate.

[0024] Optionally, the scheme provided by the embodiment of the present application can involve artificial intelligence technology, which is a comprehensive discipline and involves a wide range of fields, including both hardware-level technology and software-level technology. Artificial intelligence basic technology generally includes technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, pre-training model technology, operation / interaction system, mechatronics, etc. Among them, the pre-training model is also called large model or basic model, which can be widely applied to downstream tasks in various directions of artificial intelligence after fine-tuning. Artificial intelligence software technology mainly includes computer vision technology, speech technology, natural language processing technology, and machine learning / deep learning, etc.

[0025] Among them, the key technologies of speech technology include automatic speech recognition technology (ASR) and speech synthesis technology (TTS) and voiceprint recognition technology. Letting computers can hear, can see, can speak, can feel, is the development direction of future human-computer interaction, among which voice has become one of the most promising human-computer interaction methods in the future. Large model technology brings changes to the development of speech technology. WavLM, UniSpeech and other pre-training models using the Transformer architecture have strong generalization and universality, and can complete various speech processing tasks. The speech recognition model in the embodiment of the application can use an ASR model.

[0026] Natural language processing (NLP) is an important direction in the field of computer science and artificial intelligence. It studies various theories and methods that can realize effective communication between people and computers using natural language. Natural language processing involves natural language, i.e. the language used in daily life, and is closely related to linguistic research; at the same time, it involves computer science and mathematics. The important technology of artificial intelligence model training is the pre-training model, which is developed from the large language model (Large Language Model) in the NLP field. After fine-tuning, the large language model can be widely used in downstream tasks. Natural language processing technology usually includes text processing, semantic understanding, machine translation, robot question and answer, knowledge graph and other technologies.

[0027] Optionally, the scheme provided by the embodiment of the application can involve various fields of cloud technology, such as cloud computing and cloud storage in cloud technology. For example, the speech recognition method provided by the embodiment of the application can be executed by a server, which can be a cloud server, and the data processing (such as decoding) involved in the method can be implemented by cloud computing, and the storage of data (such as the storage of sample voice) involved in the embodiment of the application can use cloud storage.

[0028] Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on cloud computing business model application, which can form a resource pool and be used on demand, flexibly and conveniently.

[0029] Among them, cloud computing is the product of the development and integration of traditional computer and network technologies such as grid computing, distributed computing, parallel computing, utility computing, network storage, virtualization, load balancing, etc.

[0030] Cloud storage is a new concept extended and developed on the basis of the concept of cloud computing. A distributed cloud storage system (hereinafter referred to as a storage system) refers to a storage system that cooperatively provides data storage and business access functions by collecting a large number of storage devices (storage devices are also referred to as storage nodes) of various types in the network through cluster applications, grid technologies, and distributed storage file systems, etc.

[0031] It should be noted that in the optional embodiments of the present application, the data related to the object information (user voice, etc.) involved in the embodiments of the present application when applied to specific products or technologies need to obtain the permission or consent of the object, and the collection, use and processing of related data need to comply with relevant national and regional laws, regulations and standards. That is, if the embodiments of the present application involve data related to the object, the data needs to be obtained with the authorization and consent of the object, the authorization and consent of the relevant department, and in compliance with the relevant national and regional laws, regulations and standards. If personal information is involved in the embodiments, the consent of the individual needs to be obtained for the acquisition of all personal information, and the individual consent needs to be obtained for sensitive information. The embodiments also need to be implemented with the authorization and consent of the object.

[0032] In order to better understand and illustrate the method provided by the embodiments of the present application, some technical terms involved in the embodiments of the present application will be explained and described first.

[0033] Connectionist Temporal Classification (CTC): a commonly used end-to-end speech recognition system framework in the field of speech recognition.

[0034] Large Language Model (LLM): A large language model used for learning and generating natural language text, which can be used for language understanding, text classification, dialogue generation, etc. Through training on a large amount of text data, it can perform a wide range of tasks, including text summarization, translation, sentiment analysis, etc. The characteristics of large language models are large in size, containing billions of parameters, helping them learn complex patterns in text data.

[0035] Audio-LLM: A large language model based on audio input, built on the LLM model, using the basic capabilities provided by the LLM to achieve alignment between output text and audio modalities.

[0036] AutoRegressive (AR) model: A process that uses itself as a regression variable, i.e. using a linear combination of random variables at previous time points to describe the linear regression model at a certain time point, which is a common form of time series model. Real Time Factor (RTF): A value commonly used to measure the decoding speed of an automatic speech recognition system.

[0037] The technical solutions of the embodiments of the present application and the technical effects produced by the technical solutions of the present application will be described below through the description of several embodiments. It should be noted that the following implementation manners can be mutually referenced, borrowed or combined, and the same terms, similar features and similar implementation steps in different implementation manners will not be described repeatedly.

[0038] Figure 1 A flowchart of a speech recognition method provided by an embodiment of the present application is shown, which can be executed by any computer device, such as a user terminal or a server, or can be implemented by multiple electronic devices in cooperation.

[0039] As shown in Figure 1 The speech recognition method provided by the embodiment of the present application can include the following steps S110 to S130.

[0040] Step S110: Acquire the speech to be recognized.

[0041] Step S120: Obtain the first speech recognition result of the speech to be recognized by the speech recognition model.

[0042] Since the large language model may have a repetition phenomenon when generating text, when the large language model is applied to a speech recognition task, there may be a constantly circulating segment (sentence or phrase) in the output speech recognition result. Therefore, in the embodiments of the present application, a transcription tokenizer is introduced to convert the to-be-recognized speech into corresponding transcription text, and based on the number of tokens of the transcription text converted by the transcription tokenizer, the number of characters in the speech recognition result output by the subsequent large language model is limited.

[0043] In the embodiments of the present application, the transcription tokenizer is an automatic speech recognition (ASR) model, which is a classification model modeling speech features and text, and performs speech recognition based on the speech features of the to-be-recognized speech in the audio feature space.

[0044] Optionally, the ASR model can be an end-to-end model trained using a CTC loss function, which can directly predict the corresponding words from the input speech features through a neural network. Moreover, since the decoding method of the ASR model is simple and not autoregressive decoding, and the parameter amount is small, the ASR model will not cause the speech recognition result to have a segment cycle.

[0045] After obtaining the to-be-recognized speech, the to-be-recognized speech can be input to the ASR model to obtain a first speech recognition result of the to-be-recognized speech.

[0046] Step S130: based on the to-be-recognized speech and the first speech recognition result, a target speech recognition result of the to-be-recognized speech is obtained through the trained audio recognition large model.

[0047] The speech recognition method provided in the embodiments of the present application uses an audio recognition large model (also referred to as an audio large model, Audio-LLM) to recognize the to-be-recognized speech, wherein the Audio-LLM includes a speech preprocessing model and a large language model. When performing speech recognition, the speech preprocessing model is first used to extract features from the to-be-recognized speech to obtain text features corresponding to the to-be-recognized speech in a text feature space, and then based on the text features and the first speech recognition result, a target speech recognition result is obtained through the large language model.

[0048] Optionally, the speech preprocessing model comprises an encoding layer and a feature conversion layer, the encoding layer is a speech encoder, used to encode the speech to obtain speech features corresponding to the speech, the speech features being high-dimensional feature representations of the speech in an audio feature space. Optionally, the speech encoder can have the same model structure as the ASR model except for the projection layer, and the same parameters as the ASR model can be used for initialization during model training.

[0049] The feature conversion layer (also referred to as an adapter) is used to connect the high-dimensional feature representations output by the encoding layer to the large language model, and convert the high-dimensional feature representations output by the encoding layer to the feature dimensions of the embedding layer of the large language model, i.e., map the speech features output by the encoding layer to the text feature space of the large language model to obtain text features corresponding to the speech. Optionally, the feature conversion layer can comprise several one-dimensional convolutional layers and one fully connected layer.

[0050] The following steps S1301-S1304 are performed by the audio recognition large model to obtain the target speech recognition result of the speech to be recognized: Step S1301: performing feature extraction on the speech to be recognized by the speech preprocessing model to obtain text features corresponding to the speech to be recognized.

[0051] Optionally, when the speech to be recognized is obtained, the speech to be recognized can be input to the encoding layer of the speech preprocessing model to obtain speech features of the speech to be recognized, and then the speech features are input to the feature conversion layer of the speech preprocessing model to map the speech features to the text feature space corresponding to the large language model to obtain the text features of the speech to be recognized.

[0052] Step S1302: performing autoregressive decoding by the large language model based on the text features and the first speech recognition result to sequentially decode each target character until the first condition is met.

[0053] Step S1303: if the first condition met is that the target character obtained by the current decoding is the first character, determining the target speech recognition result according to the target characters that have been decoded.

[0054] Step S1304: if the first condition met is that the number of target characters that have been decoded is greater than the maximum number of characters of the speech to be recognized, performing non-autoregressive decoding by the large language model based on the text features and the first speech recognition result to obtain the target speech recognition result.

[0055] The first condition includes that the target character obtained by the current decoding is the first character, or the number of target characters that have been decoded is greater than the maximum number of characters of the speech to be recognized. The first character is a termination symbol <eos>.

[0056] Optionally, when performing the autoregressive decoding, a first target character (also referred to as a token) can be decoded based on the text feature of the speech to be recognized and the first speech recognition result. For each non-first character position, a target character of the character position can be decoded according to a first character sequence of the character position, the text feature, and the first speech recognition result. The first character sequence includes the target characters decoded before the character position.

[0057] For example, it is assumed that the speech recognition result is "today is sunny", where "today" is the first character in the speech recognition result, and "today", "day", "weather", "sunny", and "long" are non-first character positions. For the character "long", the first character sequence corresponding to the character is "today is sunny".

[0058] Since the autoregressive decoding uses the decoding results of previous time points to predict the decoding result of the next time point, i.e., uses the previously predicted tokens to predict the next token, and continues until the end-of-sentence token is predicted, this decoding method can cause the phenomenon of repeated decoding fragments (sentences or words), which can result in the inability to predict the end-of-sentence token during the decoding process and errors in the speech recognition result.

[0059] In order to avoid the problem of fragment cycle (also referred to as the problem of the repeating machine) of the large language model, the decoding result of the autoregressive decoding can be further constrained.

[0060] If the first condition satisfied by the decoding result is that the target character obtained by the current decoding is the first character, it indicates that the large language model does not have the phenomenon of fragment cycle, and the decoding result is accurate. In this case, the target speech recognition result can be obtained by combining the target characters decoded.

[0061] If the first condition satisfied by the decoding result is that the number of target characters decoded is greater than the maximum number of characters of the speech to be recognized, it indicates that the large language model has the phenomenon of fragment cycle, and the autoregressive decoding result is inaccurate. In this case, the target speech recognition result can be obtained by using a non-autoregressive (NAR) decoding method based on the text feature and the first speech recognition result.

[0062] The maximum number of characters of the speech to be recognized is determined based on the number of characters of the first speech recognition result. For example, the number of characters of the first speech recognition result can be set as the maximum number of characters of the speech to be recognized, or the maximum number of characters of the speech to be recognized can be set as a multiple of the number of characters of the first speech recognition result. The present application does not limit this, and the specific setting can be made as needed.

[0063] Figure 2 This is a schematic diagram of the structure of a speech recognition system provided in an embodiment of this application. After acquiring the speech to be recognized, the speech can be input into an ASR model to obtain a first speech recognition result. The speech is then input into the speech preprocessing model of an audio recognition large-scale model. The speech features are encoded through the encoding layer of the speech preprocessing model to obtain the speech features. Then, the feature transformation layer of the speech preprocessing model performs feature dimension transformation, mapping the speech features to the text feature space of an LLM model to obtain the text features corresponding to the speech to be recognized. Next, the first speech recognition result and the text features are input into the LLM model of the audio recognition large-scale model. Each target character is decoded sequentially using an autoregressive decoding method until a first condition is met. If the first condition is met when the currently decoded target character is the first character, then the target speech recognition result is obtained based on the currently decoded target characters. If the first condition is met when the number of currently decoded target characters is greater than the maximum number of characters in the speech to be recognized, then the target speech recognition result is obtained based on the text features and the first speech recognition result using a non-autoregressive decoding method. The maximum number of characters in the speech to be recognized is determined based on the number of characters in the first speech recognition result.

[0064] In this embodiment of the application, the first speech recognition result can be used as a transcription prompt word. After inputting it into the large language model in the large audio recognition model, the first speech recognition result can be feature extracted through the embedding layer of the large language model to obtain the feature representation (embedding) of the first speech recognition result. Based on the feature representation of the first speech recognition result and the text features of the speech to be recognized, decoding (autoregressive decoding or non-autoregressive decoding) is performed.

[0065] like Figure 3 As shown, this large language model includes an embedding layer, an LLM body, and a normalization layer (Softmax layer). During speech recognition, the first speech recognition result output by the ASR (Automatic Speech Recognition) is used as a transcription prompt. The embedding layer extracts features from the transcription prompt, obtaining corresponding feature representations (as shown by the black-filled rectangles in the figure). Based on the feature representations of the transcription prompts and the text features output by the speech preprocessing model (as shown by the gray-filled rectangles in the figure), decoding is performed, and the target speech recognition result is output through the Softmax layer. Specifically, during autoregressive decoding, the next token is predicted based on the previously decoded token sequence and transcription prompts, resulting in the target speech recognition result.

[0066] As an optional implementation, if the number of currently decoded target characters is greater than the maximum number of characters of the speech to be recognized, the first speech recognition result can also be taken as the target speech recognition result.

[0067] Based on Figure 1 The speech recognition method shown in the figure, the first speech recognition result is obtained by recognizing the speech to be recognized through the speech recognition model, the corresponding text features are obtained by extracting the features of the speech to be recognized through the trained audio recognition large model, and each target character is sequentially decoded through autoregressive decoding based on the text features and the first speech recognition result. If the first character is decoded at present, the target speech recognition result is determined based on the number of currently decoded target characters, and if the number of currently decoded target characters is greater than the maximum number of characters, the target speech recognition result is obtained in a non-autoregressive decoding manner, wherein the maximum number of characters of the speech to be recognized is determined based on the number of characters of the first speech recognition result. This method introduces a speech recognition model to recognize the speech to be recognized, and based on the first speech recognition result recognized by the speech recognition model, the number of characters of the target character obtained by autoregressive decoding of the audio recognition large model is constrained, so that the target speech recognition result obtained is more accurate, avoiding the problem of fragment cycle in the output result of the audio recognition large model, improving the robustness of the large language model in the speech recognition task, and better meeting the actual application requirements.

[0068] Optionally, in order to improve the accuracy of speech recognition, the speech to be recognized can be denoised before speech recognition to eliminate background noise, environmental noise, etc. in the speech to be recognized.

[0069] In addition, the large language model also has a common "hallucination" phenomenon. So-called "hallucination" usually refers to generating meaningless or output results that do not conform to the provided source content, including input conflict hallucination, context conflict hallucination and fact conflict hallucination. Among them, input conflict hallucination refers to the generated content deviating from the original indication of user input, context conflict hallucination refers to the generated content conflicting / contradicting with the previously generated information, and fact conflict hallucination refers to the generated content contradicting with the facts.

[0070] The causes of hallucination phenomenon are also various, mainly including: (1) The data relied on by model training may have bias, such as data annotation error, insufficient data sampling, etc., which causes the large language model to be affected by data bias and unable to generate accurate information.

[0071] (2) The knowledge absorbed by the large language model when learning text data comes from different data sources, which may have problems such as knowledge loss, confusion, and incompleteness, resulting in the large language model being unable to accurately integrate fragmented knowledge and output accurate and coherent content.

[0072] (3) The large language model is prone to information forgetting and distortion when processing complex context relationships, and is unable to accurately maintain consistent information understanding, resulting in hallucination.

[0073] In the embodiments of the present application, the first speech recognition result recognized by the speech recognition model is introduced to constrain the large language model as a transcription prompt, avoiding hallucination caused by the strong generation ability of the large language model and speech recognition errors.

[0074] In the embodiments of the present application, the large language model in the audio recognition large model is pre-trained, so when training the audio recognition large model, the model parameters of the large language model can not be updated and adjusted, but the model parameters in the encoding layer and the feature conversion layer in the speech preprocessing model are updated and adjusted, that is, only the speech preprocessing model is trained.

[0075] The speech preprocessing model is trained in the following manner: A plurality of first training samples are obtained; each first training sample includes a first sample speech and a labeled recognition text corresponding to the first sample speech.

[0076] Based on the plurality of first training samples, a training operation is continuously performed on the first neural network model to be trained until a first training end condition is met, and a trained speech preprocessing model is obtained.

[0077] The training operation includes: The first sample speech of each first training sample is input into the first neural network model to be trained, and the predicted text features of each first sample speech are obtained through the first neural network model. Wherein, the first sample speech of each first training sample is input into the encoding layer of the first neural network model to obtain the second predicted feature representation of each first sample speech, and then the second predicted feature representation of each first sample speech is input into the feature conversion layer of the first neural network model to obtain the predicted text features of each first sample speech.

[0078] According to the predicted text features of each first sample speech and the labeled recognition text corresponding to each first sample speech, a predicted recognition text corresponding to each first sample speech is obtained through the trained large language model. The large language model is a pre-trained model used to assist the speech preprocessing model in model training. The labeled recognition text is input into the large language model as a transcription prompt to constrain the predicted recognition text.

[0079] Optionally, for each first sample speech, a target character in a first position of a predicted recognition text is obtained based on a predicted text feature of the first sample speech and a corresponding annotated recognition text. For each character position other than the first position in the predicted recognition text, a first character sequence of the character position is determined according to the annotated recognition text corresponding to the first sample speech, and a target character of the character position is obtained according to the first character sequence of the character position and the text feature. The first character sequence includes each target character obtained by decoding before the character position.

[0080] A first training loss is determined according to the predicted recognition text corresponding to each first sample speech and the corresponding annotated recognition text, and model parameters in the neural network model are adjusted based on the first training loss.

[0081] The first training end condition and the loss function of the model can be configured according to requirements. For example, the first training end condition can include, but is not limited to, that the number of training reaches a preset number, the loss function converges (for example, the training loss of the model is less than a preset value, or the training loss is less than the preset value for a plurality of consecutive times, etc.), a test index of the model meets a preset index, etc.

[0082] In the embodiments of the present application, in order to avoid overfitting of the model to the transcription prompt word in the training process, a first parameter can be introduced to determine whether the annotated recognition text of the first sample speech is used as the transcription prompt word for training.

[0083] For each first sample speech, a first parameter corresponding to the first sample speech is determined. If the first parameter corresponding to the first sample speech is less than or equal to a first threshold, a transcription prompt word is input into the large language model for constraint, that is, according to the predicted text feature of the first sample speech and the annotated recognition text corresponding to the first sample speech, a first recognition text corresponding to the first sample speech is obtained through the large language model, and the first recognition text is taken as the predicted recognition text corresponding to the first sample speech.

[0084] If the first parameter corresponding to the first sample speech is greater than the first threshold, the transcription prompt word is no longer used for constraint, that is, a second recognition text corresponding to the first sample speech is directly obtained through the trained large language model according to the predicted text feature of the first sample speech, and the second recognition text is taken as the predicted recognition text corresponding to the first sample speech.

[0085] Optionally, the first parameter corresponding to the first sample speech can be a random number generated by a random generation algorithm, and the first parameter The first threshold value can be set as needed, such as 0.5. When the first parameter is less than 0.5, the annotated recognition text corresponding to the first sample voice is input as a transcription prompt word into the large language model, otherwise, the transcription prompt word is no longer input into the large language model.

[0086] Assuming that the sample voice is x, the predicted voice recognition result of the sample voice x is y, where y represents a text sequence , represents the first character, respectively represent subsequent characters arranged in order of position. The annotated recognition text of the sample voice x is taken as a transcription prompt word .

[0087] Optionally, in the model training stage, the sample voice x can also be subjected to voice recognition through the trained ASR model, and the first voice recognition result of the sample voice x is taken as a transcription prompt word .

[0088] After obtaining the sample voice x, the sample voice x can be input into the voice preprocessing model to be trained, and through the encoding layer in the voice preprocessing model, the voice feature of the sample voice x is obtained , and then through the feature conversion layer in the voice preprocessing model, the text feature of the sample voice x is obtained . Wherein, .

[0089] When the transcription prompt word is not used, the training loss function of the Audio-LLM model is represented as:

[0090] Wherein, represents the nth token, is located in the token sequence before , including , represents the model parameter in the large language model, represents that the large language model LLM maximizes the probability of the next token , , .

[0091] When the transcription prompt word is used, the training loss function of the Audio-LLM model is represented as:

[0092] Wherein, represents that the large language model LLM maximizes the probability of the next token , ,​ , , maximize the probability of the next token .

[0093] To prevent overfitting to the transcription cues during model training, a first parameter p is introduced, and a first threshold is set as , where .

[0094] The training loss function corresponding to the sample speech x is represented as:

[0095] If the first parameter p corresponding to the sample speech x is less than or equal to the first threshold , the corresponding training loss function is , otherwise the corresponding training loss function is . Wherein the first parameter p is a random number generated for the sample speech, which is uniformly distributed in [0, 1].

[0096] In the embodiments of the present application, the annotated recognition text is used as a transcription cue to constrain the predicted recognition text output by the audio recognition large model during model training, and based on the difference between the predicted recognition text and the annotated recognition text of the audio recognition large model, the parameters of the speech preprocessing model in the audio recognition large model are updated and adjusted, so that the output result of the audio recognition large model is more accurate. In the model application stage, the first speech recognition result output by the ASR model is used as a transcription cue to constrain the output result of the audio recognition large model, which reduces the occurrence of hallucination phenomenon and improves the recognition accuracy of the audio recognition large model in the speech recognition task.

[0097] For example, as shown in Figure 4 , assuming that the first speech recognition result of the speech to be recognized is "from the crystallinity of the ground", the first speech recognition result is used as a transcription cue and input into the large language model. When decoding each token in turn through autoregressive decoding, each token decoded by the LLM is constrained based on the transcription cue to generate the target speech recognition result "from the ground. The net degree of view". On the one hand, the strong generation capability of the LLM is constrained based on the first speech recognition result output by the ASR model, avoiding hallucination phenomenon and speech recognition errors. On the other hand, the first speech recognition result output by the ASR model is corrected using the rich text information and powerful text modeling capability in the LLM to obtain an accurate speech recognition result.

[0098] In another embodiment of the present application, during model training of the audio recognition large model, the speech preprocessing model and the large language model can be jointly trained, wherein the large language model for speech recognition in the embodiment of the present application can be fine-tuned based on the pre-trained language model.

[0099] The audio recognition large model can be trained in the following manner: A plurality of second training samples are obtained; each second training sample includes a second sample speech and a labeled recognition text corresponding to the second sample speech.

[0100] Based on the plurality of second training samples, the training operation is continuously performed on the second neural network model to be trained and the pre-trained large language model until the second training end condition is met, and a trained audio recognition large model is obtained.

[0101] The training operation includes: The second sample speech of each second training sample is input into the second neural network model to obtain the predicted text feature of each second sample speech through the second neural network model. The second sample speech of each second training sample is input into the encoding layer of the second neural network model to obtain the second predicted feature representation of each second sample speech, and the second predicted feature representation of each second sample speech is input into the feature conversion layer of the second neural network model to obtain the predicted text feature of each second sample speech. According to the predicted text feature of each second sample speech and the labeled recognition text corresponding to each second sample speech, the pre-trained large language model is used to obtain the predicted recognition text corresponding to each second sample speech. According to the predicted recognition text corresponding to each second sample speech and the labeled recognition text corresponding to each second sample speech, a second training loss is determined. Based on the second training loss, the model parameters in the second neural network model and the pre-trained large language model are adjusted.

[0102] The autoregressive decoding in the embodiment of the present application is to predict the next token by using the previously predicted token, which can be represented as

[0103] Wherein, represents the token predicted by the LLM from each candidate token .

[0104] The non-autoregressive (NAR) decoding in the embodiment of the present application is to take the first speech recognition result output by the ASR model as a transcription prompt word , based on and The predicted target speech recognition result can be represented as:

[0105] in, This represents the text sequence with the highest probability among the candidate recognition texts.

[0106] because Since it is pre-generated through ASR, the large language model does not need a regression process during NAR decoding. The decoding result can be obtained through a single calculation, which speeds up the decoding process. Furthermore, it does not need to predict the terminator, which can avoid the phenomenon of decoding segment loops.

[0107] Optionally, when using NAR decoding, the first character sequence corresponding to the next token can be determined from the first speech recognition result output by the ASR model. This refers to the sequence of tokens decoded before the next token. It is based on text features. Decoded token sequence Predicting the next token can be represented as:

[0108] While NAR decoding can avoid the phenomenon of decoded segment loops, the fixed length of the transcription prompt (first speech recognition result) limits the LLM model's generation capability by constraining the number of characters in the transcription prompt. Therefore, in this embodiment, the advantages of both AR and NAR decoding are combined. If the number of target characters decoded by autoregression exceeds the maximum number of characters in the speech to be recognized, the target speech recognition result is obtained through non-autoregressive decoding based on text features and the first speech recognition result.

[0109] For example, the algorithm flow for implementing hybrid decoding in a large language model is as follows: Figure 5 As shown, where: Obtain the speech to be recognized x and the second parameter ; where the second parameter It can be set to 1.5 to calculate the maximum number of characters in the speech x to be recognized; Calculate the text features of the speech x to be recognized The first speech recognition result of the speech to be recognized, x, is obtained through the ASR model and used as transcription prompt words. Identify transcription prompt words Number of tokens ; Initialize decoding result For an empty sequence, the number of characters to decode. =0; Determine the decoding result Whether it ends with a terminator, when If the token does not end with a terminator, then AR decoding is used to predict the next token. At the same time, the decoded character is incremented by 1; Determine if the number of decoded characters exceeds the maximum number of characters in the speech to be recognized, where the maximum number of characters in the speech to be recognized is... ; If the number of decoded characters is greater than If the NAR decoding method is used, the NAR decoding result is output as the target speech recognition result of the speech to be recognized. If the number of decoded characters is less than or equal to , will token spliced ​​together At the end, repeat the above steps to determine... Whether to end with a terminator, until the target speech recognition result is obtained.

[0110] For example, such as Figure 6 As shown, assuming the second parameter is 2, when recognizing a target speech, the first speech recognition result obtained by the ASR model is "Beijing is the capital of China, with a long history and splendid culture, I love Beijing very much." It can be seen that the content of the speech recognition result obtained by the ASR model is not very accurate, but the character length deviation is small. The first speech recognition result is used as a transcription prompt word and input into the audio recognition model along with the target speech. Feature extraction is performed on the target speech to obtain the corresponding text features. Based on the first speech recognition result and the text features of the target speech, autoregressive AR decoding is performed. If segment looping occurs during AR decoding and the stop character cannot be predicted, the decoding result is "Beijing is the capital of China, with a long history and splendid culture, I love Beijing very much…". When the number of characters in the AR decoding result exceeds twice the number of characters in the first speech recognition result, NAR decoding is used based on the first speech recognition result and the text features of the target speech. The decoding result is "Beijing is the capital of China, with a long history and splendid culture, I love Beijing very much," and the NAR decoding result is used as the target speech recognition result.

[0111] The speech recognition method provided in this application embodiment can be applied to, for example... Figure 7 The speech recognition system shown includes at least one speech receiving device 10 and a speech recognition device 20 (only one is shown in the figure), and the speech receiving device 10 and the speech recognition device 20 communicate with each other via a network.

[0112] The voice receiving device 10 can collect a voice signal and send the collected voice signal as a voice to be recognized to the voice recognition device 20 for recognition. The voice recognition device 20 returns a target voice recognition result to the voice receiving device 10 after the recognition is completed.

[0113] After obtaining the voice to be recognized, the voice recognition device 20 can input the voice to be recognized into a voice recognition model to obtain a first voice recognition result of the voice to be recognized. The voice to be recognized and the first voice recognition result are input into the trained audio recognition large model. The text features corresponding to the voice to be recognized are obtained by feature extraction on the voice to be recognized. Based on the text features and the first voice recognition result, each target character is obtained in sequence by a self-recurrence decoding mode until a first condition is met. If the first condition met is that the target character currently decoded is the first character, the target voice recognition result is determined according to the target characters decoded. If the first condition met is that the number of target characters currently decoded is greater than the maximum number of characters, it indicates that the self-recurrence decoding has a fragment cycle phenomenon, and the self-recurrence decoding result is inaccurate. At this time, the target voice recognition result is obtained by a non-self-recurrence decoding mode based on the text features and the first voice recognition result.

[0114] The voice receiving device 10 can be a user terminal, such as a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart voice interaction device (for example, a smart speaker), a wearable electronic device (for example, a smart watch), a vehicle terminal, a smart home appliance (for example, a smart television), an AR / VR device, and the like, but is not limited thereto. The voice receiving device 10 is installed with a voice collection component and can be used to collect a voice signal. The voice recognition device 20 can be a server, which can be a physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud computing services, and is used to recognize a voice signal into corresponding text information. The voice receiving device 10 and the voice recognition device 20 can be directly or indirectly connected through wired or wireless communication, and the embodiments of the present application do not limit this.

[0115] The voice recognition method provided by the embodiments of the present application can effectively reduce the occurrence of hallucination by inputting the first voice recognition result output by the ASR model into the audio large language model as a transcription prompt word to constrain the output result. The mixed AR and NAR coding mode is used to further solve the output fragment cycle problem.

[0116] Three sets of control experiments are set up with the Audio-LLM base model, the Audio-LLM base model + transcription prompt, and the Audio-LLM base model + transcription prompt + hybrid decoding, respectively. Through testing on the training data set, the experimental results of the word error rate are as shown in Table 1.

[0117] Table 1 As can be seen from the experimental data in the table, compared with the Audio-LLM base model, the word error rate of the speech recognition result of each test set is significantly reduced after the method of adding transcription prompts and hybrid decoding is used in the embodiment of the application. The robustness of the audio recognition large model in the embodiment of the application for the speech recognition task is obviously improved, and the real-time rate of speech recognition is also reduced.

[0118] The real-time rate is positively correlated with the time required for speech recognition. After adding transcription prompts and hybrid decoding, the time required for speech recognition is reduced, and the efficiency of speech recognition is effectively improved.

[0119] Moreover, in the test set of speech recognition errors, the experimental results of the decoding segment repetition are as shown in Table 2.

[0120] Table 2 As can be seen from the experimental data in the table, compared with the Audio-LLM base model, the decoding segment repetition rate on the two test sets is significantly reduced after the transcription prompt is added. After further using the hybrid AR and NAR decoding method, the decoding segment repetition rate on the two test sets is completely reduced to 0, verifying that the method of the embodiment of the application solves the decoding repetition problem to some extent, and the accuracy of Audio-LLM in the speech recognition task is obviously improved, which can be applied to scenarios with high accuracy requirements. The decoding segment repetition rate refers to the proportion of the number of data with decoding repetition in the test set to the total number of data in the test set.

[0121] Based on the same principle as the speech recognition method provided in the embodiment of the application, the embodiment of the application provides a speech recognition device, as shown in Figure 8 The speech recognition device 200 can include a speech acquisition module 210, a first speech recognition module 220, and a second speech recognition module 230, wherein: The speech acquisition module 210 is configured to acquire a to-be-recognized speech. The first speech recognition module 220 is configured to obtain a first speech recognition result of the to-be-recognized speech by using a speech recognition model. The second speech recognition module 230 is configured to obtain a target speech recognition result of the to-be-recognized speech by using a trained audio recognition large model based on the to-be-recognized speech and the first speech recognition result. The audio recognition large model includes a speech preprocessing model and a large language model, and the target speech recognition result is obtained by the following method. The speech preprocessing model is used to extract features of the to-be-recognized speech, and text features corresponding to the to-be-recognized speech are obtained. The large language model is used to perform self-recurrent decoding based on the text features and the first speech recognition result, and each target character is sequentially decoded until a first condition is met. The first condition includes that the target character obtained by the current decoding is a first character, or the number of target characters that have been decoded is greater than the maximum number of characters of the to-be-recognized speech. If the first condition met is that the target character obtained by the current decoding is the first character, the target speech recognition result is determined according to the target characters that have been decoded. If the first condition met is that the number of target characters that have been decoded is greater than the maximum number of characters of the to-be-recognized speech, the large language model is used to perform non-self-recurrent decoding based on the text features and the first speech recognition result, and the target speech recognition result is obtained. The maximum number of characters of the to-be-recognized speech is determined based on the number of characters of the first speech recognition result.

[0122] Optionally, the target speech recognition result is obtained by the following method: The first target character is decoded based on the text features and the first speech recognition result. For each non-first character position, the target character of the character position is decoded according to a first character sequence corresponding to the character position, the text features, and the first speech recognition result. The first character sequence includes the target characters that have been decoded before the character position.

[0123] Optionally, the speech preprocessing model includes an encoding layer and a feature conversion layer. The second speech recognition module 230 can be configured to: The encoding layer is used to encode the to-be-recognized speech to obtain speech features of the to-be-recognized speech. The feature conversion layer is used to map the speech features of the to-be-recognized speech to a text feature space to obtain the text features corresponding to the to-be-recognized speech.

[0124] Optionally, the trained audio recognition large model includes a trained speech preprocessing model and a trained large language model. The speech preprocessing model is trained by the following manner: A plurality of first training samples are obtained; wherein each first training sample comprises a first sample speech and a labeled recognition text corresponding to the first sample speech; A training operation is continuously performed on a first neural network model to be trained based on the plurality of first training samples until a first training end condition is met, and a trained speech preprocessing model is obtained; The training operation comprises: The first sample speech of each first training sample is input into the first neural network model, and a predicted text feature of each first sample speech is obtained through the first neural network model; According to the predicted text feature of each first sample speech and the labeled recognition text corresponding to each first sample speech, a predicted recognition text corresponding to each first sample speech is obtained through a trained large language model; According to the predicted recognition text corresponding to each first sample speech and the labeled recognition text corresponding to each first sample speech, a first training loss is determined; The model parameters in the first neural network model are adjusted based on the first training loss.

[0125] Optionally, for each first sample speech, the predicted recognition text corresponding to the first sample speech is obtained by the following manner: A first parameter corresponding to the first sample speech is determined; If the first parameter corresponding to the first sample speech is less than or equal to a first threshold, a first recognition text corresponding to the first sample speech is obtained through a trained large language model according to the predicted text feature of the first sample speech and the labeled recognition text corresponding to the first sample speech, and the first recognition text is taken as the predicted recognition text corresponding to the first sample speech; If the first parameter corresponding to the first sample speech is greater than the first threshold, a second recognition text corresponding to the first sample speech is obtained through a trained large language model according to the predicted text feature of the first sample speech, and the second recognition text is taken as the predicted recognition text corresponding to the first sample speech.

[0126] Optionally, for each first sample speech, the predicted recognition text corresponding to the first sample speech is obtained by the following manner: Based on the predicted text feature of the first sample speech and the corresponding labeled recognition text, a target character at a first position in the predicted recognition text is obtained; For each non-first character position in the predicted recognition text, a first character sequence corresponding to the character position is determined according to the labeled recognition text corresponding to the first sample speech; and a target character of the character position is decoded according to the first character sequence corresponding to the character position and the text feature, wherein the first character sequence includes each target character that has been decoded before the character position.

[0127] Optionally, the trained audio recognition large model comprises a trained speech preprocessing model and a trained large language model. The audio recognition large model is trained in the following manner: A plurality of second training samples are obtained; each second training sample comprises a second sample speech and a labeled recognition text corresponding to the second sample speech. The trained second neural network model and the pre-trained large language model are continuously trained based on the plurality of second training samples until a second training end condition is met, to obtain a trained audio recognition large model. The training operation comprises: The second sample speech of each second training sample is input into the second neural network model to obtain a predicted text feature of each second sample speech through the second neural network model; The predicted recognition text corresponding to each second sample speech is obtained through the pre-trained large language model according to the predicted text feature of each second sample speech and the labeled recognition text corresponding to each second sample speech; A second training loss is determined according to the predicted recognition text corresponding to each second sample speech and the labeled recognition text corresponding to each second sample speech; The model parameters in the second neural network model and the pre-trained large language model are adjusted based on the second training loss.

[0128] The device provided in the embodiments of the present application can perform the method provided in the embodiments of the present application, and the implementation principles are similar, and the same technical effects can be produced. The actions performed by each module in the device of each embodiment of the present application are corresponding to the steps in the method of each embodiment of the present application. For the detailed functions of each module of the device, refer to the description of the corresponding method in the foregoing description, which will not be repeated here.

[0129] Based on Figure 8 The speech recognition device shown in the figure, based on the first speech recognition result recognized by the speech recognition model, constrains the number of characters of the target character obtained by the audio recognition large model autoregressive decoding, so that the target speech recognition result obtained by decoding is more accurate, avoids the problem of fragment cycle in the output result of the audio recognition large model, and improves the robustness of the large language model in the speech recognition task.

[0130] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined target, and can be implemented wholly or partially by using software, hardware (such as a processing circuit or a memory) or a combination thereof. Similarly, one processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be a part of an integral module or unit that contains the function of the module or unit.

[0131] In the embodiments of the present application, an electronic device is provided, which includes a memory, a processor and a computer program stored in the memory. When the processor executes the computer program stored in the memory, the method in any of the optional embodiments of the present application can be implemented.

[0132] Figure 9 A structural schematic diagram of an electronic device to which the embodiments of the present application are applicable is shown, as shown in Figure 9 The electronic device can be a server or a user terminal, and the electronic device can be used to implement the method provided in any of the embodiments of the present application.

[0133] As shown in Figure 9 The electronic device 2000 can mainly include at least one processor 2001 (one shown in Figure 9 a memory 2002, a communication module 2003 and an input / output interface 2004, and the like. Optionally, the components can be connected and communicated through a bus 2005. It should be noted that Figure 9 The structure of the electronic device 2000 shown in

[0134] The memory 2002 can be used to store an operating system and application programs, etc. The application programs can include computer programs that implement the methods shown in the embodiments of the present application when invoked by the processor 2001, and can also include programs for implementing other functions or services. The memory 2002 can be a ROM (Read Only Memory) or other type of static storage device that can store static information and instructions, a RAM (Random Access Memory) or other type of dynamic storage device that can store information and computer programs, and can also be an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory), or other optical disc storage, optical disc storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage medium, or other magnetic storage device, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and capable of being accessed by a computer, but not limited to this.

[0135] The processor 2001 is connected to the memory 2002 through the bus 2005, and realizes corresponding functions by invoking the application programs stored in the memory 2002. The processor 2001 can be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof, which can implement or execute various exemplary logical blocks, modules and circuits described in combination with the present disclosure. The processor 2001 can also be a combination of computing functions, such as one or more microprocessor combinations, combinations of DSP and microprocessor, etc.

[0136] The electronic device 2000 can be connected to a network through the communication module 2003 (which can include, but is not limited to, a network interface and the like) to communicate with other devices (such as user terminals or servers) through the network to realize the interaction of data, such as sending data to other devices or receiving data from other devices. Among them, the communication module 2003 can include a wired network interface and / or a wireless network interface and the like, that is, the communication module can include at least one of a wired communication module or a wireless communication module.

[0137] The electronic device 2000 can be connected to the required input / output devices such as a keyboard, a display device, and the like through the input / output interface 2004, and the electronic device 2000 itself can have a display device, and other display devices can also be externally connected through the interface 2004. Optionally, the storage device such as a hard disk can also be connected through the interface 2004, so that the data in the electronic device 2000 can be stored in the storage device, or the data in the storage device can be read, and the data in the storage device can also be stored in the storage 2002. It can be understood that the input / output interface 2004 can be a wired interface or a wireless interface. According to different actual application scenarios, the device connected with the input / output interface 2004 can be a component of the electronic device 2000, or an external device connected with the electronic device 2000 when needed.

[0138] The bus 2005 for connecting various components can include a channel for transmitting information between the above components. The bus 2005 can be a PCI (Peripheral Component Interconnect, Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture, Extended Industry Standard Architecture) bus, etc. According to different functions, the bus 2005 can be divided into an address bus, a data bus, a control bus, etc.

[0139] Optionally, for the scheme provided by the embodiment of the application, the storage 2002 can be used to store a computer program for executing the scheme of the application, and the processor 2001 runs the computer program to realize the actions of the method or device provided by the embodiment of the application.

[0140] Based on the same principle as the method provided by the embodiment of the application, the embodiment of the application provides a computer readable storage medium, and the computer readable storage medium stores a computer program, and the computer program is executed by the processor to realize the corresponding content of the foregoing method embodiment.

[0141] The embodiment of the present application further provides a computer program product, which comprises a computer program, and the computer program can realize the corresponding content of the foregoing method embodiment when executed by a processor.

[0142] It should be noted that the terms "first", "second", "third", "fourth", "1", "2", and the like (if any) in the description and claims of the present application and the above drawings are used to distinguish similar objects, and do not necessarily have to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that shown or described.

[0143] It should be understood that, although the flowcharts of the embodiments of the present application indicate various operation steps by arrows, the implementation order of the steps is not limited to the order indicated by the arrows. Unless otherwise specified herein, in some implementation scenarios of the embodiments of the present application, the implementation steps in each flowchart can be executed in other orders as required. In addition, part or all of the steps in each flowchart can include multiple sub-steps or multiple stages based on the actual implementation scenario. Part or all of these sub-steps or stages can be executed at the same time, and each of these sub-steps or stages can also be executed at different times. In the scenario where the execution times are different, the execution order of these sub-steps or stages can be flexibly configured according to requirements, and the embodiments of the present application do not limit this.

[0144] The above is only an optional implementation of some implementation scenarios of the present application, and it should be pointed out that, for ordinary skilled persons in the technical field, other similar implementation means based on the technical idea of the present application without departing from the technical concept of the present application also belong to the protection scope of the embodiments of the present application.< / eos>

Claims

1. A speech recognition method, characterized in that, include: Acquire the speech to be recognized; The first speech recognition result of the speech to be recognized is obtained by using the trained speech recognition model; Based on the speech to be recognized and the first speech recognition result, the target speech recognition result of the speech to be recognized is obtained through a trained audio recognition model: The audio recognition model includes a speech preprocessing model and a large language model, and the target speech recognition result is obtained through the following methods: The speech preprocessing model is used to extract features from the speech to be recognized, and the text features corresponding to the speech to be recognized are obtained. Based on the text features and the first speech recognition result, autoregressive decoding is performed through the large language model to sequentially decode each target character until a first condition is met; the first condition includes that the currently decoded target character is the first character, or that the number of currently decoded target characters is greater than the maximum number of characters in the speech to be recognized. If the first condition is met, that is, the target character currently decoded is the first character, then the target speech recognition result is determined based on each target character currently decoded. If the first condition is met, that is, the number of currently decoded target characters is greater than the maximum number of characters in the speech to be recognized, then based on the text features and the first speech recognition result, non-autoregressive decoding is performed through the large language model to obtain the target speech recognition result; wherein, the maximum number of characters in the speech to be recognized is determined based on the number of characters in the first speech recognition result.

2. The method according to claim 1, characterized in that, Based on the text features and the first speech recognition result, autoregressive decoding is performed using the large language model to sequentially decode each target character, including: Based on the text features and the first speech recognition result, the first target character is decoded; For each non-first character position, the target character for that character position is decoded based on the first character sequence corresponding to that character position, the text features, and the first speech recognition result; wherein, the first character sequence includes all target characters that have been decoded previously for that character position.

3. The method according to claim 1, characterized in that, The speech preprocessing model includes an encoding layer and a feature transformation layer; The step of extracting features from the speech to be recognized using the speech preprocessing model to obtain the text features corresponding to the speech to be recognized includes: The speech to be recognized is encoded through the coding layer to obtain the speech features of the speech to be recognized; The feature transformation layer maps the speech features of the speech to be recognized to the text feature space to obtain the text features corresponding to the speech to be recognized.

4. The method according to claim 1, characterized in that, The trained audio recognition model includes a trained speech preprocessing model and a trained large language model; The speech preprocessing model was trained in the following way: Obtain multiple first training samples; wherein each first training sample includes a first sample speech and the labeled recognition text corresponding to the first sample speech; The training operation is continuously performed on the first neural network model to be trained based on multiple first training samples until the first training termination condition is met, and a trained speech preprocessing model is obtained. The training operations include: The first sample speech of each first training sample is input into the first neural network model, and the predicted text features of each first sample speech are obtained through the first neural network model. Based on the predicted text features of each first sample speech and the labeled recognition text corresponding to each first sample speech, the predicted recognition text corresponding to each first sample speech is obtained through the trained large language model. The first training loss is determined based on the predicted recognition text corresponding to each first sample speech and its corresponding labeled recognition text. The model parameters in the first neural network model are adjusted based on the first training loss.

5. The method according to claim 4, characterized in that, For each first sample speech, based on the predicted text features of that first sample speech and the corresponding labeled recognition text, the predicted recognition text corresponding to that first sample speech is obtained through a trained large language model, including: Determine the first parameter corresponding to the first sample speech; If the first parameter corresponding to the first sample speech is less than or equal to the first threshold, based on the predicted text features of the first sample speech and the labeled recognition text corresponding to the first sample speech, the first recognition text corresponding to the first sample speech is obtained through the trained large language model, and the first recognition text is used as the predicted recognition text corresponding to the first sample speech. If the first parameter corresponding to the first sample speech is greater than the first threshold, the second recognized text corresponding to the first sample speech is obtained through the trained large language model based on the predicted text features of the first sample speech, and the second recognized text is used as the predicted recognized text corresponding to the first sample speech.

6. The method according to claim 4, characterized in that, For each first sample speech, based on the predicted text features of that first sample speech and the corresponding labeled recognition text, the predicted recognition text corresponding to that first sample speech is obtained through a trained large language model, including: Based on the predicted text features of the first sample speech and the corresponding labeled recognition text, the first and second target characters in the predicted recognition text are obtained; For each non-first character position in the predicted recognition text, the first character sequence corresponding to the character position is determined based on the labeled recognition text corresponding to the first sample speech; the target character of the character position is decoded based on the first character sequence corresponding to the character position and the text features; wherein, the first character sequence includes each target character previously decoded for the character position.

7. The method according to claim 1, characterized in that, The trained audio recognition model includes a trained speech preprocessing model and a trained large language model; The large audio recognition model was trained in the following way: Obtain multiple second training samples; wherein each second training sample includes a second sample speech and the labeled recognition text corresponding to the second sample speech; Based on multiple second training samples, the training operation is continuously performed on the second neural network model to be trained and the pre-trained large language model until the second training termination condition is met, and a well-trained audio recognition large model is obtained. The training operations include: The second sample speech of each second training sample is input into the second neural network model, and the predicted text features of each second sample speech are obtained through the second neural network model. Based on the predicted text features of each second sample speech and the labeled recognition text corresponding to each second sample speech, the predicted recognition text corresponding to each second sample speech is obtained through a pre-trained large language model. The second training loss is determined based on the predicted recognition text corresponding to each second sample speech and its corresponding labeled recognition text. The model parameters in the second neural network model and the pre-trained large language model are adjusted based on the second training loss.

8. A voice recognition device, characterized in that, The device includes: The speech acquisition module is used to acquire the speech to be recognized; The first speech recognition module is used to obtain the first speech recognition result of the speech to be recognized by using a trained speech recognition model; The second speech recognition module is used to obtain the target speech recognition result of the speech to be recognized based on the speech to be recognized and the first speech recognition result, using a trained audio recognition model: The audio recognition model includes a speech preprocessing model and a large language model, and the target speech recognition result is obtained through the following methods: The speech preprocessing model is used to extract features from the speech to be recognized, and the text features corresponding to the speech to be recognized are obtained. Based on the text features and the first speech recognition result, autoregressive decoding is performed through the large language model to sequentially decode each target character until a first condition is met; the first condition includes that the currently decoded target character is the first character, or that the number of currently decoded target characters is greater than the maximum number of characters in the speech to be recognized. If the first condition is met, that is, the target character currently decoded is the first character, then the target speech recognition result is determined based on each target character currently decoded. If the first condition is met, that is, the number of currently decoded target characters is greater than the maximum number of characters in the speech to be recognized, then based on the text features and the first speech recognition result, non-autoregressive decoding is performed through the large language model to obtain the target speech recognition result; wherein, the maximum number of characters in the speech to be recognized is determined based on the number of characters in the first speech recognition result.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the method described in any one of claims 1 to 7.

11. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 7.

Images