Speech recognition method and device

Abstract

This disclosure provides a speech recognition method and device. The method includes: determining a speech feature vector of a speech to be recognized; acquiring associated speech and associated text related to the speech to be recognized, and determining feature vectors of the associated speech and associated text respectively to obtain associated speech feature vectors and associated text feature vectors; and recognizing the speech feature vectors based on the associated speech feature vectors and associated text feature vectors to obtain the recognized text of the speech to be recognized. This can improve the accuracy of speech recognition.

Description

Technical Field

[0001] This disclosure relates to the fields of computer and speech recognition technology, and in particular to a speech recognition method and device. Background Technology

[0002] Because voice input is easier to operate and more efficient than text input, users are increasingly using voice to input information during human-computer interaction.

[0003] When users interact with computers using voice input, the input voice information needs to be processed by Automatic Speech Recognition (ASR) to convert it into text. The accuracy of the speech recognition result significantly affects the quality of human-computer interaction. To improve the quality of human-computer interaction via voice, it is necessary to improve the accuracy of speech recognition. Summary of the Invention

[0004] This disclosure provides a speech recognition method and device to improve speech recognition accuracy.

[0005] In a first aspect, embodiments of this disclosure provide a voice information processing method, including:

[0006] Determine the speech feature vector of the speech to be recognized;

[0007] Obtain the associated speech and associated text related to the speech to be identified, and determine the feature vectors of the associated speech and associated text respectively to obtain the associated speech feature vector and associated text feature vector;

[0008] Based on the associated speech feature vector and the associated text feature vector, the speech feature vector is identified to obtain the identified text of the speech to be identified.

[0009] In a second aspect, embodiments of this disclosure provide a voice recognition device, including:

[0010] The determining unit is used to determine the speech feature vector of the speech to be recognized;

[0011] The determining unit is used to acquire associated speech and associated text related to the speech to be identified, and to determine the feature vectors of the associated speech and the associated text respectively, so as to obtain the associated speech feature vector and the associated text feature vector.

[0012] The recognition unit is used to recognize the speech feature vector based on the associated speech feature vector and the associated text feature vector to obtain the recognized text of the speech to be recognized.

[0013] Thirdly, embodiments of this disclosure provide an electronic device, including: a processor and a memory;

[0014] The memory stores computer-executed instructions;

[0015] The processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the speech recognition method as described in the first aspect and various possible designs of the first aspect.

[0016] Fourthly, embodiments of this disclosure provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the speech recognition method described in the first aspect and various possible designs of the first aspect.

[0017] Fifthly, embodiments of this disclosure provide a computer program product, including a computer program that, when executed by a processor, implements the speech recognition method as described in the first aspect and various possible designs of the first aspect.

[0018] The speech recognition method and device provided in this embodiment can improve the accuracy of speech recognition by recognizing the speech feature vector of the speech to be recognized based on the associated speech feature vector and the associated text feature vector. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 Flowchart of the speech recognition method provided in the embodiments of this disclosure Figure 1 ;

[0021] Figure 2 Flowchart of the speech recognition method provided in the embodiments of this disclosure Figure 2 ;

[0022] Figure 3 This is a schematic diagram of the model training process provided in the embodiments of this disclosure;

[0023] Figure 4 This is a structural block diagram of a speech recognition device provided in an embodiment of the present disclosure;

[0024] Figure 5 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0026] Users can interact with computers via voice. For example, users can use voice to communicate with intelligent assistants (such as chatbots) to obtain relevant information or instruct them to complete a task. When users interact with computers via voice, a voice recognition system is needed to recognize the user's input voice information (converting the voice information into corresponding text). The accuracy of voice recognition will significantly affect the outcome of the human-computer interaction.

[0027] To improve speech recognition accuracy, some related technologies extract keywords from dialogue messages within the speech recognition system and use these keywords as "hot words" to enhance recognition performance when the user's voice request hits the hot word list in the next round. Specifically, in the initial speech recognition process, the ASR system attempts to identify keywords in the speech. These keywords are usually important words related to the context, such as names of people, places, and technical terms. Once these keywords are identified, the ASR system marks them as "hot words" to increase their priority and weight in subsequent recognition processes. The ASR system dynamically adjusts its acoustic and language models, making it more inclined to recognize these hot words in subsequent recognition processes. However, in this approach, if the hot word list is inaccurate or contains incorrect keywords, the ASR system may incorrectly prioritize recognizing these words, leading to an increased probability of misrecognition and hindering the improvement of speech recognition accuracy. Furthermore, this method requires additional resources to maintain the hot word list, resulting in resource waste.

[0028] In this disclosure, when recognizing the speech to be recognized, the speech feature vector of the speech to be recognized is identified based on associated speech and associated text. The associated speech and associated text can be used as a message background for the speech to be recognized. By capturing the implicit meaning and topic shifts in the information interaction process based on the message background and the speech to be recognized, recognition errors can be reduced and the accuracy of the recognized text can be improved. In addition, this solution does not require the maintenance of a hot word list, which can reduce resource waste.

[0029] Please refer to Figure 1 , Figure 1 Flowchart of the speech recognition method provided in this disclosure Figure 1 ,like Figure 1As shown, the method includes the following steps:

[0030] S101: Determine the speech feature vector of the speech to be recognized.

[0031] The entity executing the speech recognition method can be a terminal device, specifically a preset application client running on the terminal device; or a server, specifically a preset application server running on the server to provide services to the application client running on the terminal device.

[0032] The aforementioned preset application can receive user-input voice and use that voice as the speech to be recognized.

[0033] In one example, features can be extracted from the speech to be recognized to obtain speech features, and then the speech features can be vectorized to obtain the speech feature vector of the speech to be recognized.

[0034] The aforementioned feature extraction process for the speech to be recognized includes steps such as preprocessing, framing, windowing, Fourier transform, power spectrum calculation, Mel filter bank, Discrete Cosine Transform (DCT), difference coefficient calculation, and mean normalization to obtain the speech features to be recognized. These speech features can include key attribute features of the captured speech, such as pitch, timbre, and articulation patterns. These features can be used for tasks such as speech recognition and speaker recognition.

[0035] In one example, an audio encoder can be invoked to encode the speech to be recognized, obtaining a speech feature vector. Encoding the speech using an audio encoder extracts key features from the raw audio data. These features can be used for tasks such as speech recognition and speaker recognition. Furthermore, raw audio data is typically high-dimensional time-series data, requiring significant computational resources to process. Encoding the audio data with an audio encoder compresses the data into a lower-dimensional space while retaining useful information, thus reducing computational costs.

[0036] S102: Obtain the associated speech and associated text related to the speech to be recognized, determine the feature vectors of the associated speech and associated text respectively, and obtain the associated speech feature vector and associated text feature vector.

[0037] In some examples, the context of the associated speech and associated text is related to the context of the speech to be identified. The aforementioned context (also known as situation or context) refers to the linguistic environment, which can be the surrounding context of a coherent speech event.

[0038] In one example, the context of the associated speech and associated text can be the same as the context of the speech to be recognized, where the associated speech and associated text are the contextual content of the speech to be recognized within that context. For example, in a scenario where a user interacts with a smart assistant, the speech to be recognized can be the speech currently input by the user, the associated speech can be the speech previously input by the user, and the associated text can be the text in which the smart assistant responds to the user's previously input speech.

[0039] In some application scenarios, the number of associated speech can be more than one, and the number of associated text can also be more than one.

[0040] For each associated speech segment, speech features can be extracted separately to obtain the associated speech features corresponding to each segment. When extracting features from each associated speech segment, the same feature extraction method used for the speech to be recognized can be employed.

[0041] For each associated text, text features can be extracted to obtain the associated text features corresponding to each associated text.

[0042] S103: Based on the associated speech feature vector and the associated text feature vector, the speech feature vector is identified to obtain the identified text of the speech to be identified.

[0043] This step, also known as the decoding process, can identify speech feature vectors into one or more characters, which together form the text recognized from the speech to be recognized.

[0044] In some implementations, deep learning methods can be used to build a recognition model that directly receives the original audio and text as input and outputs the final recognized text. In other implementations, the aforementioned speech feature vectors can be input into a classifier (such as a support vector machine, random forest, or other classifiers) for text prediction.

[0045] In this embodiment, associated speech feature vectors and associated text feature vectors are introduced to assist in the recognition of the speech feature vectors to be identified. During the text recognition process of the speech to be identified, the content of associated speech and associated text is referenced, which can better predict and understand the user's intent. Using associated speech and associated text to construct message background and capture implicit meanings and topic shifts during information interaction can reduce recognition errors. Due to the consistency of language used by users in the same context, recognizing users' personalized expressions based on associated speech and then recognizing the speech to be identified based on the recognized personalized expressions is beneficial to improving the accuracy of speech recognition. Furthermore, this solution does not require additional maintenance of a hot word list, thus avoiding resource waste.

[0046] In other related technologies, ASR systems are based on a language processing model. This model generates prompts including dialogue history text and the user's speech to be recognized, which are then input into the language processing model. The model then identifies the user's speech based on the text context. While this approach, which relies on the language processing model to recognize speech based on text context, achieves a certain level of accuracy in dialogue recognition scenarios, its accuracy remains low for complex scenarios. For example, if the ASR text in the dialogue history contains errors, using the erroneous ASR text may lead to misinterpretation. Furthermore, when the user's speech has an accent or dialect, different dialects sometimes have similar pronunciations when saying different things; for instance, the pronunciation of "2" in Cantonese is similar to "1" in Mandarin. The language processing model cannot distinguish between dialects using only the dialogue history text. In short, when the dialogue history text cannot provide sufficient information about the audio environment, the accuracy of the speech recognition model often decreases. Figure 2 The solutions provided in the illustrated embodiments can improve the above-mentioned problems.

[0047] Please refer to Figure 2 , Figure 2 Flowchart of the speech recognition method provided in this disclosure Figure 2 ,like Figure 2 As shown, the method includes the following steps:

[0048] S201: Determine the speech feature vector of the speech to be recognized.

[0049] In this embodiment, the entity executing the speech recognition method can be a preset application running on the terminal device. The specific implementation of step S201 can be found in [reference needed]. Figure 1 The description of step S101 in the illustrated embodiment will not be repeated here.

[0050] S202: Obtain the associated speech and associated text related to the speech to be recognized, and generate a prompt message including the associated speech and associated text.

[0051] In one example, the associated speech and associated text can be the context of the speech to be recognized. For example, the speech to be recognized can be a voice message in a conversation, the associated speech can be the voice message entered by the user in a preset conversation round before the speech to be recognized in that conversation; the associated text can be the text entered by the user in a preset conversation round before the speech to be recognized in that conversation, or the text reply from the other end (e.g., including a smart assistant or other users having a conversation with the user) to the message entered by the user (text message or voice message).

[0052] It should be understood that in the scheme disclosed herein, the speech to be identified, as well as the associated speech and associated text, are obtained with the user's authorization.

[0053] In this embodiment, a speech recognition model built on a pre-trained natural language processing model can be used to perform speech recognition.

[0054] To enable the aforementioned speech recognition model to accurately identify the speech feature vectors of the speech to be recognized, prompt information can be generated first. This prompt information can include associated speech and associated text, resulting in a prompt information that combines associated speech and associated text.

[0055] In some implementations, the aforementioned speech to be recognized, associated speech, and associated text can be simultaneously written into the prompt information. Specifically, the associated speech and associated text can be written into a reference content field, the speech to be recognized can be written into a speech content field, and the prompt information may further include relevant prompts to instruct the speech recognition model to perform speech recognition based on the associated speech and associated text.

[0056] S203: Call the speech vectorization model to process the associated speech in the prompt information to obtain the associated speech feature vector. Call the text feature vector extraction algorithm to process the associated text in the prompt information to obtain the associated text feature vector.

[0057] The speech vectorization model here can be a model that vectorizes audio signals. In one example, the speech vectorization model could be an audio encoder.

[0058] In some application scenarios, the speech to be recognized, its associated speech and associated text can be obtained. The speech vectorization model is then used to process the speech to be recognized to obtain the speech feature vector. The associated speech is then processed using the same speech vectorization model to obtain the associated speech feature vector.

[0059] Text feature vector extraction algorithms can be various algorithms that extract features from text and vectorize those features, such as bag-of-words model, term frequency-inverse document frequency, word embedding model, etc.

[0060] S204: Input the speech feature vector, associated speech feature vector, and associated text feature vector into the speech recognition model. The speech recognition model recognizes the speech feature vector based on the associated speech feature vector and associated text feature vector to obtain the recognized text of the speech to be recognized.

[0061] In one example, the associated speech feature vector, associated text feature vector, and speech feature vector can be concatenated, and the concatenated vector can be input into a speech recognition model. Understandably, the speech recognition model can identify the associated speech feature vector, associated text feature vector, and the speech feature vector of the speech to be recognized from the concatenated vector.

[0062] The aforementioned speech feature vectors are typically represented as numerical matrices, with each column or row representing a feature vector for a given time frame. Each element in the feature vector is the numerical value of a specific feature within that frame. These features include, but are not limited to: Mel-frequency cepstral coefficients (MFCC), linear prediction coefficients (LPC) representing the spectral envelope of the speech signal, perceptual linear prediction (PLP) using linear prediction cepstral coefficients (LPCC), fundamental frequency (F0), energy and short-time energy, and dynamic features obtained by first-order and second-order differencing of features such as MFCC and LPC. These features can be used to map speech signals to text.

[0063] The aforementioned speech recognition model can be obtained by fine-tuning a pre-trained natural language processing model. The speech recognition model can convert the input speech signal into a text sequence.

[0064] After inputting the speech feature vector of the speech to be recognized into the speech recognition model, the speech recognition model can generate the text corresponding to the speech feature vector, that is, the recognized text of the speech to be recognized.

[0065] When recognizing text corresponding to speech feature vectors, combining the contextual information reflected by the associated text feature vectors and the associated speech feature vectors to perform text recognition allows for personalized speech recognition. For example, if a user uses a specific language (e.g., a dialect) for information interaction, the associated speech and the speech to be recognized usually use the same language. Therefore, personalized features of the user can be learned from the associated speech feature vectors. Using these personalized features to recognize the speech feature vectors of the speech to be recognized helps improve the accuracy of speech recognition. Associating speech can identify the user's accent or dialect. When the same pronunciation corresponds to different words in different dialects or accents, using the accent or dialect identified by the associated speech to recognize the speech to be recognized can further improve the accuracy of speech recognition.

[0066] In this embodiment, by generating prompt information including associated speech and associated text, the associated speech and associated text in the prompt information are vectorized to obtain associated speech feature vectors and associated text feature vectors. The speech recognition model combines the associated speech feature vectors and associated text feature vectors to recognize the speech feature vectors of the speech to be recognized. Thus, when the speech recognition model recognizes the speech to be recognized, it can learn the contextual information provided by the associated text and associated speech, and use the contextual information to eliminate ambiguity to accurately understand polysemous words and homophones in the current speech. The associated speech and associated text can be used to recognize and correct recognition errors, thereby improving the accuracy of the recognized text obtained from the recognition of the speech to be recognized.

[0067] In some embodiments of this example, the number of associated speech items includes at least one, and the number of associated text items includes at least one; the generation of step S202 above includes prompt information for associated speech items and associated text items, including:

[0068] The associated audio and text are written into the preset prompt message template in the order of their generation time to obtain the prompt message.

[0069] As an example, in a conversational scenario, each associated voice message can be a complete message that the user has historically sent in voice format to other conversation participants. Each associated text message can be a complete message that the user has historically sent in text format to other conversation participants, or a message that other conversation participants have sent in text format to the user. Illustratively, text messages sent by other users can be reply messages responding to messages sent by the user.

[0070] As an example, in a video conferencing scenario, the speech to be identified can be the speech information entered by the current speaker in the current time period; associated speech can include the speaker's historical speech information in the video conference, and can also include the historical speech of other speakers.

[0071] Each associated voice message can correspond to a timestamp.

[0072] In this example, the video conference can use a presentation document, and the associated text mentioned above can be a portion of the text content of the presentation document, such as a portion of the text content of the presentation document shown to the user according to the time period covered by the timestamp corresponding to each associated voice message.

[0073] The associated audio and text can be written into the prompt information in the order of their generation time. This allows the prompt information to retain the order of the different associated audio and text and to reflect the logical connection between them.

[0074] In some embodiments of this example, step S204 includes the following steps:

[0075] First, following the order in the prompts, the associated speech feature vectors corresponding to each associated speech and the associated text feature vectors corresponding to each associated text are concatenated, and the resulting concatenated vector is input into the speech recognition model.

[0076] Secondly, the speech recognition model identifies the speech feature vectors based on the associated speech feature vectors and associated text feature vectors parsed from the concatenated vectors, thus obtaining the recognized text.

[0077] In these implementations, by concatenating the feature vectors of each associated speech and each associated text in the order of the prompt information, the speech recognition model can identify each associated speech feature vector and each associated text feature vector from the concatenated vector. It can also learn the speaker's timbre, speech rate, accent, and pronunciation features from the concatenated vector, and learn rich contextual information, thus aiding in the recognition of polysemous words, homophones, and sentence structures. Furthermore, it can identify historical speech recognition errors and adjust the speech recognition model parameters or mapping relationships based on these errors to reduce future recognition errors. Recognizing the speech to be recognized based on the speech recognition method of the user's historical language use (e.g., dialect) is beneficial for improving the accuracy of speech recognition.

[0078] In some embodiments, please refer to Figure 3 , Figure 3 This is a schematic diagram of the model training process provided in an embodiment of this disclosure. Figure 3 The method shown also includes multiple rounds of fine-tuning of the pre-trained natural language processing model and speech vectorization model based on the following steps to obtain the speech recognition model and speech vectorization model:

[0079] S301: Construct multiple sample prompts, each of which includes sample-related speech and sample-related text arranged in chronological order, as well as the sample speech to be recognized and the sample recognition text corresponding to the sample speech to be recognized.

[0080] In some application scenarios, the sample-associated speech and sample-associated text in each sample prompt message, as well as the sample speech to be recognized and the sample recognition text corresponding to the sample speech to be recognized, can be content generated in the actual scenario.

[0081] As an example, in a scenario where a user interacts with a smart assistant via voice, a voice message input by the user can be selected as a sample voice to be recognized; the voice from one or more rounds of dialogue preceding the sample voice to be recognized can be used as the associated voice; the text input by the user or the reply from the smart assistant to the user in one or more rounds of dialogue can be used as the associated text; the sample voice to be recognized can be labeled with real text, and the labeled real text can be used as the sample recognition text.

[0082] The sample prompt information can include multiple fields, such as the associated information field, the speech to be recognized field, and the output field. The associated speech and the associated text of each sample can be written in the associated message field in chronological order of their generation. The speech to be recognized of the sample can be written in the speech to be recognized field. The sample recognition text can be written in the output field as the target output.

[0083] As an example, one can also manually prepare sample-associated speech, sample-associated text, sample speech to be recognized, sample recognition text, etc., and generate prompt information including the above content.

[0084] As an example, the sample prompts mentioned above can also include related speech samples from historically misidentified samples, related text samples, the speech sample to be recognized, and the accurate text samples of the speech sample to be recognized. These sample prompts enable the speech recognition model to learn the ability to correct errors in subsequent speech recognition processes, even if errors occurred in past speech recognition.

[0085] S302: For each sample prompt, the sample-related speech and the sample-to-be-recognized speech in the sample prompt are input into the speech vectorization model. The feature vectors of each sample-related speech and the sample-to-be-recognized speech output by the speech vectorization model, along with the feature vectors of each sample-related text, are input into the natural language processing model.

[0086] S303: Based on the loss between the output of the natural language processing model and the target output, adjust the parameters of the natural language processing model and the speech vectorization model. The target output is the sample recognition text in the prompt information. Use the fine-tuned natural language processing model as the speech recognition model.

[0087] In this embodiment, the parameters of the pre-trained natural language processing model and the speech vectorization model can be jointly fine-tuned.

[0088] A preset loss function can be used to calculate the loss between the output of the speech recognition model and the target output. Such loss functions may include, for example, the cross-entropy loss function, the connectionist temporal classification (CTC) loss function, the negative log-likelihood (NLL) loss function, and the mean squared error (MSE) loss function.

[0089] The parameters of the natural language processing model and the speech vectorization model are adjusted according to the loss, with the aim of reducing the loss between the output obtained by the speech vectorization model and the natural language processing model based on the input and the target output.

[0090] After multiple rounds of training, training can be stopped once preset conditions are met, resulting in a fine-tuned natural language processing model and a speech vectorization model. The fine-tuned natural language processing model is then used as the speech recognition model.

[0091] During fine-tuning, the natural language processing model can learn to recognize the speech by using the alternating audio and text association information in the prompts as contextual content; it can learn how to map speech signals to corresponding text representations; it can learn to adapt to different accents and languages; and it can learn noise recognition, etc. Thus, through this fine-tuning process, the natural language processing model can perform speech recognition efficiently and accurately in speech recognition tasks.

[0092] In some implementations, the associated speech and the speech to be identified carry noise, and step S204 above includes the following steps:

[0093] First, noise features are identified from the associated speech feature vector using a speech recognition model, and then the noise features in the speech feature vector are identified based on the identified noise features.

[0094] Secondly, the speech recognition model filters out the noise features it identifies, and then identifies the speech feature vectors that have been filtered out of noise to obtain the recognized text.

[0095] In these embodiments, the speech to be identified and the associated speech can be acquired by an audio acquisition device. During the acquisition of audio signals using this device, noise is inevitably captured. This noise can refer to various types of audio signals other than the speaker's voice.

[0096] In these implementations, the speech recognition model can identify noise features from one or more associated speech feature vectors. Noise can then be filtered out from the speech feature vectors of the speech to be recognized based on these noise features.

[0097] For example, associated speech may include long-duration speech, while the current speech to be recognized is short-duration. Identifying noise from short-duration speech is difficult, so if the current speech to be recognized carries short-term noise, the noise cannot be identified, thus hindering accurate recognition. Speech recognition models can identify noise features from long-duration associated speech, then filter out the noise features identified from the associated speech feature vector into the speech feature vector of the speech to be recognized. The noise-filtered speech feature vector is then used for recognition. This achieves noise identification and filtering based on the context of the speech to be recognized, contributing to improved speech recognition accuracy.

[0098] In some implementations, the noise mentioned above includes the voices of users other than the main speaker.

[0099] In these implementations, all audio signals other than the speaker's voice can be treated as noise, and thus the noise can include the voices of other users captured by the audio acquisition device.

[0100] In some application scenarios, while the speaker is inputting voice data, other users are also speaking around them. The content of these other users' speech is unrelated to the topic being spoken and can be considered interference. If the speech of these other users is also recognized during speech recognition, it will insert irrelevant text into the recognized text, resulting in poor coherence of the speech recognition result and affecting its accuracy.

[0101] In these implementations, the voices of users other than the main speaker are also treated as noise. During speech recognition, the voices of other users can be filtered out. Recognizing the voice of the main speaker, after filtering out the voices of other users, can eliminate the interference of other users' voices, which helps to improve the coherence of the recognized text and thus improve the accuracy of the speech recognition results.

[0102] Corresponding to the speech recognition method in the above embodiments, Figure 4 This is a structural block diagram of a speech recognition device provided in an embodiment of this disclosure. For ease of explanation, only the parts relevant to the embodiments of this disclosure are shown. (Refer to...) Figure 4 The device includes: a determination unit 401, an acquisition unit 402, and an identification unit 403.

[0103] Among them, the determining unit 401 is used to determine the speech feature vector of the speech to be recognized;

[0104] The acquisition unit 402 is used to acquire the associated speech and associated text associated with the speech to be recognized, determine the feature vectors of the associated speech and associated text respectively, and obtain the associated speech feature vector and associated text feature vector.

[0105] The recognition unit 403 is used to recognize the speech feature vector based on the associated speech feature vector and the associated text feature vector to obtain the recognized text of the speech to be recognized.

[0106] In one embodiment of this disclosure, the acquisition unit 402 is further configured to:

[0107] Obtain associated speech and associated text with the speech to be recognized, and generate prompt information including associated speech and associated text;

[0108] The speech vectorization model is called to process the associated speech in the prompt information to obtain the associated speech feature vector; the text feature vector extraction algorithm is called to process the associated text in the prompt information to obtain the associated text feature vector.

[0109] In one embodiment of this disclosure, the identification unit 403 is further configured to:

[0110] The speech feature vector, associated speech feature vector, and associated text feature vector are input into the speech recognition model. The speech recognition model then uses the associated speech feature vector and associated text feature vector to identify the speech feature vector, thereby obtaining the recognized text of the speech to be recognized.

[0111] In one embodiment of this disclosure, the number of associated speech items includes at least one, and the number of associated text items includes at least one; the acquisition unit 402 is further configured to:

[0112] The associated audio and text are written into the preset prompt message template in the order of their generation time to obtain the prompt message.

[0113] In one embodiment of this disclosure, the identification unit 403 is further configured to:

[0114] Following the order in the prompts, the associated speech feature vectors corresponding to each associated speech and the associated text feature vectors corresponding to each associated text are concatenated, and the resulting concatenated vector is input into the speech recognition model.

[0115] The speech recognition model identifies the speech feature vectors based on the associated speech feature vectors and associated text feature vectors parsed from the concatenated vectors, and obtains the recognized text.

[0116] In one embodiment of this disclosure, device 40 further includes a model training unit (not shown), the model training unit being used for:

[0117] Construct multiple sample prompts, each of which includes sample-related speech and sample-related text arranged in chronological order, as well as the sample speech to be recognized and the sample recognition text corresponding to the sample speech to be recognized;

[0118] For each sample prompt, the sample-associated speech and the sample-to-be-recognized speech in the sample prompt are input into the speech vectorization model. The feature vectors of each sample-associated speech and each sample-to-be-recognized speech output by the speech vectorization model, along with the feature vector of the sample-associated text, are input into the natural language processing model.

[0119] Based on the loss between the output of the natural language processing model and the target output, the parameters of the natural language processing model and the speech vectorization model are adjusted, with the target output being the sample recognized text in the prompt information; the fine-tuned natural language processing model is used as the speech recognition model.

[0120] In one embodiment of this disclosure, the associated speech and the speech to be identified carry noise; the identification unit 403 is further configured to:

[0121] Noise features are identified from associated speech feature vectors using a speech recognition model, and noise features in the speech feature vectors are then identified based on the identified noise features.

[0122] The speech recognition model filters out the noise features it detects, and then identifies the speech feature vectors that have been filtered out of noise to obtain the recognized text.

[0123] In one embodiment of this disclosure, the noise includes the voices of users other than the main speaker.

[0124] The device provided in this embodiment can be used to execute the technical solutions of the above method embodiments. Its implementation principle and technical effect are similar, and will not be described again here.

[0125] To implement the above embodiments, this disclosure also provides an electronic device.

[0126] refer to Figure 5The diagram illustrates a structural schematic of an electronic device 500 suitable for implementing embodiments of the present disclosure. The electronic device 500 can be a terminal device or a server. The terminal device can include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, personal digital assistants (PDAs), tablet computers, portable media players (PMPs), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 5 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0127] like Figure 5 As shown, the electronic device 500 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of the electronic device 500. The processing unit 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0128] Typically, the following devices can be connected to I / O interface 505: input devices 506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 507 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 508 including, for example, magnetic tapes, hard disks, etc.; and communication devices 509. Communication device 509 allows electronic device 500 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5 An electronic device 500 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0129] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 509, or installed from a storage device 508, or installed from a ROM 502. When the computer program is executed by the processing device 501, it performs the functions defined in the methods of embodiments of this disclosure.

[0130] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0131] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0132] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods shown in the above embodiments.

[0133] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0134] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0135] The units described in the embodiments of this disclosure can be implemented in software or in hardware. The name of a unit does not necessarily limit the unit itself; for example, the first acquisition unit can also be described as "a unit that acquires at least two Internet Protocol addresses".

[0136] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0137] In a first aspect, according to one or more embodiments of the present disclosure, a speech recognition method is provided, comprising:

[0138] Determine the speech feature vector of the speech to be recognized;

[0139] Obtain the associated speech and associated text that are related to the speech to be recognized, determine the feature vectors of the associated speech and the associated text respectively, and obtain the associated speech feature vector and the associated text feature vector;

[0140] The speech feature vector is identified based on the associated speech feature vector and the associated text feature vector to obtain the identified text of the speech to be identified.

[0141] According to one or more embodiments of this disclosure, associated speech and associated text are obtained that are associated with the speech to be identified, and feature vectors of the associated speech and associated text are determined respectively to obtain associated speech feature vectors and associated text feature vectors, including:

[0142] Obtain associated speech and associated text with the speech to be recognized, and generate prompt information including associated speech and associated text;

[0143] The speech vectorization model is called to process the associated speech in the prompt information to obtain the associated speech feature vector; the text feature vector extraction algorithm is called to process the associated text in the prompt information to obtain the associated text feature vector.

[0144] According to one or more embodiments of this disclosure, speech feature vectors are identified based on associated speech feature vectors and associated text feature vectors to obtain the identified text of the speech to be identified, including:

[0145] The speech feature vector, associated speech feature vector, and associated text feature vector are input into the speech recognition model. The speech recognition model then uses the associated speech feature vector and associated text feature vector to identify the speech feature vector, thereby obtaining the recognized text of the speech to be recognized.

[0146] According to one or more embodiments of this disclosure, the number of associated speech includes at least one, and the number of associated text includes at least one;

[0147] Generate prompts that include associated speech and associated text, including:

[0148] The associated audio and text are written into the preset prompt message template in the order of their generation time to obtain the prompt message.

[0149] According to one or more embodiments of this disclosure, a speech feature vector, an associated speech feature vector, and an associated text feature vector are input into a speech recognition model, and the speech recognition model recognizes the speech feature vector based on the associated speech feature vector and the associated text feature vector, including:

[0150] Following the order in the prompts, the associated speech feature vectors corresponding to each associated speech and the associated text feature vectors corresponding to each associated text are concatenated, and the resulting concatenated vector is input into the speech recognition model.

[0151] The speech recognition model identifies the speech feature vectors based on the associated speech feature vectors and associated text feature vectors parsed from the concatenated vectors, and obtains the recognized text.

[0152] According to one or more embodiments of this disclosure, the method further includes fine-tuning a pre-trained natural language processing model and a speech vectorization model based on the following steps to obtain a speech recognition model and a speech vectorization model:

[0153] Construct multiple sample prompts, each of which includes sample-related speech and sample-related text arranged in chronological order, as well as the sample speech to be recognized and the sample recognition text corresponding to the sample speech to be recognized;

[0154] For each sample prompt, the sample-associated speech and the sample-to-be-recognized speech in the sample prompt are input into the speech vectorization model. The feature vectors of each sample-associated speech and each sample-to-be-recognized speech output by the speech vectorization model, along with the feature vector of the sample-associated text, are input into the natural language processing model.

[0155] Based on the loss between the output of the natural language processing model and the target output, the parameters of the natural language processing model and the speech vectorization model are adjusted, with the target output being the sample recognized text in the prompt information; the fine-tuned natural language processing model is used as the speech recognition model.

[0156] According to one or more embodiments of this disclosure, the associated speech and the speech to be recognized carry noise; and the speech feature vector, the associated speech feature vector, and the associated text feature vector are input into a speech recognition model, and the speech recognition model recognizes the speech feature vector based on the associated speech feature vector and the associated text feature vector to obtain the recognized text of the speech to be recognized, including:

[0157] Noise features are identified from associated speech feature vectors using a speech recognition model, and noise features in the speech feature vectors are then identified based on the identified noise features.

[0158] The speech recognition model filters out the noise features it detects, and then identifies the speech feature vectors that have been filtered out of noise to obtain the recognized text.

[0159] According to one or more embodiments of this disclosure, the noise includes the voice of other users besides the main speaker.

[0160] Secondly, according to one or more embodiments of this disclosure, a voice recognition device is provided, comprising:

[0161] The determining unit is used to determine the speech feature vector of the speech to be recognized;

[0162] The acquisition unit is used to acquire the associated speech and associated text associated with the speech to be recognized, determine the feature vectors of the associated speech and associated text respectively, and obtain the associated speech feature vector and associated text feature vector.

[0163] The recognition unit is used to recognize the speech feature vector based on the associated speech feature vector and the associated text feature vector to obtain the recognized text of the speech to be recognized.

[0164] According to one or more embodiments of this disclosure, the acquisition unit is further configured to:

[0165] Obtain associated speech and associated text with the speech to be recognized, and generate prompt information including associated speech and associated text;

[0166] The speech vectorization model is called to process the associated speech in the prompt information to obtain the associated speech feature vector; the text feature vector extraction algorithm is called to process the associated text in the prompt information to obtain the associated text feature vector.

[0167] According to one or more embodiments of this disclosure, the identification unit is further configured to:

[0168] The speech feature vector, associated speech feature vector, and associated text feature vector are input into the speech recognition model. The speech recognition model then uses the associated speech feature vector and associated text feature vector to identify the speech feature vector, thereby obtaining the recognized text of the speech to be recognized.

[0169] According to one or more embodiments of this disclosure, the number of associated speech items includes at least one, and the number of associated text items includes at least one; the acquisition unit is further configured to:

[0170] The associated audio and text are written into the preset prompt message template in the order of their generation time to obtain the prompt message.

[0171] According to one or more embodiments of this disclosure, the identification unit is further configured to:

[0172] Following the order in the prompts, the associated speech feature vectors corresponding to each associated speech and the associated text feature vectors corresponding to each associated text are concatenated, and the resulting concatenated vector is input into the speech recognition model.

[0173] The speech recognition model identifies the speech feature vectors based on the associated speech feature vectors and associated text feature vectors parsed from the concatenated vectors, and obtains the recognized text.

[0174] According to one or more embodiments of this disclosure, the device further includes a model training unit, the model training unit being used for:

[0175] Construct multiple sample prompts, each of which includes sample-related speech and sample-related text arranged in chronological order, as well as the sample speech to be recognized and the sample recognition text corresponding to the sample speech to be recognized;

[0176] For each sample prompt, the sample-associated speech and the sample-to-be-recognized speech in the sample prompt are input into the speech vectorization model. The feature vectors of each sample-associated speech and each sample-to-be-recognized speech output by the speech vectorization model, along with the feature vector of the sample-associated text, are input into the natural language processing model.

[0177] Based on the loss between the output of the natural language processing model and the target output, the parameters of the natural language processing model and the speech vectorization model are adjusted, with the target output being the sample recognized text in the prompt information; the fine-tuned natural language processing model is used as the speech recognition model.

[0178] According to one or more embodiments of this disclosure, the associated speech and the speech to be identified carry noise; the identification unit is further configured to:

[0179] Noise features are identified from associated speech feature vectors using a speech recognition model, and noise features in the speech feature vectors are then identified based on the identified noise features.

[0180] The speech recognition model filters out the noise features it detects, and then identifies the speech feature vectors that have been filtered out of noise to obtain the recognized text.

[0181] According to one or more embodiments of this disclosure, the noise includes the voice of other users besides the main speaker.

[0182] Thirdly, according to one or more embodiments of the present disclosure, an electronic device is provided, comprising: at least one processor and a memory;

[0183] The memory stores instructions that the computer executes;

[0184] At least one processor executes computer execution instructions stored in memory, causing at least one processor to perform the speech recognition method as described in the first aspect above and various possible designs of the first aspect.

[0185] Fourthly, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, which stores computer-executable instructions that, when executed by a processor, implement the speech recognition method described in the first aspect and various possible designs of the first aspect.

[0186] Fifthly, according to one or more embodiments of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the speech recognition method as described in the first aspect and various possible designs of the first aspect.

[0187] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0188] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0189] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A speech recognition method, comprising: Determine the speech feature vector of the speech to be recognized; Obtain the associated speech and associated text related to the speech to be identified, and determine the feature vectors of the associated speech and associated text respectively to obtain the associated speech feature vector and associated text feature vector; Based on the associated speech feature vector and the associated text feature vector, the speech feature vector is identified to obtain the identified text of the speech to be identified.

2. The method according to claim 1, characterized in that, The step of obtaining associated speech and associated text related to the speech to be identified, and determining the feature vectors of the associated speech and associated text respectively, to obtain the associated speech feature vector and associated text feature vector, includes: Obtain associated speech and associated text related to the speech to be identified, and generate prompt information including the associated speech and associated text; The speech vectorization model is called to process the associated speech in the prompt information to obtain the associated speech feature vector; the text feature vector extraction algorithm is called to process the associated text in the prompt information to obtain the associated text feature vector.

3. The method according to claim 2, characterized in that, The step of recognizing the speech feature vector based on the associated speech feature vector and the associated text feature vector to obtain the recognized text of the speech to be recognized includes: The speech feature vector, the associated speech feature vector, and the associated text feature vector are input into the speech recognition model. The speech recognition model then recognizes the speech feature vector based on the associated speech feature vector and the associated text feature vector to obtain the recognized text of the speech to be recognized.

4. The method according to claim 3, characterized in that, The number of associated speech items includes at least one, and the number of associated text items includes at least one. The generation of prompt information, including the associated speech and associated text, includes: The associated voice messages and associated text messages are written into a preset prompt message template in chronological order of their generation to obtain the prompt message.

5. The method according to claim 4, characterized in that, The step of inputting the speech feature vector, the associated speech feature vector, and the associated text feature vector into the speech recognition model, and having the speech recognition model recognize the speech feature vector based on the associated speech feature vector and the associated text feature vector, includes: Following the order in the prompt information, the associated speech feature vectors corresponding to each associated speech and the associated text feature vectors corresponding to each associated text are concatenated, and the resulting concatenated vector is input into the speech recognition model. The speech recognition model identifies the speech feature vectors based on the associated speech feature vectors and associated text feature vectors parsed from the concatenated vectors, thereby obtaining the recognized text.

6. The method according to any one of claims 3-5, characterized in that, The method further includes fine-tuning the pre-trained natural language processing model and the speech vectorization model based on the following steps to obtain the speech recognition model and the speech vectorization model: Multiple sample prompts are constructed, each of which includes sample-related speech and sample-related text arranged in chronological order, as well as the sample speech to be recognized and the sample recognition text corresponding to the sample speech to be recognized; For each sample prompt, the sample-associated speech and the sample-to-be-recognized speech in the sample prompt are input into the speech vectorization model. The feature vectors of each sample-associated speech and each sample-to-be-recognized speech output by the speech vectorization model, along with the feature vector of the sample-associated text, are input into the natural language processing model. Based on the loss between the output of the natural language processing model and the target output, the parameters of the natural language processing model and the speech vectorization model are adjusted, and the target output is the sample recognition text in the prompt information; the fine-tuned natural language processing model is used as the speech recognition model.

7. The method according to claim 3, characterized in that, The associated speech and the speech to be recognized carry noise; and the process of inputting the speech feature vector, the associated speech feature vector, and the associated text feature vector into a speech recognition model, and having the speech recognition model recognize the speech feature vector based on the associated speech feature vector and the associated text feature vector to obtain the recognized text of the speech to be recognized, includes: The speech recognition model is used to identify noise features from the associated speech feature vector, and the noise features in the speech feature vector are identified based on the identified noise features. The speech recognition model filters out the identified noise features and then identifies the speech feature vectors after filtering out the noise features to obtain the recognized text.

8. The method according to claim 7, characterized in that, The noise includes the voices of users other than the main speaker.

9. A voice recognition device, comprising: The determining unit is used to determine the speech feature vector of the speech to be recognized; The acquisition unit is used to acquire associated speech and associated text related to the speech to be identified, determine the feature vectors of the associated speech and the associated text respectively, and obtain the associated speech feature vector and the associated text feature vector. The recognition unit is used to recognize the speech feature vector based on the associated speech feature vector and the associated text feature vector to obtain the recognized text of the speech to be recognized.

10. An electronic device, characterized in that, include: Processor and memory; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the speech recognition method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by the processor, implement the speech recognition method as described in any one of claims 1 to 8.

12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the speech recognition method as described in any one of claims 1 to 8.

Images