A method for training a speech recognition model, a method and apparatus for speech recognition

Abstract

This invention discloses a training method for a speech recognition model, a speech recognition method, and an apparatus, relating to the field of computer technology. One specific embodiment of the method includes: constructing multimodal recognition data based on audio data and corresponding annotated text; determining a first recognition result based on the audio data in the multimodal data; identifying target keywords based on the first recognition result and a keyword list; constructing model prompt words based on the target keywords, the first recognition result, and the multimodal recognition data; and training the speech recognition model based on these prompt words. This embodiment can train the speech recognition model by constructing multimodal training data, greatly improving the accuracy of the model's recognition results.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method for training a speech recognition model, a speech recognition method, and an apparatus. Background Technology

[0002] Currently, with the development of internet technology, various terminals and clients can provide users with more convenient functions. For example, they can provide voice input services, allowing users to input information via voice. The system can automatically recognize the user's voice input data, convert it into text content, and enable the system to understand the content and perform the corresponding functions. Voice recognition plays a crucial role in human-computer interaction. Accurate recognition helps the system quickly generate accurate responses, improving user experience. The commonly used voice recognition method involves using a trained voice recognition model. Training this model uses audio data and corresponding text data. The audio data is input into the model, and the model is trained based on the output results and the text data. This training method has certain limitations in practical applications and is prone to auditory hallucinations, affecting the recognition results. Summary of the Invention

[0003] In view of this, embodiments of the present invention provide a method for training a speech recognition model, a method for speech recognition, and an apparatus for speech recognition, which can greatly improve the accuracy and efficiency of speech recognition results.

[0004] To achieve the above objectives, according to one aspect of the present invention, a method for training a speech recognition model is provided, comprising: Multimodal recognition data is constructed based on the audio data and the corresponding labeled text. A first recognition result is determined based on the audio data in the multimodal recognition data, and target keywords are determined based on the first recognition result and the keyword list; Based on the target keywords, the first recognition result, and the multimodal recognition data, model prompt words are constructed, and the speech recognition model is trained based on the model prompt words.

[0005] Optionally, the step of constructing multimodal recognition data based on audio data and the corresponding labeled text includes: Generate a text context and an audio context based on the audio data and the corresponding labeled text; Image data is generated based on the audio context; The multimodal recognition data is constructed based on the audio data, the corresponding labeled text, the text context, the audio context, and the image data.

[0006] Optionally, determining the first recognition result based on the audio data in the multimodal recognition data includes: encoding the audio data in the multimodal recognition data and decoding the audio encoding result to obtain the first recognition result; The step of determining the target keyword based on the first identification result and the keyword list includes: converting the first text in the first identification result into a pinyin sequence to obtain a conversion result; matching the conversion result with the pinyin sequence of each keyword in the keyword list, and determining the target keyword based on the matching result.

[0007] Optionally, the step of constructing model prompt words based on the target keyword, the first recognition result, and the multimodal recognition data includes: Based on the audio encoding result corresponding to the first recognition result, text recognition is performed to obtain the second recognition result; The image data in the multimodal recognition data is encoded, and text recognition is performed based on the image encoding results to obtain a third recognition result; Model prompt words are constructed based on the target keywords, the first recognition result, the second recognition result, the third recognition result, and the multimodal recognition data.

[0008] Optionally, training the speech recognition model based on the model prompts includes: The prompt word is input into the large model in the speech recognition model to obtain the initial recognition result output by the large model; The large model is trained based on the initial recognition results and the preset reward function, so as to determine the trained speech recognition model based on the trained large model.

[0009] Optionally, the preset reward function includes a keyword recall reward function for whether the target keyword is contained in the output of the large model and a pinyin sequence reward function for whether the pinyin sequence of the output of the large model is erroneous.

[0010] According to another aspect of the present invention, a speech recognition method is provided, comprising: The audio data to be recognized is input into the audio processing layer of the speech recognition model to obtain the initial recognition result output by the audio processing layer; The initial recognition result is input into the keyword recognition unit of the speech recognition model, so that the keyword recognition unit generates initial keywords based on the pinyin sequence of the initial recognition result and the pinyin sequence of the keywords in the keyword list; Based on the initial results and the initial keywords, prompt words for the audio data to be identified are constructed. The prompt words for the audio data to be identified are then input into a large model to obtain the recognition result of the audio data to be identified output by the large model.

[0011] According to another aspect of the present invention, a training apparatus for a speech recognition model is provided, comprising: The construction module is used to construct multimodal recognition data based on audio data and the corresponding labeled text. The determining module is used to determine a first recognition result based on the audio data in the multimodal recognition data, and to determine target keywords based on the first recognition result and the keyword list; The training module is used to construct model prompt words based on the target keywords, the first recognition result and the multimodal recognition data, and to train the speech recognition model based on the model prompt words.

[0012] According to another aspect of the present invention, a speech recognition apparatus is provided, comprising: The first input module is used to input the audio data to be recognized into the audio processing layer of the speech recognition model to obtain the initial recognition result output by the audio processing layer. The second input module is used to input the initial recognition result into the keyword recognition unit of the speech recognition model, so that the keyword recognition unit generates initial keywords based on the pinyin sequence of the initial recognition result and the pinyin sequence of keywords in the keyword list; The recognition module is used to construct prompt words for the audio data to be recognized based on the initial results and the initial keywords, and input the prompt words for the audio data to be recognized into the large model to obtain the recognition result of the audio data to be recognized output by the large model.

[0013] According to another aspect of the present invention, an electronic device is provided, comprising: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the training method for the speech recognition model or the speech recognition method provided in the embodiments of the present invention.

[0014] According to another aspect of the present invention, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements the training method for the speech recognition model and the speech recognition method provided in the embodiments of the present invention.

[0015] According to another aspect of the present invention, a computer program product is provided, including a computer program that, when executed by a processor, implements the training method for the speech recognition model and the speech recognition method provided in the embodiments of the present invention.

[0016] One embodiment of the above invention has the following advantages or beneficial effects: it can construct multimodal recognition data based on audio data, and train a speech recognition model using this multimodal recognition data, enabling the trained speech recognition model to output accurate recognition results. Specifically, the multimodal recognition data can be used as multimodal context data input into the model, resulting in higher accuracy of the model's output. Furthermore, during training, keyword enhancement recognition can be performed using a keyword list, further improving the accuracy of the model's output.

[0017] The further effects of the aforementioned unconventional alternative methods will be explained below in conjunction with specific implementation methods. Attached Figure Description

[0018] The accompanying drawings are provided to better understand the invention and are not intended to unduly limit the scope of the invention. Wherein: Figure 1 This is a schematic diagram illustrating the main steps of a speech recognition model training method according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the speech recognition model of the present invention; Figure 3 This is a schematic diagram illustrating the process of constructing multimodal recognition data. Figure 4 This is a schematic diagram of the main steps of a speech recognition method according to an embodiment of the present invention; Figure 5 This is a schematic diagram of the main modules of a speech recognition model training device according to an embodiment of the present invention; Figure 6 This is a schematic diagram of the main modules of a speech recognition device according to an embodiment of the present invention; Figure 7 This is an exemplary system architecture diagram in which embodiments of the present invention can be applied; Figure 8 This is a schematic diagram of the structure of a computer system suitable for implementing terminal devices or servers of the present invention. Detailed Implementation

[0019] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention, including various details to aid understanding. These details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0020] It should be noted that the technical solutions disclosed in this invention, regarding the collection, updating, analysis, processing, use, transmission, and storage of user personal information, all comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken to prevent unauthorized access to user personal information data and to safeguard user personal information security, network security, and national security.

[0021] It should be noted that the collection, use, storage, sharing and transfer of user personal information involved in the technical solution of the present invention all comply with the provisions of relevant laws and regulations, and require notification to users and obtaining their consent or authorization. When applicable, user personal information is subjected to de-identification and / or anonymization and / or encryption technical processing.

[0022] In current speech recognition scenarios, the learning capabilities of large models are typically required. A model capable of speech recognition is obtained through training. Model training primarily relies on audio data and corresponding manually labeled data, resulting in generally poor training performance. Furthermore, existing speech recognition methods mainly depend on keyword recognition. During recognition, the encoded results of the user's voice input data are compared with the word vectors of a keyword list using a cross-attention mechanism. The matching keywords are then determined based on the calculation results, and content recognition is performed based on the matched keywords. However, this recognition method requires calculating the word vectors of all keywords in a pre-set keyword list. With a long list, the calculation process increases significantly, slowing down the process and impacting the speech recognition speed. Additionally, as the list length increases, system resources may not be able to support the large-scale attention mechanism calculation process, affecting the generation of the final recognition result.

[0023] To fully consider the shortcomings of the above methods, this invention provides a training method for a speech recognition model and a speech recognition method. This method constructs multimodal data based on audio data and corresponding labeled data, and trains the speech recognition model using this multimodal data input to improve the model's recognition accuracy. Furthermore, when generating recognition results, unlike existing technologies that calculate word vectors from a keyword list for keyword matching, this invention uses a matching method based on the pinyin sequence of keywords. This significantly reduces the computational load required for matching, increases matching speed, and consequently improves the speed of generating subsequent recognition results.

[0024] Figure 1 This is a schematic diagram illustrating the main steps of a speech recognition model training method according to an embodiment of the present invention. Figure 1 As shown, the training method for the speech recognition model mainly includes steps S101 to S103.

[0025] Step S101: Construct multimodal recognition data based on audio data and the corresponding labeled text.

[0026] To obtain accurate recognition results, this invention employs a full-modal large model as the base model, which can support data input from multiple modalities to improve the accuracy of model training results. For example... Figure 2 The diagram shows the structure of the speech recognition model of this invention, which mainly includes an audio encoder, an audio-text adapter, a CTC decoder (Connectionist Temporal Classification), a keyword recognition unit (also known as a keyword filtering module), an image encoder, an image-text adapter, a cue word constructor, and an LLM (Large Language Model) structure. The speech recognition model based on this structure can support multimodal data input, enabling the model to be trained using multimodal data, thus allowing the trained model to output more accurate recognition results.

[0027] Specifically, training this speech recognition model first requires constructing multimodal recognition data to build the training dataset. This multimodal data includes text, image, and audio data. During construction, different modalities of data need to be generated based on the acquired audio files and corresponding manually annotated text. This includes generating text content from the audio files, further generating image content from the generated text, and expanding upon the manually annotated text to generate a text context containing linguistic information. The final multimodal recognition data is then constructed by combining these various data sources. For example, the audio file could be 001.wav, and the manually annotated text could be "What is pompholyx?". In addition, the acquired data may also include audio descriptive text, such as "A man around 30 years old is consulting a doctor about pompholyx; he sounds very worried."

[0028] In one embodiment, when constructing multimodal data, for a training dataset consisting of multiple audio files, different modal data can be constructed only for some audio files. That is, for one audio file, no other data can be generated, or only image content data can be generated, or image content data and text context can be generated. However, it is required that the final training dataset consisting of all audio files contains data of multiple modalities.

[0029] Step S102: Determine the first recognition result based on the audio data in the multimodal recognition data, and determine the target keywords based on the first recognition result and the keyword list.

[0030] In the training phase after generating multimodal recognition data, a speech recognition model is first used to generate a preliminary recognition result based on the audio data. This result is the initial recognition result of the model, specifically the output of the CTC decoder in the speech recognition model. The CTC encoder is trained separately to improve the accuracy of the CTC decoder's output. This preliminary recognition result is then matched with keywords in a keyword list to determine the matched target keywords. Subsequently, when generating the recognition text corresponding to the audio data, an accurate recognition result can be generated based on the determined target keywords. For example, if the preliminary recognition result is "What is a pompholyx pillow?", the matched target keyword is "pompholyx".

[0031] In one embodiment, the keyword matching operation can be implemented by the keyword recognition unit in the speech recognition model.

[0032] Step S103: Construct model prompt words based on target keywords, first recognition results and multimodal recognition data, and train speech recognition model based on model prompt words.

[0033] Before inputting the data into the large model to obtain the recognition result, prompt words need to be constructed based on the obtained data. This allows the large model to perform more accurate reasoning and thus obtain accurate recognition results. The construction process of prompt words mainly involves combining various data, combining the target keyword, the first recognition result, and multimodal recognition data into complete prompt word content. The prompt words are then input into the LLM model to obtain the output result of the LLM model. The speech recognition model is trained based on the output result and the prompt word content. In addition to the recognized text, the output result of the LLM model can also include the confidence score of the recognized text, so that the confidence score can be combined during the training process for a more accurate training process.

[0034] The speech recognition model training method provided by this invention can construct multimodal recognition data based on audio data, and train the speech recognition model using this multimodal recognition data, enabling the trained speech recognition model to output accurate recognition results. The multimodal recognition data can be used as multimodal context data input to the model, resulting in higher accuracy of the model's output. Furthermore, during training, keyword enhancement recognition can be performed using a keyword list, further improving the accuracy of the model's output.

[0035] In one embodiment, constructing multimodal recognition data based on audio data and the corresponding labeled text includes: generating text context and audio context based on audio data and the corresponding labeled text; generating image data based on the audio context; and constructing multimodal recognition data based on audio data, the corresponding labeled text, text context, audio context, and image data.

[0036] like Figure 3The diagram illustrates the construction process of multimodal recognition data. The initial training data includes audio data and corresponding labeled text. The labeled text can contain manually annotated speech-to-text and audio descriptions. The speech-to-text is the textual representation of the content in the audio data; while the audio description provides a multi-faceted description of the audio data, including the context in which the audio data was generated and the user's emotions reflected in it. For example, the audio data could be a file named 001.wav, the speech-to-text could be "What is pompholyx?", and the audio description could be "A man around 30 years old is consulting a doctor about pompholyx, and he sounds very worried." This data is input into the audio description model and the text language model, respectively, to obtain the audio description text (i.e., audio context) and the text context. The audio description text is obtained by the audio description model based on the audio description content and other data. This text can be the same as the audio description content; for example, when the audio description model determines that the audio description content is correct based on other data, it can directly output that content as the audio description text. For example, the text context could be: "speaker_a: I have small pimples on my hands. speaker_b: Xiaoning thinks you might have 'pompholyx,' which is a... I wish you a speedy recovery." Furthermore, the obtained audio description text can be input into the text-based image model, causing it to output image context, thus ensuring an image representation of the current audio data scene. Finally, by combining the audio file, labeled text, text context, audio description text, and image context, multimodal recognition data is obtained.

[0037] In one embodiment, a keyword list can also be added to the multimodal recognition data. The keywords in the list can be words that are difficult to recognize based on manually annotated transcribed text.

[0038] In one embodiment, to ensure that the model prioritizes following the audio content rather than relying entirely on the context when performing recognition, and to suppress the model's context illusion (the model directly copies the context content instead of outputting the transcribed content), the context content of 20% of the audio data samples in the training dataset is replaced with context unrelated to the current audio, or empty strings or blank images, so that the correct speech recognition result can still be output even when the context content is unrelated to the audio.

[0039] The speech recognition model training method provided by the present invention can generate multimodal recognition data based on audio data and corresponding labeled data, which helps to enrich the types of training data, improve the training effect, and thus improve the accuracy of the speech recognition model recognition results.

[0040] In one embodiment, determining a first recognition result based on audio data in multimodal recognition data includes: encoding the audio data in the multimodal recognition data and decoding the audio encoding result to obtain the first recognition result; determining a target keyword according to the first recognition result and a keyword list, including: converting the first text in the first recognition result into a pinyin sequence to obtain a conversion result; matching the conversion result with the pinyin sequence of each keyword in the keyword list, and determining the target keyword according to the matching result.

[0041] When generating the first recognition result, first input the audio data (i.e., the audio file) into the audio encoder of the speech recognition model to obtain audio encoding features, and decode the audio encoding features by using a CTC decoder to obtain the first recognition result. There may be cases of recognition errors for more difficult keywords in the first recognition result, such as homophones with different characters, and further processing is required to correct the errors. Specifically, perform a pinyin sequence conversion operation on the first text in the first recognition result to obtain a pinyin sequence, match the pinyin sequence with the pinyin sequences corresponding to each keyword in the keyword list, and use the matched keyword as the target keyword for subsequent addition of the target keyword to the prompt word to make the LLM output more accurate results. Among them, the keyword list stores the pinyin sequences corresponding to each keyword for matching. For example, the obtained pinyin sequence after conversion is shenmeshihanpaozhen, which matches the pinyin sequence hanpaozhen of the keyword "dyshidrosis", so dyshidrosis is determined as the target keyword for this first recognition result. Among them, when performing the matching, it is required that the pinyin sequence of the keyword is exactly the same as the pinyin sequence included in the first recognition result to recognize the two as a match.

[0042] In one embodiment, the pinyin sequence can also be replaced by a phonetic symbol sequence, and the effect of accurate recognition can also be achieved.

[0043] According to the training method of the speech recognition model provided by the embodiments of the present invention, a keyword enhancement recognition method can be introduced into the speech recognition model, and keyword recognition and matching are performed in the result recognized by the model based on audio data, which helps to ensure the accuracy of the model recognition result, enables it to still achieve accurate recognition of more difficult words in a more complex audio data scenario, and ensures the reliability of the recognition result. In addition, when performing keyword recognition, the recognition difficulty and the amount of data to be processed during recognition can be reduced by constructing the pinyin sequences of the recognition content and keywords, greatly improving the speed of keyword recognition, and at the same time ensuring the accuracy of recognition.

[0044] In one embodiment, constructing model prompt words based on target keywords, a first recognition result, and multimodal recognition data includes: performing text recognition based on the audio encoding result corresponding to the first recognition result to obtain a second recognition result; encoding image data in the multimodal recognition data and performing text recognition based on the image encoding result to obtain a third recognition result; and constructing model prompt words based on target keywords, the first recognition result, the second recognition result, the third recognition result, and the multimodal recognition data.

[0045] When constructing model prompts, the audio encoding results need to be recognized through an audio-text adapter to obtain an audio representation, i.e., the second recognition result. In addition to audio and text data, image data also needs to be processed. Specifically, image data is encoded using an image encoder, and the image encoding results are further recognized through an image-text adapter to obtain an image representation, i.e., the third recognition result. By combining the target keyword, image representation, audio representation, and the first recognition result, the prompts input to the LLM model are obtained. The Prompt constructor can be used to generate prompts by combining a prompt template with the aforementioned data.

[0046] In one embodiment, training a speech recognition model based on model prompts includes: inputting prompts into a large model within the speech recognition model to obtain an initial recognition result output by the large model; training the large model based on the initial recognition result and a preset reward function to determine the trained speech recognition model based on the trained large model.

[0047] After constructing the prompt words, they are input into the LLM structure to obtain the initial recognition results. These initial results are then used to train the model based on a preset reward function. During training, the GRPO (Generalized Reinforcement Learning with Policy Optimization) reinforcement learning strategy can be employed. The GRPO algorithm is a generalized reinforcement learning policy optimization method, an extension and generalization of traditional policy gradient algorithms (such as PPO). In this algorithm, the reward function plays a crucial role, directly influencing the direction of policy optimization and the final performance.

[0048] In one embodiment, the preset reward function includes a keyword recall reward function to determine whether the output of the large model contains the target keyword and a pinyin sequence reward function to determine whether the pinyin sequence of the output of the large model is erroneous.

[0049] To enhance keyword recognition capabilities while reducing the risk of recognition illusions in large models, two reward function schemes can be implemented. Reward function design schemes: Keyword Recall Reward Function: Check if the keyword appears in the model's output; if it does, the reward value is 1; otherwise, it is 0. Illusion Detection Reward Function: Extract the pinyin sequence (or phonetic symbol sequence) from the model's output and calculate whether there are replacement errors exceeding 5 characters in length. If such errors occur, the reward value is 0; otherwise, it is 1.

[0050] The speech recognition model training method provided by the embodiments of the present invention can enrich the amount and type of training data by constructing multimodal data, which helps to improve the training effect of the speech recognition model. Furthermore, during the model training process, a keyword recognition method is introduced to improve the accuracy of the model output results. Specifically, the keyword recognition method can accurately identify difficult words in the audio data, ensuring the accuracy of the model's recognition results.

[0051] Figure 4 A schematic diagram illustrating the main steps of the speech recognition method provided in an embodiment of the present invention. Figure 4 As shown, the speech recognition method mainly includes steps S401 to S403.

[0052] Step S401: Input the audio data to be recognized into the audio processing layer of the speech recognition model to obtain the initial recognition result output by the audio processing layer.

[0053] When using a trained speech recognition model, the first step is to input the audio data to be recognized into the model. The audio data is first fed into the audio processing layer of the speech recognition model, which then outputs the initial recognition result—the result before keyword enhancement—which may contain inaccurate words. The audio processing layer includes an audio encoder and a CTC decoder.

[0054] In one embodiment, the audio processing layer further includes an audio-text adapter for recognizing the audio encoding result output by the audio encoder to obtain an audio representation. This audio representation can be added to subsequently constructed cue words.

[0055] Step S402: Input the initial recognition result into the keyword recognition unit of the speech recognition model so that the keyword recognition unit generates initial keywords based on the pinyin sequence of the initial recognition result and the pinyin sequence of the keywords in the keyword list.

[0056] After the audio processing layer outputs the initial recognition result, this result is input into the keyword recognition unit for keyword enhancement and recognition processing. Specifically, this unit performs keyword recognition based on the keyword list of the initial recognition result. During recognition, the initial recognition result needs to be converted into pinyin to obtain a pinyin sequence, and then the pinyin sequence is matched with the pinyin sequence corresponding to each keyword in the keyword list to obtain the matched initial keyword.

[0057] Step S403: Construct prompt words for the audio data to be recognized based on the initial results and initial keywords, and input the prompt words for the audio data to be recognized into the large model to obtain the recognition result of the audio data to be recognized output by the large model.

[0058] The initial keywords and initial results are combined to obtain prompt words for the audio data. These prompt words are then input into the LLM (Language Probe), which performs inference based on the prompt words and outputs the speech recognition results.

[0059] In one embodiment, during speech recognition, in addition to audio data, corresponding image data can also be acquired. This image data can be an image of the scene where the audio data was collected, or an image related to that scene. For example, in a meeting scenario, the image data could be an overall image of the meeting room, or an image of a PowerPoint presentation (PowerPoint slides) that the user is presenting. After processing by an image encoder and an image-text adapter, the image data yields an image representation. This representation is then added to the prompts in the input LLM to obtain more accurate recognition results.

[0060] The speech recognition method provided by the embodiments of the present invention can introduce a keyword recognition unit during the recognition process, and use the keyword recognition unit to accurately match difficult words in the audio content. The matched keywords are then used to calibrate the recognition results, ensuring their reliability. Simultaneously, the speech recognition model supports multimodal data input; using multimodal data for speech recognition can significantly improve the accuracy of the recognition results.

[0061] Figure 5 This is a schematic diagram of the main modules of the training device for the speech recognition model provided in an embodiment of the present invention. Figure 5 As shown, the training device 500 for the speech recognition model mainly includes a construction module 501, a determination module 502, and a training module 503.

[0062] Module 501 is used to construct multimodal recognition data based on audio data and the corresponding labeled text. The determining module 502 is used to determine a first recognition result based on audio data in the multimodal recognition data, and to determine target keywords based on the first recognition result and the keyword list; Training module 503 is used to construct model prompt words based on target keywords, first recognition results and multimodal recognition data, and to train the speech recognition model based on the model prompt words.

[0063] The speech recognition model training apparatus provided in this embodiment of the invention can construct multimodal recognition data based on audio data, and train the speech recognition model using the multimodal recognition data, enabling the trained speech recognition model to output accurate recognition results. The multimodal recognition data can be used as multimodal context data input to the model, resulting in higher accuracy of the model's output results. Furthermore, during training, keyword enhancement recognition can be performed by combining a keyword list, further improving the accuracy of the model's output results.

[0064] In one embodiment, the construction module 501 is further configured to: generate text context and audio context based on audio data and the labeled text corresponding to the audio data; generate image data based on the audio context; and construct multimodal recognition data based on the audio data, the labeled text corresponding to the audio data, the text context, the audio context, and the image data.

[0065] In one embodiment, the determining module 502 is further configured to: encode audio data based on multimodal recognition data and decode the audio encoding result to obtain a first recognition result; convert the first text in the first recognition result into a pinyin sequence to obtain a conversion result; match the conversion result with the pinyin sequence of each keyword in the keyword list, and determine the target keyword based on the matching result.

[0066] In one embodiment, the training module 503 is further configured to: perform text recognition based on the audio encoding result corresponding to the first recognition result to obtain a second recognition result; encode the image data in the multimodal recognition data and perform text recognition based on the image encoding result to obtain a third recognition result; and construct model prompt words based on the target keyword, the first recognition result, the second recognition result, the third recognition result and the multimodal recognition data.

[0067] In one embodiment, the training module 503 is further configured to: input the prompt word into the large model in the speech recognition model to obtain the initial recognition result output by the large model; train the large model based on the initial recognition result and a preset reward function, so as to determine the trained speech recognition model according to the trained large model.

[0068] In one embodiment, the preset reward function includes a keyword recall reward function to determine whether the output of the large model contains the target keyword and a pinyin sequence reward function to determine whether the pinyin sequence of the output of the large model is erroneous.

[0069] The speech recognition model training apparatus provided in this embodiment of the invention can enrich the amount and type of training data by constructing multimodal data, which helps to improve the training effect of the speech recognition model. Furthermore, during the model training process, a keyword recognition method is introduced to improve the accuracy of the model's output results. Specifically, the keyword recognition method can accurately identify difficult words in the audio data, ensuring the accuracy of the model's recognition results.

[0070] Figure 6 This is a schematic diagram of the main modules of a speech recognition device provided in an embodiment of the present invention. Figure 6 As shown, the speech recognition device 600 mainly includes a first input module 601, a second input module 602, and a recognition module 603.

[0071] The first input module 601 is used to input the audio data to be recognized into the audio processing layer of the speech recognition model to obtain the initial recognition result output by the audio processing layer. The second input module 602 is used to input the initial recognition result into the keyword recognition unit of the speech recognition model, so that the keyword recognition unit generates initial keywords based on the pinyin sequence of the initial recognition result and the pinyin sequence of the keywords in the keyword list; The recognition module 603 is used to construct prompt words for the audio data to be recognized based on the initial results and initial keywords, and input the prompt words of the audio data to be recognized into the large model to obtain the recognition result of the audio data to be recognized output by the large model.

[0072] The speech recognition apparatus provided in this embodiment of the invention can introduce a keyword recognition unit during the recognition process, and use the keyword recognition unit to accurately match difficult words in the audio content. The matched keywords are then used to calibrate the recognition results, ensuring their reliability. Simultaneously, the speech recognition model supports multimodal data input; using multimodal data for speech recognition can significantly improve the accuracy of the recognition results.

[0073] Figure 7 An exemplary system architecture 700 is shown, which can be used to train a speech recognition model, a speech recognition method, or a speech recognition model training apparatus, or a speech recognition apparatus according to embodiments of the present invention.

[0074] like Figure 7As shown, system architecture 700 may include terminal devices 701, 702, and 703, a network 704, and a server 705. Network 704 serves as the medium for providing communication links between terminal devices 701, 702, and 703 and server 705. Network 704 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0075] Users can use terminal devices 701, 702, and 703 to interact with server 705 via network 704 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 701, 702, and 703, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0076] Terminal devices 701, 702, and 703 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0077] Server 705 can be a server that provides various services, such as a backend management server that supports shopping websites browsed by users using terminal devices 701, 702, and 703 (for example only). The backend management server can analyze and process data such as received speech recognition requests, and feed back the processing results (such as recognition results - for example only) to the terminal devices.

[0078] It should be noted that the speech recognition model training method or speech recognition method provided in the embodiments of the present invention is generally executed by the server 705, and correspondingly, the speech recognition model training device or speech recognition device is generally set in the server 705.

[0079] It should be understood that Figure 7 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0080] The following is for reference. Figure 8 It shows a schematic diagram of the structure of a computer system 800 suitable for implementing terminal devices or servers of the present invention. Figure 8 The terminal device or server shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of the present invention.

[0081] like Figure 8As shown, the computer system 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 802 or programs loaded from storage section 808 into random access memory (RAM) 803. The RAM 803 also stores various programs and data required for the operation of the system 800. The CPU 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0082] The following components are connected to I / O interface 805: an input section 806 including a keyboard, mouse, etc.; an output section 807 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to I / O interface 805 as needed. A removable medium 811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 810 as needed so that computer programs read from it can be installed into storage section 808 as needed.

[0083] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 809, and / or installed from removable medium 811. When the computer program is executed by central processing unit (CPU) 801, it performs the functions defined above in the system of this invention.

[0084] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0085] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0086] The units or modules described in the embodiments of the present invention can be implemented in software or hardware. The described units or modules can also be housed in a processor; for example, a processor may be described as including a construction module, a determination module, and a training module, or including a first input module, a second input module, and a recognition module. The names of these units or modules do not necessarily limit the specific unit or module itself; for example, a construction module may also be described as "a module for constructing multimodal recognition data based on audio data and corresponding labeled text."

[0087] In another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs, which, when executed by the device, cause the device to include: Multimodal recognition data is constructed based on audio data and the corresponding labeled text. The first recognition result is determined based on the audio data in the multimodal recognition data, and the target keywords are determined based on the first recognition result and the keyword list; Model prompts are constructed based on target keywords, first recognition results, and multimodal recognition data, and the speech recognition model is trained based on the model prompts.

[0088] or, The audio data to be recognized is input into the audio processing layer of the speech recognition model to obtain the initial recognition result output by the audio processing layer. The initial recognition results are input into the keyword recognition unit of the speech recognition model, so that the keyword recognition unit generates initial keywords based on the pinyin sequence of the initial recognition results and the pinyin sequence of the keywords in the keyword list; Based on the initial results and initial keywords, prompt words for the audio data to be identified are constructed. These prompt words are then input into a large model to obtain the recognition results of the audio data to be identified from the output of the large model.

[0089] According to the technical solution of this invention, multimodal recognition data can be constructed based on audio data, and a speech recognition model can be trained using this multimodal recognition data, enabling the trained speech recognition model to output accurate recognition results. The multimodal recognition data can be used as multimodal context data input to the model, resulting in higher accuracy of the model's output. Furthermore, during training, keyword enhancement recognition can be performed using a keyword list, further improving the accuracy of the model's output.

[0090] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for training a speech recognition model, characterized in that, include: Multimodal recognition data is constructed based on the audio data and the corresponding labeled text. A first recognition result is determined based on the audio data in the multimodal recognition data, and target keywords are determined based on the first recognition result and the keyword list; Based on the target keywords, the first recognition result, and the multimodal recognition data, model prompt words are constructed, and the speech recognition model is trained based on the model prompt words.

2. The method according to claim 1, characterized in that, The construction of multimodal recognition data based on audio data and the corresponding labeled text includes: Generate a text context and an audio context based on the audio data and the corresponding labeled text; Image data is generated based on the audio context; The multimodal recognition data is constructed based on the audio data, the corresponding labeled text, the text context, the audio context, and the image data.

3. The method according to claim 2, characterized in that, Determining the first recognition result based on the audio data in the multimodal recognition data includes: encoding the audio data in the multimodal recognition data and decoding the audio encoding result to obtain the first recognition result; The step of determining the target keyword based on the first identification result and the keyword list includes: converting the first text in the first identification result into a pinyin sequence to obtain a conversion result; matching the conversion result with the pinyin sequence of each keyword in the keyword list, and determining the target keyword based on the matching result.

4. The method according to claim 3, characterized in that, The construction of model prompt words based on the target keyword, the first recognition result, and the multimodal recognition data includes: Based on the audio encoding result corresponding to the first recognition result, text recognition is performed to obtain the second recognition result; The image data in the multimodal recognition data is encoded, and text recognition is performed based on the image encoding results to obtain a third recognition result; Model prompt words are constructed based on the target keywords, the first recognition result, the second recognition result, the third recognition result, and the multimodal recognition data.

5. The method according to any one of claims 1-4, characterized in that, The training of the speech recognition model based on the model prompts includes: The prompt word is input into the large model in the speech recognition model to obtain the initial recognition result output by the large model; The large model is trained based on the initial recognition results and the preset reward function, so as to determine the trained speech recognition model based on the trained large model.

6. The method according to claim 5, characterized in that, The preset reward function includes a keyword recall reward function that determines whether the target keyword is included in the output of the large model, and a pinyin sequence reward function that determines whether the pinyin sequence in the output of the large model is erroneous.

7. A method for speech recognition based on the speech recognition model according to any one of claims 1-6, characterized in that, include: The audio data to be recognized is input into the audio processing layer of the speech recognition model to obtain the initial recognition result output by the audio processing layer. The initial recognition result is input into the keyword recognition unit of the speech recognition model, so that the keyword recognition unit generates initial keywords based on the pinyin sequence of the initial recognition result and the pinyin sequence of the keywords in the keyword list; Based on the initial results and the initial keywords, prompt words for the audio data to be identified are constructed. The prompt words for the audio data to be identified are then input into a large model to obtain the recognition result of the audio data to be identified output by the large model.

8. A training device for a speech recognition model, characterized in that, include: The construction module is used to construct multimodal recognition data based on audio data and the corresponding labeled text. The determining module is used to determine a first recognition result based on the audio data in the multimodal recognition data, and to determine target keywords based on the first recognition result and the keyword list; The training module is used to construct model prompt words based on the target keywords, the first recognition result and the multimodal recognition data, and to train the speech recognition model based on the model prompt words.

9. A speech recognition device, characterized in that, include: The first input module is used to input the audio data to be recognized into the audio processing layer of the speech recognition model to obtain the initial recognition result output by the audio processing layer. The second input module is used to input the initial recognition result into the keyword recognition unit of the speech recognition model, so that the keyword recognition unit generates initial keywords based on the pinyin sequence of the initial recognition result and the pinyin sequence of keywords in the keyword list; The recognition module is used to construct prompt words for the audio data to be recognized based on the initial results and the initial keywords, and input the prompt words for the audio data to be recognized into the large model to obtain the recognition result of the audio data to be recognized output by the large model.

10. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-7.

11. A computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-7.

12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-7.

Images