A speech recognition method and related device

Abstract

This application discloses a speech recognition method and related devices, relating to the field of speech recognition technology. The speech recognition method comprises: obtaining target speech data; performing preliminary speech recognition on the target speech data to obtain word-level acoustic information of the target speech data and candidate speech recognition results; projecting the word-level acoustic information of the target speech data into the representation space of a large language model to obtain target acoustic features of the target speech data; and utilizing the large language model, supplemented by the candidate speech recognition results of the target speech data, processing the target acoustic features of the target speech data into a text sequence to obtain a speech recognition result of the target speech data. The speech recognition method disclosed in this application has good recognition effect.

Description

Technical Field

[0001] The present application relates to the field of speech recognition technology, and in particular to a speech recognition method and related devices. Background Art

[0002] Speech recognition technology converts speech into text. Language models play a crucial role in speech recognition. They provide linguistic constraints based on contextual semantics, improve recognition accuracy, and enhance the ability to process uncommon words and proper nouns. Language models, in conjunction with acoustic models, significantly enhance the quality and efficiency of speech recognition.

[0003] Current speech recognition methods include speech recognition methods based on large language models. The speech recognition methods based on large language models first use the acoustic model to obtain frame-level acoustic information for the speech to be recognized, then project the frame-level acoustic information into the representation space of the large language model, and finally input the features obtained in the representation space of the large language model into the large language model to obtain the speech recognition results of the speech to be recognized.

[0004] Although the above-mentioned speech recognition method based on the large language model can realize speech recognition, the recognition effect is not good. Summary of the Invention

[0005] In view of this, the present application provides a speech recognition method and related devices to solve the problem of poor recognition effect of existing speech recognition methods based on large language models. The technical solution is as follows:

[0006] The first aspect of the present application provides a speech recognition method, comprising:

[0007] Obtain target voice data;

[0008] Acquiring word-level acoustic information of the target speech data and candidate speech recognition results by performing preliminary speech recognition on the target speech data;

[0009] Projecting the word-level acoustic information of the target speech data into the representation space of the large language model to obtain target acoustic features of the target speech data;

[0010] The target acoustic features of the target speech data are processed into a text sequence using the large language model and supplemented by the candidate speech recognition results of the target speech data to obtain the speech recognition results of the target speech data.

[0011] In one possible implementation, obtaining word-level acoustic information of the target speech data includes:

[0012] Acquiring frame-level acoustic information for the target speech data;

[0013] Determining whether each speech frame of the target speech data is a decoding position, determining speech frames belonging to the same word;

[0014] The acoustic information of the speech frames belonging to the same word is fused to obtain word-level acoustic information of the target speech data.

[0015] In a possible implementation, performing preliminary speech recognition on the target speech data to obtain word-level acoustic information of the target speech data and candidate speech recognition results includes:

[0016] The target speech data is subjected to preliminary speech recognition using a pre-trained speech recognition model based on a monotonic block attention module to obtain word-level acoustic information of the target speech data and candidate speech recognition results.

[0017] In one possible implementation, the method of using the large language model, supplemented by the candidate speech recognition results of the target speech data, processing the target acoustic features of the target speech data into a text sequence, and obtaining the speech recognition results of the target speech data includes:

[0018] Integrating the target acoustic features of the target speech data with the candidate speech recognition results of the target speech data using a preconfigured Prompt instruction template to obtain an integrated result;

[0019] The integration result is input into the large language model to obtain a speech recognition result of the target speech data output by the large language model.

[0020] In one possible implementation, the integration of the target acoustic features of the target speech data with the candidate speech recognition results of the target speech data using a preconfigured Prompt instruction template to obtain an integration result includes:

[0021] Converting the candidate speech recognition results of the target speech data into features suitable for processing by the large language model to obtain features corresponding to the candidate speech recognition results of the target speech data;

[0022] Converting a preconfigured Prompt instruction template into features suitable for processing by the large language model to obtain features corresponding to the Prompt instruction template, wherein the Prompt instruction template includes an acoustic information slot and a recognition auxiliary information slot, and the Prompt instruction template is used to instruct the large language model to provide a speech recognition result based on the information in the acoustic information slot and the information in the recognition auxiliary information slot;

[0023] According to the positions of the acoustic information slot and the recognition auxiliary information slot, the target acoustic features of the target voice data and the features corresponding to the candidate voice recognition results of the target voice data are inserted into the features corresponding to the Prompt instruction template to obtain the combined features.

[0024] In a possible implementation, the Prompt instruction template further includes: task description information;

[0025] The task description information is used to prompt the large language model to pay attention to the information in the acoustic information slot and the recognition auxiliary information slot, and to prompt the large language model how to process the information in the acoustic information slot and the recognition auxiliary information slot.

[0026] In one possible implementation, the candidate speech recognition results of the target speech data are the best N candidate speech recognition results, where N is an integer greater than 1;

[0027] The identification auxiliary information slot includes a first auxiliary information slot and a second auxiliary information slot;

[0028] The step of inserting the target acoustic features of the target speech data and the features corresponding to the candidate speech recognition results of the target speech data into the features corresponding to the Prompt instruction template according to the positions of the acoustic information slot and the recognition auxiliary information slot to obtain the combined features includes: inserting the target acoustic features of the target speech data into the features corresponding to the Prompt instruction template according to the positions of the acoustic information slot;

[0029] Inserting a feature corresponding to the best candidate speech recognition result among the N candidate speech recognition results into the feature corresponding to the Prompt instruction template according to the position of the first auxiliary information slot;

[0030] According to the position of the second auxiliary information slot, features corresponding to other candidate speech recognition results among the N candidate speech recognition results are inserted into the features corresponding to the Prompt instruction template.

[0031] In one possible implementation, projecting the word-level acoustic information of the target speech data into the representation space of the large language model to obtain target acoustic features of the target speech data includes:

[0032] Using a pre-trained connection module, projecting the word-level acoustic information of the target speech data into the representation space of the large language model, obtaining acoustic features adapted to the large language model as target acoustic features of the target speech data;

[0033] The connection module is trained on a speech recognition task based on a large language model using training speech data annotated with text.

[0034] In one possible implementation, the large language model is obtained by training a pre-trained large language model on a speech recognition task using training speech data annotated with text;

[0035] The training process of the connection module and the large language model includes:

[0036] The parameters of the large language model are fixed, and the connection module is trained on the speech recognition task using text-annotated training speech data.

[0037] The parameters of the connection module are fixed, and the large language model is trained on the speech recognition task using text-annotated training speech data.

[0038] Using training speech data annotated with text, the connection module and the large language model are trained simultaneously on the speech recognition task.

[0039] A second aspect of the present application provides a speech recognition device, comprising: a speech data acquisition module, a speech recognition information acquisition module, an acoustic information processing module, and a speech recognition result acquisition module;

[0040] A voice data acquisition module is used to acquire target voice data;

[0041] A speech recognition information acquisition module, configured to obtain word-level acoustic information of the target speech data and candidate speech recognition results by performing preliminary speech recognition on the target speech data;

[0042] The acoustic information processing module is used to project the word-level acoustic information of the target speech data into the representation space of the large language model to obtain the target acoustic features of the target speech data;

[0043] The speech recognition result acquisition module is used to use the large language model, supplemented by the candidate speech recognition results of the target speech data, to process the target acoustic features of the target speech data into a text sequence to obtain the speech recognition results of the target speech data.

[0044] A third aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0045] The memory is used to store computer programs;

[0046] The processor is used to execute the computer program so that the electronic device can implement the steps of any one of the above-mentioned speech recognition methods.

[0047] In a fourth aspect, the present application provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the steps of any one of the above-mentioned speech recognition methods.

[0048] In a fifth aspect of the present application, a computer program product includes computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the steps of any one of the above-mentioned speech recognition methods.

[0049] By means of the above-mentioned technical solution, the speech recognition method provided by the present application, after obtaining the target speech data, first performs preliminary speech recognition on the target speech data to obtain the word-level acoustic information of the target speech data and the candidate speech recognition results, then projects the word-level acoustic information of the target speech data into the representation space of the large language model to obtain the target acoustic features, and finally uses the large language model, supplemented by the candidate speech recognition results of the target speech data, to process the target acoustic features into a text sequence to obtain the speech recognition results of the target speech data. Considering that the modeling granularity of the large language model is word, the speech recognition method provided in this application obtains word-level acoustic information for the target speech data. In order to enable the large language model to understand the information of the acoustic modality well and thus give accurate speech recognition results, the speech recognition method provided in this application further projects the word-level acoustic information into the representation space of the large language model after obtaining the word-level acoustic information to adapt it to the large language model. Taking into account the problems of hallucinations and unstoppable decoding in the large language model, the speech recognition method provided in this application introduces candidate speech recognition results to assist the large language model in limiting the decoding length, so as to effectively solve the problems of hallucinations and unstoppable decoding in the large language model. In summary, the speech recognition method provided in this application has good speech recognition effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0051] Figure 1 A schematic diagram of a system architecture involved in this application;

[0052] Figure 2 A schematic diagram of the hardware structure of a terminal provided in an embodiment of the present application;

[0053] Figure 3 A schematic diagram of the hardware structure of a server provided in an embodiment of the present application;

[0054] Figure 4 A flowchart of a speech recognition method provided in an embodiment of the present application;

[0055] Figure 5 A flowchart of an embodiment of the present application for utilizing a large language model, supplemented by candidate speech recognition results of target speech data, processing target acoustic features of the target speech data into a text sequence, and obtaining a speech recognition result of the target speech data;

[0056] Figure 6 A schematic diagram of an embodiment of the present application for implementing speech recognition using a speech recognition model based on a monotonic block attention module, a connection module, a large language model, and other modules;

[0057] Figure 7 A schematic diagram of the process of training the connection module and the large language model provided in the embodiment of the present application;

[0058] Figure 8 A schematic diagram of the structure of the speech recognition device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0059] The following describes the embodiments of the present application in conjunction with the accompanying drawings. The terms used in the implementation methods of the present application are only used to explain the specific embodiments of the present application and are not intended to limit the present application.

[0060] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0061] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, and this is merely a way of distinguishing the objects of the same attributes when describing them in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0062] In one possible implementation, Figure 1As shown, the system architecture involved in this application may include a terminal 101 and a server 102, and the terminal 101 may interact with the server 102 via a network (wired network or wireless network). The server 102 may include one or more servers ( Figure 1 (The example includes a server.) Terminal 101 can obtain speech data to be recognized and transmit the speech data to be recognized to server 102 via a network. Server 102 performs speech recognition on the speech data using the speech recognition method provided in the application and transmits the speech recognition result to terminal 101 via a network.

[0063] In another possible implementation, the system architecture involved in the present application may include a terminal. The terminal has strong data processing capabilities. The terminal can obtain speech data to be recognized and perform speech recognition on the speech data using the speech recognition method provided in the application.

[0064] Next, the product form of the above-mentioned terminal will be described.

[0065] The above-mentioned terminal can be a mobile phone, a tablet computer, a wearable device, an in-vehicle device, a robot, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc., and the embodiments of the present application do not impose any restrictions on this.

[0066] Figure 2 A schematic diagram of an optional hardware structure of a terminal is shown.

[0067] refer to Figure 2 As shown, the terminal may include a radio frequency unit 210, a memory 220, an input unit 230, a display unit 240, a camera 250 (optional), an audio circuit 260 (optional), a speaker 261 (optional), a microphone 262 (optional), an earphone jack 263 (optional), a processor 270, an external interface 280, a power supply 290 and other components. Those skilled in the art will understand that Figure 2 This is merely an example of a terminal and does not constitute a limitation on the terminal. The terminal may include more or fewer components than shown in the figure, or a combination of certain components, or different components.

[0068] The input unit 230 can be used to receive input digital or character information and generate key signal input related to the user settings and function control of the terminal. Specifically, the input unit 230 may include a touch screen 231 (optional) and / or other input devices 232. The touch screen 231 can collect user touch operations on or near it (for example, operations performed on or near the touch screen using a finger, joint, stylus, or any other suitable object) and drive corresponding connected devices according to a pre-set program. The touch screen can detect user touch actions on the touch screen, convert the touch actions into touch signals and transmit them to the processor 270, and can receive and execute commands sent by the processor 270; the touch signals include at least touch point coordinate information. The touch screen 231 can provide an input interface and an output interface between the terminal and the user. In addition, touch screens can be implemented using various types, such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch screen 231, the input unit 230 may also include other input devices. Specifically, the other input devices 232 may include, but are not limited to, one or more of a physical keyboard, function keys (such as a volume control button, a switch button, etc.), a trackball, a mouse, a joystick, and the like.

[0069] The display unit 240 may be used to display information input by a user or information provided to a user, various menus of the terminal, an interactive interface, file display, and / or playback of any multimedia file.

[0070] Memory 220 can be used to store instructions and data. Memory 220 primarily includes an instruction storage area and a data storage area. The data storage area can store various data, such as multimedia files and text. The instruction storage area can store software units such as the operating system, applications, and instructions required for at least one function, or subsets or extensions thereof. It may also include non-volatile random access memory (RAM). It provides processor 270 with management functions for the hardware, software, and data resources within the computing and processing device, supporting control software and applications. It is also used to store multimedia files and running programs and applications.

[0071] The processor 270 is the terminal's control center, connecting all components of the terminal using various interfaces and circuits. By executing instructions stored in memory 220 and accessing data stored in memory 220, it performs various terminal functions and processes data, thereby providing overall control of the terminal. Optionally, the processor 270 may include one or more processing units. Preferably, the processor 270 integrates an application processor and a modem processor, with the application processor primarily processing the operating system, user interface, and application programs, while the modem processor primarily handles wireless communications. It is understood that the modem processor may not be integrated into the processor 270. In some embodiments, the processor and memory may be implemented on a single chip; in other embodiments, they may be implemented on separate chips. The processor 270 may also generate corresponding operational control signals and send them to corresponding components of the computing and processing device. It may also read and process data in the software, particularly the data and programs in memory 220, to enable the various functional modules therein to perform their corresponding functions, thereby controlling the corresponding components to operate as instructed.

[0072] Among them, the memory 220 can be used to store software codes related to the speech recognition method, the processor 270 can execute the software codes in the memory 220, and can also schedule other units (such as the above-mentioned input unit 230 and display unit 240) to implement corresponding functions.

[0073] The RF unit 210 (optional) can be used to send and receive information or receive and send signals during a call. For example, it receives downlink information from the base station and passes it to the processor 270 for processing; in addition, it sends uplink data to the base station. Typically, the RF unit 210 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low-noise amplifier (LNA), a duplexer, etc. Furthermore, the RF unit 210 can communicate with network devices and other devices via wireless communication. This wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Messaging Service (SMS), etc.

[0074] In the embodiment of the present application, the radio frequency unit 210 can send data to other devices and can also receive data sent by other devices. It should be understood that the radio frequency unit 210 is optional and can be replaced by other communication interfaces, such as a network port.

[0075] The terminal also includes a power supply 290 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 270 through a power management system, thereby implementing functions such as charging, discharging, and power consumption management through the power management system.

[0076] The terminal further includes an external interface 280 , which may be a standard Micro USB interface or a multi-pin connector, and may be used to connect the terminal to other devices for communication, or to connect a charger for charging the terminal.

[0077] Although not shown, the terminal may further include a flashlight, a wireless fidelity (WiFi) module, a Bluetooth module, sensors with different functions, etc., which will not be described in detail here.

[0078] Next, the product form of the above server will be described.

[0079] Figure 3 A structural diagram of the above server is provided, such as Figure 3 As shown, the server may include a bus 301 , a processor 302 , a communication interface 303 , and a memory 304 . The processor 302 , the memory 304 , and the communication interface 303 communicate with each other via the bus 301 .

[0080] The bus 301 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0081] The processor 302 may be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0082] The memory 304 may include volatile memory, such as random access memory (RAM). The memory 304 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0083] The memory 304 may be used to store software codes related to the speech recognition method. The processor 302 may call the software codes stored in the memory 304 or schedule other units to implement corresponding functions.

[0084] The processors in the above-mentioned terminals and servers (such as processor 270 and processor 302) can be hardware circuits (such as application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), general-purpose processors, digital signal processors (DSP), microprocessors or microcontrollers, etc.), or a combination of these hardware circuits. For example, the processor can be a hardware system with an instruction execution function, such as a CPU, DSP, etc., or a hardware system without an instruction execution function, such as an ASIC, FPGA, etc., or a combination of the above-mentioned hardware systems without an instruction execution function and hardware systems with an instruction execution function.

[0085] The inventors of this case studied existing speech recognition methods based on large language models and found that: in existing speech recognition methods based on large language models, the acoustic information input to the large language model is frame-level acoustic information, which is inconsistent with the granularity of the modeling units of the large language model. The inconsistency between the granularity of the acoustic information and the granularity of the modeling units of the large language model will result in the large language model being unable to better understand the acoustic information and, in turn, unable to provide more accurate recognition results. In addition, existing speech recognition methods based on large language models also have problems with unstoppable decoding and large language model hallucinations. In response to the problems of existing speech recognition methods based on large language models, the inventors of this case conducted research and, through continuous research, ultimately proposed a speech recognition method with better results.

[0086] Next, the speech recognition method provided by this application is introduced through the following embodiments.

[0087] See also Figure 4 , which shows a flow chart of a speech recognition method provided by an embodiment of the present application, and which may include:

[0088] Step S401: Acquire target voice data.

[0089] The target speech data is the speech data to be recognized.

[0090] Step S402: performing preliminary speech recognition on the target speech data to obtain word-level acoustic information of the target speech data and candidate speech recognition results.

[0091] When performing preliminary speech recognition on the target speech data, word-level acoustic information is first obtained for the target speech data, and then the candidate speech recognition results are determined based on the word-level acoustic information. It should be noted that when performing preliminary speech recognition, each time a piece of word-level acoustic information is obtained, a candidate word can be determined based on the word-level acoustic information, and finally the candidate speech recognition results can be determined based on all the candidate words obtained.

[0092] Step S403: Projecting the word-level acoustic information of the target speech data into the representation space of the large language model to obtain target acoustic features of the target speech data.

[0093] After obtaining the word-level acoustic information of the target speech data, the word-level acoustic information of the target speech data is projected into the representation space of the large language model so that it can adapt to the parameters and structure of the large language model, thereby enabling the large language model to better understand the information of the acoustic modality.

[0094] Step S404: using the large language model and the candidate speech recognition results of the target speech data, the target acoustic features of the target speech data are processed into a text sequence to obtain the speech recognition results of the target speech data.

[0095] Since the target acoustic features of the target speech data are word-level acoustic features, which are consistent with the granularity of the modeling units of the large language model, and the target acoustic features of the target speech data are acoustic features adapted to the parameters and structure of the large language model, the large language model can understand the target acoustic features well and thus provide more accurate speech recognition results.

[0096] Considering that large language models usually use autoregressive decoding to decode acoustic features, and the autoregressive decoding method sometimes has the problem of unstoppable decoding, usually a certain segment appears repeatedly, such as, "The weather is good today, good, good, good, good...", "Do you think I am right, no, no, no, no...", in addition, large language models also have hallucination problems. For example, the user said "Spring sleep does not know the dawn", because the large language model is trained through a large amount of training data, it may decode "Spring sleep does not know the dawn, and birds are heard everywhere" based on the learned knowledge. To address the above problems, this application introduces candidate speech recognition results. The introduction of candidate speech recognition results can assist the large language model to limit the decoding length, thereby effectively solving the problem of unstoppable decoding of the large language model. At the same time, the introduction of candidate speech recognition results based on word-level acoustic information can suppress the occurrence of hallucinations in the large language model, so that the decoding can fully utilize the language information of the large language model while respecting the acoustic information itself.

[0097] In this embodiment, the candidate speech recognition results that assist the large language model in obtaining the final speech recognition result can be the Nbest (for example, 5best) candidate speech recognition results obtained by performing preliminary speech recognition on the target speech data, that is, the best N (N is an integer greater than 1, and the specific value of N can be determined according to the actual application scenario) candidate speech recognition results among all the candidate speech recognition results obtained by performing preliminary speech recognition on the target speech data (all the obtained candidate speech recognition results are sorted in descending order of score, and the candidate speech recognition results ranked in the top N positions are the Nbest candidate speech recognition results). Of course, this embodiment is not limited to this. For example, the candidate speech recognition result that assists the large language model in obtaining the final speech recognition result can also be the best candidate speech recognition result among all the candidate speech recognition results (that is, the candidate speech recognition result with the highest score).

[0098] The speech recognition method provided in the embodiment of the present application, after obtaining the target speech data, first performs preliminary speech recognition on the target speech data to obtain the word-level acoustic information of the target speech data and the candidate speech recognition results, then projects the word-level acoustic information of the target speech data into the representation space of the large language model to obtain the target acoustic features, and finally uses the large language model, supplemented by the candidate speech recognition results of the target speech data, to process the target acoustic features into a text sequence to obtain the speech recognition results of the target speech data. Taking into account that the modeling granularity of the large language model is word, the speech recognition method provided in this application obtains word-level acoustic information for the target speech data. In order to enable the large language model to better understand the information of the acoustic modality and thus provide accurate speech recognition results, the speech recognition method provided in this application further projects the word-level acoustic information into the representation space of the large language model after obtaining the word-level acoustic information, so that it is adapted to the parameters and structure of the large language model. Taking into account the problems of hallucinations and unstoppable decoding in the large language model, the speech recognition method provided in the embodiment of this application introduces candidate speech recognition results to assist the large language model in limiting the decoding length, so as to effectively solve the problems of hallucinations and unstoppable decoding in the large language model. In summary, the speech recognition method provided in the embodiment of this application has a good speech recognition effect.

[0099] In another embodiment of the present application, the specific implementation process of "step S402: obtaining word-level acoustic information of the target speech data and candidate speech recognition results by performing preliminary speech recognition on the target speech data" in the above embodiment is introduced.

[0100] The process of performing preliminary speech recognition on the target speech data to obtain word-level acoustic information of the target speech data and candidate speech recognition results may include: obtaining frame-level acoustic information of the target speech data; determining the speech frames belonging to the same word by determining whether each speech frame of the target speech data is a decoding position; fusing the acoustic information of the speech frames belonging to the same word to obtain word-level acoustic information; determining a candidate word based on each word-level acoustic information obtained; and determining a candidate speech recognition result based on all the candidate words obtained.

[0101] Among them, the process of determining the speech frames belonging to the same word by determining whether each speech frame of the target speech data is a decoding position includes: for each speech frame of the target speech data, it can be judged whether the speech frame is a decoding position to obtain the decoding position judgment result of the speech frame, and then the speech frames belonging to the same word can be determined based on the decoding position judgment result of the speech frame of the target speech data, and then the acoustic information of the speech frames belonging to the same word can be fused.

[0102] Exemplarily, the speech frames of the target speech data are f1, f2, f3, f4, f5, f6, f7, f8, f9, f10, f11, etc. Assuming that speech frames f1, f2, and f3 are not decoding positions, and f4 is a decoding position, it is determined that f1~f4 belong to the same word, and the acoustic features of speech frames f1~f4 are fused to obtain the first word-level acoustic information. Assuming that f5, f6, f7, f8, and f9 are not decoding positions, and f10 is a decoding position, it is determined that f5~f10 belong to the same word, and the acoustic features of f5~f10 are fused to obtain the second word-level acoustic information. The same is true for other speech frames, and the word-level acoustic information of the target speech data is obtained in this way.

[0103] In one possible implementation, a pre-trained speech recognition model based on the Monotonic Chunk Attention (MoCha) module can be used to perform preliminary speech recognition on the target speech data to obtain word-level acoustic information of the target speech data and candidate speech recognition results.

[0104] Specifically, first, speech features (such as MFCC, FilterBank, etc.) are extracted from each speech frame of the target speech data to obtain a speech feature sequence corresponding to the target speech data. The speech feature sequence corresponding to the training speech data is input into a speech recognition model based on a monotonic block attention module. The speech recognition model obtains frame-level acoustic information according to the speech feature sequence corresponding to the training speech data, obtains word-level acoustic information according to the frame-level acoustic information, and determines the candidate speech recognition results according to the word-level acoustic information.

[0105] Among them, for each speech frame of the target speech data, the monotonic block attention module in the speech recognition model can predict the probability that the speech frame can be decoded, and can determine whether the speech frame is a decoding position based on the predicted probability to obtain the decoding position judgment result of the speech frame. Furthermore, based on the decoding position judgment result of the speech frame of the target speech data, the speech frames belonging to the same word can be determined, and then the acoustic information of the speech frames belonging to the same word can be fused to obtain the word-level acoustic information of the target speech data.

[0106] Among them, when the acoustic information of speech frames belonging to the same word is fused, the energy of each speech frame can be obtained, and the energy of each speech frame is used as a weight. The acoustic information of the speech frames belonging to the same word is fused using a weighted summation fusion method. For example, if speech frames f1 to f4 belong to the same word, the energy of speech frame f1 is determined as the weight w1, the energy of speech frame f2 is determined as the weight w2, the energy of speech frame f3 is determined as the weight w3, and the energy of speech frame f4 is determined as the weight w4. The acoustic information of speech frame f1 is weighted by weight w1, the acoustic information of speech frame f2 is weighted by weight w2, the acoustic information of speech frame f3 is weighted by weight w3, and the acoustic information of speech frame f4 is weighted by weight w4. The four weighted acoustic information are summed to obtain the acoustic information at the word level.

[0107] In another embodiment of the present application, the specific implementation process of "Step S403: projecting the word-level acoustic information of the target speech data into the representation space of the large language model to obtain the target acoustic features of the target speech data" in the above embodiment is introduced.

[0108] In one possible implementation, the process of projecting the word-level acoustic information of the target speech data into the representation space of the large language model to obtain the target acoustic features of the target speech data may include: using a pre-trained connection module to project the word-level acoustic information of the target speech data into the representation space of the large language model, and obtaining acoustic features adapted to the large language model as the target acoustic features of the target speech data.

[0109] The connection module is trained on speech recognition tasks based on a large language model using text-annotated training speech data. This connection module can adapt and transform the input acoustic information to match the parameters and structure of the large language model, enabling the large language model to better understand the acoustic modal information.

[0110] Optionally, the connection module can adopt a linear layer, an attention layer, a Q-Former (Q-Former is a lightweight transformer), etc. Of course, this embodiment is not limited to this. For example, the connection module can also adopt a linear layer and a Conformer (such as a 2-layer linear layer and a 2-layer Conformer). In actual application, the specific structure of the connection module can be adjusted according to the amount of data and the complexity of the task.

[0111] In another embodiment of the present application, the specific implementation process of "Step S404: using a large language model, supplemented by the candidate speech recognition results of the target speech data, to process the target acoustic features of the target speech data into a text sequence to obtain the speech recognition results of the target speech data" in the above embodiment is introduced.

[0112] The process of using a large language model, supplemented by candidate speech recognition results of target speech data, processing target acoustic features of the target speech data into a text sequence, and obtaining speech recognition results of the target speech data may include: integrating the target acoustic features of the target speech data with the candidate speech recognition results of the target speech data through a preconfigured Prompt instruction template to obtain an integrated result; and inputting the integrated result into the large language model to obtain a speech recognition result of the target speech data.

[0113] See also Figure 5 , shows a flow chart of a more specific implementation method of using a large language model, supplemented by candidate speech recognition results of target speech data, processing target acoustic features of the target speech data into a text sequence, and obtaining speech recognition results of the target speech data, which may include:

[0114] Step S501: converting the candidate speech recognition results of the target speech data into features suitable for processing by a large language model, and obtaining features corresponding to the candidate speech recognition results of the target speech data.

[0115] like Figure 6 As shown, a large language model embedding layer (LLM embedding layer) can be used to convert the candidate speech recognition results of the target speech data into features suitable for processing by the large language model, thereby obtaining features corresponding to the candidate speech recognition results of the target speech data.

[0116] Step S502: converting the preconfigured Prompt instruction template into features suitable for processing by a large language model, and obtaining features corresponding to the preconfigured Prompt instruction template.

[0117] Among them, the preconfigured Prompt instruction template includes an acoustic information slot and a recognition auxiliary information slot. The preconfigured Prompt instruction template is used to instruct the large language model to give a speech recognition result based on the information in the acoustic information slot and the information in the recognition auxiliary information slot.

[0118] like Figure 6 As shown, the large language model embedding layer (LLM embedding layer) can be used to convert the preconfigured Prompt instruction template into features suitable for large language model processing, thereby obtaining features corresponding to the preconfigured Prompt instruction template.

[0119] In a possible implementation, the preconfigured Prompt instruction template may be a simplified Prompt instruction template, such as: “Audio content: <speechhere>\nBest hypothesis:\n <besthyphere>\nOther assumptions:\n <otherhyphere> \n <labelhere>”.

[0120] To achieve better speech recognition results, in another possible implementation, the preconfigured Prompt command template can be a complete Prompt command template, such as: "User: You are a model used to correct speech recognition results. Please correct insertion, deletion, and substitution errors in the results based on the hypotheses generated by the speech recognition system, including one line of optimal hypothesis and four lines of other hypotheses, combined with the audio content, and return only the recognition results:\nAudio content: <speechhere>\nBest hypothesis:\n <besthyphere>\nOther assumptions:\n <otherhyphere>\nAssitant:\n <labelhere>”.

[0121] in, <speechhere>is the acoustic information slot, <speechhere>Insert acoustic information at <besthyphere>and <otherhyphere>To identify the auxiliary information slot, <besthyphere>Insert the best candidate speech recognition result among the Nbest candidate speech recognition results, <otherhyphere>Insert other candidate speech recognition results in Nbest candidate speech recognition results, <labelhere>The text sequence that needs to be predicted.

[0122] Compared to the streamlined Prompt command template, the complete Prompt command template includes detailed task description information. This task description information can prompt the large language model to pay attention to the information in the acoustic information slot and the recognition auxiliary information slot, and prompt the large language model how to process the information in the acoustic information slot and the recognition auxiliary information slot. The introduction of task description information can improve the accuracy of the large language model in speech recognition based on the target acoustic features. It should be noted that the above Prompt command template is only an example. In actual application, the Prompt command template can be set according to specific application scenarios and needs.

[0123] In addition, it should be noted that this embodiment does not limit the execution order of step S501 and step S502. Step S501 may be executed first and then step S502, or step S502 may be executed first and then step S501.

[0124] Step S503: According to the positions of the acoustic information slot and the recognition auxiliary information slot, the target acoustic features and the features corresponding to the candidate speech recognition results of the target speech data are inserted into the features corresponding to the preconfigured Prompt instruction template to obtain the combined features.

[0125] like Figure 6 As shown, according to the positions of the acoustic information slot and the recognition auxiliary information slot in the preconfigured Prompt instruction template, the target acoustic features of the target speech data, the features corresponding to the preconfigured Prompt instruction template, and the features corresponding to the candidate speech recognition results of the target speech data are combined to obtain combined features.

[0126] Specifically, according to the position of the acoustic information slot in the preconfigured Prompt instruction template, the target acoustic feature is inserted into the feature corresponding to the Prompt instruction template; according to the position of the recognition auxiliary information slot in the preconfigured Prompt instruction template, the feature corresponding to the candidate speech recognition result of the target speech data is inserted into the feature corresponding to the preconfigured Prompt instruction template.

[0127] The above embodiment mentioned that the candidate speech recognition results that assist the large language model in obtaining the final speech recognition results can be the best N candidate speech recognition results. In view of this, the recognition auxiliary information slot in the preconfigured Prompt instruction template may include a first auxiliary information slot and a second auxiliary information slot. Furthermore, when inserting the features corresponding to the candidate speech recognition results of the target speech data into the features corresponding to the preconfigured Prompt instruction template according to the position of the recognition auxiliary information slot in the preconfigured Prompt instruction template, according to the position of the first auxiliary information slot, the features corresponding to the best candidate speech recognition result among the N candidate speech recognition results are inserted into the features corresponding to the Prompt instruction template, and according to the position of the second auxiliary information slot, the features corresponding to other candidate speech recognition results among the N candidate speech recognition results are inserted into the features corresponding to the Prompt instruction template.

[0128] Step S504: Input the combined features into the large language model to obtain the speech recognition result of the target speech data.

[0129] After the combined features are input into the large language model, the large language model determines the final speech recognition result of the target speech data based on the input features.

[0130] In this embodiment, the large language model can be a pre-trained large language model, or a large language model obtained by fine-tuning the pre-trained large language model on a speech recognition task. In order to obtain better speech recognition results, the latter is preferred.

[0131] From the contents provided in the above embodiments, it can be seen that the speech recognition method provided in the present application can be implemented based on a speech recognition model (such as a speech recognition model based on a monotonic block attention module), a connection module, a large language model, etc., wherein the connection module uses training speech data annotated with text and is trained on a speech recognition task based on a large language model. In order to obtain better speech recognition effects, the large language model uses training speech data annotated with text and further trains the pre-trained large language model on the speech recognition task. Next, the training process of the connection module and the large language model is introduced.

[0132] like Figure 7 As shown, the training process of the connection module and the large language model may include:

[0133] Step S701: Fix the parameters of the speech recognition model and the large language model, use the training speech data annotated with text, and train the connection module on the speech recognition task.

[0134] Specifically, using text-annotated training speech data, the process of training the connection module on the speech recognition task includes:

[0135] Step a1: extract speech features from each speech frame of the training speech data to obtain a speech feature sequence corresponding to the training speech data.

[0136] The speech features extracted from each speech frame of the training speech data may be, but are not limited to, MFCC, FilterBank, and other features. The speech feature sequence corresponding to the training speech data can be expressed as X={x1,…, x t ,…, x T }, where x t Represents the speech features of the tth speech frame of the training speech data.

[0137] Step a2: input the speech feature sequence corresponding to the training speech data into the speech recognition model to obtain word-level acoustic information of the training speech data and candidate speech recognition results.

[0138] The speech feature sequence X={x1,…, x t ,…, x T } Input a speech recognition model based on the monotonic block attention module, which obtains frame-level acoustic information according to the input speech feature sequence, and determines the speech frames belonging to the same word by judging whether each speech frame of the training speech data is a decoding position. The acoustic information of the speech frames belonging to the same word is fused to obtain the word-level acoustic information of the training speech data, and then the candidate speech recognition results of the training speech data are determined according to the word-level acoustic information of the training speech data.

[0139] Step a3: Based on the connection module, the word-level acoustic information of the training speech data is projected into the representation space of the large language model to obtain the target acoustic features of the training speech data.

[0140] The word-level acoustic information of the training speech data is input into the connection module, which processes the input word-level acoustic information into features that are adapted to the parameters and structure of the large language model to obtain the target acoustic features of the training speech data.

[0141] Step a3: Use the large language model, supplemented by the candidate speech recognition results of the training speech data (such as the 5 best candidate speech recognition results), to predict the text sequence corresponding to the training speech data (i.e., the speech recognition result) to obtain the text sequence prediction result.

[0142] It should be noted that the introduction of candidate speech recognition results can not only limit the decoding length of the large language model, but also assist the connection module in learning the mapping relationship.

[0143] Specifically, the process of using the large language model, supplemented by the candidate speech recognition results of the training speech data (such as the 5best candidate speech recognition results), to predict the text sequence corresponding to the training speech data may include: based on the large language model embedding layer, converting the candidate speech recognition results of the training speech data (such as the 5best candidate speech recognition results) into features suitable for processing by the large language model, and obtaining features corresponding to the candidate speech recognition results of the training speech data; based on the large language model embedding layer, converting the text annotated by the training speech data into features suitable for processing by the large language model, and obtaining features corresponding to the annotated text of the training speech data; based on the large language model embedding layer, converting the preconfigured Prompt instruction model into features suitable for processing by the large language model, and obtaining features corresponding to the annotated text of the training speech data. The Prompt instruction template is converted into features suitable for processing by a large language model to obtain features corresponding to the Prompt instruction template, wherein the Prompt instruction template includes an acoustic information slot, a recognition auxiliary information slot, and a text sequence information slot. The Prompt instruction template is used to instruct the large language model to give a speech recognition result based on the information in the acoustic information slot and the information in the recognition auxiliary information slot; according to the positions of the acoustic information slot, the recognition auxiliary information slot, and the text sequence information slot, the target acoustic features of the training speech data and the features corresponding to the annotated text of the training speech data are inserted into the features corresponding to the Prompt instruction template to obtain combined features; the combined features are input into the large language model to obtain a text sequence prediction result.

[0144] Step a4: Determine the prediction loss based on the text sequence prediction results and the text annotated by the training speech data.

[0145] The prediction loss may adopt cross entropy loss, and the calculation method of cross entropy loss is an existing technology, which will not be described in detail in this embodiment.

[0146] Step a5: Update the parameters of the connection module according to the prediction loss.

[0147] The connection module is trained multiple times according to the process of steps a1 to a5 above until the training end condition is met.

[0148] Step S702: Fix the parameters of the speech recognition model and the connection module, use the training speech data annotated with text, and train the large language model on the speech recognition task.

[0149] The process of training the large language model for speech recognition tasks using training speech data annotated with text is similar to the process of training the connection module for speech recognition tasks using training speech data annotated with text. Similarly, it is necessary to determine the prediction loss according to steps a1 to a4. After determining the prediction loss, the parameters of the large language model are updated based on the prediction loss. When updating the parameters of the large language model, only some parameters of the large language model (such as the parameters of the linear layer of the large language model) may be updated.

[0150] Step S703: Fix the parameters of the speech recognition model, use the training speech data annotated with text, and train the connection module and the large language model on the speech recognition task at the same time.

[0151] The process of simultaneously training the connection module and the large language model for speech recognition tasks using training speech data annotated with text is similar to the process of training the connection module for speech recognition tasks using training speech data annotated with text. It is also necessary to determine the prediction loss according to steps a1 to a4. After determining the prediction loss, the parameters of the connection module and the large language model are updated based on the prediction loss.

[0152] It should be noted that the speech recognition model is pre-trained. During the training of the connection module and the large language model on the speech recognition task, the parameters of the speech recognition model remain fixed.

[0153] It should also be noted that this embodiment is not limited to the use of the above-mentioned steps S701 to S703 to train the connection module and the large language model. For example, training speech data annotated with text can also be used to train the connection module and the large language model simultaneously on the speech recognition task. It is also possible to first fix the parameters of the speech recognition model and the connection module, use the training speech data annotated with text, and fine-tune the large language model on the speech recognition task, and then fix the parameters of the speech recognition model and the large language model, use the training speech data annotated with text, and train the connection module on the speech recognition task. Finally, fix the parameters of the speech recognition model and the connection module, use the training speech data annotated with text, and train the connection module on the speech recognition task.

[0154] After the training is completed, the target speech data can be recognized using modules such as the connection module and large language model obtained through training, as well as the speech recognition model based on the monotonic block attention module.

[0155] The following table shows the comparison of recognition effects of several speech recognition solutions:

[0156] Table 1 Comparison of recognition effects of several speech recognition schemes

[0157]

[0158] The "MoCha ASR" recognition scheme uses only a speech recognition model based on a monotonic block attention module to perform speech recognition. The "ENC+LLM" recognition scheme refers to an existing recognition scheme based on a large language model. That is, the acoustic model is used to obtain frame-level acoustic information for the speech data to be recognized. The large language model then obtains speech recognition results based on this frame-level acoustic information. 1 ”~"(MoCha+LLM) 4 Both are solutions that use a speech recognition model based on a monotonic block attention module and a large language model to perform speech recognition on the target speech. (MoCha+LLM) 1 ”~"(MoCha+LLM) 4 The difference is that (MoCha+LLM) 1 The recognition scheme does not use the candidate speech recognition results. The Prompt command template used is a simplified Prompt command template, and the large language model used is a pre-trained large language model (without fine-tuning). (MoCha+LLM) 2 The recognition scheme uses the candidate speech recognition results, the Prompt command template used is a simplified Prompt command template, and the large language model used is a pre-trained large language model (without fine-tuning), "(MoCha+LLM) 3 The recognition scheme uses the candidate speech recognition results, the Prompt command template used is the perfect Prompt command template, and the large language model used is the pre-trained large language model (without fine-tuning), "(MoCha+LLM) 4 The recognition scheme of " utilizes the candidate speech recognition results. The Prompt command template adopted is the perfect Prompt command template. The large language model used is a large language model obtained by fine-tuning the pre-trained large language model for the speech recognition task. As can be seen from the above table, speech recognition based on word-level acoustic features and supplemented by candidate speech recognition results can improve the speech recognition accuracy. In addition, compared with the streamlined Prompt command template, the use of the perfect Prompt command template can improve the speech recognition effect. Compared with the large language model that has not been fine-tuned, the use of the fine-tuned large language model for speech recognition can improve the speech recognition effect.

[0159] The above describes the speech recognition method provided in the embodiments of the present application. The following describes the device corresponding to the above speech recognition method.

[0160] See also Figure 8 , Figure 8 This is a structural diagram of a speech recognition device provided in an embodiment of the present application. The speech recognition device may include: a speech data acquisition module 801, a speech recognition information acquisition module 802, an acoustic information processing module 803 and a speech recognition result acquisition module 804.

[0161] The voice data acquisition module 801 is used to acquire target voice data.

[0162] The speech recognition information acquisition module 802 is configured to perform preliminary speech recognition on the target speech data to obtain word-level acoustic information of the target speech data and candidate speech recognition results.

[0163] The acoustic information processing module 803 is used to project the word-level acoustic information of the target speech data into the representation space of the large language model to obtain the target acoustic features of the target speech data.

[0164] The speech recognition result acquisition module 804 is used to use the large language model, supplemented by the candidate speech recognition results of the target speech data, to process the target acoustic features of the target speech data into a text sequence to obtain the speech recognition results of the target speech data.

[0165] In one possible implementation, when acquiring word-level acoustic information of target speech data, the speech recognition information acquisition module 802 is specifically configured to:

[0166] Acquiring frame-level acoustic information for target speech data;

[0167] Determining whether each speech frame of the target speech data is a decoding position, determining speech frames belonging to the same word;

[0168] The acoustic information of speech frames belonging to the same word is fused to obtain the word-level acoustic information of the target speech data.

[0169] In one possible implementation, when the speech recognition information acquisition module 802 performs preliminary speech recognition on the target speech data to obtain word-level acoustic information of the target speech data and candidate speech recognition results, it is specifically configured to:

[0170] By using a pre-trained speech recognition model based on a monotonic block attention module to perform preliminary speech recognition on the target speech data, the word-level acoustic information of the target speech data and the candidate speech recognition results are obtained.

[0171] In one possible implementation, the speech recognition result acquisition module 804 utilizes the large language model and the candidate speech recognition results of the target speech data to process the target acoustic features of the target speech data into a text sequence to obtain the speech recognition results of the target speech data, specifically for:

[0172] The target acoustic features of the target speech data are integrated with the candidate speech recognition results of the target speech data through the pre-configured Prompt command template to obtain an integrated result;

[0173] The integration result is input into the large language model to obtain the speech recognition result of the target speech data of the large language model.

[0174] In one possible implementation, the speech recognition result acquisition module 804 integrates the target acoustic features of the target speech data with the candidate speech recognition results of the target speech data using a preconfigured Prompt instruction template to obtain an integrated result, specifically for:

[0175] Converting the candidate speech recognition results of the target speech data into features suitable for processing by the large language model, thereby obtaining features corresponding to the candidate speech recognition results of the target speech data;

[0176] Converting a preconfigured Prompt instruction template into features suitable for processing by the large language model to obtain features corresponding to the Prompt instruction template, wherein the Prompt instruction template includes an acoustic information slot and an identification auxiliary information slot, and the Prompt instruction template is used to instruct the large language model to provide a speech recognition result based on the information in the acoustic information slot and the information in the identification auxiliary information slot;

[0177] According to the positions of the acoustic information slot and the recognition auxiliary information slot, the target acoustic features of the target speech data and the features corresponding to the candidate speech recognition results of the target speech data are inserted into the features corresponding to the Prompt instruction template to obtain the combined features.

[0178] In one possible implementation, the Prompt instruction template also includes: task description information, which is used to prompt the large language model to pay attention to the information in the acoustic information slot and the recognition auxiliary information slot, and to prompt the large language model how to process the information in the acoustic information slot and the recognition auxiliary information slot.

[0179] In a possible implementation, the candidate speech recognition results of the target speech data are the best N candidate speech recognition results, where N is an integer greater than 1.

[0180] The identification auxiliary information slot in the Prompt instruction template includes a first auxiliary information slot and a second auxiliary information slot.

[0181] The speech recognition result acquisition module 804 inserts the target acoustic features of the target speech data and the features corresponding to the candidate speech recognition results of the target speech data into the features corresponding to the Prompt instruction template according to the positions of the acoustic information slot and the recognition auxiliary information slot. When obtaining the combined features, the module is specifically configured to: insert the target acoustic features of the target speech data into the features corresponding to the Prompt instruction template according to the positions of the acoustic information slot;

[0182] According to the position of the first auxiliary information slot, insert the feature corresponding to the best candidate speech recognition result among the N candidate speech recognition results into the feature corresponding to the Prompt instruction template;

[0183] According to the position of the second auxiliary information slot, features corresponding to other candidate speech recognition results in the N candidate speech recognition results are inserted into the features corresponding to the Prompt instruction template.

[0184] In one possible implementation, when the acoustic information processing module 803 projects the word-level acoustic information of the target speech data into the representation space of the large language model to obtain the target acoustic features of the target speech data, it is specifically configured to:

[0185] Using the pre-trained connection module, the word-level acoustic information of the target speech data is projected into the representation space of the large language model to obtain acoustic features adapted to the large language model as the target acoustic features of the target speech data.

[0186] Among them, the connection module is trained on the speech recognition task based on the large language model using training speech data annotated with text.

[0187] In one possible implementation, the large language model is obtained by training a pre-trained large language model on a speech recognition task using training speech data annotated with text.

[0188] The speech recognition device may further include: a training module.

[0189] The training module is used to train the connection module and the large language model.

[0190] When training the connection module and the large language model, the training module is specifically used to:

[0191] The parameters of the large language model are fixed, and the connection module is trained on the speech recognition task using text-annotated training speech data.

[0192] The parameters of the connection module are fixed, and the large language model is trained on speech recognition tasks using training speech data annotated with text.

[0193] Using training speech data annotated with text, the connection module and the large language model are trained simultaneously on the speech recognition task.

[0194] The speech recognition device provided in the embodiment of the present application, after obtaining the target speech data, first performs preliminary speech recognition on the target speech data to obtain the word-level acoustic information of the target speech data and the candidate speech recognition results, then projects the word-level acoustic information of the target speech data into the representation space of the large language model to obtain the target acoustic features, and finally uses the large language model, supplemented by the candidate speech recognition results of the target speech data, to process the target acoustic features into a text sequence to obtain the speech recognition results of the target speech data. Considering that the modeling granularity of the large language model is word, the speech recognition device provided in the present application obtains word-level acoustic information for the target speech data. In order to enable the large language model to better understand the information of the acoustic modality and thus provide accurate speech recognition results, the speech recognition device provided in the present application further projects the word-level acoustic information into the representation space of the large language model after obtaining the word-level acoustic information, so that it is adapted to the parameters and structure of the large language model. Taking into account the problems of hallucinations and unstoppable decoding of the large language model, the speech recognition device provided in the present application introduces candidate speech recognition results to assist the large language model in limiting the decoding length, so as to effectively solve the problems of hallucinations and unstoppable decoding of the large language model. In summary, the speech recognition device provided in the present application has a good speech recognition effect.

[0195] An embodiment of the present application also provides an electronic device, which may include: at least one processor, at least one communication interface, at least one memory and at least one communication bus.

[0196] In the embodiment of the present application, the number of the processor, the communication interface, the memory, and the communication bus is at least one, and the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0197] The processor may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application;

[0198] The memory may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory;

[0199] The memory stores a program, and the processor can call the program stored in the memory, and the program is used to implement the steps of the speech recognition method provided in the above embodiment.

[0200] An embodiment of the present application also provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the steps of the speech recognition method provided in the above embodiment.

[0201] An embodiment of the present application also provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the steps of the speech recognition method provided in the above embodiment.

[0202] It should also be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. In addition, in the drawings of the device embodiments provided in this application, the connection relationship between the modules indicates that there is a communication connection between them, which can be specifically implemented as one or more communication buses or signal lines.

[0203] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course can also be implemented by special hardware including application-specific integrated circuits, special CPUs, special memories, special components, etc. In general, all functions performed by computer programs can be easily implemented with corresponding hardware, and the specific hardware structures used to implement the same function can also be diverse, such as analog circuits, digital circuits or special circuits, etc. However, for the present application, software program implementation is a better implementation method in most cases. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer's floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc., and includes a number of instructions to enable a computer device (which can be a personal computer, training equipment, or network equipment, etc.) to execute the methods described in each embodiment of the present application.

[0204] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0205] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, a computer, a training device or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website, a computer, a training device or a data center. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).< / labelhere> < / otherhyphere> < / besthyphere> < / otherhyphere> < / besthyphere> < / speechhere> < / speechhere> < / labelhere> < / otherhyphere> < / besthyphere> < / speechhere> < / labelhere> < / otherhyphere> < / besthyphere> < / speechhere>

Claims

1. A speech recognition method, characterized in that: include: Obtain target voice data; Acquiring word-level acoustic information of the target speech data and candidate speech recognition results by performing preliminary speech recognition on the target speech data; Projecting the word-level acoustic information of the target speech data into the representation space of the large language model to obtain target acoustic features of the target speech data; The target acoustic features of the target speech data are processed into a text sequence using the large language model and supplemented by the candidate speech recognition results of the target speech data to obtain the speech recognition results of the target speech data.

2. The speech recognition method according to claim 1, wherein: Acquiring word-level acoustic information of the target speech data includes: Acquiring frame-level acoustic information for the target speech data; Determining whether each speech frame of the target speech data is a decoding position, determining speech frames belonging to the same word; The acoustic information of the speech frames belonging to the same word is fused to obtain word-level acoustic information of the target speech data.

3. The speech recognition method according to claim 1, wherein: The step of performing preliminary speech recognition on the target speech data to obtain word-level acoustic information of the target speech data and candidate speech recognition results includes: The target speech data is subjected to preliminary speech recognition using a pre-trained speech recognition model based on a monotonic block attention module to obtain word-level acoustic information of the target speech data and candidate speech recognition results.

4. The speech recognition method according to claim 1, wherein: The method of using the large language model, supplemented by the candidate speech recognition results of the target speech data, processing the target acoustic features of the target speech data into a text sequence to obtain the speech recognition results of the target speech data includes: Integrating the target acoustic features of the target speech data with the candidate speech recognition results of the target speech data using a preconfigured Prompt instruction template to obtain an integrated result; The integration result is input into the large language model to obtain a speech recognition result of the target speech data output by the large language model.

5. The speech recognition method according to claim 4, characterized in that The method of integrating the target acoustic features of the target speech data with the candidate speech recognition results of the target speech data through the preconfigured Prompt instruction template to obtain an integration result includes: Converting the candidate speech recognition results of the target speech data into features suitable for processing by the large language model to obtain features corresponding to the candidate speech recognition results of the target speech data; Converting a preconfigured Prompt instruction template into features suitable for processing by the large language model to obtain features corresponding to the Prompt instruction template, wherein the Prompt instruction template includes an acoustic information slot and a recognition auxiliary information slot, and the Prompt instruction template is used to instruct the large language model to provide a speech recognition result based on the information in the acoustic information slot and the information in the recognition auxiliary information slot; According to the positions of the acoustic information slot and the recognition auxiliary information slot, the target acoustic features of the target speech data and the features corresponding to the candidate speech recognition results of the target speech data are inserted into the features corresponding to the Prompt instruction template to obtain the combined features.

6. The speech recognition method according to claim 5, characterized in that The Prompt instruction template also includes: task description information; The task description information is used to prompt the large language model to pay attention to the information in the acoustic information slot and the recognition auxiliary information slot, and to prompt the large language model how to process the information in the acoustic information slot and the recognition auxiliary information slot.

7. The speech recognition method according to claim 5, characterized in that The candidate speech recognition results of the target speech data are the best N candidate speech recognition results, where N is an integer greater than 1; The identification auxiliary information slot includes a first auxiliary information slot and a second auxiliary information slot; The step of inserting the target acoustic features of the target speech data and the features corresponding to the candidate speech recognition results of the target speech data into the features corresponding to the Prompt instruction template according to the positions of the acoustic information slot and the recognition auxiliary information slot to obtain the combined features includes: inserting the target acoustic features of the target speech data into the features corresponding to the Prompt instruction template according to the positions of the acoustic information slot; Inserting a feature corresponding to the best candidate speech recognition result among the N candidate speech recognition results into the feature corresponding to the Prompt instruction template according to the position of the first auxiliary information slot; According to the position of the second auxiliary information slot, features corresponding to other candidate speech recognition results among the N candidate speech recognition results are inserted into the features corresponding to the Prompt instruction template.

8. The speech recognition method according to claim 1, wherein: The step of projecting the word-level acoustic information of the target speech data into the representation space of the large language model to obtain target acoustic features of the target speech data includes: Using a pre-trained connection module, projecting the word-level acoustic information of the target speech data into the representation space of the large language model, obtaining acoustic features adapted to the large language model as target acoustic features of the target speech data; The connection module is trained on a speech recognition task based on a large language model using training speech data annotated with text.

9. The speech recognition method according to claim 8, characterized in that The large language model is obtained by training a pre-trained large language model on a speech recognition task using training speech data annotated with text; The training process of the connection module and the large language model includes: The parameters of the large language model are fixed, and the connection module is trained on the speech recognition task using text-annotated training speech data. The parameters of the connection module are fixed, and the large language model is trained on the speech recognition task using text-annotated training speech data. Using training speech data annotated with text, the connection module and the large language model are trained simultaneously on the speech recognition task.

10. A speech recognition device, characterized in that: include: Speech data acquisition module, speech recognition information acquisition module, acoustic information processing module and speech recognition result acquisition module; A voice data acquisition module is used to acquire target voice data; A speech recognition information acquisition module, configured to obtain word-level acoustic information of the target speech data and candidate speech recognition results by performing preliminary speech recognition on the target speech data; The acoustic information processing module is used to project the word-level acoustic information of the target speech data into the representation space of the large language model to obtain the target acoustic features of the target speech data; The speech recognition result acquisition module is used to use the large language model, supplemented by the candidate speech recognition results of the target speech data, to process the target acoustic features of the target speech data into a text sequence to obtain the speech recognition results of the target speech data.

11. An electronic device, characterized in that: comprising at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the steps of the speech recognition method according to any one of claims 1 to 9.

12. A computer storage medium, characterized in that The storage medium carries one or more computer programs, and when the one or more computer programs are executed by an electronic device, the electronic device can implement the steps of the speech recognition method according to any one of claims 1 to 9.

13. A computer program product, characterized in that The method comprises computer-readable instructions, which, when executed on an electronic device, enable the electronic device to implement the steps of the speech recognition method according to any one of claims 1 to 9.

Images