Speech recognition method and related device

Abstract

The invention discloses a voice recognition method and a related device, and relates to the technical field of voice recognition, and the voice recognition method comprises the steps: obtaining target voice data; the method comprises the following steps: performing preliminary voice recognition on target voice data to obtain word-level acoustic information of the target voice data and candidate voice recognition results; projecting the word-level acoustic information of the target voice data into a representation space of the large language model to obtain target acoustic features of the target voice data; and processing the target acoustic features of the target voice data into a character sequence by using a large language model and assisted by the candidate voice recognition result of the target voice data, and obtaining a voice recognition result of the target voice data. The speech recognition method disclosed by the invention has a relatively good recognition effect.

Description

Technical Field

[0001] The present application relates to the field of speech recognition technology, and in particular to a speech recognition method and related devices. Background Art

[0002] Speech recognition technology is a technology that converts speech into text. In the field of speech recognition, language models play a vital role. They can provide language constraints based on contextual semantics, improve recognition accuracy, and enhance the ability to handle uncommon words and proper nouns. They work together with acoustic models to significantly improve the quality and efficiency of speech recognition.

[0003] Current speech recognition methods include speech recognition methods based on large language models. The speech recognition methods based on large language models first use the acoustic model to obtain frame-level acoustic information for the speech to be recognized, then project the frame-level acoustic information into the representation space of the large language model, and finally input the features obtained by projecting into the representation space of the large language model into the large language model to obtain the speech recognition result of the speech to be recognized.

[0004] Although the above-mentioned speech recognition method based on the large language model can realize speech recognition, the recognition effect is not good. Summary of the invention

[0005] In view of this, the present application provides a speech recognition method and related devices, which are used to solve the problem that the existing speech recognition method based on a large language model has poor recognition effect. The technical solution is as follows:

[0006] The first aspect of the present application provides a speech recognition method, comprising:

[0007] Obtain target voice data;

[0008] By performing preliminary speech recognition on the target speech data, word-level acoustic information of the target speech data and candidate speech recognition results are obtained;

[0009] Projecting the word-level acoustic information of the target speech data into the representation space of the large language model to obtain target acoustic features of the target speech data;

[0010] The target acoustic features of the target voice data are processed into a text sequence using the large language model and the candidate voice recognition results of the target voice data to obtain the voice recognition results of the target voice data.

[0011] In a possible implementation, obtaining word-level acoustic information of the target speech data includes:

[0012] Acquiring frame-level acoustic information for the target speech data;

[0013] Determine the speech frames belonging to the same word by determining whether each speech frame of the target speech data is a decoding position;

[0014] The acoustic information of the speech frames belonging to the same word is fused to obtain the word-level acoustic information of the target speech data.

[0015] In a possible implementation, performing preliminary speech recognition on the target speech data to obtain word-level acoustic information of the target speech data and candidate speech recognition results includes:

[0016] The target speech data is subjected to preliminary speech recognition using a pre-trained speech recognition model based on a monotone block attention module, thereby obtaining word-level acoustic information of the target speech data and candidate speech recognition results.

[0017] In a possible implementation, the using of the large language model, supplemented by the candidate speech recognition result of the target speech data, processing the target acoustic features of the target speech data into a text sequence, and obtaining the speech recognition result of the target speech data includes:

[0018] Integrate the target acoustic features of the target voice data with the candidate voice recognition results of the target voice data through a preconfigured Prompt instruction template to obtain an integrated result;

[0019] The integration result is input into the large language model to obtain the speech recognition result of the target speech data output by the large language model.

[0020] In a possible implementation, the target acoustic features of the target voice data and the candidate voice recognition results of the target voice data are integrated through a preconfigured Prompt instruction template to obtain an integrated result, including:

[0021] Converting the candidate speech recognition results of the target speech data into features suitable for processing by the large language model to obtain features corresponding to the candidate speech recognition results of the target speech data;

[0022] Converting a preconfigured Prompt instruction template into features suitable for processing by the large language model to obtain features corresponding to the Prompt instruction template, wherein the Prompt instruction template includes an acoustic information slot and a recognition auxiliary information slot, and the Prompt instruction template is used to instruct the large language model to give a speech recognition result according to the information in the acoustic information slot and the information in the recognition auxiliary information slot;

[0023] According to the positions of the acoustic information slot and the recognition auxiliary information slot, the target acoustic features of the target voice data and the features corresponding to the candidate voice recognition results of the target voice data are inserted into the features corresponding to the Prompt instruction template to obtain the combined features.

[0024] In a possible implementation, the Prompt instruction template further includes: task description information;

[0025] The task description information is used to prompt the large language model to pay attention to the information in the acoustic information slot and the recognition auxiliary information slot, and to prompt the large language model how to process the information in the acoustic information slot and the recognition auxiliary information slot.

[0026] In a possible implementation, the candidate speech recognition results of the target speech data are the best N candidate speech recognition results, where N is an integer greater than 1;

[0027] The identification auxiliary information slot includes a first auxiliary information slot and a second auxiliary information slot;

[0028] The step of inserting the target acoustic features of the target speech data and the features corresponding to the candidate speech recognition results of the target speech data into the features corresponding to the Prompt instruction template according to the positions of the acoustic information slot and the recognition auxiliary information slot to obtain the combined features includes: inserting the target acoustic features of the target speech data into the features corresponding to the Prompt instruction template according to the positions of the acoustic information slot;

[0029] According to the position of the first auxiliary information slot, inserting a feature corresponding to the best candidate speech recognition result among the N candidate speech recognition results into the feature corresponding to the Prompt instruction template;

[0030] According to the position of the second auxiliary information slot, features corresponding to other candidate speech recognition results among the N candidate speech recognition results are inserted into the features corresponding to the Prompt instruction template.

[0031] In a possible implementation, projecting the word-level acoustic information of the target speech data into the representation space of the large language model to obtain the target acoustic features of the target speech data includes:

[0032] Using the pre-trained connection module, the word-level acoustic information of the target speech data is projected into the representation space of the large language model to obtain acoustic features adapted to the large language model as target acoustic features of the target speech data;

[0033] The connection module is trained on a speech recognition task based on a large language model using training speech data annotated with text.

[0034] In a possible implementation, the large language model is obtained by training a pre-trained large language model on a speech recognition task using training speech data annotated with text;

[0035] The training process of the connection module and the large language model includes:

[0036] The parameters of the large language model are fixed, and the connection module is trained on the speech recognition task using training speech data annotated with text.

[0037] The parameters of the connection module are fixed, and the large language model is trained on the speech recognition task using the training speech data annotated with text;

[0038] Using training speech data annotated with text, the connection module and the large language model are trained simultaneously on the speech recognition task.

[0039] The second aspect of the present application provides a speech recognition device, comprising: a speech data acquisition module, a speech recognition information acquisition module, an acoustic information processing module and a speech recognition result acquisition module;

[0040] A voice data acquisition module, used to acquire target voice data;

[0041] A speech recognition information acquisition module, configured to acquire word-level acoustic information of the target speech data and candidate speech recognition results by performing preliminary speech recognition on the target speech data;

[0042] The acoustic information processing module is used to project the word-level acoustic information of the target speech data into the representation space of the large language model to obtain the target acoustic features of the target speech data;

[0043] The speech recognition result acquisition module is used to use the large language model, assisted by the candidate speech recognition results of the target speech data, to process the target acoustic features of the target speech data into a text sequence to obtain the speech recognition results of the target speech data.

[0044] A third aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0045] The memory is used to store computer programs;

[0046] The processor is used to execute the computer program so that the electronic device can implement the steps of any one of the above-mentioned speech recognition methods.

[0047] In a fourth aspect, the present application provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the steps of any one of the above-mentioned speech recognition methods.

[0048] In a fifth aspect of the present application, a computer program product includes computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the steps of any one of the above-mentioned speech recognition methods.

[0049] By means of the above technical scheme, the speech recognition method provided by the present application, after obtaining the target speech data, first performs preliminary speech recognition on the target speech data to obtain the word-level acoustic information of the target speech data and the candidate speech recognition results, then projects the word-level acoustic information of the target speech data into the representation space of the large language model to obtain the target acoustic features, and finally uses the large language model, assisted by the candidate speech recognition results of the target speech data, to process the target acoustic features into a text sequence to obtain the speech recognition results of the target speech data. Considering that the modeling granularity of the large language model is word, the speech recognition method provided in the present application obtains word-level acoustic information for the target speech data. In order to enable the large language model to understand the information of the acoustic modality well and then give accurate speech recognition results, the speech recognition method provided in the present application further projects the word-level acoustic information into the representation space of the large language model after obtaining the word-level acoustic information to make it compatible with the large language model. Considering the problems of hallucinations and unstoppable decoding of the large language model, the speech recognition method provided in the present application introduces candidate speech recognition results to assist the large language model in limiting the decoding length, so as to effectively solve the problems of hallucinations and unstoppable decoding of the large language model. In summary, the speech recognition method provided in the present application has a good speech recognition effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0051] Figure 1 A schematic diagram of a system architecture involved in this application;

[0052] Figure 2 A schematic diagram of a hardware structure of a terminal provided in an embodiment of the present application;

[0053] Figure 3 A schematic diagram of a hardware structure of a server provided in an embodiment of the present application;

[0054] Figure 4 A flowchart of a speech recognition method provided in an embodiment of the present application;

[0055] Figure 5 A schematic flow chart of using a large language model provided in an embodiment of the present application, supplemented by candidate speech recognition results of target speech data, processing target acoustic features of target speech data into a text sequence, and obtaining speech recognition results of target speech data;

[0056] Figure 6 A schematic diagram of implementing speech recognition using a speech recognition model based on a monotone block attention module, a connection module, a large language model and other modules provided in an embodiment of the present application;

[0057] Figure 7 A schematic diagram of the flow of training the connection module and the large language model provided in the embodiment of the present application;

[0058] Figure 8 A schematic diagram of the structure of a speech recognition device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0059] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0060] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0061] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0062] In one possible implementation, Figure 1As shown, the system architecture involved in the present application may include a terminal 101 and a server 102, and the terminal 101 may interact with the server 102 via a network (wired network or wireless network). The server 102 may include one or more servers ( Figure 1 The terminal 101 can obtain the voice data to be recognized, and transmit the voice data to be recognized to the server 102 through the network. The server 102 uses the voice recognition method provided in the application to perform voice recognition on the voice data to be recognized, and transmits the voice recognition result to the terminal 101 through the network.

[0063] In another possible implementation, the system architecture involved in the present application may include a terminal. The terminal has strong data processing capabilities. The terminal can obtain voice data to be recognized, and use the voice recognition method provided in the application to perform voice recognition on the voice data to be recognized.

[0064] Next, the product form of the above-mentioned terminal is described.

[0065] The above-mentioned terminal can be a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, a robot, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc., and the embodiments of the present application do not impose any restrictions on this.

[0066] Figure 2 An optional hardware structure diagram of a terminal is shown.

[0067] refer to Figure 2 As shown, the terminal may include a radio frequency unit 210, a memory 220, an input unit 230, a display unit 240, a camera 250 (optional), an audio circuit 260 (optional), a speaker 261 (optional), a microphone 262 (optional), an earphone jack 263 (optional), a processor 270, an external interface 280, a power supply 290 and other components. Those skilled in the art will appreciate that Figure 2 This is merely an example of a terminal and does not constitute a limitation on the terminal. The terminal may include more or fewer components than shown in the figure, or a combination of certain components, or different components.

[0068] The input unit 230 can be used to receive input digital or character information, and generate key signal input related to the user settings and function control of the terminal. Specifically, the input unit 230 may include a touch screen 231 (optional) and / or other input devices 232. The touch screen 231 can collect the user's touch operations on or near it (such as the user's operation on or near the touch screen using any suitable object such as fingers, joints, stylus, etc.), and drive the corresponding connection device according to a pre-set program. The touch screen can detect the user's touch action on the touch screen, convert the touch action into a touch signal and send it to the processor 270, and can receive and execute the command sent by the processor 270; the touch signal at least includes the contact point coordinate information. The touch screen 231 can provide an input interface and an output interface between the terminal and the user. In addition, the touch screen can be implemented using multiple types such as resistive, capacitive, infrared, and surface acoustic waves. In addition to the touch screen 231, the input unit 230 can also include other input devices. Specifically, other input devices 232 may include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, and the like.

[0069] The display unit 240 may be used to display information input by a user or information provided to a user, various menus of the terminal, an interactive interface, file display, and / or playback of any multimedia file.

[0070] The memory 220 can be used to store instructions and data. The memory 220 can mainly include an instruction storage area and a data storage area. The data storage area can store various data, such as multimedia files, texts, etc. The instruction storage area can store software units such as operating systems, applications, and instructions required for at least one function, or their subsets and extensions. It can also include a non-volatile random access memory; it provides the processor 270 with hardware, software, and data resources including management of computing and processing equipment, and supports control software and applications. It is also used for the storage of multimedia files, and the storage of running programs and applications.

[0071] The processor 270 is the control center of the terminal. It uses various interfaces and lines to connect various parts of the entire terminal. By running or executing instructions stored in the memory 220 and calling data stored in the memory 220, it executes various functions of the terminal and processes data, thereby controlling the terminal as a whole. Optionally, the processor 270 may include one or more processing units; preferably, the processor 270 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application program, and the modem processor mainly processes wireless communication. It is understandable that the above-mentioned modem processor may not be integrated into the processor 270. In some embodiments, the processor and the memory may be implemented on a single chip, and in some embodiments, they may also be implemented separately on separate chips. The processor 270 may also be used to generate corresponding operation control signals, send them to corresponding components of the computing and processing device, read and process data in the software, especially read and process data and programs in the memory 220, so that each functional module therein performs corresponding functions, thereby controlling the corresponding components to act according to the requirements of the instructions.

[0072] Among them, the memory 220 can be used to store software codes related to the speech recognition method, the processor 270 can execute the software codes in the memory 220, and can also schedule other units (such as the above-mentioned input unit 230 and display unit 240) to achieve corresponding functions.

[0073] The radio frequency unit 210 (optional) can be used for receiving and sending information or receiving and sending signals during a call, for example, after receiving the downlink information of the base station, it is sent to the processor 270 for processing; in addition, the designed uplink data is sent to the base station. Generally, the radio frequency unit 210 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (LowNoiseAmplifier, LNA), a duplexer, etc. In addition, the radio frequency unit 210 can also communicate with network devices and other devices through wireless communication. The wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile communication (Global System of Mobile communication, GSM), General Packet Radio Service (General PacketRadio Service, GPRS), Code Division Multiple Access (Code Division Multiple Access, CDMA), Wideband Code Division Multiple Access (Wideband Code Division Multiple Access, WCDMA), Long Term Evolution (Long Term Evolution, LTE), email, Short Messaging Service (SMS), etc.

[0074] In the embodiment of the present application, the radio frequency unit 210 can send data to other devices, and can also receive data sent by other devices. It should be understood that the radio frequency unit 210 is optional, and can be replaced by other communication interfaces, such as a network port.

[0075] The terminal also includes a power supply 290 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 270 through a power management system, so that the power management system can manage charging, discharging, power consumption and other functions.

[0076] The terminal also includes an external interface 280, which may be a standard Micro USB interface or a multi-pin connector, and may be used to connect the terminal to communicate with other devices, or to connect a charger to charge the terminal.

[0077] Although not shown, the terminal may further include a flashlight, a wireless fidelity (WiFi) module, a Bluetooth module, sensors with different functions, etc., which will not be described in detail here.

[0078] Next, the product form of the above server is described.

[0079] Figure 3 A structural diagram of the above server is provided, such as Figure 3 As shown, the server may include a bus 301, a processor 302, a communication interface 303, and a memory 304. The processor 302, the memory 304, and the communication interface 303 communicate with each other via the bus 301.

[0080] The bus 301 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0081] The processor 302 may be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0082] The memory 304 may include a volatile memory, such as a random access memory (RAM). The memory 304 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0083] The memory 304 may be used to store software codes related to the speech recognition method. The processor 302 may call the software codes stored in the memory 304 or schedule other units to implement corresponding functions.

[0084] The processors in the above-mentioned terminals and servers (for example, processor 270 and processor 302) may be hardware circuits (such as application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), general-purpose processors, digital signal processors (DSP), microprocessors or microcontrollers, etc.), or combinations of these hardware circuits. For example, the processor may be a hardware system with an instruction execution function, such as a CPU, DSP, etc., or a hardware system without an instruction execution function, such as an ASIC, FPGA, etc., or a combination of the above-mentioned hardware systems without an instruction execution function and hardware systems with an instruction execution function.

[0085] The inventors of this case studied the existing speech recognition methods based on large language models and found that: in the existing speech recognition methods based on large language models, the acoustic information input into the large language model is frame-level acoustic information, which is inconsistent with the granularity of the modeling unit of the large language model. The inconsistency between the granularity of the acoustic information and the granularity of the modeling unit of the large language model will cause the large language model to be unable to understand the acoustic information well, and thus unable to give a more accurate recognition result. In addition, the existing speech recognition methods based on large language models also have the problems of unstoppable decoding and large language model hallucination. In view of the problems existing in the existing speech recognition methods based on large language models, the inventors of this case conducted research and, through continuous research, finally proposed a speech recognition method with better results.

[0086] Next, the speech recognition method provided by the present application is introduced through the following embodiments.

[0087] See also Figure 4 , shows a flow chart of a speech recognition method provided by an embodiment of the present application, the method may include:

[0088] Step S401: Acquire target voice data.

[0089] The target voice data is the voice data to be recognized.

[0090] Step S402: performing preliminary speech recognition on the target speech data to obtain word-level acoustic information of the target speech data and candidate speech recognition results.

[0091] When performing preliminary speech recognition on the target speech data, word-level acoustic information is first obtained for the target speech data, and then the candidate speech recognition results are determined based on the word-level acoustic information. It should be noted that when performing preliminary speech recognition, each time a word-level acoustic information is obtained, a candidate word can be determined based on the word-level acoustic information, and finally the candidate speech recognition results can be determined based on all the candidate words obtained.

[0092] Step S403: Project the word-level acoustic information of the target speech data into the representation space of the large language model to obtain the target acoustic features of the target speech data.

[0093] After obtaining the word-level acoustic information of the target speech data, the word-level acoustic information of the target speech data is projected into the representation space of the large language model so that it can adapt to the parameters and structure of the large language model, thereby enabling the large language model to better understand the information of the acoustic modality.

[0094] Step S404: using the large language model and the candidate speech recognition results of the target speech data, the target acoustic features of the target speech data are processed into a text sequence to obtain the speech recognition results of the target speech data.

[0095] Since the target acoustic features of the target speech data are word-level acoustic features, which are consistent with the granularity of the modeling units of the large language model, and the target acoustic features of the target speech data are acoustic features adapted to the parameters and structure of the large language model, the large language model can understand the target acoustic features well and give more accurate speech recognition results.

[0096] Considering that large language models usually use autoregressive decoding to decode acoustic features, and the autoregressive decoding method sometimes has the problem of being unable to stop decoding, usually a certain segment appears repeatedly, such as, "Today's weather is good, good, good, good, good...", "Do you think I am right, no, no, no, no...", in addition, large language models also have hallucination problems. For example, the user said "Spring sleep does not know the dawn", because the large language model is trained through a large amount of training data, it may decode "Spring sleep does not know the dawn, everywhere hear birds singing" based on the learned knowledge. In response to the above problems, this application introduces candidate speech recognition results. The introduction of candidate speech recognition results can assist the large language model to limit the decoding length, thereby effectively solving the problem of the large language model decoding cannot stop. At the same time, the introduction of candidate speech recognition results based on word-level acoustic information can suppress the occurrence of hallucinations in the large language model, so that the decoding can make full use of the language information of the large language model while respecting the acoustic information itself.

[0097] In this embodiment, the candidate speech recognition results that assist the large language model in obtaining the final speech recognition result may be the Nbest (for example, 5best) candidate speech recognition results obtained by performing preliminary speech recognition on the target speech data, that is, the best N (N is an integer greater than 1, and the specific value of N can be determined according to the actual application scenario) candidate speech recognition results among all the candidate speech recognition results obtained by performing preliminary speech recognition on the target speech data (all the obtained candidate speech recognition results are sorted in descending order of scores, and the candidate speech recognition results ranked in the top N positions are the Nbest candidate speech recognition results). Of course, this embodiment is not limited to this. For example, the candidate speech recognition result that assists the large language model in obtaining the final speech recognition result may also be the best candidate speech recognition result among all the candidate speech recognition results (that is, the candidate speech recognition result with the highest score).

[0098] The speech recognition method provided in the embodiment of the present application, after obtaining the target speech data, first performs preliminary speech recognition on the target speech data to obtain the word-level acoustic information of the target speech data and the candidate speech recognition results, then projects the word-level acoustic information of the target speech data into the representation space of the large language model to obtain the target acoustic features, and finally uses the large language model, assisted by the candidate speech recognition results of the target speech data, to process the target acoustic features into a text sequence to obtain the speech recognition results of the target speech data. Considering that the modeling granularity of the large language model is words, the speech recognition method provided in the present application obtains word-level acoustic information for the target speech data. In order to enable the large language model to better understand the information of the acoustic modality and thus provide accurate speech recognition results, the speech recognition method provided in the present application further projects the word-level acoustic information into the representation space of the large language model after obtaining the word-level acoustic information, so as to adapt it to the parameters and structure of the large language model. Considering the problems of hallucinations and unstoppable decoding in the large language model, the speech recognition method provided in the embodiment of the present application introduces candidate speech recognition results to assist the large language model in limiting the decoding length, so as to effectively solve the problems of hallucinations and unstoppable decoding in the large language model. In summary, the speech recognition method provided in the embodiment of the present application has a good speech recognition effect.

[0099] In another embodiment of the present application, the specific implementation process of "step S402: obtaining word-level acoustic information of the target speech data and candidate speech recognition results by performing preliminary speech recognition on the target speech data" in the above embodiment is introduced.

[0100] The process of performing preliminary speech recognition on the target speech data to obtain word-level acoustic information of the target speech data and candidate speech recognition results may include: obtaining frame-level acoustic information of the target speech data; determining speech frames belonging to the same word by determining whether each speech frame of the target speech data is a decoding position; fusing the acoustic information of the speech frames belonging to the same word to obtain word-level acoustic information; determining candidate words based on each word-level acoustic information obtained; and determining candidate speech recognition results based on all the obtained candidate words.

[0101] Among them, by determining whether each speech frame of the target speech data is a decoding position, the process of determining the speech frames belonging to the same word includes: for each speech frame of the target speech data, it can be judged whether the speech frame is a decoding position to obtain the decoding position judgment result of the speech frame, and then according to the decoding position judgment result of the speech frame of the target speech data, the speech frames belonging to the same word can be determined, and then the acoustic information of the speech frames belonging to the same word can be fused.

[0102] Exemplarily, the speech frames of the target speech data are f1, f2, f3, f4, f5, f6, f7, f8, f9, f10, f11, ... in sequence. Assuming that speech frames f1, f2, and f3 are not decoding positions, and f4 is a decoding position, it is determined that f1~f4 belong to the same word, and the acoustic features of speech frames f1~f4 are fused to obtain the first word-level acoustic information. Assuming that f5, f6, f7, f8, and f9 are not decoding positions, and f10 is a decoding position, it is determined that f5~f10 belong to the same word, and the acoustic features of f5~f10 are fused to obtain the second word-level acoustic information. The same is true for other speech frames, and the word-level acoustic information of the target speech data is obtained in this way.

[0103] In one possible implementation, a pre-trained speech recognition model based on a monotonic chunk attention (MoCha) module may be used to perform preliminary speech recognition on the target speech data to obtain word-level acoustic information of the target speech data and candidate speech recognition results.

[0104] Specifically, first, speech features (such as MFCC, FilterBank and other features) are extracted for each speech frame of the target speech data to obtain a speech feature sequence corresponding to the target speech data, and the speech feature sequence corresponding to the training speech data is input into a speech recognition model based on a monotonic block attention module. The speech recognition model obtains frame-level acoustic information according to the speech feature sequence corresponding to the training speech data, obtains word-level acoustic information according to the frame-level acoustic information, and determines the candidate speech recognition results according to the word-level acoustic information.

[0105] Among them, for each speech frame of the target speech data, the monotonic block attention module in the speech recognition model can predict the probability that the speech frame can be decoded, and can determine whether the speech frame is a decoding position based on the predicted probability to obtain the decoding position determination result of the speech frame, and then, according to the decoding position determination result of the speech frame of the target speech data, the speech frames belonging to the same word can be determined, and then the acoustic information of the speech frames belonging to the same word can be fused to obtain the word-level acoustic information of the target speech data.

[0106] Among them, when the acoustic information of speech frames belonging to the same word is fused, the energy of each speech frame can be obtained, and the energy of each speech frame is used as a weight. The acoustic information of the speech frames belonging to the same word is fused by a weighted summation fusion method. For example, if speech frames f1~f4 belong to the same word, the energy of speech frame f1 is determined as weight w1, the energy of speech frame f2 is determined as weight w2, the energy of speech frame f3 is determined as weight w3, and the energy of speech frame f4 is determined as weight w4. The acoustic information of speech frame f1 is weighted by weight w1, the acoustic information of speech frame f2 is weighted by weight w2, the acoustic information of speech frame f3 is weighted by weight w3, and the acoustic information of speech frame f4 is weighted by weight w4. The four weighted acoustic information are summed to obtain the acoustic information at the one-word level.

[0107] In another embodiment of the present application, the specific implementation process of "Step S403: projecting the word-level acoustic information of the target speech data into the representation space of the large language model to obtain the target acoustic features of the target speech data" in the above embodiment is introduced.

[0108] In one possible implementation, the process of projecting the word-level acoustic information of the target speech data into the representation space of the large language model to obtain the target acoustic features of the target speech data may include: using a pre-trained connection module to project the word-level acoustic information of the target speech data into the representation space of the large language model to obtain acoustic features adapted to the large language model as the target acoustic features of the target speech data.

[0109] The connection module is trained on a speech recognition task based on a large language model using training speech data annotated with text. The connection module trained on a speech recognition task based on a large language model using training speech data annotated with text can adapt and transform the input acoustic information to match the parameters and structure of the large language model, so that the large language model can better understand the information of the acoustic modality.

[0110] Optionally, the connection module may adopt a linear layer, an attention layer, a Q-Former (Q-Former is a lightweight transformer), etc. Of course, this embodiment is not limited to this. For example, the connection module may also adopt a linear layer and a Conformer (such as 2 layers of linear layers and 2 layers of Conformer). In actual application, the specific structure of the connection module can be adjusted according to the amount of data and the complexity of the task.

[0111] In another embodiment of the present application, the specific implementation process of "Step S404: using a large language model, assisted by the candidate speech recognition results of the target speech data, processing the target acoustic features of the target speech data into a text sequence, and obtaining the speech recognition results of the target speech data" in the above embodiment is introduced.

[0112] The process of processing the target acoustic features of the target voice data into a text sequence using a large language model and the candidate voice recognition results of the target voice data to obtain the voice recognition results of the target voice data may include: integrating the target acoustic features of the target voice data with the candidate voice recognition results of the target voice data through a preconfigured Prompt instruction template to obtain an integrated result; and inputting the integrated result into the large language model to obtain the voice recognition results of the target voice data.

[0113] See also Figure 5 , shows a flow chart of a more specific implementation method of using a large language model, supplemented by the candidate speech recognition results of the target speech data, processing the target acoustic features of the target speech data into a text sequence, and obtaining the speech recognition results of the target speech data, which may include:

[0114] Step S501: converting the candidate speech recognition results of the target speech data into features suitable for processing by a large language model, and obtaining features corresponding to the candidate speech recognition results of the target speech data.

[0115] like Figure 6 As shown, a large language model embedding layer (LLM embedding layer) can be used to convert the candidate speech recognition results of the target speech data into features suitable for processing by the large language model, thereby obtaining features corresponding to the candidate speech recognition results of the target speech data.

[0116] Step S502: convert the preconfigured Prompt instruction template into features suitable for processing by a large language model, and obtain features corresponding to the preconfigured Prompt instruction template.

[0117] Among them, the preconfigured Prompt instruction template includes an acoustic information slot and a recognition auxiliary information slot. The preconfigured Prompt instruction template is used to instruct the large language model to give a speech recognition result based on the information in the acoustic information slot and the information in the recognition auxiliary information slot.

[0118] like Figure 6 As shown, the large language model embedding layer (LLM embedding layer) can be used to convert the preconfigured Prompt instruction template into features suitable for large language model processing, thereby obtaining features corresponding to the preconfigured Prompt instruction template.

[0119] In a possible implementation, the preconfigured Prompt instruction template may be a simplified Prompt instruction template, such as: “Audio content: <speechhere>\nBest hypothesis:\n <besthyphere>\nOther assumptions:\n <otherhyphere> \n <labelhere>”.

[0120] In order to obtain better speech recognition effect, in another possible implementation, the preconfigured Prompt command template can be a perfect Prompt command template, such as: "User: You are a model used for speech recognition result error correction. Please correct the insertion, deletion, and substitution errors in the result according to the hypothesis generated by the speech recognition system, including 1 line of best hypothesis and 4 lines of other hypotheses, combined with the audio content, and only return the recognition result:\nAudio content: <speechhere>\nBest hypothesis:\n <besthyphere>\nOther assumptions:\n <otherhyphere>\nAssitant:\n <labelhere>”.

[0121] in, <speechhere>is the acoustic information slot, <speechhere>Insert acoustic information at <besthyphere>and <otherhyphere>To identify the auxiliary information slot, <besthyphere>Insert the best candidate speech recognition result among the Nbest candidate speech recognition results, <otherhyphere>Insert other candidate speech recognition results in Nbest candidate speech recognition results, <labelhere>The text sequence that needs to be predicted.

[0122] Compared with the simplified Prompt command template, the complete Prompt command template contains detailed task description information, which can prompt the large language model to pay attention to the information in the acoustic information slot and the recognition auxiliary information slot, and prompt the large language model how to process the information in the acoustic information slot and the recognition auxiliary information slot. The introduction of task description information can improve the accuracy of the large language model in speech recognition combined with the target acoustic features. It should be noted that the above-mentioned Prompt command template is only an example. In actual application, the Prompt command template can be set according to specific application scenarios and requirements.

[0123] In addition, it should be noted that this embodiment does not limit the execution order of step S501 and step S502. Step S501 may be executed first, and then step S502, or step S502 may be executed first, and then step S501.

[0124] Step S503: According to the positions of the acoustic information slot and the recognition auxiliary information slot, the target acoustic features and the features corresponding to the candidate speech recognition results of the target speech data are inserted into the features corresponding to the preconfigured Prompt instruction template to obtain the combined features.

[0125] like Figure 6 As shown, according to the positions of the acoustic information slot and the recognition auxiliary information slot in the preconfigured Prompt instruction template, the target acoustic features of the target speech data, the features corresponding to the preconfigured Prompt instruction template, and the features corresponding to the candidate speech recognition results of the target speech data are combined to obtain the combined features.

[0126] Specifically, according to the position of the acoustic information slot in the preconfigured Prompt command template, the target acoustic feature is inserted into the feature corresponding to the Prompt command template; according to the position of the recognition auxiliary information slot in the preconfigured Prompt command template, the feature corresponding to the candidate speech recognition result of the target speech data is inserted into the feature corresponding to the preconfigured Prompt command template.

[0127] The above embodiment mentioned that the candidate speech recognition results that assist the large language model in obtaining the final speech recognition results may be the best N candidate speech recognition results. In view of this, the recognition auxiliary information slot in the preconfigured Prompt instruction template may include a first auxiliary information slot and a second auxiliary information slot. Furthermore, when inserting the features corresponding to the candidate speech recognition results of the target speech data into the features corresponding to the preconfigured Prompt instruction template according to the position of the recognition auxiliary information slot in the preconfigured Prompt instruction template, according to the position of the first auxiliary information slot, the features corresponding to the best candidate speech recognition result among the N candidate speech recognition results are inserted into the features corresponding to the Prompt instruction template, and according to the position of the second auxiliary information slot, the features corresponding to other candidate speech recognition results among the N candidate speech recognition results are inserted into the features corresponding to the Prompt instruction template.

[0128] Step S504: input the combined features into a large language model to obtain a speech recognition result of the target speech data.

[0129] After the combined features are input into the large language model, the large language model determines the final speech recognition result of the target speech data based on the input features.

[0130] In this embodiment, the large language model can be a pre-trained large language model, or a large language model obtained by fine-tuning the pre-trained large language model on a speech recognition task. In order to obtain a better speech recognition effect, the latter is preferred.

[0131] It can be known from the contents provided in the above embodiments that the speech recognition method provided in the present application can be implemented based on a speech recognition model (such as a speech recognition model based on a monotonic block attention module), a connection module, a large language model, etc., wherein the connection module uses training speech data annotated with text and is trained on a speech recognition task based on a large language model. In order to obtain a better speech recognition effect, the large language model uses training speech data annotated with text and further trains the pre-trained large language model on a speech recognition task. Next, the training process of the connection module and the large language model is introduced.

[0132] like Figure 7 As shown, the training process of the connection module and the large language model may include:

[0133] Step S701: fix the parameters of the speech recognition model and the large language model, use the training speech data annotated with text, and train the connection module on the speech recognition task.

[0134] Specifically, using training speech data annotated with text, the process of training the connection module on the speech recognition task includes:

[0135] Step a1: extract speech features from each speech frame of the training speech data to obtain a speech feature sequence corresponding to the training speech data.

[0136] The speech features extracted from each speech frame of the training speech data may be, but are not limited to, MFCC, FilterBank, and other features. The speech feature sequence corresponding to the training speech data can be expressed as X={x1,…, x t ,…, x T }, where x t Represents the speech features of the tth speech frame of the training speech data.

[0137] Step a2: input the speech feature sequence corresponding to the training speech data into the speech recognition model to obtain the word-level acoustic information of the training speech data and the candidate speech recognition results.

[0138] The speech feature sequence X={x1,…, x t ,…, x T Input a speech recognition model based on a monotonic block attention module, which obtains frame-level acoustic information according to the input speech feature sequence, and determines the speech frames belonging to the same word by judging whether each speech frame of the training speech data is a decoding position. The acoustic information of the speech frames belonging to the same word is fused to obtain the word-level acoustic information of the training speech data, and then the candidate speech recognition results of the training speech data are determined according to the word-level acoustic information of the training speech data.

[0139] Step a3: Based on the connection module, the word-level acoustic information of the training speech data is projected into the representation space of the large language model to obtain the target acoustic features of the training speech data.

[0140] The word-level acoustic information of the training speech data is input into the connection module, and the connection module processes the input word-level acoustic information into features that are adapted to the parameters and structure of the large language model to obtain the target acoustic features of the training speech data.

[0141] Step a3: Use the large language model, supplemented by the candidate speech recognition results of the training speech data (such as the 5 best candidate speech recognition results), to predict the text sequence (i.e., speech recognition results) corresponding to the training speech data, and obtain the text sequence prediction results.

[0142] It should be noted that the introduction of candidate speech recognition results can not only limit the decoding length of the large language model, but also assist the connection module in learning the mapping relationship.

[0143] Specifically, the process of using the large language model, supplemented by the candidate speech recognition results of the training speech data (such as the 5best candidate speech recognition results), to predict the text sequence corresponding to the training speech data may include: based on the large language model embedding layer, converting the candidate speech recognition results of the training speech data (such as the 5best candidate speech recognition results) into features suitable for processing by the large language model, and obtaining features corresponding to the candidate speech recognition results of the training speech data; based on the large language model embedding layer, converting the annotated text of the training speech data into features suitable for processing by the large language model, and obtaining features corresponding to the annotated text of the training speech data; based on the large language model embedding layer, converting the preconfigured Prompt instruction model into features suitable for processing by the large language model, and obtaining features corresponding to the annotated text of the training speech data. The Prompt instruction template is converted into features suitable for processing by a large language model to obtain features corresponding to the Prompt instruction template, wherein the Prompt instruction template includes an acoustic information slot, a recognition auxiliary information slot and a text sequence information slot, and the Prompt instruction template is used to instruct the large language model to give a speech recognition result according to the information in the acoustic information slot and the information in the recognition auxiliary information slot; according to the positions of the acoustic information slot, the recognition auxiliary information slot and the text sequence information slot, the target acoustic features of the training speech data and the features corresponding to the annotated text of the training speech data are inserted into the features corresponding to the Prompt instruction template to obtain combined features; the combined features are input into the large language model to obtain a text sequence prediction result.

[0144] Step a4: Determine the prediction loss based on the text sequence prediction result and the text annotated by the training speech data.

[0145] Among them, the prediction loss can use cross entropy loss, and the calculation method of cross entropy loss is a prior art, which will not be described in detail in this embodiment.

[0146] Step a5: Update the parameters of the connection module according to the predicted loss.

[0147] The connection module is trained multiple times according to the process of steps a1 to a5 above until the training end condition is met.

[0148] Step S702: fix the parameters of the speech recognition model and the connection module, use the training speech data annotated with text, and train the large language model on the speech recognition task.

[0149] The process of training the large language model on the speech recognition task using the training speech data annotated with text is similar to the process of training the connection module on the speech recognition task using the training speech data annotated with text. It is also necessary to determine the prediction loss according to the process of steps a1 to a4. After the prediction loss is determined, the parameters of the large language model are updated according to the prediction loss. When updating the parameters of the large language model, only some parameters of the large language model (such as the parameters of the linear layer of the large language model) may be updated.

[0150] Step S703: fix the parameters of the speech recognition model, use the training speech data annotated with text, and train the connection module and the large language model on the speech recognition task at the same time.

[0151] The process of simultaneously training the connection module and the large language model on the speech recognition task using the training speech data annotated with text is similar to the process of training the connection module on the speech recognition task using the training speech data annotated with text. It is also necessary to determine the prediction loss according to the process of steps a1 to a4. After the prediction loss is determined, the parameters of the connection module and the large language model are updated according to the prediction loss.

[0152] It should be noted that the speech recognition model is pre-trained, and the parameters of the speech recognition model remain fixed during the training of the connection module and the large language model on the speech recognition task.

[0153] It should also be noted that this embodiment is not limited to the use of the above-mentioned steps S701 to S703 to train the connection module and the large language model. For example, training speech data annotated with text can also be used to train the connection module and the large language model on the speech recognition task at the same time. It is also possible to first fix the parameters of the speech recognition model and the connection module, use the training speech data annotated with text, and fine-tune the large language model on the speech recognition task, and then fix the parameters of the speech recognition model and the large language model, use the training speech data annotated with text, and train the connection module on the speech recognition task. Finally, fix the parameters of the speech recognition model and the connection module, use the training speech data annotated with text, and train the connection module on the speech recognition task at the same time.

[0154] After the training is completed, the trained connection module, large language model, and speech recognition model based on monotonic block attention module can be used to perform speech recognition on the target speech data.

[0155] The following table shows the comparison of recognition effects of several speech recognition schemes:

[0156] Table 1 Comparison of recognition effects of several speech recognition schemes

[0157]

[0158] Among them, the recognition scheme of "MoCha ASR" refers to the use of only the speech recognition model based on the monotonic block attention module to perform speech recognition on the speech to be recognized. The recognition scheme of "ENC+LLM" refers to the existing recognition scheme based on the large language model, that is, the acoustic model is used to obtain frame-level acoustic information for the speech data to be recognized, and the large language model obtains the speech recognition result based on the frame-level acoustic information. "(MoCha+LLM) 1 ”~”(MoCha+LLM) 4 "Both are solutions that use a speech recognition model based on a monotonic block attention module and a large language model to perform speech recognition on the speech to be recognized," (MoCha+LLM) 1 ”~”(MoCha+LLM) 4 The difference between "(MoCha+LLM)" is that 1 "The recognition scheme does not use the candidate speech recognition results. The Prompt command template used is a simplified Prompt command template. The large language model used is a pre-trained large language model (without fine-tuning). "(MoCha+LLM) 2 The recognition scheme of "(MoCha+LLM)" uses the candidate speech recognition results, the Prompt command template used is a simplified Prompt command template, and the large language model used is a pre-trained large language model (without fine-tuning). 3 The recognition scheme of "(MoCha+LLM)" uses the candidate speech recognition results, the Prompt command template used is the perfect Prompt command template, and the large language model used is the pre-trained large language model (without fine-tuning). 4 The recognition scheme of " utilizes the candidate speech recognition results. The Prompt command template used is the perfect Prompt command template. The large language model used is the large language model that is fine-tuned on the pre-trained large language model in the speech recognition task. As can be seen from the above table, speech recognition based on word-level acoustic features and supplemented by candidate speech recognition results can improve the accuracy of speech recognition. In addition, compared with the streamlined Prompt command template, the use of the perfect Prompt command template can improve the speech recognition effect. Compared with the large language model that has not been fine-tuned, the use of the fine-tuned large language model for speech recognition can improve the speech recognition effect.

[0159] The above introduces the speech recognition method provided in the embodiment of the present application, and the device corresponding to the above speech recognition method will be introduced below.

[0160] See also Figure 8 , Figure 8 A structural schematic diagram of a speech recognition device provided in an embodiment of the present application, the speech recognition device may include: a speech data acquisition module 801, a speech recognition information acquisition module 802, an acoustic information processing module 803 and a speech recognition result acquisition module 804.

[0161] The voice data acquisition module 801 is used to acquire target voice data.

[0162] The speech recognition information acquisition module 802 is used to acquire word-level acoustic information of the target speech data and candidate speech recognition results by performing preliminary speech recognition on the target speech data.

[0163] The acoustic information processing module 803 is used to project the word-level acoustic information of the target speech data into the representation space of the large language model to obtain the target acoustic features of the target speech data.

[0164] The speech recognition result acquisition module 804 is used to use the large language model, supplemented by the candidate speech recognition results of the target speech data, to process the target acoustic features of the target speech data into a text sequence to obtain the speech recognition results of the target speech data.

[0165] In a possible implementation, when acquiring the word-level acoustic information of the target speech data, the speech recognition information acquisition module 802 is specifically configured to:

[0166] Acquiring frame-level acoustic information for target speech data;

[0167] Determine the speech frames belonging to the same word by determining whether each speech frame of the target speech data is a decoding position;

[0168] The acoustic information of speech frames belonging to the same word is fused to obtain the word-level acoustic information of the target speech data.

[0169] In a possible implementation, when the speech recognition information acquisition module 802 performs preliminary speech recognition on the target speech data to acquire word-level acoustic information of the target speech data and candidate speech recognition results, it is specifically used to:

[0170] By using a pre-trained speech recognition model based on a monotonic block attention module to perform preliminary speech recognition on the target speech data, the word-level acoustic information of the target speech data and the candidate speech recognition results are obtained.

[0171] In a possible implementation, the speech recognition result acquisition module 804 uses the large language model, supplemented by the candidate speech recognition results of the target speech data, to process the target acoustic features of the target speech data into a text sequence to obtain the speech recognition results of the target speech data, specifically for:

[0172] By using a preconfigured Prompt command template, the target acoustic features of the target speech data are integrated with the candidate speech recognition results of the target speech data to obtain an integrated result;

[0173] The integration result is input into the large language model to obtain the speech recognition result of the target speech data of the large language model.

[0174] In a possible implementation, the speech recognition result acquisition module 804 integrates the target acoustic features of the target speech data with the candidate speech recognition results of the target speech data through the preconfigured Prompt instruction template to obtain the integrated result, specifically for:

[0175] Converting the candidate speech recognition results of the target speech data into features suitable for processing by the large language model to obtain features corresponding to the candidate speech recognition results of the target speech data;

[0176] Convert the preconfigured Prompt instruction template into features suitable for processing by the large language model to obtain features corresponding to the Prompt instruction template, wherein the Prompt instruction template includes an acoustic information slot and an identification auxiliary information slot, and the Prompt instruction template is used to instruct the large language model to give a speech recognition result according to the information in the acoustic information slot and the information in the identification auxiliary information slot;

[0177] According to the positions of the acoustic information slot and the recognition auxiliary information slot, the target acoustic features of the target voice data and the features corresponding to the candidate voice recognition results of the target voice data are inserted into the features corresponding to the Prompt instruction template to obtain the combined features.

[0178] In a possible implementation, the Prompt instruction template also includes: task description information, the task description information is used to prompt the large language model to pay attention to the information in the acoustic information slot and the recognition auxiliary information slot, and to prompt the large language model how to process the information in the acoustic information slot and the recognition auxiliary information slot.

[0179] In a possible implementation, the candidate speech recognition results of the target speech data are the best N candidate speech recognition results, where N is an integer greater than 1.

[0180] The identification auxiliary information slot in the Prompt instruction template includes a first auxiliary information slot and a second auxiliary information slot.

[0181] The speech recognition result acquisition module 804 inserts the target acoustic features of the target speech data and the features corresponding to the candidate speech recognition results of the target speech data into the features corresponding to the Prompt instruction template according to the positions of the acoustic information slot and the recognition auxiliary information slot, and when the combined features are obtained, is specifically used to: insert the target acoustic features of the target speech data into the features corresponding to the Prompt instruction template according to the positions of the acoustic information slot;

[0182] According to the position of the first auxiliary information slot, insert the feature corresponding to the best candidate speech recognition result among the N candidate speech recognition results into the feature corresponding to the Prompt instruction template;

[0183] According to the position of the second auxiliary information slot, features corresponding to other candidate speech recognition results among the N candidate speech recognition results are inserted into the features corresponding to the Prompt instruction template.

[0184] In a possible implementation, when the acoustic information processing module 803 projects the word-level acoustic information of the target speech data into the representation space of the large language model to obtain the target acoustic features of the target speech data, it is specifically used to:

[0185] Using the pre-trained connection module, the word-level acoustic information of the target speech data is projected into the representation space of the large language model to obtain acoustic features adapted to the large language model as the target acoustic features of the target speech data.

[0186] Among them, the connection module is trained on the speech recognition task based on the large language model using training speech data annotated with text.

[0187] In a possible implementation, the large language model is obtained by training a pre-trained large language model on a speech recognition task using training speech data annotated with text.

[0188] The speech recognition device may further include: a training module.

[0189] The training module is used to train the connection module and the large language model.

[0190] When training the connection module and the large language model, the training module is specifically used to:

[0191] The parameters of the large language model are fixed, and the connection module is trained on the speech recognition task using training speech data annotated with text.

[0192] The parameters of the connection module are fixed, and the large language model is trained on the speech recognition task using the training speech data annotated with text;

[0193] Using training speech data annotated with text, the connection module and the large language model are trained simultaneously on the speech recognition task.

[0194] The speech recognition device provided in the embodiment of the present application, after obtaining the target speech data, first performs preliminary speech recognition on the target speech data to obtain the word-level acoustic information of the target speech data and the candidate speech recognition results, then projects the word-level acoustic information of the target speech data into the representation space of the large language model to obtain the target acoustic features, and finally uses the large language model, assisted by the candidate speech recognition results of the target speech data, to process the target acoustic features into a text sequence to obtain the speech recognition results of the target speech data. Considering that the modeling granularity of the large language model is word, the speech recognition device provided in the present application obtains word-level acoustic information for the target speech data. In order to enable the large language model to better understand the information of the acoustic modality and thus give accurate speech recognition results, the speech recognition device provided in the present application further projects the word-level acoustic information to the representation space of the large language model after obtaining the word-level acoustic information, so as to adapt it to the parameters and structure of the large language model. Considering the problems of hallucinations and unstoppable decoding of the large language model, the speech recognition device provided in the present application introduces candidate speech recognition results to assist the large language model in limiting the decoding length, so as to effectively solve the problems of hallucinations and unstoppable decoding of the large language model. In summary, the speech recognition device provided in the present application has a good speech recognition effect.

[0195] An embodiment of the present application also provides an electronic device, which may include: at least one processor, at least one communication interface, at least one memory and at least one communication bus.

[0196] In the embodiment of the present application, the number of the processor, the communication interface, the memory, and the communication bus is at least one, and the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0197] The processor may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application;

[0198] The memory may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory;

[0199] The memory stores a program, and the processor can call the program stored in the memory, and the program is used to implement the steps of the speech recognition method provided in the above embodiment.

[0200] An embodiment of the present application also provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the steps of the speech recognition method provided in the above embodiment.

[0201] An embodiment of the present application also provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the steps of the speech recognition method provided in the above embodiment.

[0202] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0203] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0204] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0205] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.< / labelhere> < / otherhyphere> < / besthyphere> < / otherhyphere> < / besthyphere> < / speechhere> < / speechhere> < / labelhere> < / otherhyphere> < / besthyphere> < / speechhere> < / labelhere> < / otherhyphere> < / besthyphere> < / speechhere>

Claims

1. A speech recognition method, characterized in that: include: Obtain target voice data; By performing preliminary speech recognition on the target speech data, word-level acoustic information of the target speech data and candidate speech recognition results are obtained; Projecting the word-level acoustic information of the target speech data into the representation space of the large language model to obtain target acoustic features of the target speech data; The target acoustic features of the target voice data are processed into a text sequence using the large language model and the candidate voice recognition results of the target voice data to obtain the voice recognition results of the target voice data.

2. The speech recognition method according to claim 1, characterized in that: Acquiring word-level acoustic information of the target speech data includes: Acquiring frame-level acoustic information for the target speech data; Determine the speech frames belonging to the same word by determining whether each speech frame of the target speech data is a decoding position; The acoustic information of the speech frames belonging to the same word is fused to obtain the word-level acoustic information of the target speech data.

3. The speech recognition method according to claim 1, characterized in that: The step of performing preliminary speech recognition on the target speech data to obtain word-level acoustic information of the target speech data and candidate speech recognition results includes: The target speech data is subjected to preliminary speech recognition using a pre-trained speech recognition model based on a monotone block attention module, thereby obtaining word-level acoustic information of the target speech data and candidate speech recognition results.

4. The speech recognition method according to claim 1, characterized in that: The method of using the large language model, supplemented by the candidate speech recognition results of the target speech data, processing the target acoustic features of the target speech data into a text sequence to obtain the speech recognition results of the target speech data includes: Integrate the target acoustic features of the target voice data with the candidate voice recognition results of the target voice data through a preconfigured Prompt instruction template to obtain an integrated result; The integration result is input into the large language model to obtain the speech recognition result of the target speech data output by the large language model.

5. The speech recognition method according to claim 4, characterized in that: The target acoustic features of the target voice data and the candidate voice recognition results of the target voice data are integrated through the preconfigured Prompt instruction template to obtain an integrated result, including: Converting the candidate speech recognition results of the target speech data into features suitable for processing by the large language model to obtain features corresponding to the candidate speech recognition results of the target speech data; Converting a preconfigured Prompt instruction template into features suitable for processing by the large language model to obtain features corresponding to the Prompt instruction template, wherein the Prompt instruction template includes an acoustic information slot and a recognition auxiliary information slot, and the Prompt instruction template is used to instruct the large language model to give a speech recognition result according to the information in the acoustic information slot and the information in the recognition auxiliary information slot; According to the positions of the acoustic information slot and the recognition auxiliary information slot, the target acoustic features of the target voice data and the features corresponding to the candidate voice recognition results of the target voice data are inserted into the features corresponding to the Prompt instruction template to obtain the combined features.

6. The speech recognition method according to claim 5, characterized in that: The Prompt instruction template also includes: task description information; The task description information is used to prompt the large language model to pay attention to the information in the acoustic information slot and the recognition auxiliary information slot, and to prompt the large language model how to process the information in the acoustic information slot and the recognition auxiliary information slot.

7. The speech recognition method according to claim 5, characterized in that: The candidate speech recognition results of the target speech data are the best N candidate speech recognition results, where N is an integer greater than 1; The identification auxiliary information slot includes a first auxiliary information slot and a second auxiliary information slot; The step of inserting the target acoustic features of the target speech data and the features corresponding to the candidate speech recognition results of the target speech data into the features corresponding to the Prompt instruction template according to the positions of the acoustic information slot and the recognition auxiliary information slot to obtain the combined features includes: inserting the target acoustic features of the target speech data into the features corresponding to the Prompt instruction template according to the positions of the acoustic information slot; According to the position of the first auxiliary information slot, inserting a feature corresponding to the best candidate speech recognition result among the N candidate speech recognition results into the feature corresponding to the Prompt instruction template; According to the position of the second auxiliary information slot, features corresponding to other candidate speech recognition results among the N candidate speech recognition results are inserted into the features corresponding to the Prompt instruction template.

8. The speech recognition method according to claim 1, characterized in that: The step of projecting the word-level acoustic information of the target speech data into the representation space of the large language model to obtain the target acoustic features of the target speech data includes: Using the pre-trained connection module, the word-level acoustic information of the target speech data is projected into the representation space of the large language model to obtain acoustic features adapted to the large language model as target acoustic features of the target speech data; The connection module is trained on a speech recognition task based on a large language model using training speech data annotated with text.

9. The speech recognition method according to claim 8, characterized in that: The large language model is obtained by training a pre-trained large language model on a speech recognition task using training speech data annotated with text; The training process of the connection module and the large language model includes: The parameters of the large language model are fixed, and the connection module is trained on the speech recognition task using training speech data annotated with text. The parameters of the connection module are fixed, and the large language model is trained on the speech recognition task using the training speech data annotated with text; Using training speech data annotated with text, the connection module and the large language model are trained simultaneously on the speech recognition task.

10. A speech recognition device, characterized in that: include: Speech data acquisition module, speech recognition information acquisition module, acoustic information processing module and speech recognition result acquisition module; A voice data acquisition module, used to acquire target voice data; A speech recognition information acquisition module, configured to acquire word-level acoustic information of the target speech data and candidate speech recognition results by performing preliminary speech recognition on the target speech data; The acoustic information processing module is used to project the word-level acoustic information of the target speech data into the representation space of the large language model to obtain the target acoustic features of the target speech data; The speech recognition result acquisition module is used to use the large language model, assisted by the candidate speech recognition results of the target speech data, to process the target acoustic features of the target speech data into a text sequence to obtain the speech recognition results of the target speech data.

11. An electronic device, characterized in that: The method comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the steps of the speech recognition method as described in any one of claims 1 to 9.

12. A computer storage medium, characterized in that: The storage medium carries one or more computer programs, and when the one or more computer programs are executed by an electronic device, the electronic device can implement the steps of the speech recognition method as described in any one of claims 1 to 9.

13. A computer program product, characterized in that The method comprises computer-readable instructions, which, when executed on an electronic device, enable the electronic device to implement the steps of the speech recognition method as claimed in any one of claims 1 to 9.

Images