Speech recognition method and related device

Abstract

The invention discloses a voice recognition method and a related device, and relates to the technical field of voice recognition, and the method comprises the steps: obtaining a target voice; performing voice recognition on the voice recognition model at the target voice input end side, and transmitting the target voice to the cloud side, so that the voice recognition model at the cloud side performs voice recognition on the target voice; after the voice recognition result of the end side is obtained, whether the voice recognition result of the end side is reliable or not is judged through the large language model; and if the speech recognition result of the end side is reliable, determining the speech recognition result of the end side as a target speech recognition result, otherwise, determining the speech recognition result of the cloud side as the target speech recognition result. The speech recognition method disclosed by the invention is an end-cloud collaborative speech recognition method, the method can accurately decide whether the end-side speech recognition result or the cloud-side speech recognition result is finally adopted according to the reliability judgment result of the large language model to the end-side speech recognition result, and the speech recognition method disclosed by the invention has a relatively good speech recognition effect.

Description

Technical Field

[0001] The present application relates to the field of speech recognition technology, and in particular to a speech recognition method and related devices. Background Art

[0002] Speech recognition technology is a technology that converts speech into text. Speech recognition technology is widely used and has penetrated into our daily life and work.

[0003] In some scenarios (such as car-based scenarios), voice recognition is usually performed on the device side. Performing voice recognition on the device side has the advantages of low latency and protection of data privacy and security.

[0004] However, since the voice recognition model on the end side is usually small, poor recognition effect may occur at certain times, and poor voice recognition effect on the end side will affect the user experience. Summary of the invention

[0005] In view of this, the present application provides a speech recognition method and related devices, which are used to solve the problem that the speech recognition solution on the terminal side has poor recognition effect and affects the user experience. The technical solution is as follows:

[0006] The first aspect of the present application provides a speech recognition method, comprising:

[0007] Get the target voice;

[0008] Performing speech recognition on the speech recognition model on the target speech input terminal side, and transmitting the target speech to the cloud side, so that the speech recognition model on the cloud side performs speech recognition on the target speech;

[0009] After obtaining the speech recognition result on the end side, use the large language model to determine whether the speech recognition result on the end side is reliable, wherein the large language model is obtained by fine-tuning the pre-trained large language model on the speech recognition task of end-cloud collaboration, and the large language model has the ability to determine the reliability of the speech recognition result;

[0010] If the speech recognition result on the terminal side is reliable, determining the speech recognition result on the terminal side as the target speech recognition result;

[0011] If the speech recognition result on the terminal side is unreliable, after obtaining the speech recognition result on the cloud side, the speech recognition result on the cloud side is determined as the target speech recognition result.

[0012] In a possible implementation, the speech recognition method further includes:

[0013] When the speech recognition result on the terminal side is unreliable, extracting entity words from the speech recognition result on the cloud side, and determining the similarity between the speech recognition result on the terminal side and the speech recognition result on the cloud side;

[0014] Determine whether it is necessary to immediately update the hot word library on the terminal side according to the similarity between the speech recognition result on the terminal side and the speech recognition result on the cloud side;

[0015] If it is determined that the hot word library on the end side does not need to be updated immediately, the extracted entity words are stored in the entity word database; if it is determined that the hot word library on the end side needs to be updated immediately, the extracted entity words and the entity words in the entity word database are added to the hot word library on the end side.

[0016] In a possible implementation, the large language model also has the ability to identify entity words from speech recognition results and determine the similarity between two speech recognition results;

[0017] The extracting entity words from the speech recognition result on the cloud side and determining the similarity between the speech recognition result on the terminal side and the speech recognition result on the cloud side includes:

[0018] Utilizing the large language model, entity words are extracted from the speech recognition result on the cloud side, and the similarity between the speech recognition result on the terminal side and the speech recognition result on the cloud side is determined.

[0019] In a possible implementation, judging whether it is necessary to immediately update the hot word library on the terminal side according to the similarity between the speech recognition result on the terminal side and the speech recognition result on the cloud side includes:

[0020] If the similarity between the speech recognition result on the terminal side and the speech recognition result on the cloud side is greater than or equal to a preset similarity threshold, it is determined that there is no need to immediately update the hot word library on the terminal side;

[0021] If the similarity between the speech recognition result on the terminal side and the speech recognition result on the cloud side is less than a preset similarity threshold, it is determined that the hot word library on the terminal side needs to be updated immediately.

[0022] In a possible implementation, the speech recognition method further includes:

[0023] After storing the extracted entity words in the entity word database, counting the number of entity words in the entity word database;

[0024] If the number of entity words in the entity word database is greater than or equal to a preset entity word number threshold, the entity words in the entity word database are added to the hot word library on the terminal side, and the entity words in the entity word database are deleted.

[0025] In a possible implementation, using the large language model to extract entity words from the speech recognition result on the cloud side, and determining the similarity between the speech recognition result on the terminal side and the speech recognition result on the cloud side, includes:

[0026] Using the feature acquisition module of the large language model, extract features from the speech recognition result on the terminal side and the speech recognition result on the cloud side, respectively, to obtain features of the speech recognition result on the terminal side and features of the speech recognition result on the cloud side;

[0027] Using the data distribution module of the large language model, the features of the speech recognition result on the cloud side are distributed to the entity word recognition module of the large language model, and the features of the speech recognition result on the terminal side and the features of the speech recognition result on the cloud side are distributed to the similarity determination module of the large language model;

[0028] Using the entity word recognition module of the large language model, based on the characteristics of the speech recognition results on the cloud side, the entity words in the speech recognition results on the cloud side are recognized;

[0029] The similarity determination module of the large language model is utilized to determine the similarity between the speech recognition result on the terminal side and the speech recognition result on the cloud side based on the characteristics of the speech recognition result fed back by the cloud side and the characteristics of the speech recognition result on the terminal side.

[0030] A second aspect of the present application provides a speech recognition device, including: a speech acquisition module, a speech processing module, a terminal-side speech recognition result reliability determination module, and a target speech recognition result determination module;

[0031] The speech acquisition module is used to acquire the target speech;

[0032] The speech processing module is used to perform speech recognition on the speech recognition model on the target speech input terminal side, and transmit the target speech to the cloud side, so that the speech recognition model on the cloud side performs speech recognition on the target speech;

[0033] The device-side speech recognition result reliability determination module is used to use a large language model to determine whether the device-side speech recognition result is reliable after obtaining the device-side speech recognition result, wherein the large language model is obtained by fine-tuning a pre-trained large language model on a device-cloud collaborative speech recognition task, and the large language model has the ability to determine the reliability of the speech recognition result;

[0034] The target speech recognition result determination module is used to determine the speech recognition result on the terminal side as the target speech recognition result when the speech recognition result on the terminal side is reliable, and to determine the speech recognition result on the cloud side as the target speech recognition result after obtaining the speech recognition result on the cloud side when the speech recognition result on the terminal side is unreliable.

[0035] A third aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0036] The memory is used to store computer programs;

[0037] The processor is used to execute the computer program so that the electronic device can implement the steps of any one of the above-mentioned speech recognition methods.

[0038] In a fourth aspect of the present application, a computer storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the steps of any one of the above-mentioned speech recognition methods.

[0039] A fifth aspect of the present application provides a computer program product, comprising computer-readable instructions, which, when executed on an electronic device, enables the electronic device to implement the steps of any one of the above-described speech recognition methods.

[0040] By means of the above technical scheme, the speech recognition method provided by the present application first obtains the speech to be recognized, that is, the target speech, and then inputs the target speech into the speech recognition model on the terminal side for speech recognition, and transmits the target speech to the cloud side, so that the speech recognition model on the cloud side performs speech recognition on the target speech. After obtaining the speech recognition result on the terminal side, the large language model is used to determine whether the speech recognition result on the terminal side is reliable. If the speech recognition result on the terminal side is reliable, the speech recognition result on the terminal side is determined as the target speech recognition result. If the speech recognition result on the terminal side is unreliable, after obtaining the speech recognition result on the cloud side, the speech recognition result on the cloud side is determined as the target speech recognition result. Taking into account the limited capabilities of the speech recognition model on the terminal side, the present application proposes a speech recognition solution for terminal-cloud collaboration, and also proposes a decision-making strategy based on the reliability of the speech recognition results on the terminal side, that is, the target speech is provided to the speech recognition model on the terminal side and the speech recognition model on the cloud side for speech recognition at the same time. After obtaining the speech recognition results on the terminal side, the reliability of the speech recognition results on the terminal side is judged using a large language model. Based on the reliability judgment results of the speech recognition results on the terminal side, a decision is made as to whether to finally adopt the speech recognition results on the terminal side or the speech recognition results on the cloud side. Compared with a pure terminal-side speech recognition solution, the speech recognition method provided in the present application can effectively improve the speech recognition effect, thereby improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0042] Figure 1 A schematic diagram of a system architecture involved in this application;

[0043] Figure 2 A schematic diagram of a hardware structure of a terminal provided in an embodiment of the present application;

[0044] Figure 3 A schematic diagram of a hardware structure of a server provided in an embodiment of the present application;

[0045] Figure 4 A flowchart of a speech recognition method provided in an embodiment of the present application;

[0046] Figure 5 A flowchart of another speech recognition method provided in an embodiment of the present application;

[0047] Figure 6 A schematic diagram of the structure of a large language model provided in an embodiment of the present application;

[0048] Figure 7 A schematic diagram of the structure of a speech recognition device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0050] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0051] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0052] In one possible implementation, Figure 1 As shown, the system architecture involved in the present application may include a terminal 101 and a cloud-side server 102, and the terminal 101 may interact with the cloud-side server 102 through a network (wired network or wireless network). The cloud-side server 102 may include one or more servers ( Figure 1 A server is included as an example for explanation).

[0053] Next, the product form of the above-mentioned terminal is described.

[0054] The above-mentioned terminal may be a vehicle-mounted device in a vehicle-mounted scenario. Of course, this embodiment is not limited thereto, and the above-mentioned terminal may also be a terminal in other scenarios.

[0055] Figure 2 An optional hardware structure diagram of a terminal is shown.

[0056] refer to Figure 2As shown, the terminal may include a radio frequency unit 210, a memory 220, an input unit 230, a display unit 240, a camera 250 (optional), an audio circuit 260 (optional), a speaker 261 (optional), a microphone 262 (optional), an earphone jack 263 (optional), a processor 270, an external interface 280, a power supply 290 and other components. Those skilled in the art will appreciate that Figure 2 This is merely an example of a terminal and does not constitute a limitation on the terminal. The terminal may include more or fewer components than shown in the figure, or a combination of certain components, or different components.

[0057] The input unit 230 can be used to receive input digital or character information, and generate key signal input related to the user settings and function control of the terminal. Specifically, the input unit 230 may include a touch screen 231 (optional) and / or other input devices 232. The touch screen 231 can collect the user's touch operations on or near it (such as the user's operation on or near the touch screen using any suitable object such as fingers, joints, stylus, etc.), and drive the corresponding connection device according to a pre-set program. The touch screen can detect the user's touch action on the touch screen, convert the touch action into a touch signal and send it to the processor 270, and can receive and execute the command sent by the processor 270; the touch signal at least includes the contact point coordinate information. The touch screen 231 can provide an input interface and an output interface between the terminal and the user. In addition, the touch screen can be implemented using multiple types such as resistive, capacitive, infrared, and surface acoustic waves. In addition to the touch screen 231, the input unit 230 can also include other input devices. Specifically, other input devices 232 may include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, and the like.

[0058] The display unit 240 may be used to display information input by a user or information provided to a user, various menus of the terminal, an interactive interface, file display, and / or playback of any multimedia file.

[0059] The memory 220 can be used to store instructions and data. The memory 220 can mainly include an instruction storage area and a data storage area. The data storage area can store various data, such as multimedia files, texts, etc. The instruction storage area can store software units such as operating systems, applications, and instructions required for at least one function, or their subsets and extensions. It can also include a non-volatile random access memory; it provides the processor 270 with hardware, software, and data resources including management of computing and processing equipment, and supports control software and applications. It is also used for the storage of multimedia files, and the storage of running programs and applications.

[0060] The processor 270 is the control center of the terminal. It uses various interfaces and lines to connect various parts of the entire terminal. By running or executing instructions stored in the memory 220 and calling data stored in the memory 220, it executes various functions of the terminal and processes data, thereby controlling the terminal as a whole. Optionally, the processor 270 may include one or more processing units; preferably, the processor 270 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application program, and the modem processor mainly processes wireless communication. It is understandable that the above-mentioned modem processor may not be integrated into the processor 270. In some embodiments, the processor and the memory may be implemented on a single chip, and in some embodiments, they may also be implemented separately on separate chips. The processor 270 may also be used to generate corresponding operation control signals, send them to corresponding components of the computing and processing device, read and process data in the software, especially read and process data and programs in the memory 220, so that each functional module therein performs corresponding functions, thereby controlling the corresponding components to act according to the requirements of the instructions.

[0061] Among them, the memory 220 can be used to store software codes related to the speech recognition method, the processor 270 can execute the software codes in the memory 220, and can also schedule other units (such as the above-mentioned input unit 230 and display unit 240) to achieve corresponding functions.

[0062] The radio frequency unit 210 (optional) can be used for receiving and sending information or receiving and sending signals during a call, for example, after receiving the downlink information of the base station, it is sent to the processor 270 for processing; in addition, the designed uplink data is sent to the base station. Generally, the radio frequency unit 210 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (LowNoiseAmplifier, LNA), a duplexer, etc. In addition, the radio frequency unit 210 can also communicate with network devices and other devices through wireless communication. The wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile communication (Global System of Mobile communication, GSM), General Packet Radio Service (General PacketRadio Service, GPRS), Code Division Multiple Access (Code Division Multiple Access, CDMA), Wideband Code Division Multiple Access (Wideband Code Division Multiple Access, WCDMA), Long Term Evolution (Long Term Evolution, LTE), email, Short Messaging Service (SMS), etc.

[0063] In the embodiment of the present application, the radio frequency unit 210 can send data to other devices, and can also receive data sent by other devices. It should be understood that the radio frequency unit 210 is optional, and can be replaced by other communication interfaces, such as a network port.

[0064] The terminal also includes a power supply 290 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 270 through a power management system, so that the power management system can manage charging, discharging, power consumption and other functions.

[0065] The terminal also includes an external interface 280, which may be a standard Micro USB interface or a multi-pin connector, and may be used to connect the terminal to communicate with other devices, or to connect a charger to charge the terminal.

[0066] Although not shown, the terminal may further include a flashlight, a wireless fidelity (WiFi) module, a Bluetooth module, sensors with different functions, etc., which will not be described in detail here.

[0067] Next, the product form of the above-mentioned cloud-side server is described.

[0068] Figure 3 A structural diagram of the above cloud-side server is provided, such as Figure 3 As shown, the cloud-side server may include a bus 301 , a processor 302 , a communication interface 303 , and a memory 304 . The processor 302 , the memory 304 , and the communication interface 303 communicate with each other via the bus 301 .

[0069] The bus 301 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0070] The processor 302 may be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0071] The memory 304 may include a volatile memory, such as a random access memory (RAM). The memory 304 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0072] The memory 304 may be used to store software codes related to the speech recognition method. The processor 302 may call the software codes stored in the memory 304 or schedule other units to implement corresponding functions.

[0073] The processors in the above-mentioned terminals and cloud-side servers (for example, processor 270 and processor 302) may be hardware circuits (such as application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), general-purpose processors, digital signal processors (DSP), microprocessors or microcontrollers, etc.), or combinations of these hardware circuits. For example, the processor may be a hardware system with an instruction execution function, such as a CPU, DSP, etc., or a hardware system without an instruction execution function, such as an ASIC, FPGA, etc., or a combination of the above-mentioned hardware systems without an instruction execution function and hardware systems with an instruction execution function.

[0074] In view of the fact that the speech recognition model on the terminal side is usually small and may sometimes have poor recognition effects, the inventor of this case conducted research and came up with a speech recognition solution for terminal-cloud collaboration during the research process. The initially conceived speech recognition solution for terminal-cloud collaboration is as follows: input the target speech into the speech recognition model on the terminal side for speech recognition, and transmit the target speech to the cloud side so that the speech recognition model on the cloud side can perform speech recognition on the target speech. After obtaining the speech recognition result on the terminal side, the speech recognition result on the terminal side is matched with the standard text in a pre-built standard text database. If there is text in the standard text database that matches the speech recognition result on the terminal side (for example, the speech recognition result on the terminal side is "turn on the air conditioner", and "turn on the air conditioner" exists in the standard text database), the speech recognition result on the terminal side is determined as the target speech recognition result. If there is no text in the standard text database that matches the speech recognition result on the terminal side, the speech recognition result on the cloud side is determined as the target speech recognition result.

[0075] The inventor of this case studied the above-mentioned end-cloud collaborative speech recognition solution and found that the strategy of deciding whether to finally adopt the speech recognition result on the end side or the speech recognition result on the cloud side is too rough and will have a certain impact on the end-side performance (for example, the speech recognition result on the end side should have been adopted, but in the end, the speech recognition result on the cloud side was adopted due to the limitations of the above-mentioned decision-making strategy). In view of the defects of the above-mentioned end-cloud collaborative speech recognition solution, the inventor of this case continued to study and finally proposed a speech recognition method with better effect through continuous research. Next, the speech recognition method provided by this application is introduced through the following embodiments.

[0076] See also Figure 4 , shows a flow chart of a speech recognition method provided by an embodiment of the present application, the method may include:

[0077] Step S401: Acquire target speech.

[0078] The target speech is the speech to be recognized.

[0079] Step S402: Perform speech recognition on the speech recognition model on the target speech input side, and transmit the target speech to the cloud side, so that the speech recognition model on the cloud side performs speech recognition on the target speech.

[0080] After the target voice is obtained, the target voice is provided to the client side for voice recognition and is also transmitted to the cloud side for voice recognition.

[0081] In this embodiment, the cloud-side speech recognition model is more capable than the end-side speech recognition model, especially for personalized scenarios. In order to ultimately obtain a better speech recognition effect, this application combines the end-side speech recognition capability with the cloud-side speech recognition capability.

[0082] Step S403: After obtaining the speech recognition result on the terminal side, use the large language model to determine whether the speech recognition result on the terminal side is reliable.

[0083] Specifically, after obtaining the voice recognition result on the terminal side, the voice recognition result on the terminal side is input into the large language model, and the large language model determines the reliability of the voice recognition result on the terminal side. If the reliability of the voice recognition result on the terminal side is greater than or equal to a preset reliability threshold, the voice recognition result on the terminal side is determined to be reliable. If the reliability of the voice recognition result on the terminal side is less than the preset reliability threshold, the voice recognition result on the terminal side is determined to be unreliable.

[0084] The large language model in this embodiment is obtained by fine-tuning the pre-trained large language model on the end-cloud collaborative speech recognition task. The large language model has the ability to determine the reliability of speech recognition results.

[0085] Step S404a: If the speech recognition result on the terminal side is reliable, the speech recognition result on the terminal side is determined as the target speech recognition result.

[0086] When it is determined that the speech recognition result on the terminal side is reliable, the speech recognition result on the terminal side is determined as the final speech recognition result, that is, the target speech recognition result.

[0087] Step S404b: If the speech recognition result on the terminal side is unreliable, after obtaining the speech recognition result on the cloud side, the speech recognition result on the cloud side is determined as the target speech recognition result.

[0088] When it is determined that the speech recognition result on the terminal side is unreliable, the speech recognition result on the cloud side is determined as the final speech recognition result, that is, the target speech recognition result.

[0089] This embodiment uses the natural language understanding capability of the large language model to determine the reliability of the voice recognition result on the end side, and then, based on the reliability determination result of the voice recognition result on the end side, decides whether to use the voice recognition result on the end side or the voice recognition result on the cloud side as the final voice recognition result. This strategy can more accurately determine whether to use the voice recognition result on the end side as the final voice recognition result.

[0090] The speech recognition method provided in the embodiment of the present application first obtains the speech to be recognized, that is, the target speech, and then inputs the target speech into the speech recognition model on the terminal side for speech recognition, and transmits the target speech to the cloud side so that the speech recognition model on the cloud side performs speech recognition on the target speech. After obtaining the speech recognition result on the terminal side, the large language model is used to determine whether the speech recognition result on the terminal side is reliable. If the speech recognition result on the terminal side is reliable, the speech recognition result on the terminal side is determined as the target speech recognition result. If the speech recognition result on the terminal side is unreliable, the speech recognition result on the cloud side is determined as the target speech recognition result. Taking into account the limited capabilities of the speech recognition model on the terminal side, the embodiment of the present application proposes a speech recognition solution for terminal-cloud collaboration, and at the same time proposes a decision-making strategy based on the reliability of the speech recognition results on the terminal side, that is, the target speech is provided to the speech recognition model on the terminal side and the speech recognition model on the cloud side for speech recognition at the same time. After obtaining the speech recognition results on the terminal side, the reliability of the speech recognition results on the terminal side is judged using a large language model. According to the reliability judgment results of the speech recognition results on the terminal side, a decision is made as to whether to finally adopt the speech recognition results on the terminal side or the speech recognition results on the cloud side. Compared with a pure speech recognition solution on the terminal side, the speech recognition method provided in the embodiment of the present application can effectively improve the speech recognition effect, thereby improving the user experience. In addition, the decision-making strategy based on the reliability of the speech recognition results on the terminal side provided in the embodiment of the present application can more accurately determine whether to use the speech recognition results on the terminal side as the final speech recognition results.

[0091] In order to improve the voice recognition effect on the terminal side, the embodiment of the present application provides another voice recognition method, such as Figure 5 As shown, the speech recognition method includes the following steps in addition to steps S401 to S404 in the above embodiment:

[0092] Step S405b: extracting entity words from the speech recognition results on the cloud side, and determining the similarity between the speech recognition results on the terminal side and the speech recognition results on the cloud side.

[0093] When the speech recognition results on the terminal side are unreliable, a large language model can be used to extract entity words from the speech recognition results on the cloud side and determine the similarity between the speech recognition results on the terminal side and the speech recognition results on the cloud side.

[0094] The large language model in this embodiment has the ability to identify entity words from speech recognition results and determine the similarity of two speech recognition results. The pre-trained large language model can be fine-tuned using the training data set corresponding to the end-cloud collaborative speech recognition task, so that it has the ability to judge the reliability of speech recognition results, the ability to identify entity words from speech recognition results, and the ability to determine the similarity of two speech recognition results.

[0095] Step S406b: Determine whether the hot word library on the terminal side needs to be updated immediately based on the similarity between the speech recognition result on the terminal side and the speech recognition result on the cloud side.

[0096] In this embodiment, the process of determining whether the hot word library on the terminal side needs to be updated immediately based on the similarity between the voice recognition result on the terminal side and the voice recognition result on the cloud side may include: determining whether the similarity between the voice recognition result on the terminal side and the voice recognition result on the cloud side is greater than or equal to a preset similarity threshold; if the similarity between the voice recognition result on the terminal side and the voice recognition result on the cloud side is greater than or equal to the preset similarity threshold, determining that the hot word library on the terminal side does not need to be updated immediately; if the similarity between the voice recognition result on the terminal side and the voice recognition result on the cloud side is less than the preset similarity threshold, determining that the hot word library on the terminal side needs to be updated immediately.

[0097] If the similarity between the speech recognition result on the terminal side and the speech recognition result on the cloud side is greater than or equal to the preset similarity threshold, it means that the difference between the speech recognition result on the terminal side and the speech recognition result on the cloud side is not large, and it is determined that there is no need to update the hot word library on the terminal side immediately. If the similarity between the speech recognition result on the terminal side and the speech recognition result on the cloud side is less than the preset similarity threshold, it means that the difference between the speech recognition result on the terminal side and the speech recognition result on the cloud side is large, and it is determined that the hot word library on the terminal side needs to be updated immediately.

[0098] It should be noted that the speech recognition model on the end side will be combined with the hot word library on the end side for speech recognition, and the quality of the hot word library will affect the speech recognition effect on the end side. In order to improve the speech recognition effect on the end side, this embodiment updates the entity words extracted from the speech recognition results on the cloud side to the hot word library on the end side when the difference between the speech recognition results on the end side and the speech recognition results on the cloud side is large.

[0099] Taking into account the limited resources on the terminal side, in order to alleviate the resource consumption on the terminal side, after extracting the entity words from the speech recognition results on the cloud side, this embodiment does not directly add the entity words to the hot word library on the terminal side, but first determines whether the hot word update conditions are currently met, that is, determines the similarity between the speech recognition results on the terminal side and the speech recognition results on the cloud side, and then determines whether the hot word library on the terminal side needs to be updated immediately based on the similarity between the speech recognition results on the terminal side and the speech recognition results on the cloud side. When it is determined that the hot word library on the terminal side needs to be updated immediately, the entity words extracted from the speech recognition results on the cloud side are added to the hot word library on the terminal side.

[0100] Of course, this embodiment is not limited to updating the hot word library on the terminal side when the hot word update condition is met. After the entity words are extracted from the speech recognition results on the cloud side, the extracted entity words can be immediately added to the hot word library on the terminal side.

[0101] Updating the hot word library on the client side can improve the recognition accuracy of the speech recognition model on the client side the next time it performs speech recognition.

[0102] Step S407b-a: If it is determined that the hot word database on the terminal side does not need to be updated immediately, the extracted entity words are stored in the entity word database.

[0103] When it is determined that there is no need to immediately update the hot word library on the terminal side, the entity words extracted from the speech recognition results on the cloud side are temporarily stored in the entity word database.

[0104] In a possible implementation, after the entity words extracted from the speech recognition results on the cloud side are stored in the entity word database, the number of entity words in the entity word database can be counted to determine whether the number of entity words in the entity word database is greater than or equal to a preset entity word number threshold. If the number of entity words in the entity word database is greater than or equal to the preset entity word number threshold, the entity words in the entity word database are added to the hot word library on the end side, and the entity words in the entity word database are deleted. If the number of entity words in the entity word database is less than the preset entity word number threshold, the hot word library on the end side is not updated.

[0105] Step S407b-b: If it is determined that the hot word library on the terminal side needs to be updated immediately, the extracted entity words and the entity words in the entity word database are added to the hot word library on the terminal side.

[0106] It should be noted that, when it is determined that the hot word library on the terminal side needs to be updated immediately, the entity words extracted from the speech recognition results on the cloud side are added to the hot word library on the terminal side. In addition to adding the entity words extracted from the speech recognition results on the cloud side to the hot word library on the terminal side, the entity words already in the entity word database are also added to the hot word library on the terminal side. After the entity words in the entity word database are added to the hot word library on the terminal side, the entity words in the entity word database are deleted.

[0107] The speech recognition method provided in the embodiment of the present application first obtains the speech to be recognized, that is, the target speech, and then inputs the target speech into the speech recognition model on the terminal side for speech recognition, and transmits the target speech to the cloud side so that the speech recognition model on the cloud side performs speech recognition on the target speech. After obtaining the speech recognition result on the terminal side, the large language model is used to determine whether the speech recognition result on the terminal side is reliable. If the speech recognition result on the terminal side is reliable, the speech recognition result on the terminal side is determined as the target speech recognition result. If the speech recognition result on the terminal side is unreliable, the speech recognition result on the cloud side is determined as the target speech recognition result, and entity words are extracted from the speech recognition result on the cloud side. and determining the similarity between the voice recognition result on the terminal side and the voice recognition result on the cloud side, and judging whether it is necessary to immediately update the hot word library on the terminal side according to the similarity between the voice recognition result on the terminal side and the voice recognition result on the cloud side; if it is determined that the hot word library on the terminal side does not need to be updated immediately, the extracted entity words are stored in the entity word database, and when the number of entity words in the entity word database is greater than or equal to a preset entity word number threshold, the entity words in the entity word database are updated to the hot word library on the terminal side; if it is determined that the hot word library on the terminal side needs to be updated immediately, the entity words extracted from the voice recognition result on the cloud side and the entity words in the entity word database are updated to the hot word library on the terminal side. Taking into account the limited capabilities of the speech recognition model on the terminal side, the embodiment of the present application proposes a speech recognition solution for terminal-cloud collaboration, and also proposes a decision-making strategy based on the reliability of the speech recognition results on the terminal side, that is, the target speech is provided to the speech recognition model on the terminal side and the speech recognition model on the cloud side for speech recognition at the same time. After obtaining the speech recognition results on the terminal side, the reliability of the speech recognition results on the terminal side is judged using a large language model. Based on the reliability judgment results of the speech recognition results on the terminal side, a decision is made as to whether to finally adopt the speech recognition results on the terminal side or the speech recognition results on the cloud side. Compared with a pure speech recognition solution on the terminal side, the speech recognition method provided in the embodiment of the present application can effectively improve the speech recognition effect, thereby improving the user experience. In addition, based on the reliability judgment results of the speech recognition results on the terminal side, a decision is made as to whether to use the speech recognition results on the terminal side or the speech recognition results on the cloud side as the final speech recognition results, so that it can more accurately determine whether to use the speech recognition results on the terminal side as the final speech recognition results. In order to improve the subsequent speech recognition accuracy on the terminal side, the embodiment of the present application proposes to update the entity words extracted from the speech recognition results on the cloud side to the hot word library on the terminal side. In order to alleviate the resource consumption on the terminal side, after extracting the entity words from the speech recognition results on the cloud side, the embodiment of the present application does not directly add the entity words to the hot word library on the terminal side, but first determines whether the hot word update conditions are met. When the hot word update conditions are met, the extracted entity words are added to the hot word library on the terminal side.In summary, the speech recognition method provided in the embodiment of the present application has a good speech recognition effect, can improve the subsequent speech recognition accuracy on the terminal side, and can also alleviate the resource consumption on the terminal side to a certain extent.

[0108] The above embodiment mentioned that a large language model can be used to determine whether the speech recognition results on the terminal side are reliable. A large language model can be used to extract entity words from the speech recognition results on the cloud side and determine the similarity between the speech recognition results on the terminal side and the speech recognition results on the cloud side. The above process is introduced below.

[0109] In one possible implementation, Figure 6 As shown, the large language model may include: a feature acquisition module 601, a data distribution module 602, a reliability determination module 603a, an entity word recognition module 603b and a similarity determination module 603c.

[0110] The process of using the large language model to determine whether the speech recognition result on the terminal side is reliable may include:

[0111] Step a1: Use the feature acquisition module 601 of the large language model to extract features from the speech recognition result on the terminal side to obtain the features of the speech recognition result on the terminal side.

[0112] The speech recognition result on the terminal side can be input into the feature acquisition module 601 of the large language model, and the feature acquisition module 601 of the large language model extracts features from the input speech recognition result and outputs the features of the speech recognition result on the terminal side. Optionally, the feature acquisition module 601 can be, but is not limited to, a feature acquisition module based on a multi-head attention mechanism.

[0113] Step a2: Use the data distribution module 602 of the large language model to distribute the features of the speech recognition result on the terminal side to the reliability determination module 603a.

[0114] The features of the speech recognition result on the terminal side are input into the data distribution module 602 of the large language model, and the data distribution module 602 distributes the features of the speech recognition result on the terminal side to the reliability determination module 603a for processing.

[0115] Step a3: using the reliability determination module 603a of the large language model, based on the characteristics of the speech recognition result on the terminal side, predict the reliability of the speech recognition result on the terminal side.

[0116] The reliability determination module 603a predicts the reliability of the speech recognition result on the terminal side according to the characteristics of the speech recognition result on the terminal side.

[0117] Step a4: If the reliability of the speech recognition result on the terminal side is greater than or equal to the preset reliability threshold, the speech recognition result on the terminal side is determined to be reliable; if the reliability of the speech recognition result on the terminal side is less than the preset reliability threshold, the speech recognition result on the terminal side is determined to be unreliable.

[0118] Through the above process, it can be determined whether the speech recognition result on the terminal side is reliable. If the speech recognition result on the terminal side is unreliable, the speech recognition result on the terminal side is used as the target speech recognition result, and the large language model is used to extract entity words from the speech recognition result on the cloud side, and the similarity between the speech recognition result on the terminal side and the speech recognition result on the cloud side is determined. Next, the specific implementation process of using the large language model to extract entity words from the speech recognition result on the cloud side and determine the similarity between the speech recognition result on the terminal side and the speech recognition result on the cloud side is introduced.

[0119] The process of extracting entity words from the speech recognition results on the cloud side by using the large language model and determining the similarity between the speech recognition results on the client side and the speech recognition results on the cloud side may include:

[0120] Step b1: Use the feature acquisition module 601 of the large language model to extract features from the speech recognition results on the terminal side and the speech recognition results on the cloud side, respectively, to obtain features of the speech recognition results on the terminal side and features of the speech recognition results on the cloud side.

[0121] The speech recognition results on the terminal side and the speech recognition results on the cloud side are input into the feature acquisition module 601 of the large language model. The feature acquisition module 601 extracts features from the speech recognition results on the terminal side and the speech recognition results on the cloud side respectively to obtain the features of the speech recognition results on the terminal side and the features of the speech recognition results on the cloud side.

[0122] Step b2: Use the data distribution module 602 of the large language model to distribute the features of the speech recognition results on the cloud side to the entity word recognition module 603b of the large language model, and distribute the features of the speech recognition results on the terminal side and the features of the speech recognition results on the cloud side to the similarity determination module 603c of the large language model.

[0123] The features of the speech recognition results on the terminal side and the features of the speech recognition results on the cloud side are input into the data distribution module 602. The data distribution module 602 distributes data through a routing mechanism. Specifically, the features of the speech recognition results on the cloud side are distributed to the entity word recognition module 603b, and the features of the speech recognition results on the terminal side and the features of the speech recognition results on the cloud side are distributed to the similarity determination module 603c.

[0124] Step b3-a, using the entity word recognition module 603b of the large language model, based on the characteristics of the speech recognition results on the cloud side, identify the entity words in the speech recognition results on the cloud side.

[0125] Specifically, after the data distribution module 602 distributes the features of the speech recognition results on the cloud side to the entity word recognition module 603b, the entity word recognition module 603b predicts the probability that each word contained in the speech recognition results on the cloud side is an entity word based on the features of the speech recognition results on the cloud side. After obtaining the probability that each word contained in the speech recognition results on the cloud side is an entity word, the words with a probability greater than a preset probability threshold can be determined as entity words, so that the entity words in the speech recognition results on the cloud side can be obtained.

[0126] Step b3-b, using the similarity determination module 603c of the large language model, based on the characteristics of the speech recognition results fed back from the cloud side and the characteristics of the speech recognition results on the terminal side, determine the similarity between the speech recognition results on the terminal side and the speech recognition results on the cloud side.

[0127] Specifically, after the data distribution module 602 distributes the features of the speech recognition results on the terminal side and the features of the speech recognition results on the cloud side to the similarity determination module 603c, the similarity determination module 603c determines the similarity between the speech recognition results on the terminal side and the speech recognition results on the cloud side based on the features of the speech recognition results on the terminal side and the features of the speech recognition results on the cloud side. More specifically, the similarity between the features of the speech recognition results on the terminal side and the features of the speech recognition results on the cloud side can be calculated, and the obtained similarity is used as the similarity between the speech recognition results on the terminal side and the speech recognition results on the cloud side.

[0128] The above embodiment mentioned that the pre-trained large language model can be fine-tuned using the training data set corresponding to the end-cloud collaborative speech recognition task to obtain a large language model that has the ability to judge the reliability of speech recognition results, the ability to identify entity words from speech recognition results, and the ability to determine the similarity of two speech recognition results. The training data set corresponding to the end-cloud collaborative speech recognition task may include a number of first training data, a number of second training data, a number of third training data, and a number of fourth training data, wherein the first training data includes a training text with a first reliability label indicating that the text is reliable, the second training data includes a training text with a second reliability label indicating that the text is unreliable, the third training data includes two training texts with the same semantics, the two training texts with the same semantics have a hot word update label indicating that hot word update is not required immediately, and one of the two training texts with the same semantics has an entity word label indicating the location of the entity word, and the fourth training data includes two training texts with different semantics, the two training texts with different semantics have a hot word update label indicating that hot word update is required immediately, and one of the two training texts with different semantics has an entity word label indicating the location of the entity word. The above-mentioned several types of training data can be used to perform fine-tuning training on the pre-trained large language model. When fine-tuning the pre-trained large language model, the training is performed with the goal of enabling the large language model to predict corresponding labels for the input training data.

[0129] The above introduces the speech recognition method provided in the embodiment of the present application, and the device corresponding to the above speech recognition method will be introduced below.

[0130] See also Figure 7 , Figure 7 A structural schematic diagram of a speech recognition device provided in an embodiment of the present application, the speech recognition device may include: a speech acquisition module 701, a speech processing module 702, a terminal-side speech recognition result reliability determination module 703 and a target speech recognition result determination module 704.

[0131] The speech acquisition module 701 is used to acquire the target speech.

[0132] The speech processing module 702 is used to perform speech recognition on the speech recognition model on the target speech input end side, and transmit the target speech to the cloud side so that the speech recognition model on the cloud side performs speech recognition on the target speech.

[0133] The terminal-side speech recognition result reliability determination module 703 is used to determine whether the terminal-side speech recognition result is reliable by using a large language model after obtaining the terminal-side speech recognition result.

[0134] Among them, the large language model is obtained by fine-tuning the pre-trained large language model on the end-cloud collaborative speech recognition task. The large language model has the ability to judge the reliability of speech recognition results.

[0135] The target speech recognition result determination module 704 is used to determine the speech recognition result on the terminal side as the target speech recognition result when the speech recognition result on the terminal side is reliable, and to determine the speech recognition result on the cloud side as the target speech recognition result after obtaining the speech recognition result on the cloud side when the speech recognition result on the terminal side is unreliable.

[0136] In a possible implementation, the speech recognition device may further include: an entity word and similarity acquisition module, a hot word update determination module, a hot word storage module and a hot word update module.

[0137] The entity word and similarity acquisition module is used to extract entity words from the speech recognition results on the cloud side and determine the similarity between the speech recognition results on the terminal side and the speech recognition results on the cloud side when the speech recognition results on the terminal side are unreliable.

[0138] The hot word update determination module is used to determine whether the hot word library on the terminal side needs to be updated immediately based on the similarity between the speech recognition results on the terminal side and the speech recognition results on the cloud side.

[0139] The hot word storage module is used to store the extracted entity words in the entity word database when the hot word update determination module determines that the hot word library on the terminal side does not need to be updated immediately.

[0140] The hot word update module is used to add the extracted entity words and entity words in the entity word database to the hot word library on the end side when the hot word update determination module determines that the hot word library on the end side needs to be updated immediately.

[0141] In one possible implementation, the large language model also has the ability to identify entity words from speech recognition results and determine the similarity between two speech recognition results.

[0142] The entity word and similarity acquisition module extracts entity words from the speech recognition results on the cloud side and determines the similarity between the speech recognition results on the client side and the speech recognition results on the cloud side. Specifically, it is used to:

[0143] Using the large language model, entities are extracted from the speech recognition results on the cloud side, and the similarity between the speech recognition results on the client side and the speech recognition results on the cloud side is determined.

[0144] In a possible implementation, the hot word update determination module determines whether the hot word library on the terminal side needs to be updated immediately according to the similarity between the speech recognition result on the terminal side and the speech recognition result on the cloud side. Specifically, it is used to:

[0145] If the similarity between the speech recognition result on the terminal side and the speech recognition result on the cloud side is greater than or equal to the preset similarity threshold, it is determined that there is no need to update the hot word library on the terminal side immediately;

[0146] If the similarity between the speech recognition result on the terminal side and the speech recognition result on the cloud side is less than a preset similarity threshold, it is determined that the hot word library on the terminal side needs to be updated immediately.

[0147] In a possible implementation, the speech recognition device may further include: an entity word statistics module.

[0148] An entity word statistics module is used to count the number of entity words in the entity word database after the hot word storage module stores the extracted entity words in the entity word database;

[0149] The hot word update module is also used to add the entity words in the entity word database to the hot word library on the terminal side and delete the entity words in the entity word database when the number of entity words in the entity word database is greater than or equal to a preset entity word number threshold.

[0150] In a possible implementation, the entity word and similarity acquisition module uses a large language model to extract entities from the speech recognition results on the cloud side and determines the similarity between the speech recognition results on the terminal side and the speech recognition results on the cloud side. Specifically, it is used to:

[0151] The feature acquisition module of the large language model is used to extract features from the speech recognition results on the terminal side and the speech recognition results on the cloud side, respectively, to obtain the features of the speech recognition results on the terminal side and the features of the speech recognition results on the cloud side.

[0152] Using the data distribution module of the large language model, the features of the speech recognition results on the cloud side are distributed to the entity word recognition module of the large language model, and the features of the speech recognition results on the terminal side and the features of the speech recognition results on the cloud side are distributed to the similarity determination module of the large language model.

[0153] The entity word recognition module of the large language model is used to identify the entity words in the speech recognition results on the cloud side based on the characteristics of the speech recognition results on the cloud side.

[0154] The similarity determination module of the large language model is used to determine the similarity between the speech recognition results on the terminal side and the speech recognition results on the cloud side based on the characteristics of the speech recognition results fed back by the cloud side and the characteristics of the speech recognition results on the terminal side.

[0155] The speech recognition device provided in the embodiment of the present application provides the target speech to the speech recognition model on the terminal side and the speech recognition model on the cloud side for speech recognition at the same time. After obtaining the speech recognition result on the terminal side, the large language model is used to judge the reliability of the speech recognition result on the terminal side. According to the reliability judgment result of the speech recognition result on the terminal side, a decision is made as to whether to finally adopt the speech recognition result on the terminal side or the speech recognition result on the cloud side. Compared with the speech recognition solution on the terminal side alone, the speech recognition device provided in the embodiment of the present application can effectively improve the speech recognition effect, thereby improving the user experience. In addition, according to the reliability judgment result of the speech recognition result on the terminal side, a decision is made as to whether to use the speech recognition result on the terminal side or the speech recognition result on the cloud side as the final speech recognition result, so that it can more accurately determine whether to use the speech recognition result on the terminal side as the final speech recognition result. In order to improve the subsequent voice recognition accuracy on the end side, the voice recognition device provided in the embodiment of the present application updates the entity words extracted from the voice recognition results on the cloud side to the hot word library on the end side. In order to alleviate the resource consumption on the end side, the voice recognition device provided in the embodiment of the present application does not directly add the entity words to the hot word library on the end side after extracting the entity words from the voice recognition results on the cloud side, but first determines whether the hot word update conditions are met. When the hot word update conditions are met, the extracted entity words are added to the hot word library on the end side. In summary, the voice recognition device provided in the embodiment of the present application has a good voice recognition effect, can improve the subsequent voice recognition accuracy on the end side, and can also alleviate the resource consumption on the end side to a certain extent.

[0156] An embodiment of the present application also provides an electronic device, which may include: at least one processor, at least one communication interface, at least one memory and at least one communication bus.

[0157] In the embodiment of the present application, the number of the processor, the communication interface, the memory, and the communication bus is at least one, and the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0158] The processor may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application;

[0159] The memory may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory;

[0160] The memory stores a program, and the processor can call the program stored in the memory, and the program is used to implement the steps of the speech recognition method provided in the above embodiment.

[0161] An embodiment of the present application also provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the steps of the speech recognition method provided in the above embodiment.

[0162] An embodiment of the present application also provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the steps of the speech recognition method provided in the above embodiment.

[0163] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0164] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a magnetic disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0165] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0166] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. A speech recognition method, characterized in that: include: Get the target voice; Performing speech recognition on the speech recognition model on the target speech input terminal side, and transmitting the target speech to the cloud side, so that the speech recognition model on the cloud side performs speech recognition on the target speech; After obtaining the speech recognition result on the end side, use the large language model to determine whether the speech recognition result on the end side is reliable, wherein the large language model is obtained by fine-tuning the pre-trained large language model on the speech recognition task of end-cloud collaboration, and the large language model has the ability to determine the reliability of the speech recognition result; If the speech recognition result on the terminal side is reliable, determining the speech recognition result on the terminal side as the target speech recognition result; If the speech recognition result on the terminal side is unreliable, after obtaining the speech recognition result on the cloud side, the speech recognition result on the cloud side is determined as the target speech recognition result.

2. The speech recognition method according to claim 1, characterized in that: Also includes: When the speech recognition result on the terminal side is unreliable, extracting entity words from the speech recognition result on the cloud side, and determining the similarity between the speech recognition result on the terminal side and the speech recognition result on the cloud side; Determine whether it is necessary to immediately update the hot word library on the terminal side according to the similarity between the speech recognition result on the terminal side and the speech recognition result on the cloud side; If it is determined that the hot word database on the client side does not need to be updated immediately, the extracted entity words are stored in the entity word database; If it is determined that the hot word library on the terminal side needs to be updated immediately, the extracted entity words and the entity words in the entity word database are added to the hot word library on the terminal side.

3. The speech recognition method according to claim 2, characterized in that: The large language model also has the ability to identify entity words from speech recognition results and determine the similarity between two speech recognition results; The extracting entity words from the speech recognition result on the cloud side and determining the similarity between the speech recognition result on the terminal side and the speech recognition result on the cloud side includes: Utilizing the large language model, entity words are extracted from the speech recognition result on the cloud side, and the similarity between the speech recognition result on the terminal side and the speech recognition result on the cloud side is determined.

4. The speech recognition method according to claim 2, characterized in that: The determining whether it is necessary to immediately update the hot word library on the terminal side according to the similarity between the speech recognition result on the terminal side and the speech recognition result on the cloud side includes: If the similarity between the speech recognition result on the terminal side and the speech recognition result on the cloud side is greater than or equal to a preset similarity threshold, it is determined that there is no need to immediately update the hot word library on the terminal side; If the similarity between the speech recognition result on the terminal side and the speech recognition result on the cloud side is less than a preset similarity threshold, it is determined that the hot word library on the terminal side needs to be updated immediately.

5. The speech recognition method according to claim 2, characterized in that: Also includes: After storing the extracted entity words in the entity word database, counting the number of entity words in the entity word database; If the number of entity words in the entity word database is greater than or equal to a preset entity word number threshold, the entity words in the entity word database are added to the hot word library on the terminal side, and the entity words in the entity word database are deleted.

6. The speech recognition method according to claim 3, characterized in that: The extracting entity words from the speech recognition result on the cloud side by using the large language model, and determining the similarity between the speech recognition result on the terminal side and the speech recognition result on the cloud side, includes: Using the feature acquisition module of the large language model, extract features from the speech recognition result on the terminal side and the speech recognition result on the cloud side, respectively, to obtain features of the speech recognition result on the terminal side and features of the speech recognition result on the cloud side; Using the data distribution module of the large language model, the features of the speech recognition result on the cloud side are distributed to the entity word recognition module of the large language model, and the features of the speech recognition result on the terminal side and the features of the speech recognition result on the cloud side are distributed to the similarity determination module of the large language model; Using the entity word recognition module of the large language model, based on the characteristics of the speech recognition results on the cloud side, the entity words in the speech recognition results on the cloud side are recognized; The similarity determination module of the large language model is utilized to determine the similarity between the speech recognition result on the terminal side and the speech recognition result on the cloud side based on the characteristics of the speech recognition result fed back by the cloud side and the characteristics of the speech recognition result on the terminal side.

7. A speech recognition device, characterized in that: include: A speech acquisition module, a speech processing module, a terminal-side speech recognition result reliability determination module, and a target speech recognition result determination module; The speech acquisition module is used to acquire the target speech; The speech processing module is used to perform speech recognition on the speech recognition model on the target speech input terminal side, and transmit the target speech to the cloud side, so that the speech recognition model on the cloud side performs speech recognition on the target speech; The device-side speech recognition result reliability determination module is used to use a large language model to determine whether the device-side speech recognition result is reliable after obtaining the device-side speech recognition result, wherein the large language model is obtained by fine-tuning a pre-trained large language model on a device-cloud collaborative speech recognition task, and the large language model has the ability to determine the reliability of the speech recognition result; The target speech recognition result determination module is used to determine the speech recognition result on the terminal side as the target speech recognition result when the speech recognition result on the terminal side is reliable, and to determine the speech recognition result on the cloud side as the target speech recognition result after obtaining the speech recognition result on the cloud side when the speech recognition result on the terminal side is unreliable.

8. An electronic device, characterized in that: The method comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the steps of the speech recognition method as described in any one of claims 1 to 6.

9. A computer storage medium, characterized in that: The storage medium carries one or more computer programs, and when the one or more computer programs are executed by an electronic device, the electronic device can implement the steps of the speech recognition method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that The method comprises computer-readable instructions, which, when executed on an electronic device, enable the electronic device to implement the steps of the speech recognition method as claimed in any one of claims 1 to 6.

Images