A method, device, equipment and storage medium for processing speech recognition text

Abstract

The embodiments of the present application disclose a method, device, equipment and storage medium for processing speech recognition text. The relevant embodiments can be applied to various scenarios such as artificial intelligence, maps, and intelligent transportation to improve the error correction accuracy of speech recognition text. The method includes: vector conversion of the original speech to obtain an audio coding vector, vector conversion of each text unit in the speech recognition text to obtain a text coding vector corresponding to each text unit, using an attention mechanism for the audio coding vector and the i-th text coding vector to obtain a fusion vector of the i-th text unit after interaction, inputting the fusion vector of the i-th text unit and the text vector corresponding to the i-1th target text unit into a text error correction network, outputting the text probability distribution corresponding to the fusion vector of the i-th text unit through the text error correction network, and determining the i-th target text unit based on the text probability distribution corresponding to the fusion vector of the i-th text unit.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular to a method, device, equipment and storage medium for processing speech recognition text. Background Art

[0002] Automatic Speech Recognition (ASR) is the process of converting audio into text using computers.

[0003] With the continuous development of information technology and artificial intelligence, ASR still has a high word error rate (Word Error Rate, WER). Traditional solutions generally use ASR recognized text and actual manual transcription to form text pair training data, build a sequence-to-sequence model, use ASR recognized erroneous text as the input of the model, correct the ASR recognized erroneous text, and output the corrected text.

[0004] However, this error correction model is generally based on a text pair consisting of ASR-recognized text and actual manual transcription, learning the fixed pattern of ASR recognition errors and text-related error patterns. As a result, in the actual process of correcting the ASR-recognized text, it can only identify errors in specific text patterns, while some errors with non-fixed patterns, such as similar pronunciations, are often not effectively handled, resulting in a low accuracy rate in correcting the ASR-recognized text. Summary of the invention

[0005] The embodiments of the present application provide a method, apparatus, device and storage medium for processing speech recognition text, which are used to increase speech-assisted recognition text units by obtaining fusion vectors using an attention mechanism for audio coding vectors and text coding vectors, and then recognize the fusion vectors through a text error correction network, so as to better predict and correct each text unit in the speech recognition text, thereby improving the accuracy of error correction of the speech recognition text recognized by ASR.

[0006] The present application provides a method for processing speech recognition text, including:

[0007] Get the speech recognition text corresponding to the original speech;

[0008] Perform vector conversion on the original speech to obtain an audio coding vector;

[0009] Perform vector conversion on each text unit in the speech recognition text to obtain a text encoding vector corresponding to each text unit;

[0010] An attention mechanism is used for the audio encoding vector and the i-th text encoding vector to obtain a fusion vector of the i-th text unit after interaction, where i is an integer greater than or equal to 1;

[0011] The fusion vector of the i-th text unit and the text vector corresponding to the i-1-th target text unit are input into the text error correction network, and the text probability distribution corresponding to the fusion vector of the i-th text unit is output through the text error correction network;

[0012] Based on the text probability distribution corresponding to the fusion vector of the i-th text unit, the i-th target text unit is determined.

[0013] On the other hand, the present application provides a device for processing speech recognition text, comprising:

[0014] An acquisition unit, used for acquiring a speech recognition text corresponding to the original speech;

[0015] A processing unit, used for performing vector conversion on the original speech to obtain an audio coding vector;

[0016] The processing unit is further used to perform vector conversion on each text unit in the speech recognition text to obtain a text encoding vector corresponding to each text unit;

[0017] The processing unit is further used to use an attention mechanism on the audio encoding vector and the i-th text encoding vector to obtain a fusion vector of the i-th text unit after interaction, where i is an integer greater than or equal to 1;

[0018] The processing unit is further used to input the fusion vector of the i-th text unit and the text vector corresponding to the i-1-th target text unit into the text error correction network, and output the text probability distribution corresponding to the fusion vector of the i-th text unit through the text error correction network;

[0019] The determination unit is used to determine the i-th target text unit based on the text probability distribution corresponding to the fusion vector of the i-th text unit.

[0020] In a possible design, in an implementation of another aspect of the embodiment of the present application,

[0021] The processing unit is further used to input the fusion vector of the i-th text unit into the text recognition network, and output the error probability value corresponding to the fusion vector of the i-th text unit through the text recognition network;

[0022] The determination unit is further used to determine the i-th erroneous text unit when the error probability value corresponding to the fusion vector of the i-th text unit is 0;

[0023] The determination unit is further used to determine the i-th correct text unit when the error probability value corresponding to the fusion vector of the i-th text unit is 1.

[0024] In a possible design, in an implementation of another aspect of the embodiment of the present application, the processing unit may be specifically used for:

[0025] The text encoding vector corresponding to each text unit and the audio encoding vector are subjected to the attention mechanism to obtain the audio vector corresponding to each text encoding vector;

[0026] The audio vector corresponding to the i-th text encoding vector and the text encoding vector corresponding to the i-th text unit are summed to obtain a fusion vector of the i-th text unit.

[0027] In a possible design, in an implementation of another aspect of the embodiment of the present application, the processing unit may be specifically used for:

[0028] Convert the original speech into a single-frame speech signal;

[0029] Each single-frame speech signal is vector-converted to obtain an audio coding vector corresponding to each single-frame speech signal.

[0030] In a possible design, in an implementation of another aspect of the embodiment of the present application, the processing unit may be specifically used for:

[0031] Obtaining a sample speech recognition text corresponding to the sample speech and a text unit label corresponding to the sample speech recognition text;

[0032] Performing vector conversion on the sample speech to obtain a sample speech coding vector;

[0033] Performing vector conversion on each text unit in the sample speech recognition text to obtain a sample encoding vector corresponding to each text unit;

[0034] An attention mechanism is used for the sample speech coding vector and the i-th sample coding vector to obtain the sample fusion vector of the i-th text unit after interaction;

[0035] The sample fusion vector of the i-th text unit and the text vector corresponding to the i-1-th predicted text unit are input into the basic text error correction network, and the text probability distribution corresponding to the sample fusion vector of the i-th text unit is output through the basic text error correction network;

[0036] Calculate the error correction loss value based on the text unit label and the text probability distribution corresponding to the sample fusion vector of the i-th text unit;

[0037] The parameters of the basic text error correction network are adjusted based on the error correction loss value to obtain a text error correction network.

[0038] In a possible design, in an implementation of another aspect of the embodiment of the present application,

[0039] The acquisition unit is also used to obtain the error detection label corresponding to the sample speech recognition text;

[0040] The processing unit is further used to input the sample fusion vector of each text unit into the basic text recognition network, and output the error probability value corresponding to the sample fusion vector of each text unit through the basic text recognition network;

[0041] The processing unit is further used to calculate the error detection loss value based on the error detection label and the error probability value corresponding to the sample fusion vector of each text unit;

[0042] The processing unit is also used to adjust the parameters of the basic text error correction network based on the error correction loss value to obtain the text error correction network, including:

[0043] The processing unit is also used to adjust the parameters of the basic text recognition network and the basic text error correction network based on the error detection loss value and the error correction loss value to obtain the text error correction network and the text recognition network.

[0044] In a possible design, in an implementation of another aspect of the embodiment of the present application, the processing unit may be specifically used for:

[0045] Based on the loss weight, the error detection loss value and the error correction loss value are weighted and summed to obtain the total loss value;

[0046] Based on the total loss value, the parameters of the basic text recognition network and the basic text error correction network are adjusted to obtain the text recognition network and the text error correction network.

[0047] In a possible design, in an implementation of another aspect of the embodiment of the present application, the processing unit may be specifically used for:

[0048] The sample coding vector corresponding to each text unit and the sample speech coding vector are subjected to the attention mechanism to obtain the sample speech vector corresponding to each sample speech coding vector;

[0049] The sample speech vector corresponding to the i-th sample speech coding vector and the sample coding vector corresponding to the i-th text unit are summed to obtain a sample fusion vector of the i-th text unit.

[0050] In a possible design, in an implementation of another aspect of the embodiment of the present application, the processing unit may be specifically used for:

[0051] Converting the sample speech into a single-frame sample speech signal;

[0052] Each single-frame sample speech signal is vector-converted to obtain a sample speech coding vector corresponding to each single-frame sample speech signal.

[0053] In a possible design, in an implementation of another aspect of the embodiment of the present application, the acquisition unit may be specifically used to:

[0054] The original speech is input into the automatic speech recognition model, and the speech coding vector corresponding to the original speech is obtained through the pre-trained acoustic network in the automatic speech recognition model;

[0055] Pass the speech coding vector and the text vector corresponding to the i-1th recognized text unit to the text decoder in the automatic speech recognition model to obtain the recognized text probability distribution corresponding to the i-th text unit;

[0056] Based on the recognition text probability distribution corresponding to the ith text unit, the ith recognition text unit is determined to obtain the speech recognition text.

[0057] In a possible design, in an implementation of another aspect of the embodiment of the present application, the processing unit may be specifically used for:

[0058] Convert the original speech into a single-frame speech signal;

[0059] Input each single-frame speech signal into an automatic speech recognition model, and obtain a single-frame speech vector corresponding to each single-frame speech signal through a pre-trained acoustic network in the automatic speech recognition model;

[0060] Based on the audio weight corresponding to each single-frame speech signal, the single-frame speech vectors corresponding to each single-frame speech signal are weighted summed to obtain a speech coding vector.

[0061] In a possible design, in an implementation of another aspect of the embodiment of the present application, the processing unit may be specifically used for:

[0062] Input the sample speech into the basic automatic speech recognition model, and obtain the sample speech vector corresponding to the sample speech through the pre-trained acoustic network in the basic automatic speech recognition model;

[0063] Pass the sample speech vector and the text vector corresponding to the i-1th sample recognition text unit to the text decoder in the basic automatic speech recognition model to obtain the sample recognition text probability distribution corresponding to the i-th text unit;

[0064] Calculate the recognition loss value based on the sample recognition text probability distribution corresponding to the i-th text unit and the text unit label corresponding to the sample speech;

[0065] The parameters of the basic automatic speech recognition model are adjusted based on the recognition loss value to obtain an automatic speech recognition model.

[0066] In a possible design, in an implementation of another aspect of the embodiment of the present application, the processing unit may be specifically used for:

[0067] Converting the sample speech into a single-frame sample speech signal;

[0068] Input each single-frame sample speech signal into a basic automatic speech recognition model, and obtain a single-frame sample speech vector corresponding to each single-frame sample speech signal through a pre-trained acoustic network in the basic automatic speech recognition model;

[0069] Based on the audio weight corresponding to each single-frame sample speech signal, the single-frame sample speech vectors corresponding to each single-frame sample speech signal are weighted summed to obtain a sample speech vector.

[0070] In a possible design, in an implementation of another aspect of the embodiment of the present application,

[0071] The acquisition unit is also used to receive the speech recognition optimization instruction;

[0072] The acquisition unit can be specifically used to read the speech recognition text from the database according to the speech recognition optimization instruction.

[0073] On the other hand, the present application provides a computer device, including: a memory, a processor, and a bus system;

[0074] Wherein, the memory is used to store programs;

[0075] The processor is used to implement the above-mentioned methods when executing the program in the memory;

[0076] The bus system is used to connect the memory and the processor so that the memory and the processor can communicate with each other.

[0077] Another aspect of the present application provides a computer-readable storage medium, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer is enabled to execute the above-mentioned methods.

[0078] It can be seen from the above technical solutions that the embodiments of the present application have the following beneficial effects:

[0079] By obtaining the speech recognition text corresponding to the original speech, the original speech can be vectorized to obtain the audio coding vector, and each text unit in the speech recognition text can be vectorized to obtain the text coding vector corresponding to each text unit, and then the attention mechanism is used for the audio coding vector and the i-th text coding vector to interactively obtain the fusion vector of the i-th text unit, and then the fusion vector of the i-th text unit and the text vector corresponding to the i-1-th target text unit can be input into the text error correction network, and the text error correction network outputs the text probability distribution corresponding to the fusion vector of the i-th text unit, and based on the text probability distribution corresponding to the fusion vector of the i-th text unit, the i-th target text unit is determined. In the above manner, the speech-assisted recognition text unit can be increased by using the fusion vector obtained by the attention mechanism for the audio coding vector corresponding to the original speech and the text coding vector corresponding to each text unit in the speech recognition text recognized by ASR, so that each text unit in the speech recognition text can be better predicted and corrected based on the fusion vector through the text error correction network, so as to improve the accuracy of error correction of the speech recognition text recognized by ASR. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 This is a schematic diagram of the structure of the text data control system in the embodiment of the present application;

[0081] Figure 2 is a flow chart of an embodiment of a method for processing speech recognition text in an embodiment of the present application;

[0082] Figure 3 is another embodiment flow chart of the method for processing speech recognition text in the embodiment of the present application;

[0083] Figure 4 is another embodiment flow chart of the method for processing speech recognition text in the embodiment of the present application;

[0084] Figure 5 is another embodiment flow chart of the method for processing speech recognition text in the embodiment of the present application;

[0085] Figure 6 is another embodiment flow chart of the method for processing speech recognition text in the embodiment of the present application;

[0086] Figure 7 is another embodiment flow chart of the method for processing speech recognition text in the embodiment of the present application;

[0087] Figure 8 is another embodiment flow chart of the method for processing speech recognition text in the embodiment of the present application;

[0088] Fig. 9 is another embodiment flow chart of the method for processing speech recognition text in the embodiment of the present application;

[0089] Fig.10 is another embodiment flow chart of the method for processing speech recognition text in the embodiment of the present application;

[0090] Fig.11 is another embodiment flow chart of the method for processing speech recognition text in the embodiment of the present application;

[0091] Fig.12 is another embodiment flow chart of the method for processing speech recognition text in the embodiment of the present application;

[0092] Fig.13 is another embodiment flow chart of the method for processing speech recognition text in the embodiment of the present application;

[0093] Fig.14 is another embodiment flow chart of the method for processing speech recognition text in the embodiment of the present application;

[0094] Fig.15 is another embodiment flow chart of the method for processing speech recognition text in the embodiment of the present application;

[0095] Fig.16 It is a schematic diagram of a principle flow chart of a method for processing speech recognition text in an embodiment of the present application;

[0096] Fig.17 It is another principle flow chart of the method for processing speech recognition text in the embodiment of the present application;

[0097] Fig.18 It is another principle flow chart of the method for processing speech recognition text in the embodiment of the present application;

[0098] Fig.19 It is a schematic diagram of the principle flow of an automatic speech recognition model in the method for processing speech recognition text in an embodiment of the present application;

[0099] Fig. 20 It is a schematic diagram of an error detection label in the method for processing speech recognition text in an embodiment of the present application;

[0100] FIG. 21( a ) is a schematic diagram of an interface for obtaining original speech in a method for processing speech recognition text in an embodiment of the present application;

[0101] FIG. 21( b ) is a schematic diagram of an interface for completing original speech acquisition in a method for processing speech recognition text in an embodiment of the present application;

[0102] FIG. 22( a ) is a schematic diagram of an interface of a speech recognition text of a method for processing speech recognition text in an embodiment of the present application;

[0103] FIG. 22( b ) is a schematic diagram of an interface for completing speech recognition text optimization in the method for processing speech recognition text in an embodiment of the present application;

[0104] Fig.23 It is a schematic diagram comparing error correction results of the method for processing speech recognition text in an embodiment of the present application;

[0105] Fig.24 It is a schematic diagram of error correction effect analysis of the method for processing speech recognition text in the embodiment of the present application;

[0106] Fig.25 It is a schematic diagram of an embodiment of a device for processing speech recognition text in an embodiment of the present application;

[0107] Fig.26 It is a schematic diagram of an embodiment of a computer device in an embodiment of the present application. DETAILED DESCRIPTION

[0108] The embodiments of the present application provide a method, apparatus, device and storage medium for processing speech recognition text, which are used to increase speech-assisted recognition text units by obtaining fusion vectors using an attention mechanism for audio coding vectors and text coding vectors, and then recognize the fusion vectors through a text error correction network, so as to better predict and correct each text unit in the speech recognition text, thereby improving the accuracy of error correction of the speech recognition text recognized by ASR.

[0109] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims and drawings of the present application are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "corresponding to" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0110] To facilitate understanding, some terms or concepts involved in the embodiments of the present application are first explained.

[0111] 1. Automatic Speech Recognition (ASR)

[0112] Automatic speech recognition refers to the process of converting audio into text.

[0113] 2. Word Error Rate (WER)

[0114] The word error rate is an indicator to measure the degree of ASR errors, including insertion, deletion and substitution errors generated after ASR recognition.

[0115] 3. ASR error correction

[0116] ASR error correction refers to the process of transcribing audio based on ASR and correcting incorrect words in the transcribed text into correct words.

[0117] 4. Sequence-to-Sequence Model

[0118] A sequence-to-sequence model is a model consisting of an encoder and a decoder, which can input a text sequence into the encoder and output a text sequence from the decoder.

[0119] It can be understood that in the specific implementation of this application, related data such as original speech and speech recognition text are involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0120] It can be understood that the processing method of speech recognition text as disclosed in the present application specifically relates to an Intelligent Vehicle Infrastructure Cooperative System (IVICS), and the Intelligent Vehicle Infrastructure Cooperative System is further introduced below. The Intelligent Vehicle Infrastructure Cooperative System is referred to as the Vehicle Infrastructure Cooperative System, which is a development direction of the Intelligent Transportation System (ITS). The vehicle-road cooperative system adopts advanced wireless communications and new generation Internet technologies to implement dynamic real-time information interaction between vehicles and roads in all directions, and carries out active vehicle safety control and road cooperative management based on the collection and integration of dynamic traffic information in all time and space, fully realizing the effective coordination of people, vehicles and roads, ensuring traffic safety, and improving traffic efficiency, thereby forming a safe, efficient and environmentally friendly road traffic system.

[0121] It is understandable that the processing method of speech recognition text disclosed in this application also involves artificial intelligence (AI) technology, and artificial intelligence technology is further introduced below. Artificial intelligence is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that the machines have the functions of perception, reasoning and decision-making.

[0122] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0123] Secondly, natural language processing (NLP) is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can achieve effective communication between people and computers using natural language. Natural language processing is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field will involve natural language, that is, the language people use in daily life, so it is closely related to the study of linguistics. Natural language processing technology usually includes text processing, semantic understanding, machine translation, robot question answering, knowledge graph and other technologies.

[0124] Secondly, Machine Learning (ML) is a multi-disciplinary interdisciplinary subject, involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0125] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, drones, robots, smart medical care, smart customer service, etc. I believe that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0126] It should be understood that the speech recognition text processing method provided in the present application can be applied to various scenarios, including but not limited to artificial intelligence, maps, smart transportation, cloud technology, etc., and is used to optimize the speech recognition text by correcting errors in the recognized speech recognition text, so as to be applied to scenarios such as oral practice, intelligent voice interaction, intelligent voice recognition or intelligent voice translation.

[0127] In order to solve the above problems, this application proposes a method for processing speech recognition text, which is applied to Figure 1 Text data control system shown, see Figure 1 , Figure 1 Schematic diagram of a text data control system in an embodiment of the present application. Figure 1 As shown, by acquiring the speech recognition text corresponding to the original speech provided by the terminal device, the server can perform vector conversion on the original speech to obtain the audio coding vector, and perform vector conversion on each text unit in the speech recognition text to obtain the text coding vector corresponding to each text unit, and then use the attention mechanism for the audio coding vector and the i-th text coding vector to interactively obtain the fusion vector of the i-th text unit, and then the fusion vector of the i-th text unit and the text vector corresponding to the i-1-th target text unit can be input into the text correction network, and the text correction network outputs the text probability distribution corresponding to the fusion vector of the i-th text unit, and based on the text probability distribution corresponding to the fusion vector of the i-th text unit, the i-th target text unit is determined. In the above manner, the speech-assisted recognition text unit can be increased by using the fusion vector obtained by the attention mechanism for the audio coding vector corresponding to the original speech and the text coding vector corresponding to each text unit in the speech recognition text recognized by ASR, so that each text unit in the speech recognition text can be better predicted and corrected based on the fusion vector through the text correction network, so as to improve the accuracy of correcting the speech recognition text recognized by ASR.

[0128] Understandably, Figure 1Only one type of terminal device is shown in the figure. In actual scenarios, more types of terminal devices may participate in the data processing process. Terminal devices include but are not limited to mobile phones, computers, intelligent voice interaction devices, smart home appliances, car terminals, etc. The specific number and type depend on the actual scenario and are not limited here. Figure 1 One server is shown in the figure, but in actual scenarios, multiple servers may also be involved, especially in the scenario of multi-model training interaction. The number of servers depends on the actual scenario and is not limited here.

[0129] It should be noted that in this embodiment, the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The terminal device and the server can be directly or indirectly connected through wired or wireless communication, and the terminal device and the server can be connected to form a blockchain network, which is not limited in this application.

[0130] Combined with the above introduction, the following will introduce the processing method of speech recognition text in this application, please refer to Figure 2 In one embodiment of the present application, a method for processing speech recognition text includes:

[0131] In step S101, a speech recognition text corresponding to the original speech is obtained;

[0132] In this embodiment, the server may input the acquired original speech into a natural language recognition model (such as an ASR model) for recognition to obtain text recognized by the ASR model, that is, speech recognition text corresponding to the original speech.

[0133] Specifically, since the target object can perform a voice recording operation through the client, the client can respond to the recording operation and collect the content spoken by the target object (such as long sentences, short sentences or phrases, etc.) through the recording interface of the recording program to obtain the original voice of the target object. Alternatively, when the target object is interacting with a terminal device (such as an intelligent voice interaction system, an intelligent voice recognition system, etc.), the audio collection device in the terminal device automatically collects the sound to obtain the original voice. Alternatively, the original voice can also be obtained in other ways, which are not specifically limited here.

[0134] Furthermore, the terminal device may input the acquired original speech into a natural language recognition model (such as an ASR model) for recognition, so as to obtain a speech recognition text corresponding to the original speech recognized by the ASR model, or Fig.16 As shown, the terminal device can upload the acquired original speech to the server and store it accordingly with the target object identifier for subsequent use or query. Then, the server can input the acquired original speech into a natural language recognition model (such as an ASR model) for recognition to obtain the speech recognition text corresponding to the original speech recognized by the natural language recognition model (such as an ASR model).

[0135] For example, in an oral practice scenario, as shown in FIG21(a), to obtain the speech recognition text corresponding to the original speech, you can first use the recording interface of the original speech. The target object can give an oral answer according to the question displayed in the recording interface, and click the start recording button to start the audio collection device to collect the oral answer of the target object. After the recording is completed, click the end recording button in the original speech acquisition interface as shown in FIG21(b) to complete the oral collection and obtain the original speech. Furthermore, the obtained original speech can be input into a natural language recognition model (such as an ASR model) for recognition to obtain the speech recognition text recognized by the natural language recognition model (such as an ASR model), and the obtained speech recognition text is displayed in the speech recognition text interface as shown in FIG22(a).

[0136] In step S102, the original speech is vector-converted to obtain an audio coding vector;

[0137] In this embodiment, after the original speech is acquired, the original speech may be vector-converted to acquire an audio coding vector that is easy for a computer to recognize.

[0138] Specifically, Fig.16 As shown, after obtaining the original speech, the server can input the original speech into a model (such as a multimodal error correction model) for processing, where Fig.17 As shown, the multimodal error correction model includes a coding part (such as Fig.17 A1 part as shown) and decoding part (as shown Fig.17 The encoding part consists of two encoders, namely the audio encoder (such as Fig.17 The Wav2vetor encoder of the A1 part shown) and the text encoder (such as Fig.17The BERT encoder of the A1 part shown in the figure) can be used to convert the original speech into a vector through an audio encoder. Specifically, the original speech can be converted into a single-frame speech signal, and each single-frame speech signal can be vectorized through an audio encoder to obtain an audio coding vector corresponding to each single-frame speech signal.

[0139] In step S103, vector conversion is performed on each text unit in the speech recognition text to obtain a text encoding vector corresponding to each text unit;

[0140] In this embodiment, after the speech recognition text corresponding to the original speech is obtained, each text unit in the speech recognition text may be vector-converted to obtain a text encoding vector corresponding to each text unit that is easy for a computer to recognize.

[0141] The text unit may be specifically expressed as a word (such as "you", "is", etc.), a character (such as "I", "you", etc.), or a word (such as "student", "reading", etc.).

[0142] Specifically, Fig.16 As shown, after obtaining the speech recognition text corresponding to the original speech, the server can input the speech recognition text into a model (such as a multimodal error correction model) for processing, where Fig.17 As shown, the text encoder of the multimodal error correction model (such as Fig.17 The BERT encoder of part A1 shown in the figure) performs vector conversion on the speech recognition text, specifically, it can be through word segmentation processing on the speech recognition text to obtain text units, and then each text unit is vector converted through a text encoder to obtain a text encoding vector corresponding to each text unit.

[0143] In step S104, an attention mechanism is used on the audio encoding vector and the i-th text encoding vector to obtain a fusion vector of the i-th text unit after interaction, where i is an integer greater than or equal to 1;

[0144] In this embodiment, after obtaining the audio coding vector and the text coding vector, the audio coding vector and the text coding vector can be interacted, that is, the attention mechanism is used for the audio coding vector and the i-th text coding vector to obtain the fusion vector of the i-th text unit after the interaction.

[0145] Specifically, Fig.17 As shown, due to the encoding part of the multimodal error correction model (such as Fig.17The A1 part shown in the figure can obtain the output representation of two modes, namely the audio coding vector and the text coding vector. After obtaining the audio coding vector and the text coding vector, the audio coding vector and the text coding vector can be fused, that is, the attention mechanism (such as Fig.17 The Multi-headAttention of the A1 part shown in the figure is used to obtain multimodal information, that is, the fusion vector of the i-th text unit, to achieve the fusion between the original speech and the speech recognition text, so that the speech recognition text can be recognized and corrected based on the fusion vector in the future to increase speech-assisted recognition, thereby improving the accuracy of error correction of the corresponding speech recognition text to a certain extent.

[0146] Among them, the attention mechanism is widely used in natural language processing, speech recognition and other scenarios. The attention mechanism is used to model the relationship between query, key and value. The attention mechanism is defined as the following formula (1):

[0147]

[0148] Among them, Q refers to the query vector, which is the deep feature representation corresponding to each text unit, that is, the text encoding vector; K is the key vector, and V is the value vector; d k is the vector dimension of K and Q; key and value are the deep feature representation corresponding to each frame of speech signal in the original speech, that is, the audio coding vector.

[0149] In step S105, the fusion vector of the i-th text unit and the text vector corresponding to the i-1-th target text unit are input into a text error correction network, and the text probability distribution corresponding to the fusion vector of the i-th text unit is output through the text error correction network;

[0150] In this embodiment, after obtaining the fusion vector of the i-th text unit, the fusion vector of the i-th text unit and the text vector corresponding to the i-1-th target text unit can be input into the text error correction network. The i-th target text unit is predicted by the text error correction network, and the text probability distribution corresponding to the fusion vector of the i-th text unit can be output.

[0151] Specifically, Fig.17 As shown, the multimodal error correction model also includes a decoding part (such as Fig.17 The decoding part consists of a decoder (such as Fig.17The BERT decoder of the A2 part shown in the figure is composed of a text error correction network. Therefore, after obtaining the fusion vector of the ith text unit, the fusion vector of the ith text unit and the text vector corresponding to the i-1th target text unit can be input into the decoder, i.e., the text error correction network. The ith target text unit is predicted by the text error correction network, and the text probability distribution corresponding to the fusion vector of the ith text unit can be output.

[0152] For example, assuming that the fusion vector of the i-th text unit is the fusion vector corresponding to the first text unit (such as a word "he"), the fusion vector corresponding to the first text unit (such as a word "he") and a pre-set random vector such as y0 can be input into a text error correction network (such as a BERT decoder), and the probability distribution corresponding to each character or word in the character table that the first text unit can be corrected can be output, that is, the text probability distribution.

[0153] Similarly, for example, assuming that the fusion vector of the i-th text unit is the fusion vector corresponding to the second text unit (such as a word "is"), the fusion vector corresponding to the second text unit (such as a word "is") and the text vector corresponding to the first target text unit (such as the word "he"), such as y1, can be input into the text error correction network (such as the BERT decoder), and the probability distribution corresponding to each character or word in the character table that can be corrected to the second text unit can be output, that is, the text probability distribution.

[0154] In step S106, the i-th target text unit is determined based on the text probability distribution corresponding to the fusion vector of the i-th text unit.

[0155] In this embodiment, after obtaining the text probability distribution corresponding to the fusion vector of the i-th text unit, each character or word in the character table can be filtered or screened based on the text probability distribution corresponding to the fusion vector of the i-th text unit, and then the screened characters or words can be determined as the i-th target text unit.

[0156] Specifically, the speech recognition text can be expressed as an article, paragraph or sentence in Chinese, English or other languages ​​to be corrected, or other representations, which are not specifically limited here. Furthermore, after obtaining the text probability distribution corresponding to the fusion vector of the i-th text unit, the maximum text probability corresponding to the i-th text unit in the speech recognition text can be screened out from the text probability distribution, and the character or word corresponding to the maximum text probability can be used as the target text unit, so that the target corrected text corresponding to the speech recognition text can be sorted out, so that the corrected target corrected text can be subsequently applied to scenarios such as intelligent recognition, intelligent retrieval or intelligent translation.

[0157] For example, in an oral practice scenario, the acquired speech recognition text is displayed in the speech recognition text interface as shown in FIG22(a). Then, the speech recognition text can be input into the multimodal error correction model for error correction to obtain the optimized target semantic recognition text as shown in FIG22(b).

[0158] In an embodiment of the present application, a method for processing speech recognition text is provided. Through the above-mentioned method, the speech-assisted recognition text unit can be increased by using the fusion vector obtained by the attention mechanism for the audio coding vector corresponding to the original speech and the text coding vector corresponding to each text unit in the speech recognition text recognized by ASR, so that each text unit in the speech recognition text can be better predicted and corrected based on the fusion vector through the text error correction network, so as to improve the accuracy of error correction of the speech recognition text recognized by ASR.

[0159] Optionally, in the above Figure 2 Based on the corresponding embodiment, in another optional embodiment of the method for processing speech recognition text provided in the embodiment of the present application, as Figure 3 As shown, in step S104, after the attention mechanism is used for the audio encoding vector and the i-th text encoding vector respectively to obtain the fusion vector of the i-th text unit after interaction, the method further includes:

[0160] In step S301, the fusion vector of the i-th text unit is input into the text recognition network, and the error probability value corresponding to the fusion vector of the i-th text unit is output through the text recognition network;

[0161] In step S302, when the error probability value corresponding to the fusion vector of the i-th text unit is 0, the i-th error text unit is determined;

[0162] In step S303, when the error probability value corresponding to the fusion vector of the i-th text unit is 1, the i-th correct text unit is determined.

[0163] In this embodiment, after obtaining the fusion vector of the i-th text unit, the fusion vector of the i-th text unit can be input into the text recognition network, and each text unit is predicted by the text recognition network, and the error probability value corresponding to the fusion vector of the i-th text unit is output. The error probability value corresponding to the fusion vector of each text unit can be used to determine whether the text unit has an error. If the error probability value corresponding to the fusion vector of the i-th text unit is 0, the i-th erroneous text unit can be determined. Conversely, if the error probability value corresponding to the fusion vector of the i-th text unit is 1, the i-th correct text unit can be determined. An auxiliary task, namely, a text recognition network, can be introduced in the decoding part to judge the errors of the speech recognition text recognized by the ASR, that is, to recognize each word or character in a sentence incorrectly, thereby further improving the accuracy of error correction of the speech recognition text.

[0164] Among them, the text recognition network can be specifically expressed as a model obtained by pre-training based on the natural language model BERT, or can be expressed as other models. There is no specific limitation here. The text recognition network can be used as a model for classifying and predicting the fusion vector of each text unit.

[0165] Specifically, as shown in 17, after obtaining the fusion vector of the i-th text unit, the fusion vector of the i-th text unit can be input into the text recognition network, and the i-th text unit can be predicted by the text recognition network, and the error probability value corresponding to the fusion vector of the i-th text unit is output. Then, the error probability value corresponding to the fusion vector of each text unit can be used to determine whether the text unit has an error. For example, if the error probability value corresponding to the fusion vector of the i-th text unit is 0, it can be understood that there is an error in the text unit, and the text unit can be determined to be an erroneous text unit. On the contrary, if the error probability value corresponding to the fusion vector of the i-th text unit is 1, it can be understood that there is no error in the text unit, and the text unit can be determined to be a correct text unit.

[0166] For example, Fig.18 As shown, assuming that a speech recognition text has three text units such as "He is grandparent", then, after obtaining the fusion vectors corresponding to the three text units, they can be input into the text recognition network. The three text units are predicted by the text recognition network respectively. Assuming that the error probability value corresponding to the text unit "He" is "1", the error probability value corresponding to the text unit "is" is "1", and the error probability value corresponding to the text unit "grandparent" is "0", it can be determined that the text unit "He" is the correct text unit, the text unit "is" is the correct text unit, and the text unit "grandparent" is the wrong text unit.

[0167] Optionally, in the above Figure 2 Based on the corresponding embodiment, in another optional embodiment of the method for processing speech recognition text provided in the embodiment of the present application, as Figure 4 As shown, step S104 uses the attention mechanism for the audio encoding vector and the i-th text encoding vector respectively to obtain the fusion vector of the i-th text unit after interaction, including:

[0168] In step S401, the text encoding vector corresponding to each text unit and the audio encoding vector are combined using an attention mechanism to obtain an audio vector corresponding to each text encoding vector;

[0169] In step S402, the audio vector corresponding to the ith text unit and the text encoding vector corresponding to the ith text unit are summed to obtain a fusion vector of the ith text unit.

[0170] In this embodiment, after obtaining the audio coding vector and the text coding vector, the audio coding vector and the text coding vector can be interacted to realize the fusion between the original speech and the speech recognition text, that is, the attention mechanism is adopted for the text coding vector and the audio coding vector corresponding to each text unit to obtain the audio vector corresponding to each text unit, and then, the audio vector corresponding to the i-th text unit and the text coding vector corresponding to the i-th text unit can be summed to obtain the fusion vector of the i-th text unit. In this way, the speech recognition text can be recognized and corrected based on the fusion vector in the future to increase speech-assisted recognition, thereby improving the accuracy of error correction of the corresponding speech recognition text to a certain extent.

[0171] Specifically, Fig.18 As shown, due to the encoding part of the multimodal error correction model ( Fig.18 The Wav2vector encoder and BERT encoder shown in the figure can obtain output representations of two modes, namely, audio encoding vectors and text encoding vectors. After obtaining the audio encoding vectors and text encoding vectors, the audio encoding vectors and text encoding vectors can be fused, that is, the text encoding vector corresponding to each text unit and the audio encoding vector are combined using an attention mechanism (such as Fig.18 The following formula (2) can be used to obtain the audio vector corresponding to each text unit:

[0172]

[0173] in, Used to represent the audio vector corresponding to the i-th text encoding vector; Used to represent the text encoding vector corresponding to the i-th text unit; Hspeech is the set of audio vectors corresponding to the i-th text unit.

[0174] Furthermore, the following formula (3) can be used to sum the audio vector corresponding to the i-th text unit and the text encoding vector corresponding to the i-th text unit to obtain the fusion vector of the i-th text unit:

[0175]

[0176] in, Used to represent the audio vector corresponding to the i-th text encoding vector; Used to represent the text encoding vector corresponding to the i-th text unit; h i Used to represent the fusion vector corresponding to the i-th text unit.

[0177] Optionally, in the above Figure 2 Based on the corresponding embodiment, in another optional embodiment of the method for processing speech recognition text provided in the embodiment of the present application, as Figure 5 As shown, step S102 performs vector conversion on the original speech to obtain an audio coding vector, including:

[0178] In step S501, the original speech is converted into a single-frame speech signal;

[0179] In step S502, each single-frame speech signal is vector-converted to obtain an audio coding vector corresponding to each single-frame speech signal.

[0180] In this embodiment, after obtaining the original speech, the original speech can be first converted into a single-frame speech signal, and each single-frame speech signal can be vector-converted to obtain the audio coding vector corresponding to each single-frame speech signal, so that the audio coding vector corresponding to each single-frame speech signal can be better interacted with the text coding vector corresponding to each text unit to obtain a more accurate fusion vector, so that speech can better assist text recognition, thereby improving the accuracy of speech recognition text correction to a certain extent.

[0181] Specifically, Fig.18 As shown, before the original speech is converted into a vector, the original speech can be converted into a single-frame speech signal. For example, an original speech can be divided into “x1, x2, …, x m Then, each single-frame speech signal can be input into an audio encoder (such as Fig.18 In the Wav2vector encoder shown in FIG. Fig.18The Wav2vector encoder shown in the figure performs vector conversion on each single-frame speech signal to obtain the audio coding vector corresponding to each single-frame speech signal.

[0182] Optionally, in the above Figure 2 Based on the corresponding embodiment, in another optional embodiment of the method for processing speech recognition text provided in the embodiment of the present application, as Figure 6 As shown, in step S105, the fusion vector of the i-th text unit and the text vector corresponding to the i-1-th target text unit are input into the text error correction network, and before the text error correction network outputs the text probability distribution corresponding to the fusion vector of the i-th text unit, the training of the text error correction network includes the following steps:

[0183] In step S601, a sample speech recognition text corresponding to the sample speech and a text unit label corresponding to the sample speech recognition text are obtained;

[0184] In this embodiment, the server can input the acquired sample speech into a natural language recognition model (such as an ASR model) for recognition to obtain the text recognized by the ASR model, that is, the sample speech recognition text corresponding to the sample speech. At the same time, the text unit label corresponding to the sample speech recognition text can be obtained.

[0185] Among them, the text unit label refers to the label of the actual word or character, which is the result of aligning the text obtained by ASR recognition with the manually transcribed text.

[0186] Specifically, during the interaction through terminal devices (such as intelligent voice interaction systems, intelligent voice recognition systems, etc.), the audio acquisition device in the terminal device can automatically collect massive voices to obtain massive sample voices, or sample voices can be obtained through other methods, which are not specifically limited here.

[0187] Furthermore, the terminal device may input the acquired sample speech into a natural language recognition model (such as an ASR model) for recognition, so as to obtain the sample speech recognition text corresponding to the sample speech recognized by the ASR model, or Fig.16 As shown, the terminal device can upload the acquired sample voice to the server, and then the server can input the acquired sample voice into a natural language recognition model (such as an ASR model) for recognition to obtain the voice recognition text corresponding to the sample voice recognized by the ASR model.

[0188] In step S602, the sample speech is vector-converted to obtain a sample speech coding vector;

[0189] In this embodiment, after the sample speech is acquired, the sample speech may be vector-converted to acquire a sample coding vector that is easy for a computer to recognize.

[0190] Specifically, after obtaining the sample speech, the server may input the sample speech into a model (such as a multimodal error correction model) for processing, wherein, Fig.17 As shown, the audio encoder in the multimodal error correction model (such as Fig.17 The Wav2vector encoder in the A1 part shown in the figure performs vector conversion on the sample speech, specifically, by converting the sample speech into a single-frame sample speech signal, and performing vector conversion on each single-frame sample speech signal through an audio encoder (such as the Wav2vector encoder) to obtain a sample speech coding vector corresponding to each single-frame sample speech signal.

[0191] In step S603, vector conversion is performed on each text unit in the sample speech recognition text to obtain a sample encoding vector corresponding to each text unit;

[0192] In this embodiment, after obtaining the sample speech recognition text corresponding to the sample speech, each text unit in the sample speech recognition text may be vector-converted to obtain a sample encoding vector corresponding to each text unit that is easy for a computer to recognize.

[0193] Specifically, Fig.17 As shown, after obtaining the sample speech recognition text corresponding to the sample speech, the server can input the sample speech recognition text into a model (such as a multimodal error correction model) for processing, and can use the text encoder of the multimodal error correction model (such as Fig.17 The BERT encoder in the A1 part shown in the figure) performs vector conversion on the sample speech recognition text, specifically, it can be through word segmentation processing on the sample speech recognition text to obtain text units, and each text unit is vector converted through a text encoder (such as a BERT encoder) to obtain a sample encoding vector corresponding to each text unit.

[0194] In step S604, an attention mechanism is used on the sample speech coding vector and the i-th sample coding vector to obtain a sample fusion vector of the i-th text unit after interaction;

[0195] In this embodiment, after obtaining the sample speech coding vector and the sample coding vector, the sample speech coding vector and the sample coding vector can be interacted, that is, the attention mechanism is adopted for the sample speech coding vector and the i-th sample coding vector to obtain the sample fusion vector of the i-th text unit after the interaction.

[0196] Specifically, Fig.17As shown in FIG. 1 , since the sample speech and the sample speech recognition text are used as the input of the multimodal error correction model, the encoding part of the multimodal error correction model (such as Fig.17 The output representation of the two modes can be obtained, namely, the sample speech coding vector and the sample coding vector. After obtaining the sample speech coding vector and the sample coding vector, the sample speech coding vector and the sample coding vector can be fused, that is, the sample speech coding vector and the i-th sample coding vector are subjected to an attention mechanism (such as Fig.17 The Multi-head Attention of part A1 shown in the figure is used to obtain multimodal information, that is, the sample fusion vector of the i-th text unit, to achieve the fusion between the sample speech and the sample speech recognition text, to increase the speech-assisted recognition, so that the subsequent fusion vector can assist the basic text correction network to better learn the ability to recognize speech-like text units, thereby improving the accuracy of the correction of the corresponding speech recognition text to a certain extent.

[0197] In step S605, the sample fusion vector of the i-th text unit and the text vector corresponding to the i-1-th predicted text unit are input into the basic text error correction network, and the text probability distribution corresponding to the sample fusion vector of the i-th text unit is output through the basic text error correction network;

[0198] In this embodiment, after obtaining the sample fusion vector of the i-th text unit, the sample fusion vector of the i-th text unit and the text vector corresponding to the i-1-th predicted text unit can be input into the basic text error correction network. The i-th predicted text unit is predicted by the basic text error correction network, and the text probability distribution corresponding to the sample fusion vector of the i-th text unit can be output.

[0199] Specifically, Fig.17 As shown, the multimodal error correction model also includes a decoding part (such as Fig.17 The decoding part consists of a decoder (such as Fig.17 Therefore, after obtaining the sample fusion vector of the ith text unit, the sample fusion vector of the ith text unit and the text vector corresponding to the i-1th predicted text unit can be input into the decoder, i.e., the text correction network (such as the BERT decoder). The ith predicted text unit is predicted by the text correction network (such as the BERT decoder), and the text probability distribution corresponding to the sample fusion vector of the ith text unit can be output.

[0200] For example, assuming that the sample fusion vector of the i-th text unit is the sample fusion vector corresponding to the third text unit (such as a word "our"), the sample fusion vector corresponding to the third text unit (such as a word "our") and the text vector corresponding to the second predicted text unit (such as the word "is"), such as y2, can be input into the text error correction network (such as the BERT decoder), and the probability distribution corresponding to each character or word in the character table that the third text unit can be corrected can be output, that is, the text probability distribution.

[0201] In step S606, based on the text unit label and the text probability distribution corresponding to the sample fusion vector of the i-th text unit, the error correction loss value is calculated;

[0202] In this embodiment, after obtaining the text probability distribution corresponding to the sample fusion vector of the i-th text unit, the error correction loss value can be calculated based on the text unit label and the text probability distribution corresponding to the sample fusion vector of the i-th text unit, so that the parameters of the basic text correction network can be better adjusted based on the obtained error correction loss value until the basic text correction network converges, so as to better obtain a text correction network that can be used for speech recognition text correction.

[0203] Specifically, after obtaining the text probability distribution corresponding to the sample fusion vector of the i-th text unit, the error correction loss value can be calculated based on the text unit label and the text probability distribution corresponding to the sample fusion vector of the i-th text unit using the following formula (4), that is, the cross entropy loss of the actual word or character label and the predicted text unit probability:

[0204]

[0205] Among them, L cor Used to indicate the error correction loss value, The text unit label used to represent the i-th text unit, Used to represent the text probability distribution corresponding to the sample fusion vector of the i-th text unit.

[0206] In step S607, parameters of the basic text error correction network are adjusted based on the error correction loss value to obtain a text error correction network.

[0207] Specifically, based on the error correction loss function in the above formula (4), the text unit label and the text probability distribution corresponding to the sample fusion vector of the i-th text unit are calculated to obtain the error correction loss value, and based on the error correction loss value, the parameters of the basic text error correction network are adjusted, such as by using the back propagation gradient descent method, until the model parameters of the basic text error correction network tend to be stable and the basic text error correction network converges, so that a trained text error correction network can be obtained to achieve accurate error correction of speech recognition text.

[0208] Optionally, in the above Figure 6 Based on the corresponding embodiment, in another optional embodiment of the method for processing speech recognition text provided in the embodiment of the present application, as Figure 7 As shown, after the attention mechanism is applied to the sample speech coding vector and the i-th sample coding vector in step S604 to obtain the sample fusion vector of the i-th text unit after interaction, the method further includes: steps S701 to S703, and step S607 includes: step S704;

[0209] In step S701, an error detection label corresponding to the sample speech recognition text is obtained;

[0210] Specifically, when the sample speech recognition text is obtained, an error detection label corresponding to the sample speech recognition text can also be obtained.

[0211] Among them, Fig. 20 As shown, the error detection label refers to the detection label of the sample speech text recognized by ASR with recognition errors, which is the result obtained by aligning the text recognized by ASR with the manually transcribed text.

[0212] In step S702, the sample fusion vector of each text unit is input into the basic text recognition network, and the error probability value corresponding to the sample fusion vector of each text unit is output through the basic text recognition network;

[0213] In this embodiment, after obtaining the sample fusion vector of each text unit, the sample fusion vector of each text unit can be input into the basic text recognition network. Each text unit can be predicted separately by the basic text recognition network, and the error probability value corresponding to the sample fusion vector of each text unit can be output.

[0214] Specifically, the sample fusion vector of each text unit is input into the basic text recognition network, and after linear transformation, the result of whether each text unit is recognized incorrectly can be obtained. Among them, the error probability value, including 0 and 1, is used to represent the prediction result of the basic text recognition network.

[0215] Among them, the generated data detection label is determined as follows Fig. 20 As shown in the figure, ASR is the ASR recognized text, that is, the sample speech recognition text, and the truth value is Truth, which is used to represent the corresponding correct text unit. For example, the first text unit in the sample speech recognition text recognized by ASR is the word "He", and the corresponding error detection label is Truth. Therefore, the error probability value corresponding to the first text unit is 1, which can be understood as no error, that is, the text unit is a correct text unit.

[0216] In step S703, based on the error detection label and the error probability value corresponding to the sample fusion vector of each text unit, the error detection loss value is calculated;

[0217] In this embodiment, after obtaining the error probability value corresponding to the sample fusion vector of each text unit, the error detection loss value can be calculated based on the error detection label and the error probability value corresponding to the sample fusion vector of each text unit, so that the parameters of the basic text error correction network and the text recognition network can be better adjusted based on the obtained error detection loss value until the basic text error correction network and the basic text recognition network converge, so as to better obtain a text error correction network that can be used for speech recognition text error correction and a text recognition network for speech recognition text error detection.

[0218] Specifically, after obtaining the error probability value corresponding to the sample fusion vector of each text unit, the error detection loss value can be calculated based on the error detection label and the error probability value corresponding to the sample fusion vector of each text unit using the following formula (5), that is, the cross entropy loss of the detection label of the text recognition error and the error probability value of the predicted text unit:

[0219]

[0220] Among them, L detec Used to indicate the error correction loss value, It is used to represent the error detection label of the i-th text unit. Used to represent the error probability value corresponding to the sample fusion vector of the i-th text unit.

[0221] In step S704, based on the error detection loss value and the error correction loss value, the parameters of the basic text recognition network and the basic text error correction network are adjusted to obtain a text error correction network and a text recognition network.

[0222] Specifically, based on the error correction loss function in the above formula (4), the text unit label and the text probability distribution corresponding to the sample fusion vector of the i-th text unit are calculated to obtain the error correction loss value. At the same time, based on the error detection loss function in the above formula (5), the error detection label and the error probability value corresponding to the sample fusion vector of each text unit are calculated to obtain the error detection loss value. Then, based on the error correction loss value and the error detection loss value, the basic text error correction network and the basic text recognition network can be jointly adjusted in parameters, such as by using the back propagation gradient descent method, until the model parameters of the basic text error correction network and the model parameters of the basic text recognition network tend to be stable, and the basic text error correction network and the text recognition network converge, and the trained text error correction network and the text recognition network can be obtained to further realize the accurate error correction of speech recognition text.

[0223] Furthermore, experiments show that if Fig.23 As shown, this embodiment compares the error correction results of the multimodal error correction model with multiple ASR error correction baselines, and compares them based on the word error rate WER. The compared work includes previous work by others, as well as an error correction model that only uses text (Text-only), a method that uses audio to fine-tune the ASR model (Audio-only), and traditional ASR results (E2E ASR). It can be seen from the results that by combining multimodal information, the WER can be reduced to 10.8, which is a significant improvement compared to the single-modal form of traditional ASR and previous work.

[0224] Furthermore, experiments show that if Fig.24 As shown, this embodiment conducts a case analysis on the error correction effect of the speech recognition text recognized by ASR, such as Fig.24 As shown, it is understandable that the traditional ASR recognition effect is poor for proper nouns, fixed phrases, and words with similar pronunciations. It can be used to solve the problems of proper nouns and fixed phrases, such as shan hai->shanghai, but the traditional ASR recognition cannot solve the errors of words with similar pronunciations, such as plane->place, clothes->close, etc.

[0225] Optionally, in the above Figure 7 Based on the corresponding embodiment, in another optional embodiment of the method for processing speech recognition text provided in the embodiment of the present application, as Figure 8 As shown, step S704 adjusts the parameters of the basic text recognition network and the basic text error correction network based on the error detection loss value and the error correction loss value to obtain the text recognition network and the text error correction network, including:

[0226] In step S801, based on the loss weight, the error detection loss value and the error correction loss value are weighted and summed to obtain a total loss value;

[0227] In step S802, based on the total loss value, the parameters of the basic text recognition network and the basic text error correction network are adjusted to obtain the text recognition network and the text error correction network.

[0228] In this embodiment, after obtaining the error detection loss value and the error correction loss value, in order to make the model parameters converge better and faster, this embodiment can perform weighted summation of the error detection loss value and the error correction loss value based on the loss weight to obtain the total loss value, and based on the total loss value, adjust the parameters of the basic text recognition network and the basic text error correction network to obtain the text recognition network and the text error correction network.

[0229] Specifically, after obtaining the error detection loss value and the error correction loss value, the following formula (6) can be used to perform weighted summation of the error detection loss value and the error correction loss value based on the loss weight to obtain the total loss value:

[0230] L total =L cor +α*L detec (6);

[0231] Among them, L detec Used to represent the error correction loss value, L cor Used to represent the error correction loss value, L total It is used to represent the total loss value. α is a hyperparameter set according to the actual application requirements, that is, the loss weight. There is no specific restriction here and it can usually be set to 0.05.

[0232] Furthermore, after obtaining the total loss value, the basic text error correction network and the basic text recognition network can be jointly adjusted in parameters, such as by using back-propagation gradient descent, until the model parameters of the basic text error correction network and the model parameters of the basic text recognition network tend to be stable and the basic text error correction network and the text recognition network converge, and the trained text error correction network and text recognition network can be obtained to further achieve accurate error correction of speech recognition text.

[0233] Optionally, in the above Figure 6 Based on the corresponding embodiment, in another optional embodiment of the method for processing speech recognition text provided in the embodiment of the present application, as Fig. 9 As shown, step S604 uses an attention mechanism on the sample speech coding vector and the i-th sample coding vector to obtain a sample fusion vector of the i-th text unit after interaction, including:

[0234] In step S901, the sample coding vector corresponding to each text unit and the sample speech coding vector are combined using an attention mechanism to obtain a sample speech vector corresponding to each sample speech coding vector;

[0235] In step S902, the sample speech vector corresponding to the i-th sample speech coding vector and the sample coding vector corresponding to the i-th text unit are summed to obtain a sample fusion vector of the i-th text unit.

[0236] In this embodiment, after obtaining the sample speech coding vector and the sample coding vector corresponding to each text unit, the sample speech coding vector can be interacted with the sample coding vector corresponding to each text unit respectively to realize the fusion between the sample speech and the sample speech recognition text, that is, the sample coding vector corresponding to each text unit and the sample speech coding vector adopt an attention mechanism to obtain the sample speech vector corresponding to each sample speech coding vector, and then, the sample speech vector corresponding to the i-th text unit and the sample coding vector corresponding to the i-th text unit can be summed to obtain the fusion vector of the i-th text unit, so that the sample speech recognition text can be subsequently recognized and corrected based on the sample fusion vector, so as to increase speech-assisted recognition and improve the ability of the basic text correction network and the basic text recognition network to learn to recognize text units with similar pronunciations, thereby improving the accuracy of the correction of the corresponding speech recognition text to a certain extent.

[0237] Specifically, Fig.18 As shown in FIG. 1 , since the sample speech and the sample speech recognition text are used as the input of the multimodal error correction model, the encoding part of the multimodal error correction model ( Fig.18 The Wav2vector encoder and BERT encoder shown in the figure can obtain the output representation of two modes, namely, the sample coding vector and the sample speech coding vector corresponding to each text unit. After obtaining the sample speech coding vector and the sample coding vector, the sample speech coding vector and the sample coding vector can be fused, that is, the text coding vector corresponding to each text unit and the audio coding vector are combined by using the attention mechanism (such as Fig.18 For the Muti-head Attention shown in the figure, the above formula (2) can be used to obtain the sample speech vector corresponding to each sample speech coding vector. Furthermore, the above formula (3) can be used to sum the sample speech vector corresponding to the i-th sample speech coding vector and the sample coding vector corresponding to the i-th text unit to obtain the sample fusion vector of the i-th text unit.

[0238] Optionally, in the above Figure 6Based on the corresponding embodiment, in another optional embodiment of the method for processing speech recognition text provided in the embodiment of the present application, as Fig.10 As shown, step S602 performs vector conversion on the sample speech to obtain a sample speech coding vector, including:

[0239] In step S1001, the sample speech is converted into a single-frame sample speech signal;

[0240] In step S1002, each single-frame sample speech signal is vector-converted to obtain a sample speech coding vector corresponding to each single-frame sample speech signal.

[0241] In this embodiment, after obtaining the sample speech, the sample speech can be first converted into a single-frame sample speech signal, and each single-frame sample speech signal can be vector-converted to obtain a sample speech coding vector corresponding to each single-frame sample speech signal, so that the sample speech coding vector corresponding to each single-frame sample speech signal can be better interacted with the sample coding vector corresponding to each text unit to obtain a more accurate sample fusion vector, so that speech can better assist text recognition, and the basic text error correction network and text recognition network can better learn the recognition ability of text units with similar pronunciations, thereby improving the accuracy of speech recognition text correction to a certain extent.

[0242] Specifically, Fig.18 As shown, before the sample speech is vectorized, the sample speech can be converted into a single-frame sample speech signal. For example, a sample speech can be divided into single-frame sample speech signals such as "x1, x2, ..., xm", and then each single-frame sample speech signal can be input into an audio encoder (such as a wav2vec encoder), and each single-frame sample speech signal can be vectorized by the audio encoder (such as a wav2vec encoder) to obtain a sample speech coding vector corresponding to each single-frame sample speech signal.

[0243] Optionally, in the above Figure 2 Based on the corresponding embodiment, in another optional embodiment of the method for processing speech recognition text provided in the embodiment of the present application, as Fig.11 As shown, step S101 obtains the speech recognition text corresponding to the original speech, including:

[0244] In step S1101, the original speech is input into the automatic speech recognition model, and the speech coding vector corresponding to the original speech is obtained through the pre-trained acoustic network in the automatic speech recognition model;

[0245] In this embodiment, after the original speech is obtained, the obtained original speech can be input into the automatic speech recognition model for text unit prediction, and the speech coding vector corresponding to the original speech can be obtained through the pre-trained acoustic network in the automatic speech recognition model.

[0246] The automatic speech recognition model can be specifically expressed as an ASR model, or other models, which are not specifically limited here. The ASR model is composed of an audio encoder and a text decoder. The ASR model takes the original speech as input and outputs the speech recognition text. The pre-trained acoustic network can be specifically expressed as a wav2vec 2.0 model, or other models, which are not specifically limited here. Wav2vec 2.0 is a model for self-supervised pre-training based on a large amount of unlabeled data, which is composed of a multi-layer convolutional feature encoding layer and a multi-layer transformer.

[0247] Specifically, Fig.16 As shown, after the original voice is obtained through the voice recording interface of the client, the obtained original voice can be uploaded to the server for storage, so that the server can input the received original voice into the automatic speech recognition model for text unit prediction, wherein the voice coding vector corresponding to the original voice can be obtained first through the pre-trained acoustic network in the automatic speech recognition model, that is, the audio encoder, specifically, by converting the original voice into a single-frame voice signal, and inputting each single-frame voice signal into the automatic speech recognition model, and then the single-frame voice vector corresponding to each single-frame voice signal can be obtained through the pre-trained acoustic network in the automatic speech recognition model, and based on the audio weight corresponding to each single-frame voice signal, the single-frame voice vector corresponding to each single-frame voice signal is weighted summed to obtain the voice coding vector.

[0248] In step S1102, the speech coding vector and the text vector corresponding to the i-1th recognized text unit are passed to a text decoder in the automatic speech recognition model to obtain a recognized text probability distribution corresponding to the i-th text unit;

[0249] In this embodiment, after obtaining the speech coding vector, the speech coding vector and the text vector corresponding to the i-1th recognized text unit can be input into the text decoder in the automatic speech recognition model. The i-th text unit is predicted by the text decoder in the automatic speech recognition model, and the recognized text probability distribution corresponding to the i-th text unit can be output.

[0250] Specifically, Fig.19As shown, after obtaining the speech coding vector, the speech coding vector and the text vector corresponding to the i-1th recognized text unit can be input into the text decoder in the automatic speech recognition model. The i-th recognized text unit is predicted by the text decoder in the automatic speech recognition model, and the recognized text probability distribution corresponding to the i-th text unit can be output.

[0251] For example, Fig.19 As shown, assuming that the text vector corresponding to the i-1th recognized text unit is the text vector corresponding to the 0th text unit, a random vector is pre-set such as y0, so the speech encoding vector and the text vector y0 corresponding to the 0th recognized text unit can be input into the text decoder (such as BERT decoder) in the automatic speech recognition model, and the probability distribution corresponding to each character or word in the character table that the first text unit can be recognized can be output, that is, the recognized text probability distribution.

[0252] Similarly, for example, assuming that the text vector corresponding to the i-1th recognized text unit is the text vector corresponding to the first text unit, such as y1, the speech encoding vector and the text vector y1 corresponding to the first recognized text unit can be input into the text decoder (such as BERT decoder) in the automatic speech recognition model, and the probability distribution corresponding to each character or word in the character table that the second text unit can be recognized can be output, that is, the recognized text probability distribution.

[0253] In step S1103, based on the recognition text probability distribution corresponding to the ith text unit, the ith recognition text unit is determined to obtain the speech recognition text.

[0254] In this embodiment, after obtaining the recognition text probability distribution corresponding to the i-th text unit, each character or word in the character table can be filtered or screened based on the recognition text probability distribution corresponding to the i-th text unit, and then the screened characters or words can be determined as the i-th recognition text unit, so that the speech recognition text can be sorted out.

[0255] Specifically, after obtaining the recognition text probability distribution corresponding to the i-th text unit, the maximum text probability corresponding to the i-th text unit can be screened out from the recognition text probability distribution, and the character or word corresponding to the maximum text probability can be used as the recognition text unit, so that the speech recognition text can be sorted out, so that the recognized speech recognition text can be corrected later.

[0256] For example, in an oral practice scenario, the acquired speech recognition text is displayed in an interface for acquiring the original speech as shown in FIG21(a), and in an interface for completing the acquisition of the original speech as shown in FIG21(b). After the original speech is acquired, the original speech can be input into the ASR model for recognition to obtain an interface for the semantically recognized text obtained by recognition as shown in FIG22(a).

[0257] Optionally, in the above Fig.11 Based on the corresponding embodiment, in another optional embodiment of the method for processing speech recognition text provided in the embodiment of the present application, as Fig.12 As shown, step S1101 inputs the original speech into the automatic speech recognition model, and obtains the speech coding vector corresponding to the original speech through the pre-trained acoustic network in the automatic speech recognition model, including:

[0258] In step S1201, the original speech is converted into a single-frame speech signal;

[0259] In step S1202, each single-frame speech signal is input into an automatic speech recognition model, and a single-frame speech vector corresponding to each single-frame speech signal is obtained through a pre-trained acoustic network in the automatic speech recognition model;

[0260] In step S1203, based on the audio weight corresponding to each single-frame speech signal, the single-frame speech vectors corresponding to each single-frame speech signal are weighted summed to obtain a speech coding vector.

[0261] In this embodiment, after obtaining the original speech, the original speech can be first converted into a single-frame speech signal, and then each single-frame speech signal can be input into the automatic speech recognition model. The single-frame speech vector corresponding to each single-frame speech signal is obtained through the pre-trained acoustic network in the automatic speech recognition model, and based on the audio weight corresponding to each single-frame speech signal, the single-frame speech vector corresponding to each single-frame speech signal is weighted and summed, so as to better and more accurately obtain the speech coding vector that is convenient for computer equipment to recognize and analyze.

[0262] Specifically, after obtaining the original voice through the voice recording interface of the client, the obtained original voice can be uploaded to the server for storage, so that the server can divide the received original voice into single-frame voice signals. Then, the obtained single-frame voice signals such as "x1, x2,..., xm" can be input into the pre-trained acoustic network in the automatic speech recognition model, i.e., the audio encoder (such as wav2vec 2.0 encoder), and the single-frame voice signal is encoded through the pre-trained acoustic network, i.e., the audio encoder (such as wav2vec 2.0 encoder), that is, each single-frame voice signal is converted into a corresponding single-frame voice vector, and based on the preset audio weight corresponding to each single-frame voice signal, the single-frame voice vector corresponding to each single-frame voice signal is weighted and summed to obtain a voice coding vector.

[0263] Optionally, in the above Fig.11 Based on the corresponding embodiment, in another optional embodiment of the method for processing speech recognition text provided in the embodiment of the present application, as Fig.13 As shown, in step S1101, the original speech is input into the automatic speech recognition model, and before the speech coding vector corresponding to the original speech is obtained through the pre-trained acoustic network in the automatic speech recognition model, the training of the automatic speech recognition model includes the following steps:

[0264] In step S1301, the sample speech is input into a basic automatic speech recognition model, and a sample speech vector corresponding to the sample speech is obtained through a pre-trained acoustic network in the basic automatic speech recognition model;

[0265] In this embodiment, after the sample speech is obtained, the obtained sample speech can be input into the basic automatic speech recognition model for text unit prediction, and the sample speech vector corresponding to the sample speech can be obtained through the pre-trained acoustic network in the basic automatic speech recognition model.

[0266] Specifically, Fig.19 As shown, after the sample speech is read in the database, the obtained sample speech can be input into the pre-trained acoustic network in the automatic speech recognition model, that is, the audio encoder (such as wav2vec 2.0 encoder) to obtain the sample speech vector corresponding to the sample speech. Specifically, the sample speech can be converted into a single-frame sample speech signal, and each single-frame sample speech signal is input into the basic automatic speech recognition model. The single-frame sample speech vector corresponding to each single-frame sample speech signal is obtained through the pre-trained acoustic network in the basic automatic speech recognition model. Then, based on the audio weight corresponding to each single-frame sample speech signal, the single-frame sample speech vector corresponding to each single-frame sample speech signal can be weighted summed to obtain the sample speech vector.

[0267] In step S1302, the sample speech vector and the text vector corresponding to the i-1th sample recognition text unit are passed to the text decoder in the basic automatic speech recognition model to obtain the sample recognition text probability distribution corresponding to the i-th text unit;

[0268] In this embodiment, after obtaining the sample speech vector, the sample speech vector and the text vector corresponding to the i-1th recognized text unit can be input into the text decoder in the automatic speech recognition model. The i-th text unit is predicted by the text decoder in the automatic speech recognition model, and the sample recognized text probability distribution corresponding to the i-th text unit can be output.

[0269] Specifically, Fig.19 As shown, after obtaining the sample speech vector, the sample speech vector and the text vector corresponding to the i-1th sample text unit can be input into the text decoder in the automatic speech recognition model. The i-th sample text unit is predicted by the text decoder in the automatic speech recognition model, and the sample recognition text probability distribution corresponding to the i-th text unit can be output.

[0270] In step S1303, the recognition loss value is calculated based on the sample recognition text probability distribution corresponding to the i-th text unit and the text unit label corresponding to the sample speech;

[0271] In this embodiment, after obtaining the sample recognition text probability distribution corresponding to the i-th text unit, the recognition loss value can be calculated based on the text unit label corresponding to the sample speech and the sample recognition text probability distribution of the i-th text unit, so that the parameters of the basic automatic speech recognition model can be better adjusted based on the obtained recognition loss value until the basic automatic speech recognition model converges, so as to better obtain an automatic speech recognition model that can be used to recognize text units.

[0272] Specifically, after obtaining the sample recognition text probability distribution of the i-th text unit, the cross entropy loss calculation can be performed on the label of the actual word or character and the predicted probability of the text unit based on the text unit label corresponding to the sample speech and the sample recognition text probability distribution of the i-th text unit to obtain the recognition loss value.

[0273] In step S1304, parameters of the basic automatic speech recognition model are adjusted based on the recognition loss value to obtain an automatic speech recognition model.

[0274] In this embodiment, the cross entropy loss is calculated for the text unit label corresponding to the sample speech and the sample recognition text probability distribution of the i-th text unit to obtain the recognition loss value, and based on the recognition loss value, the parameters of the basic automatic speech recognition model are adjusted, such as by using the back propagation gradient descent method, until the model parameters of the basic automatic speech recognition model tend to be stable and the basic automatic speech recognition model converges, so that a trained automatic speech recognition model can be obtained.

[0275] Optionally, in the above Fig.13 Based on the corresponding embodiment, in another optional embodiment of the method for processing speech recognition text provided in the embodiment of the present application, as Fig.14 As shown, step S1301 inputs the sample speech into the basic automatic speech recognition model, and obtains the sample speech vector corresponding to the sample speech through the pre-trained acoustic network in the basic automatic speech recognition model, including:

[0276] In step S1401, the sample speech is converted into a single-frame sample speech signal;

[0277] In step S1402, each single-frame sample speech signal is input into a basic automatic speech recognition model, and a single-frame sample speech vector corresponding to each single-frame sample speech signal is obtained through a pre-trained acoustic network in the basic automatic speech recognition model;

[0278] In step S1403, based on the audio weight corresponding to each single-frame sample speech signal, the single-frame sample speech vectors corresponding to each single-frame sample speech signal are weighted summed to obtain a sample speech vector.

[0279] In this embodiment, after obtaining the sample speech, the sample speech can be first converted into a single-frame sample speech signal, and then each single-frame sample speech signal can be input into the basic automatic speech recognition model. The single-frame sample speech vector corresponding to each single-frame sample speech signal is obtained through the pre-trained acoustic network in the basic automatic speech recognition model, and based on the audio weight corresponding to each single-frame sample speech signal, the single-frame sample speech vector corresponding to each single-frame sample speech signal is weighted and summed, so as to better and more accurately obtain the sample speech vector that is convenient for computer equipment to recognize and analyze.

[0280] Specifically, after reading the sample speech in the database, the obtained sample speech can be divided into single-frame sample speech signals, and then the obtained single-frame sample speech signals such as "x1, x2,..., xm" can be input into the pre-trained acoustic network, i.e., the audio encoder (such as wav2vec 2.0 encoder) in the basic automatic speech recognition model, and the single-frame sample speech signal is encoded by the pre-trained acoustic network, i.e., the audio encoder (such as wav2vec 2.0 encoder), that is, each single-frame sample speech signal is converted into a corresponding single-frame sample speech vector, and based on the preset audio weight corresponding to each single-frame sample speech signal, the single-frame sample speech vector corresponding to each single-frame sample speech signal is weighted and summed to obtain the sample speech vector.

[0281] Optionally, in the above Figure 2 Based on the corresponding embodiment, in another optional embodiment of the method for processing speech recognition text provided in the embodiment of the present application, as Fig.15 As shown, before step S101 acquires the speech recognition text corresponding to the original speech, the method further includes: step S1501; step S101 includes: S1502;

[0282] In step S1501, a speech recognition optimization instruction is received;

[0283] In step S1502, according to the speech recognition optimization instruction, the speech recognition text is read from the database.

[0284] In this embodiment, when a speech recognition optimization instruction is received from a target object, the speech recognition text can be read from a database according to the speech recognition optimization instruction, so that the target object can optimize the acquired speech recognition text at any time, thereby improving the user experience.

[0285] Specifically, Fig.16 As shown, in the target correspondence, the application installed on the terminal device can obtain oral answers to given questions in the oral practice scenario, and send the collected original speech and corresponding questions to the server through the application, so that the server can input the original speech into the ASR model for recognition, so as to obtain the speech recognition text for storage, and display the semantic recognition text obtained by the recognition of the application in the interface as shown in Figure 22 (a).

[0286] Furthermore, the target object can click a one-key optimization button according to the needs through the interface of the semantic recognition text obtained by recognition as shown in FIG22(a), so that the terminal device can generate a corresponding speech recognition optimization instruction in response to the target object's selection operation, and send the speech recognition optimization instruction to the server, so that the server reads the speech recognition text and the corresponding original speech from the database, and inputs the read speech recognition text and the corresponding original speech into the multimodal error correction model for error detection and correction, and combines the original speech and the speech recognition text through the multimodal error correction model, outputs the corrected speech recognition text, returns it to the server, sends it to the terminal device through the server, and displays it in the speech recognition text optimization interface of the application as shown in FIG22(b), so that the target object can obtain the optimized speech recognition text in a timely and intuitive manner.

[0287] The following is a detailed description of the speech recognition text processing device in this application. Fig.25 , Fig.25 This is a schematic diagram of an embodiment of a device for processing speech recognition text in an embodiment of the present application. The device 20 for processing speech recognition text includes:

[0288] An acquisition unit 201 is used to acquire a speech recognition text corresponding to the original speech;

[0289] The processing unit 202 is used to perform vector conversion on the original speech to obtain an audio coding vector;

[0290] The processing unit 202 is further used to perform vector conversion on each text unit in the speech recognition text to obtain a text encoding vector corresponding to each text unit;

[0291] The processing unit 202 is further configured to use an attention mechanism on the audio encoding vector and the i-th text encoding vector to obtain a fusion vector of the i-th text unit after interaction, where i is an integer greater than or equal to 1;

[0292] The processing unit 202 is further used to input the fusion vector of the i-th text unit and the text vector corresponding to the i-1-th target text unit into the text error correction network, and output the text probability distribution corresponding to the fusion vector of the i-th text unit through the text error correction network;

[0293] The determination unit 203 is used to determine the i-th target text unit based on the text probability distribution corresponding to the fusion vector of the i-th text unit.

[0294] Optionally, in the above Fig.25 Based on the corresponding embodiment, in another embodiment of the apparatus for processing speech recognition text provided by the embodiment of the present application,

[0295] The processing unit 202 is further configured to input the fusion vector of the i-th text unit into the text recognition network, and output the error probability value corresponding to the fusion vector of the i-th text unit through the text recognition network;

[0296] The determination unit 203 is further configured to determine the i-th erroneous text unit when the error probability value corresponding to the fusion vector of the i-th text unit is 0;

[0297] The determination unit 203 is further configured to determine the i-th correct text unit when the error probability value corresponding to the fusion vector of the i-th text unit is 1.

[0298] Optionally, in the above Fig.25 On the basis of the corresponding embodiment, in another embodiment of the apparatus for processing speech recognition text provided by the embodiment of the present application, the processing unit 202 can be specifically used for:

[0299] The text encoding vector corresponding to each text unit and the audio encoding vector are subjected to the attention mechanism to obtain the audio vector corresponding to each text encoding vector;

[0300] The audio vector corresponding to the i-th text encoding vector and the text encoding vector corresponding to the i-th text unit are summed to obtain a fusion vector of the i-th text unit.

[0301] Optionally, in the above Fig.25 On the basis of the corresponding embodiment, in another embodiment of the apparatus for processing speech recognition text provided by the embodiment of the present application, the processing unit 202 can be specifically used for:

[0302] Convert the original speech into a single-frame speech signal;

[0303] Each single-frame speech signal is vector-converted to obtain an audio coding vector corresponding to each single-frame speech signal.

[0304] Optionally, in the above Fig.25 On the basis of the corresponding embodiment, in another embodiment of the apparatus for processing speech recognition text provided by the embodiment of the present application, the processing unit 202 can be specifically used for:

[0305] Obtaining a sample speech recognition text corresponding to the sample speech and a text unit label corresponding to the sample speech recognition text;

[0306] Performing vector conversion on the sample speech to obtain a sample speech coding vector;

[0307] Performing vector conversion on each text unit in the sample speech recognition text to obtain a sample encoding vector corresponding to each text unit;

[0308] An attention mechanism is used for the sample speech coding vector and the i-th sample coding vector to obtain the sample fusion vector of the i-th text unit after interaction;

[0309] The sample fusion vector of the i-th text unit and the text vector corresponding to the i-1-th predicted text unit are input into the basic text error correction network, and the text probability distribution corresponding to the sample fusion vector of the i-th text unit is output through the basic text error correction network;

[0310] Calculate the error correction loss value based on the text unit label and the text probability distribution corresponding to the sample fusion vector of the i-th text unit;

[0311] The parameters of the basic text error correction network are adjusted based on the error correction loss value to obtain a text error correction network.

[0312] Optionally, in the above Fig.25 Based on the corresponding embodiment, in another embodiment of the apparatus for processing speech recognition text provided by the embodiment of the present application,

[0313] The acquisition unit 201 is further used to acquire an error detection label corresponding to the sample speech recognition text;

[0314] The processing unit 202 is further used to input the sample fusion vector of each text unit into the basic text recognition network, and output the error probability value corresponding to the sample fusion vector of each text unit through the basic text recognition network;

[0315] The processing unit 202 is further used to calculate the error detection loss value based on the error detection label and the error probability value corresponding to the sample fusion vector of each text unit;

[0316] The processing unit 202 is further used to adjust the parameters of the basic text error correction network based on the error correction loss value to obtain the text error correction network, including:

[0317] The processing unit 202 is further used to adjust parameters of the basic text recognition network and the basic text error correction network based on the error detection loss value and the error correction loss value to obtain the text error correction network and the text recognition network.

[0318] Optionally, in the above Fig.25 On the basis of the corresponding embodiment, in another embodiment of the apparatus for processing speech recognition text provided by the embodiment of the present application, the processing unit 202 can be specifically used for:

[0319] Based on the loss weight, the error detection loss value and the error correction loss value are weighted and summed to obtain the total loss value;

[0320] Based on the total loss value, the parameters of the basic text recognition network and the basic text error correction network are adjusted to obtain the text recognition network and the text error correction network.

[0321] Optionally, in the above Fig.25 On the basis of the corresponding embodiment, in another embodiment of the apparatus for processing speech recognition text provided by the embodiment of the present application, the processing unit 202 can be specifically used for:

[0322] The sample coding vector corresponding to each text unit and the sample speech coding vector are subjected to the attention mechanism to obtain the sample speech vector corresponding to each sample speech coding vector;

[0323] The sample speech vector corresponding to the i-th sample speech coding vector and the sample coding vector corresponding to the i-th text unit are summed to obtain a sample fusion vector of the i-th text unit.

[0324] Optionally, in the above Fig.25 On the basis of the corresponding embodiment, in another embodiment of the apparatus for processing speech recognition text provided by the embodiment of the present application, the processing unit 202 can be specifically used for:

[0325] Converting the sample speech into a single-frame sample speech signal;

[0326] Each single-frame sample speech signal is vector-converted to obtain a sample speech coding vector corresponding to each single-frame sample speech signal.

[0327] Optionally, in the above Fig.25 On the basis of the corresponding embodiment, in another embodiment of the apparatus for processing speech recognition text provided by the embodiment of the present application, the acquisition unit 201 can be specifically used for:

[0328] The original speech is input into the automatic speech recognition model, and the speech coding vector corresponding to the original speech is obtained through the pre-trained acoustic network in the automatic speech recognition model;

[0329] Pass the speech coding vector and the text vector corresponding to the i-1th recognized text unit to the text decoder in the automatic speech recognition model to obtain the recognized text probability distribution corresponding to the i-th text unit;

[0330] Based on the recognition text probability distribution corresponding to the ith text unit, the ith recognition text unit is determined to obtain the speech recognition text.

[0331] Optionally, in the above Fig.25 On the basis of the corresponding embodiment, in another embodiment of the apparatus for processing speech recognition text provided by the embodiment of the present application, the processing unit 202 can be specifically used for:

[0332] Convert the original speech into a single-frame speech signal;

[0333] Input each single-frame speech signal into an automatic speech recognition model, and obtain a single-frame speech vector corresponding to each single-frame speech signal through a pre-trained acoustic network in the automatic speech recognition model;

[0334] Based on the audio weight corresponding to each single-frame speech signal, the single-frame speech vectors corresponding to each single-frame speech signal are weighted summed to obtain a speech coding vector.

[0335] Optionally, in the above Fig.25 On the basis of the corresponding embodiment, in another embodiment of the apparatus for processing speech recognition text provided by the embodiment of the present application, the processing unit 202 can be specifically used for:

[0336] Input the sample speech into the basic automatic speech recognition model, and obtain the sample speech vector corresponding to the sample speech through the pre-trained acoustic network in the basic automatic speech recognition model;

[0337] Pass the sample speech vector and the text vector corresponding to the i-1th sample recognition text unit to the text decoder in the basic automatic speech recognition model to obtain the sample recognition text probability distribution corresponding to the i-th text unit;

[0338] Calculate the recognition loss value based on the sample recognition text probability distribution corresponding to the i-th text unit and the text unit label corresponding to the sample speech;

[0339] The parameters of the basic automatic speech recognition model are adjusted based on the recognition loss value to obtain an automatic speech recognition model.

[0340] Optionally, in the above Fig.25 On the basis of the corresponding embodiment, in another embodiment of the apparatus for processing speech recognition text provided by the embodiment of the present application, the processing unit 202 can be specifically used for:

[0341] Converting the sample speech into a single-frame sample speech signal;

[0342] Input each single-frame sample speech signal into a basic automatic speech recognition model, and obtain a single-frame sample speech vector corresponding to each single-frame sample speech signal through a pre-trained acoustic network in the basic automatic speech recognition model;

[0343] Based on the audio weight corresponding to each single-frame sample speech signal, the single-frame sample speech vectors corresponding to each single-frame sample speech signal are weighted summed to obtain a sample speech vector.

[0344] Optionally, in the above Fig.25 Based on the corresponding embodiment, in another embodiment of the apparatus for processing speech recognition text provided by the embodiment of the present application,

[0345] The acquisition unit 201 is further used to receive a speech recognition optimization instruction;

[0346] The acquisition unit 201 may be specifically configured to read the speech recognition text from the database according to the speech recognition optimization instruction.

[0347] On the other hand, the present application provides another schematic diagram of a computer device, such as Fig.26 As shown, Fig.26 3 is a schematic diagram of a computer device structure provided in an embodiment of the present application. The computer device 300 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 310 (for example, one or more processors) and a memory 320, and one or more storage media 330 (for example, one or more mass storage devices) storing application programs 331 or data 332. Among them, the memory 320 and the storage medium 330 may be temporary storage or permanent storage. The program stored in the storage medium 330 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the computer device 300. Furthermore, the central processing unit 310 may be configured to communicate with the storage medium 330 to execute a series of instruction operations in the storage medium 330 on the computer device 300.

[0348] The computer device 300 may also include one or more power supplies 340, one or more wired or wireless network interfaces 350, one or more input and output interfaces 360, and / or one or more operating systems 333, such as Windows Server 2000, Windows Server 2003, Windows Server 2003E, Windows Server 2003R ... TM , Mac OS X TM , Unix TM ,Linux TM , FreeBSD TM etc.

[0349] The computer device 300 is also used to perform the following steps: Figures 2 to 15 The steps in the corresponding embodiments.

[0350] Another aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the following Figures 2 to 15 The illustrated embodiments describe steps in a method.

[0351] Another aspect of the present application provides a computer program product comprising a computer program, which, when executed by a processor, implements the following Figures 2 to 15 The illustrated embodiments describe steps in a method.

[0352] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0353] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0354] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0355] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0356] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

Claims

1. A method for processing speech recognition text, characterized in that: include: Get the speech recognition text corresponding to the original speech; Performing vector conversion on the original speech to obtain an audio coding vector; Performing vector conversion on each text unit in the speech recognition text to obtain a text encoding vector corresponding to each text unit; An attention mechanism is applied to the audio encoding vector and the i-th text encoding vector to obtain a fusion vector of the i-th text unit after interaction, wherein i is an integer greater than or equal to 1; Inputting the fusion vector of the i-th text unit and the text vector corresponding to the i-1-th target text unit into a text error correction network, and outputting the text probability distribution corresponding to the fusion vector of the i-th text unit through the text error correction network; Based on the text probability distribution corresponding to the fusion vector of the i-th text unit, the i-th target text unit is determined.

2. The method according to claim 1, characterized in that After the attention mechanism is used for the audio encoding vector and the i-th text encoding vector respectively to obtain the fusion vector of the i-th text unit after interaction, the method further includes: Inputting the fusion vector of the i-th text unit into a text recognition network, and outputting the error probability value corresponding to the fusion vector of the i-th text unit through the text recognition network; When the error probability value corresponding to the fusion vector of the i-th text unit is 0, determining the i-th error text unit; When the error probability value corresponding to the fusion vector of the i-th text unit is 1, the i-th correct text unit is determined.

3. The method according to claim 1, characterized in that The step of using an attention mechanism on the audio encoding vector and the i-th text encoding vector to obtain a fusion vector of the i-th text unit after interaction includes: The text encoding vector corresponding to each text unit and the audio encoding vector are subjected to an attention mechanism to obtain an audio vector corresponding to each text encoding vector; The audio vector corresponding to the i-th text encoding vector and the text encoding vector corresponding to the i-th text unit are summed to obtain a fusion vector of the i-th text unit.

4. The method according to claim 1, characterized in that The step of performing vector conversion on the original speech to obtain an audio coding vector includes: Converting the original speech into a single-frame speech signal; Each of the single-frame speech signals is vector-converted to obtain the audio coding vector corresponding to each of the single-frame speech signals.

5. The method according to claim 1, characterized in that The training of the text error correction network includes the following steps: Acquire a sample speech recognition text corresponding to the sample speech, and a text unit label corresponding to the sample speech recognition text; Performing vector conversion on the sample speech to obtain a sample speech coding vector; Performing vector conversion on each text unit in the sample speech recognition text to obtain a sample encoding vector corresponding to each text unit; Using an attention mechanism on the sample speech coding vector and the i-th sample coding vector to obtain a sample fusion vector of the i-th text unit after interaction; Inputting the sample fusion vector of the i-th text unit and the text vector corresponding to the i-1-th predicted text unit into a basic text error correction network, and outputting the text probability distribution corresponding to the sample fusion vector of the i-th text unit through the basic text error correction network; Calculate the error correction loss value based on the text unit label and the text probability distribution corresponding to the sample fusion vector of the i-th text unit; The parameters of the basic text error correction network are adjusted based on the error correction loss value to obtain the text error correction network.

6. The method according to claim 5, characterized in that After the attention mechanism is used on the sample speech coding vector and the i-th sample coding vector to obtain the sample fusion vector of the i-th text unit after interaction, the method further includes: Obtaining an error detection label corresponding to the sample speech recognition text; Inputting the sample fusion vector of each text unit into a basic text recognition network, and outputting the error probability value corresponding to the sample fusion vector of each text unit through the basic text recognition network; Calculating an error detection loss value based on the error detection label and the error probability value corresponding to the sample fusion vector of each text unit; The step of adjusting parameters of the basic text error correction network based on the error correction loss value to obtain the text error correction network includes: Based on the error detection loss value and the error correction loss value, the parameters of the basic text recognition network and the basic text error correction network are adjusted to obtain the text error correction network and the text recognition network.

7. The method according to claim 6, characterized in that The step of adjusting parameters of the basic text recognition network and the basic text error correction network based on the error detection loss value and the error correction loss value to obtain the text recognition network and the text error correction network includes: Based on the loss weight, weighted summing the error detection loss value and the error correction loss value is performed to obtain a total loss value; Based on the total loss value, parameters of the basic text recognition network and the basic text error correction network are adjusted to obtain the text recognition network and the text error correction network.

8. The method according to claim 5, characterized in that The adopting the attention mechanism to the sample speech coding vector and the i-th sample coding vector to obtain the sample fusion vector of the i-th text unit after interaction includes: The sample coding vector corresponding to each text unit and the sample speech coding vector are subjected to an attention mechanism to obtain a sample speech vector corresponding to each sample speech coding vector; The sample speech vector corresponding to the i-th sample speech coding vector and the sample coding vector corresponding to the i-th text unit are summed to obtain a sample fusion vector of the i-th text unit.

9. The method according to claim 5, characterized in that The step of performing vector conversion on the sample speech to obtain a sample speech coding vector includes: Converting the sample speech into a single-frame sample speech signal; Each of the single-frame sample speech signals is vector-converted to obtain the sample speech coding vector corresponding to each of the single-frame sample speech signals.

10. The method according to claim 1, characterized in that The step of obtaining the speech recognition text corresponding to the original speech comprises: Inputting the original speech into an automatic speech recognition model, and obtaining a speech coding vector corresponding to the original speech through a pre-trained acoustic network in the automatic speech recognition model; Passing the speech coding vector and the text vector corresponding to the i-1th recognized text unit to a text decoder in the automatic speech recognition model to obtain a recognized text probability distribution corresponding to the i-th text unit; Based on the recognition text probability distribution corresponding to the ith text unit, the ith recognition text unit is determined to obtain the speech recognition text.

11. The method according to claim 10, characterized in that The step of inputting the original speech into an automatic speech recognition model and obtaining a speech coding vector corresponding to the original speech through a pre-trained acoustic network in the automatic speech recognition model includes: Converting the original speech into a single-frame speech signal; Inputting each of the single-frame speech signals into the automatic speech recognition model, and obtaining a single-frame speech vector corresponding to each of the single-frame speech signals through a pre-trained acoustic network in the automatic speech recognition model; Based on the audio weight corresponding to each of the single-frame speech signals, weighted summation is performed on the single-frame speech vectors corresponding to each of the single-frame speech signals to obtain the speech coding vector.

12. The method according to claim 10, characterized in that The training of the automatic speech recognition model includes the following steps: Inputting the sample speech into a basic automatic speech recognition model, and obtaining a sample speech vector corresponding to the sample speech through a pre-trained acoustic network in the basic automatic speech recognition model; Passing the sample speech vector and the text vector corresponding to the i-1th sample recognition text unit to the text decoder in the basic automatic speech recognition model to obtain the sample recognition text probability distribution corresponding to the i-th text unit; Calculating a recognition loss value based on a sample recognition text probability distribution corresponding to the i-th text unit and a text unit label corresponding to the sample speech; The parameters of the basic automatic speech recognition model are adjusted based on the recognition loss value to obtain the automatic speech recognition model.

13. The method according to claim 12, characterized in that The step of inputting the sample speech into a basic automatic speech recognition model and obtaining a sample speech vector corresponding to the sample speech through a pre-trained acoustic network in the basic automatic speech recognition model includes: Converting the sample speech into a single-frame sample speech signal; Inputting each of the single-frame sample speech signals into the basic automatic speech recognition model, and obtaining a single-frame sample speech vector corresponding to each of the single-frame sample speech signals through a pre-trained acoustic network in the basic automatic speech recognition model; Based on the audio weight corresponding to each of the single-frame sample speech signals, weighted summation is performed on the single-frame sample speech vectors corresponding to each of the single-frame sample speech signals to obtain the sample speech vector.

14. The method according to claim 1, characterized in that Before acquiring the speech recognition text corresponding to the original speech, the method further includes: Receive speech recognition optimization instructions; The step of obtaining the speech recognition text corresponding to the original speech comprises: According to the speech recognition optimization instruction, the speech recognition text is read from a database.

15. A speech recognition text processing device, characterized in that: include: An acquisition unit, used for acquiring a speech recognition text corresponding to the original speech; A processing unit, used for performing vector conversion on the original speech to obtain an audio coding vector; The processing unit is further used to perform vector conversion on each text unit in the speech recognition text to obtain a text encoding vector corresponding to each text unit; The processing unit is further used to use an attention mechanism on the audio encoding vector and the i-th text encoding vector to obtain a fusion vector of the i-th text unit after interaction, wherein i is an integer greater than or equal to 1; The processing unit is further used to input the fusion vector of the i-th text unit and the text vector corresponding to the i-1-th target text unit into the text error correction network, and output the text probability distribution corresponding to the fusion vector of the i-th text unit through the text error correction network; A determination unit is used to determine the i-th target text unit based on the text probability distribution corresponding to the fusion vector of the i-th text unit.

16. A computer device comprising a memory, a processor and a bus system, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 14 are implemented; The bus system is used to connect the memory and the processor so that the memory and the processor can communicate with each other.

17. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 14 are implemented.

18. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 14 are implemented.

Images