Speech recognition method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application provide a speech recognition method and device, electronic equipment and storage medium, at least applied to the field of artificial intelligence and speech recognition, wherein the method comprises: performing vector coding processing on the audio feature vector of the speech to be recognized to obtain an audio coding vector; performing classification processing on the audio coding vector to obtain a prediction probability distribution of each predicted character in a preset vocabulary corresponding to each speech frame in the speech to be recognized; performing pruning processing on the audio coding vector based on the prediction probability distribution to obtain a pruned audio coding vector; and performing speech recognition on the speech to be recognized based on the pruned audio coding vector to obtain a speech recognition result. Through the present application, the decoding calculation amount in the speech recognition process can be reduced, the decoding efficiency can be improved, and thus the speech recognition efficiency can be improved.

Description

Technical Field

[0001] This application relates to the Internet field, and includes, but is not limited to, a voice recognition method, device, electronic device, and storage medium. Background Technology

[0002] Speech recognition technology is a computer technology that identifies given audio as its corresponding text. Modern speech recognition technology extensively uses neural networks as its main structure. CTC (Concurrent Transformation) is a common training loss in neural network training. In speech recognition systems using CTC, an encoder-decoder structure is commonly used. That is, the encoder's output h, besides being used to predict the posterior probability y of CTC, often serves as both the training and decoding input for the decoder network.

[0003] In the inference process of speech recognition (i.e., the decoding process of the decoder), computational load or real-time efficiency is a crucial indicator of a speech recognition system. In related technologies, the encoder's output h is entirely input into the decoder for training and decoding. However, for many speech recognition systems, the computational load or real-time efficiency is highly correlated with the number of frames output by the encoder h. Therefore, speech recognition methods in related technologies suffer from high decoding computational load and low decoding efficiency, resulting in low speech recognition efficiency. Summary of the Invention

[0004] This application provides a speech recognition method, apparatus, electronic device, and storage medium, which can be applied to at least the fields of artificial intelligence and speech recognition. It can reduce the amount of decoding computation in the speech recognition process, improve decoding efficiency, and thus improve speech recognition efficiency.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] This application provides a speech recognition method, comprising: vectorizing the speech to be recognized to obtain an audio feature vector; performing vector encoding on the audio feature vector to obtain an audio encoding vector; classifying the audio encoding vector based on a preset vocabulary to obtain a prediction probability distribution of each predicted character in the preset vocabulary corresponding to each speech frame in the speech to be recognized; pruning the audio encoding vector based on the prediction probability distribution to obtain a pruned audio encoding vector; wherein, the pruning process refers to removing at least one audio encoding element corresponding to a speech frame in the audio encoding vector; and performing speech recognition on the speech to be recognized based on the pruned audio encoding vector to obtain a speech recognition result.

[0007] This application provides a speech recognition device, comprising: a vectorization processing module for vectorizing the speech to be recognized to obtain an audio feature vector; a vector encoding module for vector encoding the audio feature vector to obtain an audio encoding vector; a classification processing module for classifying the audio encoding vector based on a preset vocabulary to obtain a prediction probability distribution of each predicted character in the preset vocabulary corresponding to each speech frame in the speech to be recognized; a pruning processing module for pruning the audio encoding vector based on the prediction probability distribution to obtain a pruned audio encoding vector; wherein the pruning process refers to removing at least one audio encoding element corresponding to a speech frame in the audio encoding vector; and a speech recognition module for performing speech recognition on the speech to be recognized based on the pruned audio encoding vector to obtain a speech recognition result.

[0008] In some embodiments, the preset vocabulary includes multiple non-empty characters and one empty character; the classification processing module is further configured to: determine the predicted probability of each speech frame corresponding to each non-empty character and the empty character based on the non-empty characters and the empty character in the preset vocabulary, thereby forming the predicted probability distribution.

[0009] In some embodiments, the pruning processing module is further configured to: determine the predicted character with the highest prediction probability for each speech frame based on the prediction probability distribution; the predicted character includes multiple non-empty characters and an empty character; determine the speech frame with the predicted character being an empty character as the target speech frame; and perform a removal process on the audio coding element corresponding to the target speech frame in the audio coding vector to obtain the pruned audio coding vector.

[0010] In some embodiments, the pruning module is further configured to: sequentially remove the audio coding elements corresponding to the target speech frame in the audio coding vector according to the reverse order of the audio coding elements in the audio coding vector, to obtain the pruned audio coding vector.

[0011] In some embodiments, the apparatus further includes: a first determining module, configured to determine the audio coding element corresponding to the target speech frame in the audio coding vector as the target audio coding element when the pruning process is stopped; a second determining module, configured to determine a preset number of audio coding elements as boundary audio coding elements, starting from the target audio coding element, according to the ascending order of the audio coding elements in the audio coding vector; and a control module, configured to prohibit the removal of the boundary audio coding elements.

[0012] In some embodiments, the predicted character includes multiple non-empty characters and one empty character; the pruning processing module is further configured to: obtain the empty character prediction probability for each speech frame corresponding to the empty character based on the prediction probability distribution; determine the audio coding elements corresponding to the speech frames whose empty character prediction probability is greater than the probability threshold in the audio coding vector as audio coding elements to be removed; and remove the audio coding elements to be removed sequentially according to the reverse order of the audio coding elements in the audio coding vector to obtain the pruned audio coding vector.

[0013] In some embodiments, the predicted character includes multiple non-empty characters and one empty character; the pruning processing module is further configured to: determine, based on the prediction probability distribution, the predicted character with the highest prediction probability for each speech frame and the empty character prediction probability for each speech frame corresponding to the empty character; determine speech frames whose predicted character is an empty character and whose empty character prediction probability is greater than a probability threshold as speech frames to be pruned; and sequentially remove the corresponding audio coding elements of the speech frames to be pruned in the audio coding vector according to the reverse order of the audio coding elements in the audio coding vector to obtain the pruned audio coding vector.

[0014] In some embodiments, the speech to be identified is single-channel audio, which is represented by a one-dimensional waveform; the vectorization processing module is further configured to: extract audio features from the sub-waveform corresponding to each speech frame in the one-dimensional waveform to obtain a fixed-length vector corresponding to each speech frame; wherein, the fixed-length vectors corresponding to all speech frames constitute the audio feature vector, which is a two-dimensional matrix.

[0015] In some embodiments, the speech recognition method is implemented through a speech recognition model; the apparatus further includes a model training device, which is used to: input sample data into the speech recognition model; the sample data includes sample audio feature vectors and label sequences of sample speech; perform vector encoding processing on the sample audio feature vectors through the encoder of the speech recognition model to obtain sample audio encoding vectors; perform classification processing on the sample audio encoding vectors through the classifier of the speech recognition model to obtain the sample prediction probability distribution of each predicted character in the preset vocabulary corresponding to each sample speech frame in the sample speech; input the sample prediction probability distribution into the CTC loss model to obtain the CTC loss result; input the sample audio encoding vectors and the label sequences into the decoder to obtain the sample prediction sequence; determine the cross-entropy loss result based on the sample prediction sequence and the label sequence; and correct the parameters in the speech recognition model based on the CTC loss result and the cross-entropy loss result to obtain the trained speech recognition model.

[0016] In some embodiments, the model training device is further configured to: obtain the CTC loss weights corresponding to the CTC loss result and the cross-entropy loss weights corresponding to the cross-entropy loss result; perform weighted calculation on the CTC loss result and the cross-entropy loss result based on the CTC loss weights and the cross-entropy loss weights to obtain a weighted loss result; and correct the parameters in the speech recognition model based on the weighted loss result to obtain the trained speech recognition model.

[0017] In some embodiments, the model training device is further configured to: perform model fine-tuning on the trained speech recognition model; wherein, the model training device is specifically configured to: input fine-tuning sample data into the trained speech recognition model; the fine-tuning sample data includes fine-tuning sample audio feature vectors and fine-tuning label sequences of the fine-tuning sample speech; perform vector encoding processing on the fine-tuning sample audio feature vectors through the encoder of the trained speech recognition model to obtain fine-tuning sample audio encoding vectors; and perform classification processing on the fine-tuning sample audio encoding vectors through the classifier of the trained speech recognition model to obtain each predicted character in the preset vocabulary corresponding to each fine-tuning sample speech frame in the fine-tuning sample speech. Fine-tune the sample prediction probability distribution; input the fine-tuned sample prediction probability distribution into the CTC loss model to obtain the fine-tuned CTC loss result; based on the fine-tuned sample prediction probability distribution, prune the fine-tuned sample audio encoding vector to obtain the pruned fine-tuned sample audio encoding vector; input the pruned fine-tuned sample audio encoding vector and the fine-tuned label sequence into the decoder to obtain the fine-tuned sample prediction sequence; based on the sample prediction sequence and the fine-tuned label sequence, determine the fine-tuned cross-entropy loss result; based on the fine-tuned CTC loss result and the fine-tuned cross-entropy loss result, correct the parameters in the trained speech recognition model to obtain the fine-tuned speech recognition model.

[0018] In some embodiments, the encoder is a hierarchical encoder, which is composed of multiple neural networks with the same structure; the model training device is further configured to: perform vector encoding processing on the sample audio feature vectors through the neural networks of different levels of the hierarchical encoder, corresponding to multiple different sample audio encoding vectors; perform classification processing on the different sample audio encoding vectors respectively through the classifier, corresponding to multiple different sample prediction probability distributions; input the multiple different sample prediction probability distributions into the CTC loss model respectively, corresponding to multiple CTC loss results; perform loss fusion processing on the multiple CTC loss results to obtain a fused loss result; and based on the fused loss result, correct the parameters in the speech recognition model to obtain the trained speech recognition model.

[0019] This application provides an electronic device, including: a memory for storing executable instructions; and a processor for executing the executable instructions stored in the memory to implement the above-described speech recognition method.

[0020] This application provides a computer program product including executable instructions stored in a computer-readable storage medium; wherein, when the processor of an electronic device reads the executable instructions from the computer-readable storage medium and executes the executable instructions, the above-mentioned speech recognition method is implemented.

[0021] This application provides a computer-readable storage medium storing executable instructions, which, when executed by a processor, implement the above-described speech recognition method.

[0022] The embodiments of this application have the following beneficial effects: Based on a preset vocabulary, the audio encoding vector of the speech to be recognized is classified to obtain the prediction probability distribution of each predicted character in the preset vocabulary corresponding to each speech frame in the speech to be recognized; and based on the prediction probability distribution, the audio encoding vector is pruned to obtain a pruned audio encoding vector; wherein, the pruning process refers to removing the audio encoding elements corresponding to at least one speech frame in the audio encoding vector; and then, based on the pruned audio encoding vector, speech recognition is performed on the speech to be recognized to obtain the speech recognition result. Thus, by pruning the audio encoding vector, the number of elements in the audio encoding vector during the speech recognition inference process is greatly reduced, the decoding computation load during the speech recognition process is reduced, the decoding efficiency is improved, and consequently, the speech recognition efficiency is greatly improved. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of an optional architecture of the speech recognition system provided in the embodiments of this application;

[0024] Figure 2 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application;

[0025] Figure 3 This is an optional flowchart illustrating the speech recognition method provided in an embodiment of this application;

[0026] Figure 4 This is another optional flowchart illustrating the speech recognition method provided in the embodiments of this application;

[0027] Figure 5 This is a schematic diagram illustrating the implementation process of the pruning process provided in the embodiments of this application;

[0028] Figure 6This is a flowchart illustrating the pre-training stage of the speech recognition model training method provided in this application embodiment;

[0029] Figure 7 This is a schematic diagram illustrating the implementation process of correcting parameters in a speech recognition model according to an embodiment of this application;

[0030] Figure 8 This is a flowchart illustrating the fine-tuning stage of the training method for the speech recognition model provided in this application embodiment;

[0031] Figure 9 This is a schematic diagram of the predicted distribution of the standard CTC provided in the embodiments of this application;

[0032] Figure 10 This is a schematic diagram of the predicted distribution of BCTC provided in an embodiment of this application;

[0033] Figure 11 This is a schematic diagram of the model during the pre-training stage provided in an embodiment of this application;

[0034] Figure 12 This is a schematic diagram of the AED framework provided in the embodiments of this application;

[0035] Figure 13 This is a schematic diagram of the NT system provided in the embodiments of this application. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0037] In the following description, references to "some embodiments" refer to a subset of all possible embodiments. However, it is understood that "some embodiments" may be the same or different subsets of all possible embodiments and may be combined with each other without conflict. Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art to which the embodiments of this application pertain. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit the application.

[0038] This application provides a speech recognition method that, by adjusting the loss function of connectionist temporal classification (CTC), allows the speech recognition model to achieve effective downsampling at the feature level during encoder inference, thereby reducing the model inference cost and maintaining almost no change in system performance.

[0039] The speech recognition method provided in this application firstly vectorizes the speech to be recognized to obtain an audio feature vector; then, it performs vector encoding on the audio feature vector to obtain an audio encoded vector; next, based on a preset vocabulary, it classifies the audio encoded vector to obtain the prediction probability distribution of each predicted character in the preset vocabulary corresponding to each speech frame in the speech to be recognized; then, based on the prediction probability distribution, it prunes the audio encoded vector to obtain a pruned audio encoded vector; wherein, pruning refers to removing at least one audio encoded element corresponding to a speech frame in the audio encoded vector; finally, it performs speech recognition on the speech to be recognized based on the pruned audio encoded vector to obtain the speech recognition result. Thus, by pruning the audio encoded vector, the number of elements in the audio encoded vector during the speech recognition inference process is greatly reduced, the decoding computation load during the speech recognition process is reduced, and the decoding efficiency is improved, thereby greatly improving the speech recognition efficiency.

[0040] Here, we first describe an exemplary application of the speech recognition device in this application embodiment. This speech recognition device is an electronic device used to implement a speech recognition method. In one implementation, the speech recognition device (i.e., electronic device) provided in this application embodiment can be implemented as a terminal or as a server. In one implementation, the speech recognition device provided in this application embodiment can be implemented as any terminal with voice data processing capabilities, such as a laptop, tablet, desktop computer, mobile phone, portable music player, personal digital assistant, dedicated messaging device, portable gaming device, intelligent robot, smart home appliance, and smart vehicle device. In another implementation, the speech recognition device provided in this application embodiment can also be implemented as a server. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The terminal and the server can be directly or indirectly connected through wired or wireless communication methods, which is not limited in this application embodiment. The following will describe an exemplary application of a speech recognition device implemented as a server.

[0041] See Figure 1 , Figure 1 This is a schematic diagram of an optional architecture of the speech recognition system provided in this application embodiment. This application embodiment uses the application of the speech recognition method to any instant messaging application as an example for illustration. In an instant messaging application, a speech recognition function is provided to perform speech recognition on the user-input speech to be recognized, obtaining a speech recognition result. This speech recognition result is used to realize instant communication between the user and friends. In this application embodiment, the speech recognition system includes at least a terminal 100, a network 200, and a server 300. The server 300 can be the server of the instant messaging application. The server 300 can constitute the speech recognition device of this application embodiment. The terminal 100 is connected to the server 300 through the network 200, which can be a wide area network (WAN), a local area network (LAN), or a combination of both.

[0042] In this embodiment, during speech recognition, terminal 100 receives the user's speech to be recognized through an instant messaging application client; encapsulates the speech to be recognized into a speech recognition request, and sends the speech recognition request to server 300 through network 200. Upon receiving the speech recognition request, server 300, in response, performs vectorization processing on the speech to be recognized to obtain an audio feature vector; then, it performs vector encoding processing on the audio feature vector to obtain an audio encoding vector; next, based on a preset vocabulary, it performs classification processing on the audio encoding vector to obtain the prediction probability distribution of each predicted character in the preset vocabulary corresponding to each speech frame in the speech to be recognized; then, based on the prediction probability distribution, it performs pruning processing on the audio encoding vector to obtain a pruned audio encoding vector; wherein, pruning processing refers to removing at least one audio encoding element corresponding to a speech frame in the audio encoding vector; finally, it performs speech recognition on the speech to be recognized based on the pruned audio encoding vector to obtain the speech recognition result. After obtaining the speech recognition result, server 300 sends the speech recognition result to terminal 100 so that terminal 100 displays the speech recognition result on the current interface.

[0043] In other embodiments, the speech recognition device can also be implemented as a terminal, that is, the speech recognition method of the embodiments of this application is implemented by the terminal as the execution subject. In the implementation process, the terminal collects the speech to be recognized through the client of an instant messaging application. Furthermore, the terminal performs vectorization processing on the speech to be recognized to obtain an audio feature vector; the terminal performs vector encoding processing on the audio feature vector to obtain an audio encoded vector; the terminal performs classification processing on the audio encoded vector based on a preset vocabulary to obtain the prediction probability distribution of each predicted character in the preset vocabulary corresponding to each speech frame in the speech to be recognized; the terminal performs pruning processing on the audio encoded vector based on the prediction probability distribution to obtain a pruned audio encoded vector; the terminal performs speech recognition on the speech to be recognized based on the pruned audio encoded vector to obtain the speech recognition result.

[0044] The speech recognition method provided in this application embodiment can also be implemented based on a cloud platform and through cloud technology. For example, the server 300 mentioned above can be a cloud server. The cloud server performs vectorization processing on the speech to be recognized, or performs vector encoding processing on the audio feature vector, or performs classification processing on the audio encoded vector, or performs pruning processing on the audio encoded vector, and performs speech recognition on the speech to be recognized based on the pruned audio encoded vector.

[0045] In some embodiments, a cloud storage system may also be included, where information such as a preset vocabulary and the predicted probability distribution of the speech to be recognized can be stored. Alternatively, the speech recognition results can also be stored in the cloud storage system. This way, when a speech recognition request is received, the preset vocabulary can be retrieved from the cloud storage system for speech recognition, thereby improving speech recognition efficiency.

[0046] It's important to clarify that cloud technology refers to a hosting technology that unifies hardware, software, and network resources within a wide area network (WAN) or local area network (LAN) to achieve data computation, storage, processing, and sharing. Cloud technology is a collective term for network technologies, information technologies, integration technologies, management platform technologies, and application technologies applied in the cloud computing business model. It can form resource pools, providing flexible and convenient on-demand access. Cloud computing technology will become a crucial support. Backend services of technical network systems require substantial computing and storage resources, such as video websites, image websites, and many portal websites. With the rapid development and application of the internet industry, every item may have its own identification mark in the future, requiring transmission to backend systems for logical processing. Data at different levels will be processed separately, and various industry data will require robust system support, which can be achieved through cloud computing.

[0047] Figure 2 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Figure 2 The illustrated electronic device may be a voice recognition device, which includes at least one processor 310, a memory 350, at least one network interface 320, and a user interface 330. The various components of the voice recognition device are coupled together via a bus system 340. It is understood that the bus system 340 is used to implement communication between these components. In addition to a data bus, the bus system 340 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2 The general labeled all buses as Bus System 340.

[0048] The processor 310 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0049] User interface 330 includes one or more output devices 331 that enable the presentation of media content, and one or more input devices 332.

[0050] Memory 350 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. Memory 350 may optionally include one or more storage devices physically located remote from processor 310. Memory 350 may include volatile memory or non-volatile memory, or both. Non-volatile memory may be read-only memory (ROM), and volatile memory may be random access memory (RAM). The memory 350 described in this application embodiment is intended to include any suitable type of memory. In some embodiments, memory 350 is capable of storing data to support various operations, examples of which include programs, modules, and data structures, or subsets or supersets thereof, as illustrated below.

[0051] Operating system 351 includes system programs for handling various basic system services and performing hardware-related tasks, such as framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks; network communication module 352 is used to reach other computing devices via one or more (wired or wireless) network interfaces 320, exemplary network interfaces 320 include: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.; input processing module 353 is used to detect and translate one or more user inputs or interactions from one or more input devices 332.

[0052] In some embodiments, the apparatus provided in this application may be implemented in software. Figure 2 A speech recognition device 354 stored in memory 350 is shown. This speech recognition device 354 can be a speech recognition device in an electronic device, and can be software in the form of programs and plug-ins, including the following software modules: a vectorization processing module 3541, a vector encoding module 3542, a classification processing module 3543, a pruning processing module 3544, and a speech recognition module 3545. These modules are logically connected and can therefore be arbitrarily combined or further divided according to the functions they implement. The functions of each module will be described below.

[0053] In other embodiments, the apparatus provided in this application can be implemented in hardware. As an example, the apparatus provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the speech recognition method provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0054] The speech recognition methods provided in the embodiments of this application can be executed by an electronic device, which can be a server or a terminal. That is, the speech recognition methods in the embodiments of this application can be executed by a server, by a terminal, or through interaction between a server and a terminal.

[0055] Figure 3 This is an optional flowchart illustrating the speech recognition method provided in this application embodiment. The following will be combined with... Figure 3 The steps shown will be explained. It should be noted that... Figure 3 The speech recognition method in this example is illustrated using a server as the execution entity. Figure 3 As shown, the method includes the following steps S101 to S105:

[0056] Step S101: The speech to be recognized is vectorized to obtain audio feature vectors.

[0057] Here, the speech to be recognized can be speech acquired using any speech acquisition method. For example, the speech to be recognized can be real-time speech input by the user captured by the microphone on the terminal, or speech downloaded from the network, or speech extracted from a video, or speech synthesized using speech synthesis technology.

[0058] In this embodiment, the speech to be recognized can be speech that requires text conversion processing. For example, the speech to be recognized can be recognized to obtain a speech recognition result, and the speech recognition result can be output in the form of text. Alternatively, the speech to be recognized can also be speech that requires information retrieval. For example, the speech to be recognized can be recognized to obtain a speech recognition result, and the recognized content in the speech recognition result can be retrieved for information retrieval. Alternatively, the speech to be recognized can also be speech that requires information editing. For example, the speech to be recognized can be the subtitle speech for making a video. The speech to be recognized can be recognized to obtain a speech recognition result, and the recognized content in the speech recognition result can be converted and edited into subtitles.

[0059] In this embodiment, the speech to be recognized can be single-channel audio or multi-channel audio. When the speech to be recognized is multi-channel audio, it can be converted into single-channel audio first, and then the speech recognition method of this embodiment can be used for speech recognition. In the implementation process, single-channel audio can be represented by a one-dimensional waveform.

[0060] It's important to clarify that single-channel audio refers to audio with only one sound channel, meaning it's recorded using a single microphone and played back by a single speaker; multi-channel audio, on the other hand, refers to audio with multiple sound channels. Single-channel audio mixes audio signals from different directions, records them uniformly, and then plays them back through a single speaker. Multi-channel audio, however, utilizes the principle of stereo sound, placing multiple speakers at specific angles in space, with each speaker receiving its own signal from a single channel.

[0061] The speech to be recognized consists of multiple speech frames. Vectorization processing refers to extracting audio features from the sub-waveform corresponding to each speech frame in a one-dimensional waveform, thereby obtaining a fixed-length vector corresponding to each speech frame. The fixed-length vectors corresponding to all speech frames constitute the audio feature vector, which is a two-dimensional matrix. The number of speech frames in the speech to be recognized is the same as the number of fixed-length vectors in the audio feature vector, and the dimensions of the fixed-length vectors of all speech frames are the same.

[0062] In this embodiment of the application, when extracting audio features from sub-waveforms, an audio conversion processing method can be used, that is, the audio can be converted into a two-dimensional matrix. Here, the audio conversion processing method can be implemented, for example, through convolutional neural networks, Mel Frequency Cepstrum Coefficients (MFCC), FBAN K features, etc.

[0063] Step S102: Perform vector encoding processing on the audio feature vector to obtain the audio encoded vector.

[0064] Here, vector encoding refers to using a neural network (i.e., an encoder) with a certain structure to encode audio feature vectors. This application does not limit the structure and network parameters of the encoder. The encoder's network parameters can be trained during the model training phase, and during speech recognition, the trained encoder can be directly used to perform vector encoding on the audio feature vectors.

[0065] Step S103: Based on the preset vocabulary, classify the audio encoding vector to obtain the prediction probability distribution of each predicted character in the preset vocabulary for each speech frame in the speech to be recognized.

[0066] Here, the preset vocabulary includes multiple non-empty characters and one empty character. Each non-empty character corresponds to a non-empty character, and the non-empty character can be determined based on all characters in the vocabulary. For example, a non-empty character can be any character from A to Z, and the preset vocabulary can include each character corresponding to A to Z; or, a non-empty character can be any Chinese character, and the non-empty character corresponding to each Chinese character can be preset to form the preset vocabulary. An empty character refers to the character corresponding to blank content, that is, a character that does not contain any character information.

[0067] In this embodiment, a classifier can be used to predict the distribution of each speech frame under a preset vocabulary based on the audio coding vector, thus obtaining a predicted probability distribution. Here, the predicted probability distribution includes the predicted probability of each speech frame under each predicted character in the preset vocabulary. In one implementation, the predicted probability distribution can be represented by an N×M table. In the table, the horizontal list represents each speech frame, totaling M speech frames; the vertical list represents each predicted character in the preset vocabulary, totaling N predicted characters, where M and N are both integers greater than 1. Each table cell can be filled with the predicted probability of the corresponding speech frame under the corresponding predicted character. Based on this N×M table, the predicted probability distribution of each speech frame in the speech to be recognized corresponding to each predicted character in the preset vocabulary can be obtained intuitively.

[0068] Step S104: Based on the predicted probability distribution, the audio coding vector is pruned to obtain the pruned audio coding vector; wherein, the pruning process refers to removing the audio coding element corresponding to at least one speech frame in the audio coding vector.

[0069] In this embodiment, pruning can reduce the number of audio coding elements in the audio coding vector. This reduces the amount of data required for decoding calculations during subsequent speech recognition, thereby improving the computational efficiency of decoding and reducing computational overhead.

[0070] In this embodiment, since the speech frame corresponding to the empty character is a speech frame in which no valid information can be recognized during speech recognition, the audio coding elements of the speech frame corresponding to the empty character can be removed during pruning. Typically, there are many speech frames corresponding to empty characters in a speech segment; therefore, removing the audio coding elements of these speech frames can significantly reduce the amount of data required for encoding calculations.

[0071] It should be noted that the pruning process in this embodiment refers to deleting the audio coding element corresponding to at least one speech frame from the audio coding vector. In other words, it involves deleting the audio coding element of the speech frame corresponding to the empty character from the audio coding vector. Thus, the number of audio coding elements in the pruned audio coding vector is less than the number of audio coding elements in the original audio coding vector of the speech to be recognized.

[0072] In some embodiments, the audio coding vector can be pruned based on the predicted probability distribution to obtain a pruned audio coding vector. During implementation, the speech frame corresponding to the empty character can be determined by analyzing an N×M table of the predicted probability distribution. For example, for any speech frame K, the predicted probabilities of different predicted characters for speech frame K are analyzed. If the predicted character with the highest predicted probability is an empty character, then speech frame K can be determined to be an empty character that does not contain any character information; or, if the predicted probability corresponding to the empty character is greater than a probability threshold, then speech frame K can be determined to be an empty character that does not contain any character information.

[0073] Step S105: Perform speech recognition on the speech to be recognized based on the pruned audio coding vector to obtain the speech recognition result.

[0074] In this embodiment of the application, after pruning the audio coding vector to obtain the pruned audio coding vector, the number of audio coding elements in the pruned audio coding vector is significantly reduced compared to the number of audio coding elements in the original audio coding vector. That is, the number of audio coding elements in the pruned audio coding vector is much smaller than the number of audio coding elements in the original audio coding vector.

[0075] In speech recognition, the pruned audio encoding vector can be decoded to obtain the speech recognition result. In this embodiment, the decoding process can be implemented using a decoder, and the structure and network parameters of the decoder are not limited. The network parameters of the decoder can be trained during the model training phase, and during speech recognition, the trained decoder can be directly used to decode the pruned audio encoding vector.

[0076] The speech recognition method provided in this application, based on a preset vocabulary, classifies the audio encoding vector of the speech to be recognized to obtain the prediction probability distribution of each predicted character in the preset vocabulary corresponding to each speech frame in the speech to be recognized; and based on the prediction probability distribution, prunes the audio encoding vector to obtain a pruned audio encoding vector; wherein, pruning refers to removing at least one audio encoding element corresponding to a speech frame in the audio encoding vector; and then, based on the pruned audio encoding vector, speech recognition is performed on the speech to be recognized to obtain the speech recognition result. Thus, by pruning the audio encoding vector, the number of elements in the audio encoding vector during the speech recognition inference process is greatly reduced, the decoding computation load during the speech recognition process is reduced, the decoding efficiency is improved, and consequently, the speech recognition efficiency is greatly improved.

[0077] In some embodiments, the speech recognition system includes at least a terminal and a server. The terminal may have any application with speech recognition functionality installed. For example, an instant messaging application may be installed on the terminal. The terminal collects the speech to be recognized through the client of the instant messaging application and performs speech recognition using the speech recognition method provided in this application embodiment to obtain a speech recognition result. This speech recognition result is used for real-time communication within the instant messaging application. As another example, a search application may be installed on the terminal. The terminal collects the speech to be recognized through the client of the search application and performs speech recognition using the speech recognition method provided in this application embodiment to obtain a speech recognition result. This speech recognition result is used for real-time searching within the search application. The speech recognition method of this application embodiment will now be described using an instant messaging application installed on the terminal as an example.

[0078] Figure 4 This is another optional flowchart illustrating the speech recognition method provided in the embodiments of this application, such as... Figure 4 As shown, the method includes the following steps S201 to S213:

[0079] In step S201, the terminal collects the user's voice input through the client of the instant messaging application, and the user's voice input constitutes the voice to be recognized.

[0080] In step S202, the terminal encapsulates the speech to be recognized into a speech recognition request.

[0081] Here, a speech recognition request is used to request the speech to be recognized to obtain the corresponding text, which is then displayed in the instant messaging application as text.

[0082] In step S203, the terminal sends a voice recognition request to the server.

[0083] In step S204, the server responds to the speech recognition request by performing vectorization processing on the speech to be recognized to obtain audio feature vectors.

[0084] In this embodiment, the speech to be identified is single-channel audio, which is represented by a one-dimensional waveform. During vectorization, audio features can be extracted from the sub-waveform corresponding to each speech frame in the one-dimensional waveform to obtain a fixed-length vector corresponding to each speech frame; wherein, the fixed-length vectors corresponding to all speech frames constitute the audio feature vector, which is a two-dimensional matrix.

[0085] In step S205, the server performs vector encoding on the audio feature vector to obtain the audio encoded vector.

[0086] Here, vector encoding processing refers to using an encoder to perform vector encoding on audio feature vectors to obtain audio encoded vectors. In this embodiment, the network parameters of the encoder can be trained during the model training phase, and during speech recognition, the trained encoder can be directly used to perform vector encoding on the audio feature vectors.

[0087] In step S206, the server performs classification processing on the audio encoding vector based on a preset vocabulary to obtain the prediction probability distribution of each predicted character in the preset vocabulary for each speech frame in the speech to be recognized.

[0088] In some embodiments, the preset vocabulary includes multiple non-empty characters and one empty character. Step S206 can be implemented by determining the prediction probability of each speech frame corresponding to each non-empty character and the empty character based on the non-empty characters and the empty character in the preset vocabulary, thereby forming a prediction probability distribution.

[0089] Here, a classifier can be used to classify each audio coding element in the audio coding vector, determining the predicted probability of each audio coding element corresponding to each non-empty character and each empty character, that is, determining the predicted probability of each speech frame corresponding to each non-empty character and each empty character. The predicted probabilities of all speech frames constitute the predicted probability distribution.

[0090] Step S207: The server performs pruning on the audio coding vector based on the predicted probability distribution to obtain the pruned audio coding vector; wherein, pruning refers to removing the audio coding element corresponding to at least one speech frame in the audio coding vector.

[0091] In one implementation, see Figure 5 , Figure 5 Step S207 can be achieved through the following steps S2071 to S2073:

[0092] Step S2071: Based on the prediction probability distribution, determine the prediction character with the highest prediction probability for each speech frame; the prediction character includes multiple non-empty characters and one empty character.

[0093] Here, each speech frame has a prediction probability for each predicted character in the preset vocabulary. That is, each speech frame has multiple prediction probabilities. The predicted character corresponding to the maximum prediction probability in each speech frame can be obtained from the prediction probability distribution. This predicted character is the predicted character corresponding to the corresponding speech frame.

[0094] Step S2072: Determine the speech frame whose predicted character is an empty character as the target speech frame.

[0095] Step S2073: Remove the audio coding elements corresponding to the target speech frame in the audio coding vector to obtain the pruned audio coding vector.

[0096] Here, the audio coding elements corresponding to the target speech frame in the audio coding vector can be removed sequentially in reverse order to obtain the pruned audio coding vector.

[0097] An audio encoding vector contains multiple audio encoding elements, which are arranged sequentially according to the order of the corresponding speech frames in the speech to be recognized. Here, the order of multiple audio encoding elements in the audio encoding vector from left to right can be defined as ascending order, and the order of multiple audio encoding elements in the audio encoding vector from right to left can be defined as descending order.

[0098] In this embodiment, the audio coding elements of speech frames where the predicted character is an empty character are removed, resulting in a pruned audio coding vector. This method of removing audio coding elements from speech frames where the predicted character is an empty character can eliminate speech frames corresponding to empty characters with a very high probability.

[0099] For another implementation, please refer to [link / reference]. Figure 5 , Figure 5 Step S207 can also be achieved through the following steps S2074 to S2076:

[0100] Step S2074: Based on the predicted probability distribution, obtain the predicted probability of the empty character corresponding to the empty character in each speech frame.

[0101] In this embodiment of the application, the prediction probability of the row where the predicted character is an empty character can be obtained from the N-M table corresponding to the prediction probability distribution, thereby obtaining the empty character prediction probability of each speech frame corresponding to the empty character.

[0102] Step S2075: The audio coding elements corresponding to speech frames with a predicted probability of empty characters greater than the probability threshold in the audio coding vector are identified as audio coding elements to be removed.

[0103] Here, a speech frame with a predicted probability of an empty character greater than the probability threshold means that the speech frame has a relatively high probability of containing an empty character. These speech frames can be identified as speech frames to be removed. Correspondingly, the audio coding elements corresponding to these speech frames in the audio coding vector are identified as audio coding elements to be removed.

[0104] Step S2076: Following the reverse order of the audio coding elements in the audio coding vector, remove the audio coding elements to be removed one by one to obtain the pruned audio coding vector.

[0105] In this embodiment of the application, the audio coding elements corresponding to speech frames with a high probability of predicting the null character (greater than the probability threshold) are removed, which can remove speech frames corresponding to null characters with a very high probability.

[0106] In another implementation, please refer to [link / reference]. Figure 5 , Figure 5 Step S207 can also be achieved through the following steps S2077 to S2079:

[0107] Step S2077: Based on the prediction probability distribution, determine the prediction character with the highest prediction probability in each speech frame and the prediction probability of the empty character corresponding to the empty character in each speech frame.

[0108] Step S2078: Speech frames whose predicted character is an empty character and whose predicted probability of the empty character is greater than the probability threshold are identified as speech frames to be pruned.

[0109] Step S2079: Following the reverse order of the audio coding elements in the audio coding vector, the audio coding elements corresponding to the speech frames to be pruned are removed sequentially to obtain the pruned audio coding vector.

[0110] In this embodiment of the application, the audio encoding elements of the speech frames whose predicted character is an empty character, and the audio encoding elements corresponding to the speech frames with a high probability of predicting an empty character (greater than the probability threshold) are removed, which can remove the speech frames corresponding to empty characters with a very high probability.

[0111] In step S208, when the server stops the pruning process, the audio coding element corresponding to the target speech frame in the audio coding vector is determined as the target audio coding element.

[0112] In step S209, the server determines a preset number of audio encoding elements as boundary audio encoding elements, starting from the target audio encoding element, according to the ascending order of the audio encoding elements in the audio encoding vector.

[0113] Step S210: The server prohibits the removal of boundary audio encoding elements.

[0114] In this embodiment, the audio coding elements of the first few audio frames when the pruning process is triggered are not removed, thereby preventing information loss at the boundaries caused by excessive pruning.

[0115] In step S211, the server performs speech recognition on the speech to be recognized based on the pruned audio coding vector, and obtains the speech recognition result.

[0116] In step S212, the server sends the speech recognition result to the terminal.

[0117] Step S213: The terminal displays the speech recognition result on the current interface.

[0118] The speech recognition method provided in this application, based on a preset vocabulary, classifies the audio encoding vector of the speech to be recognized to obtain the prediction probability distribution of each predicted character in the preset vocabulary for each speech frame in the speech to be recognized. Based on the prediction probability distribution, audio encoding elements of speech frames that satisfy at least one of the following conditions are removed: audio encoding elements corresponding to speech frames where the predicted character is an empty character, and audio encoding elements corresponding to speech frames where the predicted probability of an empty character is greater than a probability threshold. Thus, speech recognition is performed on the speech to be recognized based on the pruned audio encoding vector to obtain the speech recognition result. In this way, by pruning the audio encoding vector, the number of elements in the audio encoding vector during the speech recognition inference process is greatly reduced, the decoding computation load during speech recognition is reduced, and the decoding efficiency is improved. Furthermore, since the audio encoding elements of speech frames corresponding to empty characters are removed during pruning, the accuracy of speech recognition is not affected when performing speech recognition based on the pruned audio encoding vector.

[0119] In some embodiments, the speech recognition method can be implemented using a speech recognition model, which includes an encoder, a classifier, and a decoder. The encoder performs vector encoding on the audio feature vector of the speech to be recognized, obtaining an audio encoded vector. The classifier classifies the audio encoded vector, obtaining the predicted probability distribution of each predicted character in a preset vocabulary corresponding to each speech frame in the speech to be recognized. The decoder implements the inference process of speech recognition, that is, it decodes and predicts based on the audio encoded vector to obtain the speech recognition result.

[0120] The training process of the speech recognition model in this application embodiment will be described below. The training process of the speech recognition model in this application embodiment includes two stages: the first stage is the model pre-training stage, which is used to pre-train the initial parameters of the speech recognition model based on sample data; the second stage is the model fine-tuning stage, which is used to fine-tune and update the parameters in the speech recognition model obtained after pre-training.

[0121] Figure 6 This is a flowchart illustrating the pre-training stage of the speech recognition model training method provided in this application embodiment. The speech recognition model training method can be executed by a model training device. The model training device can be a component within a speech recognition device (i.e., an electronic device), meaning it can be a server or a terminal; alternatively, it can be a separate device independent of the speech recognition device, i.e., an electronic device other than the server and terminal used to implement the speech recognition method. Figure 6 As shown, the pre-training phase of the speech recognition model includes the following steps S301 to S307:

[0122] Step S301: Input the sample data into the speech recognition model; the sample data includes the sample audio feature vector and label sequence of the sample speech.

[0123] In this embodiment of the application, the sample speech can be vectorized to obtain the sample audio feature vector.

[0124] The sample speech can be single-channel audio, and it includes multiple sample speech frames. Vectorization processing refers to extracting audio features from the sub-waveform corresponding to each sample speech frame in a one-dimensional waveform, thereby obtaining a fixed-length vector corresponding to each sample speech frame.

[0125] Step S302: The sample audio feature vector is processed by the encoder of the speech recognition model to obtain the sample audio encoded vector.

[0126] Step S303: The sample audio encoding vector is classified by the classifier of the speech recognition model to obtain the sample prediction probability distribution of each predicted character in the preset vocabulary for each sample speech frame in the sample speech.

[0127] Here, the sample prediction probability distribution includes the sample prediction probability of each sample speech frame for each predicted character in a preset vocabulary. In one implementation, the sample prediction probability distribution can be represented by an N×M table. In the table, the horizontal list represents each sample speech frame, for a total of M sample speech frames; the vertical list represents each predicted character in the preset vocabulary, for a total of N predicted characters, where M and N are both integers greater than 1. Each cell in the table can be filled with the sample prediction probability of the corresponding sample speech frame for the corresponding predicted character. Based on this N×M table, the sample prediction probability distribution of each sample speech frame corresponding to each predicted character in the preset vocabulary can be intuitively obtained.

[0128] Step S304: Input the sample prediction probability distribution into the CTC loss model to obtain the CTC loss result.

[0129] Here, the CTC loss model can calculate the CTC loss result of the sample prediction probability distribution based on the CTC loss function.

[0130] Step S305: Input the sample audio encoding vector and tag sequence into the decoder to obtain the sample prediction sequence.

[0131] Here, the decoder decodes the sample audio encoding vector to obtain the sample prediction sequence.

[0132] Step S306: Determine the cross-entropy loss result based on the sample prediction sequence and the label sequence.

[0133] Here, the cross-entropy loss result is obtained by calculating the cross-entropy loss value between the sample predicted sequence and the label sequence. The smaller the cross-entropy loss value, the smaller the difference between the sample predicted sequence and the label sequence; the larger the cross-entropy loss value, the greater the difference between the sample predicted sequence and the label sequence.

[0134] Step S307: Based on the CTC loss results and cross-entropy loss results, the parameters in the speech recognition model are corrected to obtain the trained speech recognition model.

[0135] In some embodiments, see Figure 7 , Figure 7 Step S307 can be achieved through the following steps S3071 to S3073:

[0136] Step S3071: Obtain the CTC loss weights corresponding to the CTC loss results and the cross-entropy loss weights corresponding to the cross-entropy loss results.

[0137] In this embodiment, the CTC loss weight and cross-entropy loss weight can be preset weight values. For example, the CTC loss weight can be set to 30%, and the cross-entropy loss weight to 70%.

[0138] Step S3072: Based on the CTC loss weight and the cross-entropy loss weight, the CTC loss result and the cross-entropy loss result are weighted and calculated to obtain the weighted loss result.

[0139] Here, the CTC loss result can be weighted based on the CTC loss weight to obtain the weighted CTC loss result; the cross-entropy loss result can be weighted based on the cross-entropy loss weight to obtain the weighted cross-entropy loss result. Then, the weighted CTC loss result and the weighted cross-entropy loss result are summed to obtain the weighted loss result.

[0140] Step S3073: Based on the weighted loss result, the parameters in the speech recognition model are corrected to obtain the trained speech recognition model.

[0141] The speech recognition model training method provided in this application corrects the parameters in the speech recognition model during the model pre-training stage using CTC loss results and cross-entropy loss results, thereby enabling accurate training of the speech recognition model.

[0142] Figure 8 This is a flowchart illustrating the fine-tuning stage of the training method for the speech recognition model provided in this application embodiment. The fine-tuning stage of the speech recognition model can also be executed using a model training device. Figure 8 As shown, the fine-tuning stage of the speech recognition model includes the following steps S401 to S408:

[0143] Step S401: Input the fine-tuning sample data into the trained speech recognition model; the fine-tuning sample data includes the fine-tuning sample audio feature vector and the fine-tuning label sequence of the fine-tuning sample speech.

[0144] In some embodiments, the trained speech recognition model can be fine-tuned during the speech recognition process. That is, the fine-tuned sample speech can be the speech to be recognized, and the fine-tuned label sequence can be the label sequence corresponding to the speech to be recognized. In this way, the trained speech recognition model can be continuously fine-tuned during use, thereby gradually improving the recognition accuracy of the speech recognition model.

[0145] Step S402: The encoder of the trained speech recognition model performs vector encoding on the fine-tuned sample audio feature vector to obtain the fine-tuned sample audio encoding vector.

[0146] Step S403: The fine-tuned sample audio encoding vector is classified by the classifier of the trained speech recognition model to obtain the fine-tuned sample prediction probability distribution of each predicted character in the preset vocabulary for each fine-tuned sample speech frame.

[0147] Step S404: Input the predicted probability distribution of the fine-tuned samples into the CTC loss model to obtain the fine-tuned CTC loss result.

[0148] Step S405: Based on the predicted probability distribution of the fine-tuned samples, the audio coding vector of the fine-tuned samples is pruned to obtain the pruned audio coding vector of the fine-tuned samples.

[0149] In this embodiment, the fine-tuning sample audio encoding vector is pruned during the fine-tuning stage to obtain a pruned fine-tuning sample audio encoding vector, thereby reducing the number of fine-tuning sample audio encoding elements in the fine-tuning sample audio encoding vector and improving the recognition efficiency of the subsequent speech recognition process.

[0150] Step S406: Input the pruned fine-tuned sample audio encoding vector and fine-tuned label sequence into the decoder to obtain the fine-tuned sample prediction sequence.

[0151] Step S407: Based on the sample prediction sequence and the fine-tuned label sequence, determine the fine-tuned cross-entropy loss result.

[0152] Step S408: Based on the fine-tuned CTC loss results and the fine-tuned cross-entropy loss results, the parameters in the trained speech recognition model are corrected to obtain the fine-tuned speech recognition model.

[0153] In some embodiments, the encoder in the speech recognition model can be a hierarchical encoder, which consists of multiple neural networks with the same structure. In implementation, steps S302 to S304 can be implemented as follows: The sample audio feature vectors are vector-encoded using different levels of the hierarchical encoder's neural networks, resulting in multiple different sample audio encoding vectors; the different sample audio encoding vectors are classified using a classifier, resulting in multiple different sample prediction probability distributions; and the multiple different sample prediction probability distributions are input into the CTC loss model, resulting in multiple CTC loss results.

[0154] In this embodiment of the application, after obtaining multiple CTC loss results, loss fusion processing can be performed on the multiple CTC loss results to obtain a fused loss result; then, based on the fused loss result, the parameters in the speech recognition model are corrected to obtain the trained speech recognition model.

[0155] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0156] Speech recognition technology is a computer technology that identifies given audio as corresponding text. In this application's embodiments, the speech to be recognized refers to single-channel audio, but obviously, the recognition of multi-channel audio is generally converted into single-channel audio as well. For the input speech to be recognized, it is generally considered to be a one-dimensional waveform. Using a certain method, the speech to be recognized can be converted into a two-dimensional matrix x = [x1, ..., x...]. T Methods such as convolutional neural networks, MFCC, and FBANK features are used. Each element x... t Each vector is a fixed-length vector. T represents the number of vectors, hereinafter referred to as the frame number. A neural network with a certain structure (hereinafter referred to as the encoder) is used to process the two-dimensional matrix x = [x1, ..., x2]. T Encode the vector h = [h1, ..., h2] to obtain the encoded vector h = [h1, ..., h2]. T The number of frames is consistent with the two-dimensional feature input x.

[0157] Typically, speech recognition systems use a fixed vocabulary V. In CTC, an additional character set is also present. <blk>This represents the concept of "empty". Using a classifier, based on h, we predict a distribution y = [y1, ..., y2] where an empty character is merged into a vocabulary V. T ],in This represents the probability of the k-th predicted character in the t-th frame.

[0158] Meanwhile, the data used to train the speech recognition system is labeled. Define l = [l1, ..., l...] U ], where each element l u Each character represents a non-empty character in the vocabulary V. U represents the total length of the labeled data. CTC requires U ≤ T. This assumption is generally satisfied for normal speech recognition systems.

[0159] CTC attempts to maximize the posterior probability of the labeled sequence l based on a given feature x, i.e., maximize the probability p(l|x). However, directly optimizing this posterior probability is difficult, so a transformation is required. Let the path π = [π1,…,π] T [ is a sequence of length T, where any π] t It can be a word in the vocabulary V or an empty character. Define another mapping B to process π: it removes all consecutive repeating non-empty characters, removes all empty characters, but retains repeating characters separated by empty characters. For example, B(A <blk>ABB <blk>= AAB. Based on the above definition, maximizing the posterior probability p(l|x) is transformed into maximizing the sum of probabilities of all possible paths that lead to the labeled sequence l, i.e., the following formula (1):

[0160]

[0161] Among them, the summation object B -1 (l) represents the set of all paths that, after mapping B, yield the labeled sequence l. The posterior probability of each path is expressed as follows: (2)

[0162]

[0163] The approximately equal sign in the middle reflects the conditional independence assumption widely used in CTC.

[0164] For a given x and l, it is impractical to enumerate all possible paths, calculate their probabilities individually, and then sum them up for optimization, because the number of possible paths explodes exponentially as x and l increase. However, an equivalent computation can be performed using a forward-backward algorithm.

[0165] However, in speech recognition systems using CTC (Continuous Transmission Theory), a common architecture is the encoder-decoder structure. That is, the encoder's output h, besides being used to predict the posterior probability y of the CTC, often serves as training and decoding data for the decoder network. The computational cost or real-time performance during inference (decoding) is a crucial indicator of a speech recognition system. For many systems, the computational cost or real-time performance is highly correlated with the number of frames in the encoder's output h; that is, fewer frames result in lower decoding computation. In general practice, the encoder's output h is entirely input into the decoder for training and decoding. Since T is generally much larger than U, it's possible to reduce the number of frames in the encoder's output h to lower the decoding cost. Reducing the number of frames in h is equivalent to reducing the sequence length, which can be understood as a form of downsampling.

[0166] The speech recognition method provided in this application is a solution used in the AI ​​Lab end-to-end speech recognition framework, which can reduce the model inference cost and keep the system performance almost unchanged.

[0167] In optimizing the CTC loss, all paths in related techniques are considered equally. However, in this embodiment, more attention is paid to those paths that end non-empty predictions earlier. For example, compared to sequence A... <blk> <blk>B, on the other hand, prefers sequence A. <blk> B <blk>This will receive more emphasis. In this way, it is expected that the CTC prediction distribution y will concentrate all non-empty predictions within earlier timestamps.

[0168] In the implementation process, the function τ=f(π) can be defined as the timestamp of the last non-empty prediction in the path, and the corresponding training objective can be modified as follows (3):

[0169]

[0170] Here, λ is a hyperparameter greater than zero, and its value can take on a preset range. For example, for Chinese characters, the value of λ can be between 10 and 50; for English characters, the value of λ can be between 100 and 1000. This method can reuse the forward and backward algorithms in CTC to calculate the training objective, but unlike the standard CTC, it does not have a closed-form solution for the gradient. However, since both the forward and backward algorithms only use addition and multiplication, their gradients can still be obtained using the naive chain rule. The modified CTC described above is called Bayesian CTC (BCTC).

[0171] The following is an example comparing BCTC and standard CTC. Figure 9 and Figure 10 As shown, in the prediction distribution y of the standard CTC, non-empty predictions are randomly scattered along the time axis; while the BCTC allows the model to concentrate non-empty predictions mainly on the left side of the time axis. It is also pointed out that BCTC, like CTC, has the ability to correctly predict sequences. Figure 9 and Figure 10 In the two distributions shown, different line styles represent different non-whitespace characters. Figure 9 and Figure 10 The non-empty character sequences in these two images are consistent.

[0172] The following section explains the application of Bayesian CTC as a downsampling method.

[0173] In this embodiment, it can be assumed that by observing distribution y, important information in h can be located. If no non-empty characters are predicted in the subsequent frames of distribution y, the latter half of h is considered meaningless and can be deleted. Based on this, this embodiment proposes a two-stage method to downsample h.

[0174] The first stage is the pre-training stage. The model in the pre-training stage is as follows: Figure 11 As shown, the input x is encoded by encoder 111 (Enc) to obtain encoder output h, which is then processed by classifier 112 to obtain distribution y, and BCTC loss 113 is calculated. Simultaneously, h and the corresponding label sequence l (i.e., labeled sequence l) are input to attention-based decoder 114, and cross-entropy loss 115 is calculated. The model is trained using multi-task learning. During training, the BCTC weights and cross-entropy loss weights can be adjusted appropriately according to the needs of the task; for example, the BCTC weight can be 30%, and the cross-entropy loss weight can be 70%.

[0175] The second stage is the fine-tuning stage. This application's embodiments consider two end-to-end speech recognition technology frameworks: Attention-based Encoder-Decoder (AED) and Transducer (an autoregressive model).

[0176] For AEDs, the working framework is as follows: Figure 12 As shown. The AED framework and the structure of the one-stage training are basically the same, but with an additional pruning step 121. In the implementation process, based on the known distribution prediction y, the frames corresponding to h that predict empty characters with a prediction probability greater than 99% are deleted from right to left until a frame no longer meets this condition and the deletion stops. To prevent the loss of boundary information due to over-pruning, this embodiment stops deletion 5 frames before the stop condition is triggered. The encoder 111 outputs h after deletion, which becomes h'. h' and the label sequence l are input into the attention-based decoder 114 and the cross-entropy loss 115 is calculated. The multi-task learning training mode remains unchanged. Note that for the AED model, all trainable parameters during pre-training can be retained and reused in the fine-tuning stage. Also note that h and y used for BCTC do not involve pruning operations.

[0177] For NT (Neural Transducer) systems, the working framework is as follows: Figure 13 As shown. Similar to the AED system, the NT system uses the encoder output h', which has undergone pruning (131), for downstream decoder training. However, due to differences between the NT and AED systems, the joint network (132) and prediction network (133) of the NT system require random initialization for optimization. The NT system calculates the Transducer loss (134). The optimization ratio of the BCTC loss (113) and the Transducer loss (134) can be 1:2.

[0178] In some embodiments, the use of intermediate CTC (intermediate CTC) may also be provided. For example, Figures 11 to 13 The encoder in the model consists of several layers of a neural network with the same structure. When using hierarchical CTC, a classifier is still used to predict a distribution and calculate the BCTC loss at the output of the encoder at half the number of layers. The BCTC loss calculated in the middle layer and the BCTC loss calculated in the last layer can be weighted to form the overall BCTC loss, where the BCTC loss of the middle layer can account for 30% and the BCTC loss of the last layer can account for 70%.

[0179] Here, the scheme of this application embodiment is verified on the open-source Chinese speech recognition datasets AISHELL-1 and AISHELL-2, and the results are shown in Table 1 below.

[0180] Table 1

[0181]

[0182]

[0183] As shown in Table 1, the models using the BCTC two-stage training method (Experiments 2, 3, 5, 6) and the original scheme (Experiments 1, 4) exhibit similar recognition performance (word error rate CER, lower is better). Simultaneously, the experiments revealed that the model using BCTC has a lower real-time rate (lower is better), indicating a significant reduction in computational cost. For the NT model, the number of decoding steps is directly related to the encoder output length, and the corresponding number of decoding steps also decreases significantly, demonstrating the superiority of the proposed method.

[0184] It is understood that in the embodiments of this application, if the content involves user information, such as the voice to be recognized, the voice recognition result, etc., and if it involves data related to user information or enterprise information, when the embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0185] The following continues to describe the exemplary structure of the speech recognition device 354 provided in the embodiments of this application as a software module. In some embodiments, such as Figure 2 As shown, the speech recognition device 354 includes: a vectorization processing module 3541, used to perform vectorization processing on the speech to be recognized to obtain an audio feature vector; a vector encoding module 3542, used to perform vector encoding processing on the audio feature vector to obtain an audio encoding vector; a classification processing module 3543, used to perform classification processing on the audio encoding vector based on a preset vocabulary to obtain a prediction probability distribution of each predicted character in the preset vocabulary corresponding to each speech frame in the speech to be recognized; a pruning processing module 3544, used to perform pruning processing on the audio encoding vector based on the prediction probability distribution to obtain a pruned audio encoding vector; wherein, the pruning processing refers to removing the audio encoding element corresponding to at least one speech frame in the audio encoding vector; and a speech recognition module 3545, used to perform speech recognition on the speech to be recognized based on the pruned audio encoding vector to obtain a speech recognition result.

[0186] In some embodiments, the preset vocabulary includes multiple non-empty characters and one empty character; the classification processing module is further configured to: determine the predicted probability of each speech frame corresponding to each non-empty character and the empty character based on the non-empty characters and the empty character in the preset vocabulary, thereby forming the predicted probability distribution.

[0187] In some embodiments, the pruning processing module is further configured to: determine the predicted character with the highest prediction probability for each speech frame based on the prediction probability distribution; the predicted character includes multiple non-empty characters and an empty character; determine the speech frame with the predicted character being an empty character as the target speech frame; and perform a removal process on the audio coding element corresponding to the target speech frame in the audio coding vector to obtain the pruned audio coding vector.

[0188] In some embodiments, the pruning module is further configured to: sequentially remove the audio coding elements corresponding to the target speech frame in the audio coding vector according to the reverse order of the audio coding elements in the audio coding vector, to obtain the pruned audio coding vector.

[0189] In some embodiments, the apparatus further includes: a first determining module, configured to determine the audio coding element corresponding to the target speech frame in the audio coding vector as the target audio coding element when the pruning process is stopped; a second determining module, configured to determine a preset number of audio coding elements as boundary audio coding elements, starting from the target audio coding element, according to the ascending order of the audio coding elements in the audio coding vector; and a control module, configured to prohibit the removal of the boundary audio coding elements.

[0190] In some embodiments, the predicted character includes multiple non-empty characters and one empty character; the pruning processing module is further configured to: obtain the empty character prediction probability for each speech frame corresponding to the empty character based on the prediction probability distribution; determine the audio coding elements corresponding to the speech frames whose empty character prediction probability is greater than the probability threshold in the audio coding vector as audio coding elements to be removed; and remove the audio coding elements to be removed sequentially according to the reverse order of the audio coding elements in the audio coding vector to obtain the pruned audio coding vector.

[0191] In some embodiments, the predicted character includes multiple non-empty characters and one empty character; the pruning processing module is further configured to: determine, based on the prediction probability distribution, the predicted character with the highest prediction probability for each speech frame and the empty character prediction probability for each speech frame corresponding to the empty character; determine speech frames whose predicted character is an empty character and whose empty character prediction probability is greater than a probability threshold as speech frames to be pruned; and sequentially remove the corresponding audio coding elements of the speech frames to be pruned in the audio coding vector according to the reverse order of the audio coding elements in the audio coding vector to obtain the pruned audio coding vector.

[0192] In some embodiments, the speech to be identified is single-channel audio, which is represented by a one-dimensional waveform; the vectorization processing module is further configured to: extract audio features from the sub-waveform corresponding to each speech frame in the one-dimensional waveform to obtain a fixed-length vector corresponding to each speech frame; wherein, the fixed-length vectors corresponding to all speech frames constitute the audio feature vector, which is a two-dimensional matrix.

[0193] In some embodiments, the speech recognition method is implemented through a speech recognition model; the apparatus further includes a model training device, which is used to: input sample data into the speech recognition model; the sample data includes sample audio feature vectors and label sequences of sample speech; perform vector encoding processing on the sample audio feature vectors through the encoder of the speech recognition model to obtain sample audio encoding vectors; perform classification processing on the sample audio encoding vectors through the classifier of the speech recognition model to obtain the sample prediction probability distribution of each predicted character in the preset vocabulary corresponding to each sample speech frame in the sample speech; input the sample prediction probability distribution into the CTC loss model to obtain the CTC loss result; input the sample audio encoding vectors and the label sequences into the decoder to obtain the sample prediction sequence; determine the cross-entropy loss result based on the sample prediction sequence and the label sequence; and correct the parameters in the speech recognition model based on the CTC loss result and the cross-entropy loss result to obtain the trained speech recognition model.

[0194] In some embodiments, the model training device is further configured to: obtain the CTC loss weights corresponding to the CTC loss result and the cross-entropy loss weights corresponding to the cross-entropy loss result; perform weighted calculation on the CTC loss result and the cross-entropy loss result based on the CTC loss weights and the cross-entropy loss weights to obtain a weighted loss result; and correct the parameters in the speech recognition model based on the weighted loss result to obtain the trained speech recognition model.

[0195] In some embodiments, the model training device is further configured to: perform model fine-tuning on the trained speech recognition model; wherein, the model training device is specifically configured to: input fine-tuning sample data into the trained speech recognition model; the fine-tuning sample data includes fine-tuning sample audio feature vectors and fine-tuning label sequences of the fine-tuning sample speech; perform vector encoding processing on the fine-tuning sample audio feature vectors through the encoder of the trained speech recognition model to obtain fine-tuning sample audio encoding vectors; and perform classification processing on the fine-tuning sample audio encoding vectors through the classifier of the trained speech recognition model to obtain each predicted character in the preset vocabulary corresponding to each fine-tuning sample speech frame in the fine-tuning sample speech. Fine-tune the sample prediction probability distribution; input the fine-tuned sample prediction probability distribution into the CTC loss model to obtain the fine-tuned CTC loss result; based on the fine-tuned sample prediction probability distribution, prune the fine-tuned sample audio encoding vector to obtain the pruned fine-tuned sample audio encoding vector; input the pruned fine-tuned sample audio encoding vector and the fine-tuned label sequence into the decoder to obtain the fine-tuned sample prediction sequence; based on the sample prediction sequence and the fine-tuned label sequence, determine the fine-tuned cross-entropy loss result; based on the fine-tuned CTC loss result and the fine-tuned cross-entropy loss result, correct the parameters in the trained speech recognition model to obtain the fine-tuned speech recognition model.

[0196] In some embodiments, the encoder is a hierarchical encoder, which is composed of multiple neural networks with the same structure; the model training device is further configured to: perform vector encoding processing on the sample audio feature vectors through the neural networks of different levels of the hierarchical encoder, corresponding to multiple different sample audio encoding vectors; perform classification processing on the different sample audio encoding vectors respectively through the classifier, corresponding to multiple different sample prediction probability distributions; input the multiple different sample prediction probability distributions into the CTC loss model respectively, corresponding to multiple CTC loss results; perform loss fusion processing on the multiple CTC loss results to obtain a fused loss result; and based on the fused loss result, correct the parameters in the speech recognition model to obtain the trained speech recognition model.

[0197] It should be noted that the description of the apparatus in this application embodiment is similar to the description of the method embodiment described above, and has similar beneficial effects as the method embodiment; therefore, it will not be repeated. For technical details not disclosed in this apparatus embodiment, please refer to the description of the method embodiment of this application for understanding.

[0198] This application provides a computer program product including executable instructions, which are computer instructions stored in a computer-readable storage medium. When the processor of an electronic device reads the executable instructions from the computer-readable storage medium and executes the executable instructions, the electronic device performs the method described in this application.

[0199] This application provides a storage medium storing executable instructions. When these executable instructions are executed by a processor, they cause the processor to perform the method provided in this application, for example... Figure 3 The method shown.

[0200] In some embodiments, the storage medium may be a computer-readable storage medium, such as a ferromagnetic random access memory (FRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEROM), flash memory, magnetic surface memory, optical disk, or a compact disk-read-only memory (CD-ROM); or it may be a device that includes one or any combination of the above-mentioned memories.

[0201] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0202] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file containing other programs or data, for example, in one or more scripts within a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files storing one or more modules, subroutines, or code sections). As an example, executable instructions may be deployed to execute on a single electronic device, or on multiple electronic devices located in one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0203] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.< / blk> < / blk> < / blk> < / blk> < / blk> < / blk> < / blk>

Claims

1. A speech recognition method, characterized in that, The method includes: The speech to be recognized is vectorized to obtain audio feature vectors; The audio feature vector is subjected to vector encoding processing to obtain the audio encoded vector; Based on a preset vocabulary, the audio encoding vector is classified to obtain the prediction probability distribution of each predicted character in the preset vocabulary for each speech frame in the speech to be recognized. Based on the predicted probability distribution, the audio coding vector is pruned to obtain a pruned audio coding vector; wherein, the pruning of the audio coding vector based on the predicted probability distribution to obtain a pruned audio coding vector includes: determining, based on the predicted probability distribution, the predicted character with the highest predicted probability for each speech frame; the predicted character includes multiple non-empty characters and one empty character; determining the speech frame with the predicted character being an empty character as the target speech frame; and removing the corresponding audio coding elements of the target speech frame from the audio coding vector to obtain the pruned audio coding vector; Based on the pruned audio coding vector, speech recognition is performed on the speech to be recognized to obtain the speech recognition result.

2. The method according to claim 1, characterized in that, The preset vocabulary includes multiple non-empty characters and one empty character; The step of classifying the audio encoding vector based on a preset vocabulary to obtain the prediction probability distribution of each predicted character in the preset vocabulary for each speech frame in the speech to be recognized includes: Based on the non-empty characters and empty characters in the preset vocabulary, the predicted probability of each speech frame corresponding to each non-empty character and the empty character is determined, forming the predicted probability distribution.

3. The method according to claim 1, characterized in that, The step of removing the audio coding elements corresponding to the target speech frame in the audio coding vector to obtain the pruned audio coding vector includes: Following the reverse order of the audio coding elements in the audio coding vector, the corresponding audio coding elements of the target speech frame in the audio coding vector are sequentially removed to obtain the pruned audio coding vector.

4. The method according to claim 1, characterized in that, The method further includes: When the pruning process is stopped, the audio coding element corresponding to the target speech frame in the audio coding vector is determined as the target audio coding element; Following the ascending order of the audio coding elements in the audio coding vector, starting from the target audio coding element, a preset number of audio coding elements are determined as boundary audio coding elements; The removal of the boundary audio encoding elements is prohibited.

5. The method according to claim 1, characterized in that, The step of pruning the audio coding vector based on the predicted probability distribution to obtain the pruned audio coding vector includes: Based on the predicted probability distribution, the predicted probability of the empty character corresponding to the empty character in each speech frame is obtained; The audio coding elements corresponding to the speech frames whose predicted probability of the empty character is greater than the probability threshold in the audio coding vector are determined as audio coding elements to be removed. The audio coding elements in the audio coding vector are sequentially removed in reverse order to obtain the pruned audio coding vector.

6. The method according to claim 1, characterized in that, The step of pruning the audio coding vector based on the predicted probability distribution to obtain the pruned audio coding vector includes: Based on the predicted probability distribution, the predicted character with the highest predicted probability in each speech frame and the predicted probability of the empty character corresponding to the empty character in each speech frame are determined. Speech frames whose predicted character is an empty character and whose predicted probability of the empty character is greater than the probability threshold are identified as speech frames to be pruned. Following the reverse order of the audio coding elements in the audio coding vector, the audio coding elements corresponding to the speech frames to be pruned in the audio coding vector are sequentially removed to obtain the pruned audio coding vector.

7. The method according to any one of claims 1 to 6, characterized in that, The speech to be recognized is single-channel audio, which is represented by a one-dimensional waveform; the vectorization process of the speech to be recognized to obtain an audio feature vector includes: Audio features are extracted from the sub-waveform corresponding to each speech frame in the one-dimensional waveform to obtain a fixed-length vector corresponding to each speech frame; wherein, the fixed-length vectors corresponding to all speech frames constitute the audio feature vector, and the audio feature vector is a two-dimensional matrix.

8. The method according to any one of claims 1 to 6, characterized in that, The speech recognition method is implemented through a speech recognition model; the speech recognition model is trained through the following steps: The sample data is input into the speech recognition model; the sample data includes the sample audio feature vector and label sequence of the sample speech. The encoder of the speech recognition model performs vector encoding on the sample audio feature vector to obtain the sample audio encoded vector. The classifier of the speech recognition model is used to classify the sample audio encoding vector to obtain the sample prediction probability distribution of each predicted character in the preset vocabulary for each sample speech frame in the sample speech. The predicted probability distribution of the samples is input into the CTC loss model to obtain the CTC loss result; The sample audio encoding vector and the tag sequence are input into the decoder to obtain the sample prediction sequence; Based on the sample prediction sequence and the label sequence, determine the cross-entropy loss result; Based on the CTC loss result and the cross-entropy loss result, the parameters in the speech recognition model are corrected to obtain the trained speech recognition model.

9. The method according to claim 8, characterized in that, The step of correcting the parameters in the speech recognition model based on the CTC loss result and the cross-entropy loss result to obtain the trained speech recognition model includes: Obtain the CTC loss weights corresponding to the CTC loss results and the cross-entropy loss weights corresponding to the cross-entropy loss results; Based on the CTC loss weight and the cross-entropy loss weight, the CTC loss result and the cross-entropy loss result are weighted and calculated to obtain the weighted loss result. Based on the weighted loss result, the parameters in the speech recognition model are corrected to obtain the trained speech recognition model.

10. The method according to claim 8, characterized in that, The method further includes: performing model fine-tuning on the trained speech recognition model; wherein the model fine-tuning includes the following steps: The fine-tuning sample data is input into the trained speech recognition model; the fine-tuning sample data includes the fine-tuning sample audio feature vector and the fine-tuning label sequence of the fine-tuning sample speech. The encoder of the trained speech recognition model performs vector encoding on the fine-tuned sample audio feature vector to obtain the fine-tuned sample audio encoding vector. The fine-tuned sample audio encoding vector is classified by the classifier of the trained speech recognition model to obtain the fine-tuned sample prediction probability distribution of each predicted character in the preset vocabulary for each fine-tuned sample speech frame. The fine-tuned sample prediction probability distribution is input into the CTC loss model to obtain the fine-tuned CTC loss result; Based on the predicted probability distribution of the fine-tuned sample, the audio coding vector of the fine-tuned sample is pruned to obtain the pruned audio coding vector of the fine-tuned sample. The fine-tuned sample audio encoding vector after pruning and the fine-tuned label sequence are input into the decoder to obtain the fine-tuned sample prediction sequence; Based on the sample prediction sequence and the fine-tuned label sequence, determine the fine-tuned cross-entropy loss result; Based on the fine-tuned CTC loss results and the fine-tuned cross-entropy loss results, the parameters in the trained speech recognition model are corrected to obtain the fine-tuned speech recognition model.

11. The method according to claim 8, characterized in that, The encoder is a hierarchical encoder, which is composed of multiple neural networks with the same structure; The speech recognition model's encoder performs vector encoding on the sample audio feature vector to obtain a sample audio encoded vector; the speech recognition model's classifier performs classification on the sample audio encoded vector to obtain the sample prediction probability distribution of each predicted character in the preset vocabulary for each sample speech frame in the sample speech. The predicted probability distribution of the samples is input into the CTC loss model to obtain the CTC loss result, including: The sample audio feature vector is vector encoded by the neural network of different levels of the hierarchical encoder, resulting in multiple different sample audio encoded vectors. The classifier is used to classify the different sample audio encoding vectors respectively, resulting in multiple different sample prediction probability distributions. The multiple different sample prediction probability distributions are input into the CTC loss model respectively, resulting in multiple CTC loss results; The method further includes: The multiple CTC loss results are subjected to loss fusion processing to obtain a fused loss result; Based on the fusion loss result, the parameters in the speech recognition model are corrected to obtain the trained speech recognition model.

12. A voice recognition device, characterized in that, The device includes: The vectorization processing module is used to perform vectorization processing on the speech to be recognized to obtain audio feature vectors; The vector encoding module is used to perform vector encoding processing on the audio feature vector to obtain the audio encoded vector; The classification processing module is used to classify the audio encoding vector based on a preset vocabulary to obtain the prediction probability distribution of each predicted character in the preset vocabulary for each speech frame in the speech to be recognized. The pruning module is used to prune the audio encoding vector based on the predicted probability distribution to obtain a pruned audio encoding vector. The pruning process includes: determining the predicted character with the highest prediction probability for each speech frame based on the predicted probability distribution; the predicted character includes multiple non-empty characters and one empty character; identifying speech frames where the predicted character is an empty character as target speech frames; and removing the corresponding audio encoding elements of the target speech frame from the audio encoding vector to obtain the pruned audio encoding vector. The speech recognition module is used to perform speech recognition on the speech to be recognized based on the pruned audio encoding vector, and obtain the speech recognition result.

13. An electronic device, characterized in that, include: Memory, used to store executable instructions; A processor, when executing executable instructions stored in the memory, implements the speech recognition method according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, The device stores executable instructions for causing a processor to execute the executable instructions to implement the speech recognition method according to any one of claims 1 to 11.

Images