Speech recognition model processing methods, devices, computer equipment and storage media
By performing human voice recognition and feature merging on speech frames, and using the wenet and trtorch frameworks for batch inference of the speech recognition model, the problem of slow speech recognition speed is solved, achieving faster speech recognition and higher computational efficiency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- PING AN TECH (SHENZHEN) CO LTD
- Filing Date
- 2023-04-11
- Publication Date
- 2026-05-26
AI Technical Summary
Existing speech recognition models have low recognition speed and poor real-time performance, failing to meet the need for rapid user response.
By acquiring the speech features of speech frames, human voice recognition is performed, human voice speech frames are filtered out, and the merged features are input into a speech recognition model based on Wenet for batch inference. The inference is accelerated using the Trtorch framework.
It improves the speed and computational efficiency of speech recognition, reduces the processing of useless data, and enhances real-time performance and user experience.
Smart Images
Figure CN116453509B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, and in particular to a speech recognition model processing method, apparatus, computer equipment, and storage medium. Background Technology
[0002] With the development of artificial intelligence technology, speech recognition is being used more and more widely in daily life. Speech recognition often operates in the downstream stages of these applications, directly interacting with users. For example, in the field of intelligent customer service, it involves receiving and recognizing customers' voices in real time, and then responding to users based on the recognition results. Therefore, the speed of speech recognition is particularly important and profoundly impacts the user experience.
[0003] Streaming speech recognition refers to a type of speech recognition model that can return recognition results in real time while processing audio streams. However, the inventors found that current speech recognition models still have low recognition speed and poor real-time performance. Summary of the Invention
[0004] The purpose of this application is to provide a speech recognition model processing method, apparatus, computer equipment, and storage medium to solve the problem of low speech recognition speed in speech recognition models.
[0005] To address the aforementioned technical problems, this application provides a speech recognition model processing method, which employs the following technical solution:
[0006] Obtain the speech to be recognized, which contains multiple speech frames;
[0007] Feature extraction is performed on each of the speech frames to obtain the speech features of each speech frame;
[0008] Based on the speech features of each speech frame, human voice recognition is performed on each speech frame to obtain a human voice speech frame;
[0009] The speech features of each human voice speech frame are obtained, and the speech features are merged to obtain the merged features;
[0010] The merged features are input into a speech recognition model reconstructed by trtorch, and the merged features are subjected to batch inference through the speech recognition model to obtain the speech recognition result of the speech to be recognized. The speech recognition model is built based on wenet.
[0011] To address the aforementioned technical problems, this application also provides a speech recognition model processing device, which employs the following technical solution:
[0012] The speech acquisition module is used to acquire the speech to be recognized, which contains multiple speech frames;
[0013] The feature extraction module is used to extract features from each of the speech frames to obtain the speech features of each speech frame.
[0014] A voice recognition module is used to perform voice recognition on each speech frame based on the speech features of each speech frame to obtain a human voice speech frame;
[0015] The feature merging module is used to obtain the speech features of each human voice speech frame and merge the speech features to obtain the merged features.
[0016] The speech recognition module is used to input the merged features into a speech recognition model reconstructed by trtorch, so as to perform batch inference on the merged features through the speech recognition model to obtain the speech recognition result of the speech to be recognized, wherein the speech recognition model is built based on wenet.
[0017] To address the aforementioned technical problems, this application also provides a computer device that employs the following technical solution:
[0018] Obtain the speech to be recognized, which contains multiple speech frames;
[0019] Feature extraction is performed on each of the speech frames to obtain the speech features of each speech frame;
[0020] Based on the speech features of each speech frame, human voice recognition is performed on each speech frame to obtain a human voice speech frame;
[0021] The speech features of each human voice speech frame are obtained, and the speech features are merged to obtain the merged features;
[0022] The merged features are input into a speech recognition model reconstructed by trtorch, and the merged features are subjected to batch inference through the speech recognition model to obtain the speech recognition result of the speech to be recognized. The speech recognition model is built based on wenet.
[0023] To address the aforementioned technical problems, this application also provides a computer-readable storage medium, employing the technical solution described below:
[0024] Obtain the speech to be recognized, which contains multiple speech frames;
[0025] Feature extraction is performed on each of the speech frames to obtain the speech features of each speech frame;
[0026] Based on the speech features of each speech frame, human voice recognition is performed on each speech frame to obtain a human voice speech frame;
[0027] The speech features of each human voice speech frame are obtained, and the speech features are merged to obtain the merged features;
[0028] The merged features are input into a speech recognition model reconstructed by trtorch, and the merged features are subjected to batch inference through the speech recognition model to obtain the speech recognition result of the speech to be recognized. The speech recognition model is built based on wenet.
[0029] Compared with the prior art, the embodiments of this application have the following advantages: acquiring speech to be recognized containing multiple speech frames; performing human voice recognition on each speech frame based on the speech features of each speech frame to determine whether the speech frame contains a human voice, thus obtaining a human voice speech frame; acquiring the speech features of each human voice speech frame and merging the speech features to obtain merged features, thereby processing only the speech features containing human voices and avoiding processing useless data; inputting the merged features into a speech recognition model, which is built based on Wenet and reconstructed by Trtorch, thereby accelerating inference through the Trtorch framework; and performing batch inference on the merged features by the speech recognition model to further improve computational efficiency, thereby greatly improving the speed of speech recognition. Attached Figure Description
[0030] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0031] Figure 1 This is an exemplary system architecture diagram to which this application can be applied;
[0032] Figure 2 This is a flowchart of an embodiment of the speech recognition model processing method according to this application;
[0033] Figure 3 This is a schematic diagram of the structure of one embodiment of the speech recognition model processing apparatus according to this application;
[0034] Figure 4 This is a schematic diagram of the structure of one embodiment of the computer device according to this application. Detailed Implementation
[0035] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.
[0036] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.
[0037] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.
[0038] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.
[0039] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social media platform software, etc.
[0040] Terminal devices 101, 102, and 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 players (Moving Picture Experts Group Audio Layer IV), laptops, and desktop computers, etc.
[0041] Server 105 can be a server that provides various services, such as a backend server that supports the pages displayed on terminal devices 101, 102, and 103.
[0042] It should be noted that the speech recognition model processing method provided in this application embodiment is generally executed by a server, and correspondingly, the speech recognition model processing device is generally located in the server.
[0043] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.
[0044] Continue to refer to Figure 2 The diagram illustrates a flowchart of an embodiment of the speech recognition model processing method according to this application. The speech recognition model processing method includes the following steps:
[0045] Step S201: Obtain the speech to be recognized, which contains multiple speech frames.
[0046] In this embodiment, the speech recognition model processing method runs on an electronic device (e.g., Figure 1 The server shown can communicate with the terminal via wired or wireless connection. It should be noted that the aforementioned wireless connection methods may include, but are not limited to, 3G / 4G / 5G connections, WiFi connections, Bluetooth connections, WiMAX connections, Zigbee connections, UWB (ultra wideband) connections, and other currently known or future wireless connection methods.
[0047] Specifically, the process involves acquiring the speech to be recognized. This speech can come from a terminal, which collects the user's voice to obtain the speech to be recognized and sends it to the server. The speech to be recognized consists of multiple speech frames, and processing is performed on a frame-by-frame basis.
[0048] Step S202: Extract features from each speech frame to obtain the speech features of each speech frame.
[0049] Specifically, feature extraction is performed on each speech frame to obtain the speech features of each speech frame. The speech features can be feature parameters of a preset type. For example, the speech features can be at least one of linear prediction cepstral coefficients (LPCC) and Mel-scale frequency cepstral coefficients (MFCC).
[0050] Step S203: Perform human voice recognition on each speech frame based on the speech features of each speech frame to obtain human voice speech frames.
[0051] Specifically, speech analysis can be performed on speech frames based on speech features. For each speech frame, human voice recognition is performed based on the speech features of the speech frame to determine whether the speech frame contains a human voice. Speech frames containing human voices are marked as human voice speech frames.
[0052] Step S204: Obtain the speech features of each human voice speech frame, and merge the speech features to obtain merged features.
[0053] Specifically, the speech features of each human voice speech frame are obtained, and the obtained speech features are merged to obtain merged features. For speech frames that do not contain human voices, in order to avoid invalid speech data processing, the speech frames that do not contain human voices can be discarded.
[0054] Step S205: Input the merged features into the speech recognition model reconstructed by trtorch, so as to perform batch inference on the merged features through the speech recognition model to obtain the speech recognition result of the speech to be recognized. The speech recognition model is built based on wenet.
[0055] Specifically, the merged features are input into the speech recognition model. The speech recognition model can be built on a neural network; in one embodiment, it is built on wenet. Wenet is a unified model for streaming and non-streaming speech recognition, originally based on the PyTorch speech processing framework. In this application, wenet is refactored using trtorch. trtorch is a PyTorch / TorchScript compiler, and trtorch is an NVIDIA TensorRT deep learning optimizer and a compiler for the NVIDIA GPU runtime. Unlike PyTorch's Just-In-Time (JIT) compiler, trtorch is an Ahead-of-Time (AOF) compiler, meaning that before deploying TorchScript code, standard TorchScript programs need to be converted into modules for the TensorRT engine through explicit compilation steps. trtorch, as a PyTorch extension, is a module that seamlessly integrates with the JIT runtime for runtime execution and compilation. This application uses the trtorch framework to accelerate wenet for GPUs; wenet has been refactored based on trtorch, and the refactored wenet improves inference speed when running on trtorch.
[0056] The speech recognition model performs speech recognition processing on merged features. During this process, the speech recognition model performs batch inference on the merged features based on a batch inference model. Batch inference means that the model can choose to process multiple data points at once within the allowed latency. The model can infer N (N is a positive integer greater than 1) data points simultaneously and then distribute them to N threads, thereby improving computational efficiency and accelerating inference. The number of data points processed each time, i.e., the batch size, can be preset. Alternatively, the batch size can be determined adaptively by the mode.
[0057] After completing inference, the speech recognition model outputs the speech recognition result, converting the speech to be recognized into text.
[0058] It should be emphasized that, to further ensure the privacy and security of the above-mentioned speech recognition results, the speech recognition results can also be stored in a blockchain node.
[0059] The blockchain referred to in this application is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.
[0060] In this embodiment, a speech to be recognized containing multiple speech frames is acquired. Human voice recognition is performed on each speech frame based on its speech features to determine whether it contains a human voice, thus obtaining a human voice speech frame. The speech features of each human voice speech frame are acquired and merged to obtain merged features, thereby processing only the speech features containing human voices and avoiding the processing of useless data. The merged features are input into a speech recognition model, which is built based on Wenet and reconstructed using Trtorch, thereby accelerating inference through the Trtorch framework. The speech recognition model performs batch inference on the merged features, further improving computational efficiency and significantly increasing the speed of speech recognition.
[0061] Furthermore, step S203 may include: for each speech frame, acquiring the speech features of the speech frame, wherein the speech features are the Mel frequency cepstral coefficients of the speech frame; inputting the speech features into the speech endpoint detection model to obtain the speech endpoint detection result; when it is determined from the speech endpoint detection result that the speech frame contains a speech endpoint, the speech frame is identified as a human voice speech frame.
[0062] Specifically, for each speech frame, the Mel-scale frequency cepstral coefficients (MFCCs) of the speech frame can be extracted, and the set of Mel-scale frequency cepstral coefficients can be used as the speech features of the speech frame.
[0063] Mel-frequency cepstral coefficients are input into the speech endpoint detection model for speech endpoint detection (VAD), yielding the VAD results. The VAD results indicate whether a speech frame contains a speech endpoint. When a person speaks, significant fluctuations in the audio signal occur within the speech frame. If the VAD results show that a speech frame contains a speech endpoint, then the speech frame is considered to contain a human voice and is marked as a human speech frame.
[0064] In this embodiment, Mel frequency cepstral coefficients are used as speech features. Speech endpoint detection is performed on speech frames based on speech features. The presence or absence of speech endpoints determines whether a speech frame is a human voice speech frame, thereby filtering out speech frames containing human voices.
[0065] Furthermore, the steps described above for inputting speech features into the speech endpoint detection model to obtain speech endpoint detection results may include: inputting speech features into the speech endpoint detection model built based on Kaldi to obtain a speech endpoint information form; processing the speech endpoint information form through select-voiced-frames to obtain an endpoint detection form; and determining the endpoint detection form as the speech endpoint detection result.
[0066] Specifically, the speech endpoint detection model can be built based on Kaldi, an Automatic Speech Recognition (ASR) tool that provides training tools for various ASR models and offers multiple neural network models. When performing speech endpoint detection, the Kaldi-based speech endpoint detection model first generates a speech endpoint information form `vad.scp`, then calls a specific bin: `select-voiced-frames` to process the `vad.scp` form, generating the endpoint detection form `vad.ark`. In application, the endpoint detection form `vad.ark` exists as a binary file. That is, the endpoint detection form `vad.ark` is a binary form of 0-1; when its value is 0, it indicates that there is no human voice; when its value is 1, it indicates that there is human voice. The endpoint detection form will serve as the speech endpoint detection result.
[0067] In this embodiment, the voice endpoint detection model can be built based on Kaldi, and a voice endpoint detection form can be generated through select-voiced-frames to obtain the voice endpoint detection results.
[0068] Furthermore, in another embodiment of this application, step S203 may include: for each speech frame, starting a speech processing thread corresponding to each speech frame; and performing human voice recognition on each speech frame based on each speech processing thread to obtain a human voice speech frame.
[0069] Specifically, for each speech frame, a dedicated speech processing thread can be started. Each speech processing thread generates the Mel-frequency cepstral coefficients for the corresponding speech frame and performs speech endpoint detection to achieve human voice recognition, thus realizing parallel processing. When the speech frame is a human voice frame, the corresponding speech processing thread sends the speech frame to aggregate and merge the speech features of each human voice frame. This enables multi-threaded acquisition of speech endpoint detection results and transmission of speech frames, improving data processing efficiency.
[0070] In this embodiment, each voice frame corresponds to a separate voice processing thread, thereby improving the data processing efficiency of voice frames.
[0071] Furthermore, the steps described above for performing batch inference on the merged features using a speech recognition model to obtain the speech recognition result of the speech to be recognized may include: performing batch inference on the merged features using the Encoder-CTC model in the speech recognition model to obtain a first encoding result and a first CTC decoding result; decoding the first CTC decoding result according to a preset decoding method to obtain a second CTC decoding result; merging the first encoding result and the second CTC decoding result to obtain a second encoding result; and performing batch inference on the second encoding result using a decoder to obtain the speech recognition result of the speech to be recognized.
[0072] Specifically, the Wenet model consists of three parts: an encoder, a CTC decoder, and a decoder. The encoder is a shared encoder, which contains multiple transformer or conformer layers. The CTC decoder includes a fully connected layer and a softmax layer. The decoder is an attention decoder, which contains multiple transformer layers.
[0073] In this application, the encoder and CTC decoder are combined into an Encoder-CTC model. After merging features into the speech recognition model, the Encoder-CTC model first performs batch inference, where the encoder outputs the first encoded result and the CTC decoder outputs the first CTC decoded result.
[0074] Then, the first CTC decoding result is decoded according to a preset decoding method to obtain the second CTC decoding result. The first encoding result is updated based on the second CTC decoding result, specifically by merging the first encoding result with the second CTC decoding result to obtain the second encoding result. The second encoding result is then input into the decoder for batch inference to obtain the speech recognition result in text form.
[0075] The CTC decoder outputs the decoding result in a streaming manner. During the second decoding pass, the Attention decoder utilizes the global context's Attention information to obtain better recognition results. There are two methods for the second decoding pass:
[0076] Method 1: Attention decoder mode. Ignoring the decoding result of CTC, the Attention decoder directly decodes based on the output of the shared Encoder in an autoregressive manner, outputting the final recognition result;
[0077] Method 2: Rescoring Model. Based on the n-best hypothesis of CTC decoding, the Attention decoder uses teacher-forcing to re-score, using the best-rescored hypothesis as the final recognition result. This method avoids the autoregressive process and achieves a good real-time performance (RTF).
[0078] During batch inference, the dynamic block size can be uniformly distributed from 1 to the maximum utterance length, meaning the attention ranges from left-context attention to full-context attention. The model captures different information for various block sizes and learns how to make accurate predictions when provided with different limited right contexts. Blocks of size from 1 to X (where X can be 25) are called streaming blocks in the streaming model, and the maximum utterance length is called the non-streaming block in the non-streaming model. Furthermore, the preset decoding methods include shallow decoding, wfst decoding, or prefix_beam_search decoding. Any of these methods can be selected when decoding the first decoding result of CTC.
[0079] In this embodiment, multiple decoding methods can be used to decode the first CTC decoding result, enriching the diversity of decoding methods.
[0080] Furthermore, the speech recognition model can operate in three modes: FP32, FP16, and INT8. Among them, the FP32 mode, also known as Tensorrt FP32 mode, uses a merging technique and a CBR / CBM / CBL network structure. In the horizontal merging direction, parts with the same input and the same operation are merged into one operator op; in the vertical merging direction, the concat operation can be directly ignored, and the input or output can be directly processed.
[0081] FP16 mode, or Tensorrt FP16 mode, uses float16 data precision instead of float32. The actual performance improvement comes from half2mode, also known as interleaved storage. The advantage of interleaved storage is that it allows for parallel access of multiple modules by transferring consecutive words in blocks, thus increasing memory bandwidth compared to sequential storage.
[0082] In normal FP32 computation, the computation process of a convolution is as follows: the input bottom_blob(fp32) and the weights bottom_blob(fp32) are convolved to obtain inner_blob(fp32); then, inner_blob(fp32) is added to the bias_blob(fp32) to obtain top_blob(fp32).
[0083] During computation in INT8 mode in TensorRTr conv, the input bottom_blob(fp32) is quantized online to obtain bottom_blob(int8), and the weights bottom_blob(fp32) are also quantized to obtain weights bottom_blob(int8). A convolution operation is then performed on bottom_blob(int8) and weights bottom_blob(int8) to obtain inner_blob(int32), which is then dequantized online to fp32 to obtain inner_blob(fp32). Finally, it is added to the bias bias_blob(fp32) to obtain top_blob(fp32).
[0084] The quantization formulas for the input and weights can be expressed as follows:
[0085] bottom_blob(int8)=bottom_blob_int8_scale*bottom_blob(fp32) (1)
[0086] weight_blob(int8)=weight_data_int8_scale*weight_blob(fp32) (2)
[0087] Where bottom_blob_int8_scale represents the quantization scale of the input; weight_data_int8_scale represents the quantization scale of the weights.
[0088] The purpose of dequantization is to map int8 back to the original fp32, ensuring the range remains consistent, since bottom_blob(int8) and bottom_blob(int8) are multiplied:
[0089]
[0090] To achieve the inverse mapping, the dequantize_scale should be:
[0091] dequantize_scale=1 / (bottom_blob_int8_scale*weight_data_int8_scale)(4)
[0092] Then we have:
[0093] inner_blob(fp32)=dequantize_scale*inner_blob(int32) (5)
[0094] In the tests, the inference speed, from highest to lowest, was: INT8, FP16, and FP32. However, INT8 has relatively poor accuracy, so in applications, FP16 mode is usually used.
[0095] In this application, a speech recognition model based on Wenet is reconstructed using TrTorch. Compared to PyTorch, the TrTorch framework lacks operators such as `masked_fill`, `layer_norm`, `select`, `conv1d`, and `glu`. Therefore, these operators need to be rewritten within the TrTorch framework to ensure the Wenet-based speech recognition model functions correctly. Specifically, the `masked_fill` operator replaces the indices of elements with a value of 1 in the mask with the corresponding values in the tensor, while leaving the indices of elements with a value of 0 unchanged. The `layer_norm` operator normalizes the channel directions to calculate the mean of the CHW (CxHxW dimension). The `select` operator slices the tensor according to the selected dimension in the index. The `conv1d` operator is a convolutional layer that uses a one-dimensional convolutional kernel of a specified size to perform convolution operations on the multi-channel one-dimensional input signal, and can be represented as:
[0096]
[0097] Among them, C in Indicates the number of input channels, C out This represents the number of output channels; N is the batch size; k is the kernel size; and bias represents the bias.
[0098] The Gated Linear Unit (GLU) is a gating mechanism in convolutional neural networks. Unlike the Gated Recurrent Unit (GRU) in recurrent neural networks, it facilitates gradient propagation and is less prone to gradient vanishing or exploding. The computation formula for each layer in a GLU can be expressed as follows:
[0099]
[0100] Where X represents the input; W, V, b, and c are the parameters to be learned; δ() represents the sigmoid function; This represents element-wise product, also known as the Hadamard product.
[0101] In this embodiment, the merged features are batch-inferred using the Encoder-CTC model in the speech recognition model, and the second encoding result of the Encoder-CTC model is batch-inferred using the decoder, thereby improving the speed of speech recognition.
[0102] Furthermore, the above-mentioned speech recognition model processing method may also include: recording the recognition inference time of the speech recognition model for the merged features; when the recognition inference time is greater than a preset time threshold, setting the merged features as negative samples; and supplementing the speech recognition model with training based on the negative samples.
[0103] Specifically, when the speech recognition model infers from merged features, the inference time of the speech recognition model can be recorded. Then, a preset time threshold is obtained. If the inference time is greater than the preset time threshold, it indicates that the speech recognition model is inferring from the merged feature too slowly. In this case, the merged feature and its corresponding speech to be recognized can be used as negative samples. Then, the speech recognition model is further trained based on these negative samples to optimize the speech recognition model.
[0104] In this embodiment, the inference time of the speech recognition model is recorded. When the inference time is greater than the preset time threshold, it indicates that the speech recognition model is slow in reasoning about merged features. The speech recognition model can be supplemented with training based on the merged features to optimize the speech recognition model.
[0105] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.
[0106] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.
[0107] This application can be applied to fields such as smart communities, smart healthcare, and smart transportation, thereby promoting the construction of smart cities.
[0108] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware with computer-readable instructions. These computer-readable instructions can be stored in a computer-readable storage medium. When executed, the program can include the processes of the embodiments of the above methods. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, optical disk, or read-only memory (ROM), or random access memory (RAM).
[0109] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.
[0110] Further reference Figure 3 As a response to the above Figure 2 The implementation of the method shown in this application provides an embodiment of a speech recognition model processing device, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.
[0111] like Figure 3 As shown, the speech recognition model processing device 300 described in this embodiment includes: a speech acquisition module 301, a feature extraction module 302, a human voice recognition module 303, a feature merging module 304, and a speech recognition module 305, wherein:
[0112] The speech acquisition module 301 is used to acquire the speech to be recognized, which contains multiple speech frames;
[0113] The feature extraction module 302 is used to extract features from each of the speech frames to obtain the speech features of each speech frame.
[0114] The voice recognition module 303 is used to perform voice recognition on each speech frame based on the speech features of each speech frame to obtain a voice speech frame.
[0115] The feature merging module 304 is used to acquire the speech features of each human voice speech frame and merge the speech features to obtain merged features.
[0116] The speech recognition module 305 is used to input the merged features into a speech recognition model reconstructed by trtorch, so as to perform batch inference on the merged features through the speech recognition model to obtain the speech recognition result of the speech to be recognized, wherein the speech recognition model is built based on wenet.
[0117] In this embodiment, a speech to be recognized containing multiple speech frames is acquired. Human voice recognition is performed on each speech frame based on its speech features to determine whether it contains a human voice, thus obtaining a human voice speech frame. The speech features of each human voice speech frame are acquired and merged to obtain merged features, thereby processing only the speech features containing human voices and avoiding the processing of useless data. The merged features are input into a speech recognition model, which is built based on Wenet and reconstructed using Trtorch, thereby accelerating inference through the Trtorch framework. The speech recognition model performs batch inference on the merged features, further improving computational efficiency and significantly increasing the speed of speech recognition.
[0118] In some optional implementations of this embodiment, the voice recognition module 303 may include: a feature acquisition submodule, a feature input submodule, and a voice determination submodule, wherein:
[0119] The feature acquisition submodule is used to acquire the speech features of each speech frame, wherein the speech features are the Mel frequency cepstral coefficients of the speech frame.
[0120] The feature input submodule is used to input speech features into the speech endpoint detection model to obtain speech endpoint detection results.
[0121] The voice determination submodule is used to determine a voice frame as a human voice frame when the voice frame contains a voice endpoint based on the voice endpoint detection results.
[0122] In this embodiment, Mel frequency cepstral coefficients are used as speech features. Speech endpoint detection is performed on speech frames based on speech features. The presence or absence of speech endpoints determines whether a speech frame is a human voice speech frame, thereby filtering out speech frames containing human voices.
[0123] In some optional implementations of this embodiment, the feature input submodule may include: a feature input unit, a form processing unit, and a result determination unit, wherein:
[0124] The feature input unit is used to input speech features into the speech endpoint detection model built based on Kaldi to obtain a speech endpoint information form.
[0125] The form processing unit is used to process the voice endpoint information form through select-voiced frames to obtain the endpoint detection form.
[0126] The result determination unit is used to determine the endpoint detection form as the voice endpoint detection result.
[0127] In this embodiment, the voice endpoint detection model can be built based on Kaldi, and a voice endpoint detection form can be generated through select-voiced-frames to obtain the voice endpoint detection results.
[0128] In some alternative implementations of this embodiment, the voice recognition module 303 may include: a thread initiation submodule and a voice recognition submodule, wherein:
[0129] The thread-starting submodule is used to start a corresponding speech processing thread for each speech frame.
[0130] The voice recognition submodule is used to perform voice recognition on each speech frame based on each speech processing thread to obtain the voice speech frame.
[0131] In this embodiment, each voice frame corresponds to a separate voice processing thread, thereby improving the data processing efficiency of voice frames.
[0132] In some optional implementations of this embodiment, the speech recognition module 305 may include: a merging and inference submodule, a first decoding submodule, a result merging submodule, and a decoding and inference submodule, wherein:
[0133] The merge inference submodule is used to perform batch inference on the merged features through the Encoder-CTC model in the speech recognition model to obtain the first encoding result and the first CTC decoding result.
[0134] The first decoding submodule is used to decode the first decoding result of CTC according to a preset decoding method to obtain the second decoding result of CTC.
[0135] The result merging submodule is used to merge the first encoding result and the CTC second decoding result to obtain the second encoding result.
[0136] The decoding and inference submodule is used to perform batch inference on the second encoding result through the decoder to obtain the speech recognition result of the speech to be recognized.
[0137] In this embodiment, the merged features are batch-inferred using the Encoder-CTC model in the speech recognition model, and the second encoding result of the Encoder-CTC model is batch-inferred using the decoder, thereby improving the speed of speech recognition.
[0138] In some optional implementations of this embodiment, the preset decoding methods include shallow decoding, wfst decoding, or prefix_beam_search decoding.
[0139] In this embodiment, multiple decoding methods can be used to decode the first CTC decoding result, enriching the diversity of decoding methods.
[0140] In some optional implementations of this embodiment, the speech recognition model processing device may further include: a time recording module, a feature setting module, and a supplementary training module, wherein:
[0141] The time recording module is used to record the inference time of the speech recognition model for recognizing merged features.
[0142] The feature setting module is used to set the merged features as negative samples when the recognition and inference time exceeds a preset time threshold.
[0143] The supplementary training module is used to supplement the training of the speech recognition model based on negative samples.
[0144] In this embodiment, the inference time of the speech recognition model is recorded. When the inference time is greater than the preset time threshold, it indicates that the speech recognition model is slow in reasoning about merged features. The speech recognition model can be supplemented with training based on the merged features to optimize the speech recognition model.
[0145] To address the aforementioned technical problems, embodiments of this application also provide a computer device. Please refer to [link / reference needed]. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.
[0146] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected via a system bus. It should be noted that only the computer device 4 with components 41-43 is shown in the figure; however, it should be understood that it is not required to implement all the shown components, and more or fewer components can be implemented alternatively. Those skilled in the art will understand that the computer device described here is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.
[0147] The computer device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device can interact with the user via a keyboard, mouse, remote control, touchpad, or voice control.
[0148] The memory 41 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 41 may be an internal storage unit of the computer device 4, such as the hard disk or memory of the computer device 4. In other embodiments, the memory 41 may also be an external storage device of the computer device 4, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 4. Of course, the memory 41 may also include both the internal storage unit and its external storage device of the computer device 4. In this embodiment, the memory 41 is typically used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions for speech recognition model processing methods. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or will be output.
[0149] In some embodiments, the processor 42 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. The processor 42 is typically used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to execute computer-readable instructions stored in the memory 41 or to process data, for example, to execute computer-readable instructions for the speech recognition model processing method.
[0150] The network interface 43 may include a wireless network interface or a wired network interface, which is typically used to establish communication connections between the computer device 4 and other electronic devices.
[0151] The computer device provided in this embodiment can execute the above-described speech recognition model processing method. The speech recognition model processing method here can be any of the speech recognition model processing methods described in the various embodiments above.
[0152] In this embodiment, a speech to be recognized containing multiple speech frames is acquired. Human voice recognition is performed on each speech frame based on its speech features to determine whether it contains a human voice, thus obtaining a human voice speech frame. The speech features of each human voice speech frame are acquired and merged to obtain merged features, thereby processing only the speech features containing human voices and avoiding the processing of useless data. The merged features are input into a speech recognition model, which is built based on Wenet and reconstructed using Trtorch, thereby accelerating inference through the Trtorch framework. The speech recognition model performs batch inference on the merged features, further improving computational efficiency and significantly increasing the speed of speech recognition.
[0153] This application also provides another embodiment, namely, providing a computer-readable storage medium storing computer-readable instructions that can be executed by at least one processor to cause the at least one processor to perform the steps of the speech recognition model processing method described above.
[0154] In this embodiment, a speech to be recognized containing multiple speech frames is acquired. Human voice recognition is performed on each speech frame based on its speech features to determine whether it contains a human voice, thus obtaining a human voice speech frame. The speech features of each human voice speech frame are acquired and merged to obtain merged features, thereby processing only the speech features containing human voices and avoiding the processing of useless data. The merged features are input into a speech recognition model, which is built based on Wenet and reconstructed using Trtorch, thereby accelerating inference through the Trtorch framework. The speech recognition model performs batch inference on the merged features, further improving computational efficiency and significantly increasing the speed of speech recognition.
[0155] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.
[0156] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, the purpose of providing these embodiments is to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application.
Claims
1. A speech recognition model processing method, characterized in that, Includes the following steps: Obtain the speech to be recognized, which contains multiple speech frames; Feature extraction is performed on each of the aforementioned speech frames to obtain the speech features of each speech frame; Human voice recognition is performed on each of the speech frames based on their speech features to obtain human voice speech frames. The speech features of each human voice speech frame are obtained, and the speech features are merged to obtain the merged features; The merged features are input into a speech recognition model reconstructed by trtorch, and batch inference is performed on the merged features through the speech recognition model to obtain the speech recognition result of the speech to be recognized. The speech recognition model is built based on wenet. The step of performing batch inference on the merged features using the speech recognition model to obtain the speech recognition result of the speech to be recognized includes: The merged features are batch-inferred using the Encoder-CTC model in the speech recognition model to obtain the first encoding result and the first CTC decoding result. The first decoding result of CTC is decoded according to a preset decoding method to obtain the second decoding result of CTC; The first encoding result and the second decoding result of CTC are combined to obtain the second encoding result; The speech recognition result of the speech to be recognized is obtained by performing batch inference on the second encoding result by the decoder.
2. The speech recognition model processing method according to claim 1, characterized in that, The step of performing human voice recognition on each of the speech frames based on the speech features of each speech frame to obtain human voice speech frames includes: For each speech frame, the speech features of the speech frame are obtained, wherein the speech features are the Mel frequency cepstral coefficients of the speech frame; The speech features are input into the speech endpoint detection model to obtain the speech endpoint detection results; When it is determined from the voice endpoint detection results that the voice frame contains a voice endpoint, the voice frame is identified as a human voice voice frame.
3. The speech recognition model processing method according to claim 2, characterized in that, The step of inputting the speech features into the speech endpoint detection model to obtain the speech endpoint detection result includes: The speech features are input into a speech endpoint detection model built on Kaldi to obtain a speech endpoint information form; The voice endpoint information form is processed by select-voiced-frames to obtain the endpoint detection form; The endpoint detection form is identified as the voice endpoint detection result.
4. The speech recognition model processing method according to claim 1, characterized in that, The step of performing human voice recognition on each of the speech frames based on the speech features of each speech frame to obtain human voice speech frames includes: For each speech frame, start the speech processing thread corresponding to each speech frame; Each speech processing thread performs human voice recognition on each of the aforementioned speech frames to obtain human voice speech frames.
5. The speech recognition model processing method according to claim 1, characterized in that, The preset decoding methods include shallow decoding, wfst decoding, or prefix_beam_search decoding.
6. The speech recognition model processing method according to claim 1, characterized in that, The method further includes: Record the inference time of the speech recognition model for the merged features; When the recognition and reasoning time exceeds a preset time threshold, the merged feature is set as a negative sample. The speech recognition model is further trained based on the negative samples.
7. A speech recognition model processing device, characterized in that, include: The speech acquisition module is used to acquire the speech to be recognized, which contains multiple speech frames; The feature extraction module is used to extract features from each of the speech frames to obtain the speech features of each speech frame. A voice recognition module is used to perform voice recognition on each of the speech frames based on the speech features of each speech frame, so as to obtain a voice speech frame; The feature merging module is used to obtain the speech features of each human voice speech frame and merge the speech features to obtain the merged features. A speech recognition module is used to input the merged features into a speech recognition model reconstructed by trtorch, so as to perform batch inference on the merged features through the speech recognition model to obtain the speech recognition result of the speech to be recognized, wherein the speech recognition model is built based on wenet; The speech recognition module specifically includes: The merging inference submodule is used to perform batch inference on the merged features through the Encoder-CTC model in the speech recognition model to obtain a first encoding result and a first CTC decoding result. The first decoding submodule is used to decode the first decoding result of CTC according to a preset decoding method to obtain the second decoding result of CTC; The result merging submodule is used to merge the first encoding result and the CTC second decoding result to obtain the second encoding result; The decoding and inference submodule is used to perform batch inference on the second encoding result through the decoder to obtain the speech recognition result of the speech to be recognized.
8. A computer device, characterized in that, The method includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of the speech recognition model processing method as described in any one of claims 1 to 6.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the speech recognition model processing method as described in any one of claims 1 to 6.