A method, apparatus, device, and storage medium for speech recognition.

Abstract

This application provides a speech recognition method, apparatus, device, and storage medium. It constructs a speech recognition engine for batch processing speech recognition tasks; concatenates speech data corresponding to the speech recognition task to obtain batch features and a batch cache; transposes the batch cache using the speech recognition engine to obtain a transposed cache whose dimensions match the dimensions corresponding to the batch features; obtains batch recognition results based on the transposed cache and batch features; and splits the batch recognition results to obtain the speech recognition results corresponding to the speech recognition task. This application, by transposing dimensions, ensures that the dimensions of the batch features and the batch cache match, avoiding frequent splitting and re-concatenation operations, thereby reducing processing time and improving the speed of speech recognition. Furthermore, by reducing frequent splitting and concatenation operations, it reduces errors that may be introduced during data processing, thereby improving the accuracy and reliability of speech recognition.

Description

Technical Field

[0001] This application relates to the field of speech recognition technology, and in particular to a speech recognition method and apparatus, an electronic device, and a storage medium. Background Technology

[0002] In related technologies, during real-time speech recognition, to fully utilize computing power and improve efficiency, it is typically necessary to combine scattered speech data into batch data for batch recognition. This batch data includes batch features and batch cache. Because the dimensions of the batch features and batch cache do not match, the batch data needs to be continuously split and reassembled to achieve dimensionality matching. However, continuously splitting and reassembling the batch data directly impacts the speed of real-time speech recognition, thus reducing its efficiency. Summary of the Invention

[0003] This application provides a speech recognition method to solve the problem of low real-time speech recognition efficiency caused by the continuous splitting and splicing of batch data.

[0004] Accordingly, embodiments of this application also provide a voice recognition device, an electronic device, and a storage medium to ensure the implementation and application of the above methods.

[0005] To address the aforementioned problems, this application discloses a speech recognition method, the method comprising:

[0006] Build a speech recognition engine for batch processing of speech recognition tasks; the speech recognition tasks have corresponding speech data;

[0007] Speech data corresponding to a speech recognition task is concatenated to obtain batch features and batch cache; the batch features and the batch cache each have corresponding dimensions.

[0008] After transposing the batch cache using the speech recognition engine, a transposed cache whose dimensions match the dimensions corresponding to the batch features is obtained;

[0009] Based on the transposed cache and the batch features, the batch recognition results are obtained;

[0010] The batch recognition results are split to obtain the speech recognition results corresponding to the speech recognition task.

[0011] Optionally, the speech recognition engine includes an encoder, a decoder, and a connector, and obtaining batch recognition results based on the transposed buffer and the batch features includes:

[0012] The encoder is used to encode the transposed buffer and the batch features to obtain the encoded buffer and the encoded features;

[0013] The decoder is used to generate language feature vectors;

[0014] The connector is used to process the encoded features and the language feature vector to obtain the connection result corresponding to the speech recognition task;

[0015] The encoding cache and the connection result are used as the batch recognition result.

[0016] Optionally, the speech recognition task supports multiple executions, and the step of encoding the transposed buffer and the batch features using the encoder to obtain the encoded buffer and encoded features includes:

[0017] The transposed buffer is input into the encoder to obtain the encoding buffer of the speech recognition task in the current execution process;

[0018] The batch features are input into the encoder to obtain the encoded features;

[0019] The encoding cache of the speech recognition task in the current execution process is used as the input of the encoder of the speech recognition task in the next execution process to obtain the encoding cache of the speech recognition task in the next execution process.

[0020] Optionally, the encoded features have corresponding time frames, and the process of using the connector to process the encoded features and the language feature vector to obtain the connection result includes:

[0021] The encoding features and language feature vectors corresponding to the current time frame are input into the connector to obtain the character probability data corresponding to the current time frame; the character probability data includes the probabilities corresponding to multiple characters respectively.

[0022] Decode the character probability data to obtain the predicted character corresponding to the current time frame;

[0023] The predicted characters are connected according to the time frames corresponding to the predicted characters to obtain the connection result.

[0024] Optionally, generating language feature vectors using the decoder includes:

[0025] The decoder is used to generate the language feature vector corresponding to the current time frame;

[0026] The predicted character corresponding to the current time frame is input into the decoder to generate the language feature vector corresponding to the next time frame.

[0027] Optionally, before obtaining batch features and batch caching from the speech data corresponding to the concatenated speech recognition task, the method further includes:

[0028] Create speech recognition tasks corresponding to real-time audio streams and place the speech recognition tasks in a task pool for management;

[0029] Extract speech features from the real-time audio stream;

[0030] Create a speech cache for storing the encoded features, the language feature vectors, and the character probability data;

[0031] The speech features and the speech cache are determined as the speech data corresponding to the speech recognition task.

[0032] Optionally, the speech recognition result includes a target speech cache and a target connection result. The speech recognition task has a result queue for storing the target connection result. After splitting the batch recognition results to obtain the speech recognition result corresponding to the speech recognition task, the method further includes:

[0033] Update the speech cache corresponding to the speech recognition task using the target speech cache corresponding to the speech recognition task;

[0034] The target connection result corresponding to the speech recognition task is stored in the result queue corresponding to the speech recognition task.

[0035] This application also discloses a speech recognition device, the device comprising:

[0036] A speech recognition engine module is used to build a speech recognition engine for batch processing of speech recognition tasks; the speech recognition tasks have corresponding speech data.

[0037] The speech data module is used to concatenate speech data corresponding to the speech recognition task to obtain batch features and batch cache; the batch features and the batch cache each have corresponding dimensions.

[0038] The dimension transpose module is used to transpose the batch cache using the speech recognition engine to obtain a transpose cache whose dimensions match the dimensions corresponding to the batch features.

[0039] A batch recognition result module is used to obtain batch recognition results based on the transposed cache and the batch features;

[0040] The speech recognition result module is used to split the batch recognition results to obtain the speech recognition result corresponding to the speech recognition task.

[0041] This application also discloses an electronic device, including: a processor; and a memory storing executable code thereon, which, when executed, causes the processor to perform any of the speech recognition methods described in the embodiments of this application.

[0042] This application also discloses one or more machine-readable media storing executable code thereon, which, when executed, causes a processor to perform a speech recognition method as described in any of the embodiments of this application.

[0043] Compared with the prior art, the embodiments of this application have the following advantages:

[0044] In this embodiment, a speech recognition engine is constructed for batch processing of speech recognition tasks. The speech recognition task has corresponding speech data. The speech data corresponding to the speech recognition task is concatenated to obtain batch features and a batch cache. The batch features and the batch cache each have corresponding dimensions. The speech recognition engine transposes the dimensions of the batch cache to obtain a transposed cache whose dimensions match the dimensions corresponding to the batch features. Based on the transposed cache and the batch features, batch recognition results are obtained. The batch recognition results are split to obtain the speech recognition results corresponding to the speech recognition task. This embodiment, by transposing dimensions, ensures that the dimensions of the batch features and the batch cache match, avoiding frequent splitting and re-concatenation operations, thereby reducing processing time and improving the speed of speech recognition. Furthermore, by reducing frequent splitting and concatenation operations, the errors that may be introduced during data processing are reduced, thereby improving the accuracy and reliability of speech recognition. Attached Figure Description

[0045] Figure 1 This is a flowchart illustrating the steps of an embodiment of a speech recognition method according to this application;

[0046] Figure 2 This is a flowchart illustrating one approach to establishing a speech recognition task according to this application;

[0047] Figure 3 This is a flowchart illustrating the process of neural network model recognition based on the Transducer architecture;

[0048] Figure 4 This is a schematic diagram of a neural network model recognition process based on the Transducer architecture according to this application;

[0049] Figure 5 This is a flowchart illustrating a speech recognition task implemented based on a speech recognition engine according to this application;

[0050] Figure 6This is a flowchart illustrating a speech recognition system according to this application;

[0051] Figure 7 This is a structural block diagram of an embodiment of a speech recognition device according to this application;

[0052] Figure 8 This is a schematic diagram of the structure of a device provided in an embodiment of this application. Detailed Implementation

[0053] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0054] Reference Figure 1 This is a flowchart illustrating the steps of an embodiment of a speech recognition method according to this application, including the following steps:

[0055] Step 101: Construct a speech recognition engine for batch processing of speech recognition tasks; the speech recognition tasks have corresponding speech data;

[0056] In this embodiment, a speech recognition engine based on the ONNX (Open Neural Network Exchange) framework and a Transducer architecture is constructed to process multiple speech recognition tasks in batches. The speech recognition engine treats a speech recognition task as the smallest unit of operation.

[0057] The speech recognition task in this application refers to the process of converting a real-time audio stream into corresponding text. Each speech recognition task has corresponding speech data, specifically, speech data refers to speech features and speech buffers. Speech features refer to the features obtained after feature extraction from the real-time audio stream; speech buffers refer to the data structure used to store intermediate results generated during the speech recognition process. These intermediate results are usually generated when processing speech features and are used in subsequent recognition steps.

[0058] Reference Figure 2 This is a flowchart illustrating one approach to establishing a speech recognition task according to this application. When a speech recognition task is established, a feature queue, a result queue, and a speech buffer corresponding to the task are also created simultaneously. The feature queue is used to store speech features.

[0059] The speech recognition engine in this application uses a neural network model based on the Transducer architecture. Transducer is a model widely used in sequence-to-sequence tasks such as machine translation, speech recognition, and question answering systems. It consists of three parts: Encoder, Decoder, and Joiner.

[0060] Reference Figure 3 This is a flowchart illustrating the process of neural network model recognition based on the Transducer architecture, as described below:

[0061] The encoder is primarily responsible for processing the input sequence and transforming it into a continuous, high-dimensional representation. This representation captures the features and contextual information of the input sequence. In speech recognition, the encoder can be analogous to a traditional acoustic model, converting the input audio feature sequence into a high-dimensional representation to provide a foundation for subsequent processing.

[0062] The decoder is primarily responsible for generating vectors that characterize the features of the current output sequence based on previously predicted output characters. It can capture the statistical properties and linguistic structure of the output sequence, helping the entire model generate coherent and grammatically correct output. In speech recognition, the decoder can predict the next word based on the already recognized words, thus forming a complete sentence or paragraph.

[0063] The Joiner is primarily responsible for combining the outputs of the Encoder and Decoder to produce the probability distribution of the final output. The Joiner aims to fuse the acoustic information of the input sequence with the linguistic information of the output sequence so that the model can determine which character should be output at each time frame.

[0064] Step 102: Concatenate the speech data corresponding to the speech recognition task to obtain batch features and batch cache; the batch features and the batch cache each have corresponding dimensions;

[0065] Speech data from multiple speech recognition tasks are concatenated to obtain batch features and batch cache. Specifically, individual speech features and speech caches obtained from different speech recognition tasks are then concatenated to construct batch features and batch cache.

[0066] Batch features and batch caches each have corresponding dimensions, which will be matched and processed in subsequent steps. By concatenating speech data to form batch features and batch caches, computing power can be fully utilized to improve recognition efficiency. The concatenation operation simplifies the data processing flow, reduces intermediate steps, and makes the entire speech recognition process more concise and efficient.

[0067] Step 103: After transposing the batch cache using the speech recognition engine, a transposed cache whose dimensions match the dimensions corresponding to the batch features is obtained;

[0068] like Figure 3 As shown, the dimensions of the batch features input to the Encoder are B×T×D. i The dimensions of the batch cache for the input encoder are T×B×D. c Where B is the batch size, T is the number of feature frames, and D is the number of feature channels.

[0069] For the batch features input to the Encoder, since the batch size B is in the first dimension, it is only necessary to place the speech features from different speech recognition tasks sequentially in the memory space; that is, the memory addresses of the speech features from different speech recognition tasks are contiguous. However, for the batch cache, since the batch size B is in the second dimension, the speech cache T×D from the speech recognition task must first be processed. c Decomposing along the first dimension yields T dimensions, each with a dimension of 1×D. c The cache is then used to construct a batch single-frame cache with dimensions B×D by concatenating the corresponding single-frame caches from multiple speech recognition tasks. c Finally, the batch of single-frame caches are concatenated frame by frame to obtain the final batch cache that can be recognized by the model, with dimensions T×B×D. c Accordingly, the new batch cache dimension obtained after model recognition is T×B×D. c To update the new batch cache to each session, the batch cache needs to be decomposed along the first and second dimensions to obtain a 1×D dimension. c The cache is then processed frame by frame, concatenating the cache corresponding to each session to obtain T×D. c Therefore, in real-time speech recognition, the speech recognition engine needs to continuously split and reassemble the speech cache. However, constantly splitting and reassembling batch caches directly reduces the processing efficiency of the speech recognition engine, thus affecting the recognition efficiency of the speech recognition task.

[0070] Batch cache splitting and piecing can be performed on either the CPU (Central Processing Unit) or the NPU (Neural Processing Unit). If batch cache splitting and piecing are performed on the CPU, the pieced batch cache must first be migrated from the CPU's memory to the NPU's video memory before the model inference. After the model inference is complete, the new inference cache also needs to be moved from the NPU's video memory to the CPU's memory for further splitting. This back-and-forth migration process is very time-consuming and consumes significant CPU resources.

[0071] Therefore, in this embodiment of the application, when creating a new speech recognition task, the speech cache is directly initialized on the NPU's video memory. Subsequent splitting and splicing operations can be performed directly on the NPU, which can greatly improve the engine's real-time processing speed and reduce the meaningless occupation of CPU resources.

[0072] However, even so, splitting and reassembling batch caches on the NPU is still performed through a naive single-threaded address copy, which is still very slow.

[0073] Therefore, in order to further accelerate the inference speed of the model in the speech recognition engine, this application embodiment modifies the original ONNX model by adding, deleting or modifying operators in the model to optimize the model's processing performance.

[0074] Specifically, in the ONNX model, a transpose operator is added before the batch buffer is input into the Encoder and after the Encoder outputs the batch buffer to transpose the first and second dimensions of the batch buffer, making the model's batch buffer input-output dimensions B×T×D. c This allows speech caches from multiple speech recognition tasks to be directly concatenated into a batch cache and input into the model in a manner similar to sequence features, without the need for redundant splitting steps. Correspondingly, the model's output only needs to be split along the first dimension to update each speech recognition task, eliminating the need for a subsequent concatenation process.

[0075] That is, in the speech recognition engine of this application embodiment, a transpose operator is added to perform a dimension transpose operation on the batch cache, so that the transposed cache dimension (B×T×D) is... c ) and the dimensions of batch features (B×T×D) i This ensures a match, thus avoiding frequent splitting and reassembly operations.

[0076] Reference Figure 4This is a flowchart illustrating a neural network model recognition process based on the Transducer architecture of this application. The input Encoder's cache is a transposed cache (transposed cache) with dimensions B×T×D. c The output transpose buffer dimension is also B×T×D. c .

[0077] Therefore, in this embodiment, when a transpose operator is added to the ONNX model, the ONNX model is converted into an OM (Optimized Model) so that the newly added transpose operator can call the corresponding acceleration algorithm on the NPU to accelerate the transpose operation, thereby improving the model inference speed and thus improving the recognition efficiency of the speech recognition task.

[0078] By transposing dimensions, batch features and batch caches can be directly matched, avoiding frequent splitting and reassembly operations, significantly reducing processing time and improving the speed of speech recognition.

[0079] Step 104: Obtain the batch recognition result based on the transposed cache and the batch features;

[0080] Based on the transposed cache and batch features, the speech recognition engine's model performs inference to obtain batch recognition results. Efficient batch inference is achieved by utilizing the transposed cache with dimension matching and batch features.

[0081] Step 105: Split the batch recognition results to obtain the speech recognition results corresponding to the speech recognition task.

[0082] The batch recognition results are broken down into individual task results. This step decomposes the batch recognition results into the results of individual tasks, which facilitates subsequent applications and processing.

[0083] This application's embodiments achieve dimension matching between batch features and batch cache by transposing dimensions, avoiding frequent splitting and re-merging operations, thereby reducing processing time and improving speech recognition speed. Furthermore, by reducing frequent splitting and merging operations, potential errors introduced during data processing are reduced, thus improving the accuracy and reliability of speech recognition.

[0084] In one embodiment of this application, the speech recognition engine includes an encoder, a decoder, and a connector. Obtaining batch recognition results based on the transposed buffer and the batch features includes:

[0085] The encoder is used to encode the transposed buffer and the batch features to obtain the encoded buffer and the encoded features;

[0086] The decoder is used to generate language feature vectors;

[0087] The connector is used to process the encoded features and the language feature vector to obtain the connection result corresponding to the speech recognition task;

[0088] The encoding cache and the connection result are used as the batch recognition result.

[0089] like Figure 4 As shown, the Encoder is the encoder in this embodiment, mainly responsible for buffering the transposed input (B×T×D). c ) and batch features (B×T×D) i Transform it into a high-dimensional representation, and output the encoded buffer (B×T×D). c ) and coding features (B×T×D o ).

[0090] The encoded buffer contains intermediate results from the encoder output, which will be used as input during the next model inference. The encoded features are high-dimensional representations of the encoder output, which will be used as input to the connector later.

[0091] like Figure 4 As shown, the Decoder is the decoder in this embodiment, which generates the current language feature vector (B×D) based on the previously predicted language feature vector. d ).

[0092] However, when processing the first time frame, since the decoder does not have previously predicted language feature vectors as input, a special initial state or initial vector is introduced as the input to the decoder in the first time frame.

[0093] like Figure 4 As shown, the Joiner is the connector in this embodiment of the application, which is mainly responsible for combining the encoding features and language feature vectors to generate the predicted character corresponding to each time frame.

[0094] After obtaining the predicted character corresponding to each time frame, the predicted characters are concatenated in the order of the time frames to obtain a complete concatenation result, which is the text corresponding to the real-time audio stream.

[0095] The encoded buffer and connection results are output together as batch recognition results. Specifically, the first and second dimensions of the encoded buffer need to be transposed again using the transpose operator, so that the buffer dimensions output to the speech recognition task are transformed back to T×B×D. c .

[0096] This application's embodiments achieve efficient speech recognition processing through the collaborative work of an encoder, decoder, and connector. The encoder transforms the input transposed buffer and batch features into a high-dimensional representation, the decoder generates language feature vectors, and the connector combines the two to generate the final connection result. The entire process is efficient and coherent.

[0097] In one embodiment of this application, the speech recognition task supports multiple executions, and the step of encoding the transposed buffer and the batch features using the encoder to obtain the encoded buffer and encoded features includes:

[0098] The transposed buffer is input into the encoder to obtain the encoding buffer of the speech recognition task in the current execution process;

[0099] The batch features are input into the encoder to obtain the encoded features;

[0100] The encoding cache of the speech recognition task in the current execution process is used as the input of the encoder of the speech recognition task in the next execution process to obtain the encoding cache of the speech recognition task in the next execution process.

[0101] In the embodiments of this application, such as Figure 4 As shown, the dimensions are B×T×D i The batch features and dimensions are B×T×D c The transposed buffer is input into the Encoder, which outputs two parts, one with dimensions B×T×D. o The encoding features and dimensions are B×T×D c A new transposed buffer is needed, and the first and second dimensions of the new transposed buffer also need to be transposed again using the transpose operator to obtain a new encoded buffer, so that the buffer dimension output to the speech recognition task is transformed back to T×B×D. c The encoded features will be used as input to the Joiner, and the new transposed buffer will be used as input during the next inference by the Encoder.

[0102] Specifically, the difference between the encoding buffer as input and output of the encoder lies in the execution process of the speech recognition task. The encoding buffer obtained in the next execution of the speech recognition task is obtained after inputting the encoding buffer obtained in the current execution process into the encoder. Figure 4The "previous transposed cache" refers to the encoded cache obtained in the previous execution process after dimensional transposition, and the "current transposed cache" refers to the encoded cache obtained in the current execution process after dimensional transposition. For example, suppose that during the first execution of the speech recognition task, the encoder encodes the transposed cache to generate encoded cache A. During the second execution of the speech recognition task, the encoder uses encoded cache A as input to generate a new encoded cache B.

[0103] In one embodiment of this application, the encoded features have corresponding time frames, and the process of using the connector to process the encoded features and the language feature vector to obtain a connection result includes:

[0104] The encoding features and language feature vectors corresponding to the current time frame are input into the connector to obtain the character probability data corresponding to the current time frame; the character probability data includes the probabilities corresponding to multiple characters respectively.

[0105] Decode the character probability data to obtain the predicted character corresponding to the current time frame;

[0106] The predicted characters are connected according to the time frames corresponding to the predicted characters to obtain the connection result.

[0107] In the speech recognition process, the joiner mainly fuses the acoustic information of the input sequence and the linguistic information of the output sequence so that the model can decide which character should be output at each time frame.

[0108] In the embodiments of this application, such as Figure 4 As shown, since the encoded features output by the Encoder contain T consecutive time frames, the Decoder and Joiner need to process the encoded features output by the Encoder frame by frame. For each time frame, the split encoded features (B×D) are... o ) and the language feature vector (B×D) output by the decoder d The Joiner outputs a dictionary-based probability distribution for subsequent Beam Search decoding. The character probability data output by the Joiner has a dimension of B×D. j , where D j This is the size of the dictionary.

[0109] Beam Search is a heuristic graph search algorithm used to find the most likely output sequence in sequence generation tasks. In the speech recognition task of this application embodiment, Beam Search is used to select the most likely predicted character from the character probability data output by the Joiner.

[0110] For example, for each time frame, Beam Search selects the character with the highest probability from the character probability data output by the Joiner as the predicted character for the current time frame.

[0111] Subsequently, the predicted characters corresponding to each time frame are concatenated in chronological order to form the final output sequence, which is the connection result in this embodiment.

[0112] In this embodiment, the character probability data corresponding to the current time frame is obtained by inputting the encoded features and the language feature vector corresponding to the current time frame into a connector. Then, the character with the highest probability is selected from the character probability data as the predicted character for the current time frame. Finally, the predicted characters corresponding to each time frame are concatenated in chronological order to form the final output sequence. Through these steps, an efficient speech recognition process is achieved.

[0113] In one embodiment of this application, generating language feature vectors using the decoder includes:

[0114] The decoder is used to generate the language feature vector corresponding to the current time frame;

[0115] The predicted character corresponding to the current time frame is input into the decoder to generate the language feature vector corresponding to the next time frame.

[0116] In the speech recognition process, the decoder can predict the next word based on the already recognized words, thereby predicting the current character. In the embodiments of this application, such as... Figure 4 As shown, the Decoder takes the previously predicted character ID as input, with a dimension of B×1, and its output is used as the input of the Joiner, with a dimension of B×D. d .

[0117] Specifically, Figure 4 The character Id t-1 (B) is the predicted character corresponding to the current time frame, and the character Id is the predicted character corresponding to the next time frame.

[0118] In one embodiment of this application, before obtaining batch features and batch caching from the speech data corresponding to the concatenated speech recognition task, the method further includes:

[0119] Create speech recognition tasks corresponding to real-time audio streams and place the speech recognition tasks in a task pool for management;

[0120] Extract speech features from the real-time audio stream;

[0121] Create a speech cache for storing the encoded features, the language feature vectors, and the character probability data;

[0122] The speech features and the speech cache are determined as the speech data corresponding to the speech recognition task.

[0123] First, a speech recognition task needs to be established, such as... Figure 2 As shown, a corresponding speech recognition task is created based on the real-time audio stream. Furthermore, when a speech recognition task is created, a feature queue, a result queue, and a speech buffer corresponding to that task are also created simultaneously. The feature queue is used to store speech features.

[0124] Once a speech recognition task is completed, it will be placed in the task pool. The task pool is used to manage all speech recognition tasks and ensure that each task can be processed correctly.

[0125] like Figure 2 As shown, when a real-time audio stream is received, speech features are extracted from it, and a feature queue, a result queue, and a speech buffer are created. The speech buffer stores the encoded features, language feature vectors, and character probability data; the feature queue stores the extracted speech features; the result queue stores the target connection results; and the target connection results refer to the text content corresponding to the real-time audio stream.

[0126] Specifically, speech features are extracted from real-time audio streams. Feature extraction algorithms (such as Mel-frequency cepstral coefficients (MFCC), filter bank features, etc.) are typically used to convert the audio signal into a high-dimensional feature vector. The extracted speech features are stored in a feature queue for subsequent model inference.

[0127] In one embodiment of this application, the speech recognition result includes a target speech cache and a target connection result. The speech recognition task has a result queue for storing the target connection result. After splitting the batch recognition result to obtain the speech recognition result corresponding to the speech recognition task, the method further includes:

[0128] Update the speech cache corresponding to the speech recognition task using the target speech cache corresponding to the speech recognition task;

[0129] The target connection result corresponding to the speech recognition task is stored in the result queue corresponding to the speech recognition task.

[0130] The speech recognition result in this embodiment includes a target speech cache and a target connection result. The target speech cache is an intermediate result generated during model inference and is used as input in the next model inference. The target connection result is the final recognized text sequence.

[0131] The speech recognition task has a results queue for storing the results of the target connection.

[0132] After splitting the batch recognition results, the speech recognition results corresponding to each speech recognition task are obtained.

[0133] The speech cache corresponding to the speech recognition task is updated using the target speech cache corresponding to the speech recognition task. The updated speech cache will be used for the next model inference.

[0134] The target connection results corresponding to the speech recognition task are stored in the result queue corresponding to the speech recognition task. The result queue is used to store the target connection results (text sequence corresponding to real-time audio stream) of each speech recognition task for subsequent processing and output.

[0135] In this embodiment, efficient speech recognition processing is achieved by updating the speech cache and storing the target connection results. The updated speech cache ensures that intermediate results generated during model inference can be correctly stored and used, and will be used for the next model inference, ensuring that the model can continue to process subsequent speech data efficiently, further improving the recognition speed.

[0136] Reference Figure 5 This diagram illustrates a process for implementing a speech recognition task based on a speech recognition engine, as described in this application. Specifically, when the number of speech features in the feature queue is sufficient, a corresponding speech recognition task is selected from the task pool and placed into the task queue. The speech recognition engine monitors the status of the task queue in real time and attempts to retrieve multiple speech recognition tasks from the queue at once. When the speech recognition engine retrieves a batch of speech recognition tasks from the task queue, it simultaneously retrieves the corresponding speech data (speech features and speech buffer) from the feature queue, forming batch data. Next, speech recognition is performed on the batch data (batch features and batch buffer) to obtain batch recognition results. Finally, the batch recognition results are decomposed and pushed to the speech recognition tasks.

[0137] In the embodiments of this application, the entire speech recognition process achieves efficient speech recognition task management, optimizes feature extraction, intermediate result storage and data preparation, and improves the overall efficiency and recognition accuracy of speech recognition.

[0138] The speech recognition method provided in this application is applicable to various application scenarios, such as smart homes, in-vehicle systems, and customer service robots. Therefore, in order to deploy speech recognition capabilities in various application scenarios, this application encapsulates a Web service based on the Websocket standard protocol for the speech recognition engine, forming a speech recognition system to enable remote access and interaction with the speech recognition engine.

[0139] Reference Figure 6 This is a flowchart illustrating a speech recognition system according to this application. The system consists of two parts: a Web service based on the WebSocket protocol and a speech recognition engine.

[0140] Specifically, the Web service provides a standard WebSocket interface for clients to call, accepts and parses client requests, and interacts with the inference engine; the speech recognition engine is used to infer the audio data in the client request, obtain the recognition result, and then return it to the service.

[0141] The interaction between the web service and the speech recognition engine mainly involves the following four operations:

[0142] 1) Create a speech recognition task: The web service calls the speech recognition engine's task creation interface. This operation will create a new speech recognition task in the speech recognition engine's task pool. The speech recognition task includes a feature queue, a result queue, and a speech cache.

[0143] 2) Real-time audio stream writing: The web service calls the real-time audio stream writing interface of the speech recognition engine to write the audio data parsed from the client request into the speech recognition engine.

[0144] 3) Real-time acquisition of recognition results: The Web service monitors the result queue of the speech recognition task in real time. When the queue is not empty, the target connection result is retrieved and returned to the client.

[0145] 4) End Session: The Web service calls the speech recognition engine's end task interface to release the resources occupied by the current task, including the feature queue, result queue, and speech cache.

[0146] The speech recognition engine built in this embodiment can be adapted to the Atlas 300IPro inference card. The Atlas 300IPro is an AI inference card based on an NPU that integrates a general-purpose processor, AI Core, and codec.

[0147] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of this application are not limited to the described order of actions, because according to the embodiments of this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of this application.

[0148] Based on the above embodiments, this embodiment also provides a voice recognition device that can be applied to electronic devices such as terminal devices and servers.

[0149] Reference Figure 7 The diagram shows a structural block diagram of an embodiment of a speech recognition device according to this application, which may specifically include the following modules:

[0150] The speech recognition engine module 701 is used to build a speech recognition engine for batch processing of speech recognition tasks; the speech recognition tasks have corresponding speech data.

[0151] The speech data module 702 is used to concatenate speech data corresponding to the speech recognition task to obtain batch features and batch cache; the batch features and the batch cache each have corresponding dimensions.

[0152] The dimension transpose module 703 is used to transpose the batch cache using the speech recognition engine to obtain a transpose cache whose dimensions match the dimensions corresponding to the batch features.

[0153] The batch recognition result module 704 is used to obtain batch recognition results based on the transposed cache and the batch features;

[0154] The speech recognition result module 705 is used to split the batch recognition results to obtain the speech recognition result corresponding to the speech recognition task.

[0155] This application also provides a non-volatile readable storage medium storing one or more modules (programs). When these modules are applied to a device, they enable the device to execute the instructions for the method steps in this application.

[0156] This application provides one or more machine-readable media storing instructions that, when executed by one or more processors, cause an electronic device to perform one or more of the methods described in the above embodiments. In this application, the electronic device includes various types of devices such as terminal devices and servers (clusters).

[0157] The embodiments of this disclosure can be implemented as an apparatus configured as desired using any suitable hardware, firmware, software, or any combination thereof, including electronic devices such as terminal devices, servers (clusters), etc. Figure 8 An exemplary apparatus 800 is schematically shown that can be used to implement the various embodiments described in this application.

[0158] In one embodiment, Figure 8An exemplary device 800 is shown, which includes one or more processors 802, a control module (chipset) 804 coupled to at least one of the processors 802, a memory 806 coupled to the control module 804, a non-volatile memory (NVM) / storage device 808 coupled to the control module 804, one or more input / output devices 810 coupled to the control module 804, and a network interface 812 coupled to the control module 804.

[0159] Processor 802 may include one or more single-core or multi-core processors, and processor 802 may include any combination of general-purpose processors or special-purpose processors (e.g., graphics processors, application processors, baseband processors, etc.). In some embodiments, device 800 can serve as a terminal device, server (cluster), or other device as described in the embodiments of this application.

[0160] In some embodiments, apparatus 800 may include one or more computer-readable media (e.g., memory 806 or NVM / storage device 808) having instructions 814 and one or more processors 802 that are combined with the one or more computer-readable media and configured to execute the instructions 814 to implement the module and thus perform the actions described in this disclosure.

[0161] In one embodiment, the control module 804 may include any suitable interface controller to provide any suitable interface to at least one of the processors 802 and / or any suitable device or component communicating with the control module 804.

[0162] The control module 804 may include a memory controller module to provide an interface to the memory 806. The memory controller module may be a hardware module, a software module, and / or a firmware module.

[0163] Memory 806 may be used, for example, to load and store data and / or instructions 814 for device 800. In one embodiment, memory 806 may include any suitable volatile memory, such as suitable DRAM. In some embodiments, memory 806 may include double data rate type quad synchronous dynamic random access memory (DDR4 SDRAM).

[0164] In one embodiment, the control module 804 may include one or more input / output controllers to provide an interface to the NVM / storage device 808 and (one or more) input / output devices 810.

[0165] For example, NVM / storage device 808 may be used to store data and / or instructions 814. NVM / storage device 808 may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable (one or more) non-volatile storage devices (e.g., one or more hard disk drives (HDDs), one or more optical disc drives (CDs), and / or one or more digital universal optical disc (DVD) drives).

[0166] NVM / storage device 808 may include storage resources that are physically part of a device on which device 800 is mounted, or that are accessible to the device but do not necessarily have to be part of the device. For example, NVM / storage device 808 may be accessed via a network via one or more input / output devices 810.

[0167] One or more input / output devices 810 may provide an interface for device 800 to communicate with any other suitable device. Input / output devices 810 may include communication components, audio components, sensor components, etc. A network interface 812 may provide an interface for device 800 to communicate via one or more networks. Device 800 may wirelessly communicate with one or more components of a wireless network according to any of one or more wireless network standards and / or protocols, such as accessing a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G, 5G, etc., or a combination thereof.

[0168] In one embodiment, at least one of the processors 802 may be logically packaged with one or more controllers (e.g., memory controller modules) of the control module 804. In one embodiment, at least one of the processors 802 may be logically packaged with one or more controllers of the control module 804 to form a system-in-package (SiP). In one embodiment, at least one of the processors 802 may be integrated with the logic of one or more controllers of the control module 804 on the same die. In one embodiment, at least one of the processors 802 may be integrated with the logic of one or more controllers of the control module 804 on the same die to form a system-on-a-chip (SoC).

[0169] In various embodiments, device 800 may be, but is not limited to, a server, desktop computing device, or mobile computing device (e.g., laptop, handheld computing device, tablet, netbook, etc.). In various embodiments, device 800 may have more or fewer components and / or different architectures. For example, in some embodiments, device 800 includes one or more cameras, a keyboard, a liquid crystal display (LCD) screen (including a touchscreen display), a non-volatile memory port, multiple antennas, a graphics chip, an application-specific integrated circuit (ASIC), and a speaker.

[0170] The detection device can use a main control chip as a processor or control module, and sensor data, position information, etc. can be stored in a memory or NVM / storage device. The sensor group can be used as an input / output device, and the communication interface can include a network interface.

[0171] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0172] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0173] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable speech recognition terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable speech recognition terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0174] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable speech recognition terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0175] These computer program instructions can also be loaded onto a computer or other programmable speech recognition terminal device, causing a series of operational steps to be performed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0176] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0177] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0178] The above provides a detailed description of a speech recognition method and apparatus, an electronic device, and a storage medium provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and its core ideas. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A speech recognition method, characterized in that, The method includes: A speech recognition engine is constructed for batch processing of speech recognition tasks; the speech recognition tasks have corresponding speech data, and the speech recognition engine includes an encoder, a decoder, and a connector. Create speech recognition tasks corresponding to real-time audio streams and place the speech recognition tasks in a task pool for management; Extract speech features from the real-time audio stream; Create a speech cache to store encoding features, language feature vectors, and character probability data; The speech features and the speech cache are determined as the speech data corresponding to the speech recognition task; Speech data corresponding to a speech recognition task is concatenated to obtain batch features and batch cache; the batch features and the batch cache each have corresponding dimensions, and the speech recognition task supports multiple executions. After transposing the batch cache using the speech recognition engine, a transposed cache whose dimensions match the dimensions corresponding to the batch features is obtained; The transposed buffer is input into the encoder to obtain the encoding buffer of the speech recognition task in the current execution process; The batch features are input into the encoder to obtain the encoded features; The encoding cache of the speech recognition task in the current execution process is used as the input of the encoder of the speech recognition task in the next execution process to obtain the encoding cache of the speech recognition task in the next execution process; The decoder is used to generate language feature vectors; The connector is used to process the encoded features and the language feature vector to obtain the connection result corresponding to the speech recognition task; The encoded cache and the connection result are used as the batch recognition result; The batch recognition results are split to obtain the speech recognition results corresponding to the speech recognition task.

2. The method according to claim 1, characterized in that, The encoded features have corresponding time frames. The process of using the connector to process the encoded features and the language feature vector to obtain the connection result includes: The encoding features and language feature vectors corresponding to the current time frame are input into the connector to obtain the character probability data corresponding to the current time frame; the character probability data includes the probabilities corresponding to multiple characters respectively. Decode the character probability data to obtain the predicted character corresponding to the current time frame; The predicted characters are connected according to the time frames corresponding to the predicted characters to obtain the connection result.

3. The method according to claim 2, characterized in that, The process of generating language feature vectors using the decoder includes: The decoder is used to generate the language feature vector corresponding to the current time frame; The predicted character corresponding to the current time frame is input into the decoder to generate the language feature vector corresponding to the next time frame.

4. The method according to claim 3, characterized in that, The speech recognition result includes a target speech cache and a target connection result. The speech recognition task has a result queue for storing the target connection result. After splitting the batch recognition result to obtain the speech recognition result corresponding to the speech recognition task, the method further includes: Update the speech cache corresponding to the speech recognition task using the target speech cache corresponding to the speech recognition task; The target connection result corresponding to the speech recognition task is stored in the result queue corresponding to the speech recognition task.

5. A voice recognition device, characterized in that, The device includes: A speech recognition engine module is used to build a speech recognition engine for batch processing of speech recognition tasks; the speech recognition task has corresponding speech data, and the speech recognition engine includes an encoder, a decoder, and a connector. The speech data module is used to create speech recognition tasks corresponding to real-time audio streams and place the speech recognition tasks in a task pool for management; extract speech features from the real-time audio streams; create a speech cache for storing encoding features, language feature vectors, and character probability data; determine the speech features and the speech cache as the speech data corresponding to the speech recognition task; concatenate the speech data corresponding to the speech recognition task to obtain batch features and batch cache; the batch features and the batch cache each have corresponding dimensions, and the speech recognition task supports multiple executions. The dimension transpose module is used to transpose the batch cache using the speech recognition engine to obtain a transpose cache whose dimensions match the dimensions corresponding to the batch features. The batch recognition result module is used to input the transposed buffer into the encoder to obtain the encoding buffer of the speech recognition task in the current execution process; input the batch features into the encoder to obtain the encoded features; use the encoding buffer of the speech recognition task in the current execution process as the input of the encoder in the next execution process of the speech recognition task to obtain the encoding buffer of the speech recognition task in the next execution process; use the decoder to generate language feature vectors; use the connector to process the encoded features and the language feature vectors to obtain the connection result corresponding to the speech recognition task; and use the encoding buffer and the connection result as the batch recognition result. The speech recognition result module is used to split the batch recognition results to obtain the speech recognition result corresponding to the speech recognition task.

6. An electronic device, characterized in that, include: processor; and A memory having executable code stored thereon, which, when executed, causes the processor to perform the speech recognition method as described in any one of claims 1-4.

7. One or more machine-readable media having executable code stored thereon, which, when executed, causes a processor to perform the speech recognition method as described in any one of claims 1-4.

Images