Voice recognition method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a voice recognition method and device, equipment and a storage medium. The method comprises the following steps: constructing a voice recognition engine for processing voice recognition tasks in batches; splicing voice data corresponding to the voice recognition task to obtain batch features and batch caches; performing dimension transposition on the batch cache by adopting a voice recognition engine to obtain a transposition cache of which the dimension is matched with the dimension corresponding to the batch feature; according to the transpose cache and the batch characteristics, obtaining a batch identification result; and splitting the batch recognition result to obtain a voice recognition result corresponding to the voice recognition task. According to the embodiment of the invention, through a dimension transposition mode, the batch features are matched with the dimension of the batch cache, and frequent splitting and re-splicing operations are avoided, so that the processing time is shortened, and the speed of voice recognition is improved. Moreover, due to the fact that frequent splitting and splicing operation is reduced, errors possibly introduced in the data processing process are reduced, and therefore the accuracy and reliability of voice recognition are improved.

Description

Technical Field

[0001] The present application relates to the field of speech recognition technology, and in particular to a speech recognition method and device, an electronic device and a storage medium. Background Art

[0002] In the related art, in the process of real-time speech recognition, in order to make full use of computing power to improve the efficiency of speech recognition, it is usually necessary to combine scattered speech data into batch data for batch recognition. Batch data includes batch features and batch cache. Since the dimensions of batch features and batch cache do not match, batch data needs to be continuously split and reassembled to make the dimensions of batch features and batch cache match. However, continuously splitting and reassembling batch data will directly affect the speed of real-time speech recognition, thereby reducing the efficiency of speech recognition. Summary of the invention

[0003] The embodiment of the present application provides a speech recognition method to solve the problem of low efficiency of real-time speech recognition caused by continuous splitting and splicing of batch data.

[0004] Correspondingly, the embodiments of the present application also provide a speech recognition device, an electronic device and a storage medium to ensure the implementation and application of the above method.

[0005] In order to solve the above problems, the present application discloses a method for speech recognition, which includes:

[0006] Constructing a speech recognition engine for batch processing of speech recognition tasks; the speech recognition tasks have corresponding speech data;

[0007] Concatenating speech data corresponding to the speech recognition task to obtain batch features and batch buffers; the batch features and the batch buffers have corresponding dimensions respectively;

[0008] After performing dimension transposition on the batch buffer using the speech recognition engine, a transposed buffer having a dimension matching a dimension corresponding to the batch feature is obtained;

[0009] Obtaining batch recognition results according to the transposed buffer and the batch features;

[0010] The batch recognition results are split to obtain speech recognition results corresponding to the speech recognition task.

[0011] Optionally, the speech recognition engine includes an encoder, a decoder, and a connector, and obtaining batch recognition results according to the transposed buffer and the batch features includes:

[0012] Encoding the transposed buffer and the batch features using the encoder to obtain an encoded buffer and encoded features;

[0013] generating a language feature vector using the decoder;

[0014] Processing the coding feature and the language feature vector using the connector to obtain a connection result corresponding to the speech recognition task;

[0015] The encoding cache and the connection result are used as the batch recognition result.

[0016] Optionally, the speech recognition task supports multiple executions, and the encoder is used to encode the transposed buffer and the batch features to obtain the encoded buffer and the encoded features, including:

[0017] Inputting the transposed buffer into the encoder to obtain the encoding buffer of the speech recognition task during the current execution process;

[0018] Inputting the batch features into the encoder to obtain the encoded features;

[0019] The encoding cache of the speech recognition task in the current execution process is used as the input of the encoder in the next execution process of the speech recognition task to obtain the encoding cache of the speech recognition task in the next execution process.

[0020] Optionally, the coding feature has a corresponding time frame, and the using the connector to process the coding feature and the language feature vector to obtain a connection result includes:

[0021] Inputting the coding feature corresponding to the current time frame and the language feature vector corresponding to the current time frame into the connector to obtain character probability data corresponding to the current time frame; the character probability data includes probabilities corresponding to a plurality of characters respectively;

[0022] Decoding the character probability data to obtain a predicted character corresponding to the current time frame;

[0023] The predicted characters are connected according to the time frames corresponding to the predicted characters to obtain the connection result.

[0024] Optionally, the using the decoder to generate a language feature vector includes:

[0025] Using the decoder to generate a language feature vector corresponding to the current time frame;

[0026] The predicted characters corresponding to the current time frame are input into the decoder to generate a language feature vector corresponding to the next time frame.

[0027] Optionally, before concatenating the speech data corresponding to the speech recognition task to obtain batch features and batch cache, the method further includes:

[0028] Create a speech recognition task corresponding to the real-time audio stream, and place the speech recognition task in a task pool for management;

[0029] Extracting speech features from the real-time audio stream;

[0030] Creating a speech buffer for storing the coding feature, the language feature vector and the character probability data;

[0031] The speech feature and the speech buffer are determined as speech data corresponding to the speech recognition task.

[0032] Optionally, the speech recognition result includes a target speech cache and a target connection result, and the speech recognition task has a result queue for storing the target connection result. After splitting the batch recognition results to obtain the speech recognition result corresponding to the speech recognition task, the method further includes:

[0033] Using the target speech buffer corresponding to the speech recognition task to update the speech buffer corresponding to the speech recognition task;

[0034] The target connection result corresponding to the speech recognition task is stored in the result queue corresponding to the speech recognition task.

[0035] The present application also discloses a speech recognition device, which includes:

[0036] A speech recognition engine module, used to construct a speech recognition engine for batch processing of speech recognition tasks; the speech recognition tasks have corresponding speech data;

[0037] A speech data module, used for splicing speech data corresponding to the speech recognition task to obtain batch features and batch buffers; the batch features and the batch buffers have corresponding dimensions respectively;

[0038] A dimension transposition module, configured to perform dimension transposition on the batch buffer using the speech recognition engine to obtain a transposed buffer whose dimension matches the dimension corresponding to the batch feature;

[0039] A batch recognition result module, used for obtaining batch recognition results according to the transposed buffer and the batch features;

[0040] The speech recognition result module is used to split the batch recognition results to obtain the speech recognition results corresponding to the speech recognition task.

[0041] An embodiment of the present application also discloses an electronic device, including: a processor; and a memory, on which executable code is stored. When the executable code is executed, the processor executes the speech recognition method as described in any one of the embodiments of the present application.

[0042] The embodiments of the present application also disclose one or more machine-readable media on which executable codes are stored. When the executable codes are executed, a processor executes the speech recognition method as described in any one of the embodiments of the present application.

[0043] Compared with the prior art, the embodiments of the present application have the following advantages:

[0044] In an embodiment of the present application, a speech recognition engine for batch processing of speech recognition tasks is constructed; the speech recognition task has corresponding speech data; the speech data corresponding to the speech recognition task is spliced ​​to obtain batch features and batch cache; the batch features and the batch cache have corresponding dimensions respectively; after the speech recognition engine performs dimension transposition on the batch cache, a transposed cache whose dimension matches the dimension corresponding to the batch features is obtained; batch recognition results are obtained based on the transposed cache and the batch features; the batch recognition results are split to obtain the speech recognition results corresponding to the speech recognition task. The embodiment of the present application matches the dimensions of the batch features and the batch cache by means of dimension transposition, avoids frequent splitting and re-splicing operations, thereby reducing processing time and improving the speed of speech recognition. In addition, since frequent splitting and splicing operations are reduced, the errors that may be introduced in the data processing process are reduced, thereby improving the accuracy and reliability of speech recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 This is a flowchart of a speech recognition method embodiment of the present application;

[0046] Figure 2 It is a flowchart of establishing a speech recognition task in this application;

[0047] Figure 3 It is a flowchart of neural network model recognition based on Transducer architecture;

[0048] Figure 4 This is a flowchart of a neural network model recognition based on the Transducer architecture of the present application;

[0049] Figure 5 This is a flowchart of a speech recognition task based on a speech recognition engine in the present application;

[0050] Figure 6It is a flow chart of a speech recognition system of the present application;

[0051] Figure 7 It is a structural block diagram of an embodiment of a speech recognition device of the present application;

[0052] Figure 8 It is a schematic diagram of the structure of a device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0053] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0054] Reference Figure 1 , is a flowchart of a speech recognition method embodiment of the present application, comprising the following steps:

[0055] Step 101: construct a speech recognition engine for batch processing of speech recognition tasks; the speech recognition tasks have corresponding speech data;

[0056] In the embodiment of the present application, a speech recognition engine based on the Transducer architecture is constructed based on the ONNX (Open Neural Network Exchange, an open neural network exchange format) framework, and multiple speech recognition tasks are processed in batches through the speech recognition engine. The speech recognition engine uses the speech recognition task as a minimum task unit.

[0057] The speech recognition task in the embodiment of the present application refers to the process of converting a real-time audio stream into corresponding text. Each speech recognition task has corresponding speech data. Specifically, the speech data refers to speech features and speech cache. Speech features refer to the features obtained after feature extraction of the real-time audio stream; speech cache refers to the data structure used to store the intermediate results generated in the speech recognition process. These intermediate results are usually generated when processing speech features and are used in subsequent recognition steps.

[0058] Reference Figure 2 , is a flowchart of establishing a speech recognition task in the present application. When the speech recognition task is established, the feature queue, result queue and speech buffer corresponding to the speech recognition task are also created at the same time. Among them, the feature queue is used to store speech features.

[0059] The recognition model adopted by the speech recognition engine in the embodiment of the present application is a neural network model based on the Transducer architecture. Transducer is a model widely used in sequence-to-sequence tasks such as machine translation, speech recognition, and question-answering systems. It includes three parts: Encoder, Decoder, and Joiner.

[0060] Reference Figure 3 , is a flowchart of neural network model recognition based on the Transducer architecture, as described below:

[0061] The encoder is mainly responsible for processing the input sequence and converting it into a continuous high-dimensional representation, which captures the characteristics and contextual information of the input sequence. In the speech recognition process, the encoder can be compared to the traditional acoustic model, converting the input audio feature sequence into a high-dimensional representation to provide a basis for subsequent processing.

[0062] The decoder is mainly responsible for generating a vector that can characterize the current output sequence based on the previously predicted output characters. It can capture the statistical characteristics and language structure of the output sequence and help the entire model generate coherent and grammatically correct output. During speech recognition, the decoder can predict the next word based on the recognized words to form a complete sentence or paragraph.

[0063] Joiner is mainly responsible for combining the output of Encoder with the output of Decoder to produce the probability distribution of the final output. The purpose of Joiner is to fuse the acoustic information of the input sequence with the linguistic information of the output sequence so that the model can decide which character should be output at each time frame.

[0064] Step 102: concatenate the speech data corresponding to the speech recognition task to obtain batch features and batch buffers; the batch features and the batch buffers have corresponding dimensions respectively;

[0065] The speech data corresponding to multiple speech recognition tasks are concatenated to obtain batch features and batch caches. Specifically, the single speech features and speech caches obtained from different speech recognition tasks are concatenated to obtain batch features and batch caches.

[0066] Batch features and batch caches have corresponding dimensions, which will be matched and processed in subsequent steps. By concatenating speech data to form batch features and batch caches, computing power can be fully utilized to improve recognition efficiency. The concatenation operation simplifies the data processing flow, reduces intermediate steps, and makes the entire speech recognition process more concise and efficient.

[0067] Step 103: After performing dimension transposition on the batch buffer using the speech recognition engine, a transposed buffer having a dimension matching a dimension corresponding to the batch feature is obtained;

[0068] like Figure 3 As shown, the dimension of the batch feature input to the Encoder is B×T×D i , the dimension of the batch buffer input to the Encoder is T×B×D c , where B is the batch size, T is the number of feature frames, and D is the number of feature channels.

[0069] For the batch features input to the encoder, since the batch size B is in the first dimension, it is only necessary to put the speech features from different speech recognition tasks in the memory space in order, that is, the memory addresses of the speech features of different speech recognition tasks are continuous. However, for the batch cache, since the batch size B is in the second dimension, the speech cache T×D in the speech recognition task must first be c Split along the first dimension to obtain T dimensions of 1×D c The corresponding single-frame buffers in multiple speech recognition tasks are then concatenated together to form a batch single-frame buffer with a dimension of B×D c Finally, the batch single frame buffers are spliced ​​frame by frame to obtain the final batch buffer that can be recognized by the model, with a dimension of T×B×D c Correspondingly, the new batch cache dimension obtained after model recognition is T×B×D c To update the new batch cache to each session, you need to first decompose the batch cache into a 1×D dimension on the first and second dimensions. c Then, the cache corresponding to each session is concatenated frame by frame to obtain T×D c Therefore, in the process of real-time speech recognition, the speech recognition engine needs to continuously split and reassemble the speech buffer. However, continuously splitting and reassembling the batch buffer will directly reduce the processing efficiency of the speech recognition engine, thereby affecting the recognition efficiency of the speech recognition task.

[0070] The splitting and splicing of batch caches can be performed on both the CPU (Central Processing Unit) and the NPU (Neural Processing Unit). If the batch cache is split and spliced ​​on the CPU, the spliced ​​batch cache must first be migrated from the CPU's memory to the NPU's video memory before the model is inferred. After the model's inference is completed, the new inference cache must also be moved from the NPU's video memory to the CPU's memory for subsequent splitting. This back-and-forth migration process is very time-consuming and occupies CPU resources.

[0071] Therefore, in an embodiment of the present application, when a new speech recognition task is created, the speech cache is initialized directly on the video memory of the NPU, and subsequent splitting and splicing operations can be performed directly on the NPU, which can greatly improve the real-time processing speed of the engine and reduce the meaningless occupation of CPU resources.

[0072] However, even so, splitting and reassembling bulk caches on the NPU is still performed by naive single-threaded address copying, which is still slow.

[0073] Therefore, in order to further speed up the inference speed of the model in the speech recognition engine, the embodiment of the present application modifies the native ONNX model. The modification operation specifically includes adding, deleting or modifying operators in the model to optimize the processing performance of the model.

[0074] Specifically, in the ONNX model, before the batch buffer is input into the Encoder and after the Encoder outputs the batch buffer, a transpose operator is added to perform matrix transposition on the first and second dimensions of the batch buffer, so that the batch buffer input and output dimensions of the model become B×T×D c In this way, the speech buffers in multiple speech recognition tasks can be directly spliced ​​into batch buffers in a similar way to sequence features and then input into the model without the need for extra splitting steps. Correspondingly, the output of the model only needs to be split on the first dimension before it can be updated to each speech recognition task, without the need for the splicing process after splitting.

[0075] That is, the speech recognition engine in the embodiment of the present application performs a dimension transposition operation on the batch cache by adding a transposition operator, so that the dimension of the transposed cache (B×T×D c ) and the dimension of batch features (B×T×D i ), thus avoiding frequent splitting and re-joining operations.

[0076] Reference Figure 4, is a flowchart of a neural network model recognition based on the Transducer architecture of the present application. The cache input to the Encoder is a transposed cache (transposed cache) with a dimension of B×T×D c , the output transposed buffer dimension is also B×T×D c .

[0077] Therefore, in the embodiment of the present application, after the transposition operator is added to the ONNX model, the ONNX model will be converted into an OM (Optimized Model) so that the newly added transposition operator calls the corresponding acceleration algorithm on the NPU to accelerate the transposition operation, thereby improving the model reasoning speed and further improving the recognition efficiency of the speech recognition task.

[0078] By transposing the dimension, batch features and batch caches can be directly matched, avoiding frequent splitting and re-splicing operations, significantly reducing processing time, and improving the speed of speech recognition.

[0079] Step 104: Obtain batch recognition results according to the transposed buffer and the batch features;

[0080] Based on the transposed cache and batch features, the model in the speech recognition engine is used to infer them and obtain batch recognition results. The dimensionally matched transposed cache and batch features are used to achieve efficient batch inference.

[0081] Step 105: split the batch recognition results to obtain the speech recognition results corresponding to the speech recognition task.

[0082] The batch recognition results are split to obtain the speech recognition results corresponding to each speech recognition task. This step decomposes the batch recognition results into the results of a single task, which is convenient for subsequent application and processing.

[0083] The embodiment of the present application uses the method of dimensional transposition to match the dimensions of batch features and batch cache, avoiding frequent splitting and re-joining operations, thereby reducing processing time and improving the speed of speech recognition. In addition, since frequent splitting and joining operations are reduced, the errors that may be introduced in the data processing process are reduced, thereby improving the accuracy and reliability of speech recognition.

[0084] In one embodiment of the present application, the speech recognition engine includes an encoder, a decoder, and a connector, and obtaining a batch recognition result according to the transposed buffer and the batch features includes:

[0085] Encoding the transposed buffer and the batch features using the encoder to obtain an encoded buffer and encoded features;

[0086] generating a language feature vector using the decoder;

[0087] Processing the coding feature and the language feature vector using the connector to obtain a connection result corresponding to the speech recognition task;

[0088] The encoding cache and the connection result are used as the batch recognition result.

[0089] like Figure 4 As shown, Encoder is the encoder in the embodiment of the present application, which is mainly responsible for converting the input transposed buffer (B×T×D c ) and batch features (B×T×D i ) is converted into a high-dimensional representation and the output encoding buffer (B×T×D c ) and encoding features (B×T×D o ).

[0090] The encoding cache is the intermediate result output by the encoder, which will be used as input in the next model inference. The encoding feature is the high-dimensional representation output by the encoder, which will be used as the input of the connector later.

[0091] like Figure 4 As shown, Decoder is the decoder in the embodiment of the present application, which generates the current language feature vector (B×D d ).

[0092] However, when processing the first time frame, since the decoder does not have the previously predicted language feature vector as input, a special initial state or initial vector is introduced as the input of the decoder in the first time frame.

[0093] like Figure 4 As shown, Joiner is the connector in the embodiment of the present application, which is mainly responsible for combining the coding features and the language feature vectors to generate the predicted characters corresponding to each time frame.

[0094] After obtaining the predicted characters corresponding to each time frame, each predicted character will be connected in the order of the time frames to obtain a complete connection result, that is, the text corresponding to the real-time audio stream.

[0095] The encoding cache and the connection result are output together as batch recognition results. Specifically, it is necessary to use the transposition operator to perform matrix transposition on the first dimension and the second dimension of the encoding cache again, so that the cache dimension output to the speech recognition task is converted back to T×B×D c .

[0096] The embodiment of the present application realizes efficient speech recognition processing through the collaborative work of the encoder, decoder and connector. The encoder converts the input transposed buffer and batch features into high-dimensional representations, the decoder generates language feature vectors, and the connector combines the two to generate the final connection result. The whole process is efficient and coherent.

[0097] In one embodiment of the present application, the speech recognition task supports multiple executions, and the encoder is used to encode the transposed buffer and the batch features to obtain the encoded buffer and the encoded features, including:

[0098] Inputting the transposed buffer into the encoder to obtain the encoding buffer of the speech recognition task during the current execution process;

[0099] Inputting the batch features into the encoder to obtain the encoded features;

[0100] The encoding cache of the speech recognition task in the current execution process is used as the input of the encoder in the next execution process of the speech recognition task to obtain the encoding cache of the speech recognition task in the next execution process.

[0101] In the embodiments of the present application, Figure 4 As shown, the dimension is B×T×D i The batch features and dimensions are B×T×D c The transposed buffer is input to the Encoder, and the Encoder will output two parts of content, which are B×T×D o The encoding features and dimensions are B×T×D c The new transposed buffer is also required to use the transposition operator to perform matrix transposition on the first dimension and the second dimension of the new transposed buffer again to obtain a new encoding buffer, so that the buffer dimension output to the speech recognition task is converted back to T×B×D c The encoded features will be used as input to the Joiner, and the new transposed buffer will be used as input the next time the Encoder performs inference.

[0102] Specifically, the difference between the coding cache as the input and output of the Encoder lies in the execution process of the speech recognition task. The coding cache obtained by the speech recognition task in the next execution process is obtained after the coding cache obtained in the current execution process is input into the encoder. Figure 4The "previous transposed buffer" in the above is the encoding buffer obtained in the previous execution process after dimension transposition, and the "current transposed buffer" is the encoding buffer obtained in the current execution process after dimension transposition. For example, assume that during the first execution of the speech recognition task, the encoder encodes the transposed buffer to generate encoding buffer A. During the second execution of the speech recognition task, the encoder uses encoding buffer A as input to generate a new encoding buffer B.

[0103] In one embodiment of the present application, the coding feature has a corresponding time frame, and the connector is used to process the coding feature and the language feature vector to obtain a connection result, including:

[0104] Inputting the coding feature corresponding to the current time frame and the language feature vector corresponding to the current time frame into the connector to obtain character probability data corresponding to the current time frame; the character probability data includes probabilities corresponding to a plurality of characters respectively;

[0105] Decoding the character probability data to obtain a predicted character corresponding to the current time frame;

[0106] The predicted characters are connected according to the time frames corresponding to the predicted characters to obtain the connection result.

[0107] In the speech recognition process, the Joiner mainly combines the acoustic information of the input sequence with the linguistic information of the output sequence so that the model can decide which character should be output in each time frame.

[0108] In the embodiments of the present application, Figure 4 As shown in Figure 1, since the encoded features output by the Encoder contain T consecutive time frames, the Decoder and Joiner need to process the encoded features output by the Encoder frame by frame. For each time frame, the split encoded features (B×D o ) and the language feature vector (B×D) output by the decoder d ) is input into Joiner, which will output the probability distribution based on the dictionary for subsequent Beam Search decoding. The dimension of the character probability data output by Joiner is B×D j , where D j is the dictionary size.

[0109] Beam Search is a heuristic graph search algorithm used to find the most likely output sequence in a sequence generation task. In the speech recognition task of the embodiment of the present application, Beam Search is used to select the most likely predicted character from the character probability data output by the Joiner.

[0110] Exemplarily, for each time frame, Beam Search selects the character with the highest probability from the character probability data output by Joiner as the predicted character for the current time frame.

[0111] Subsequently, the predicted characters corresponding to each time frame are connected in chronological order to form a final output sequence, which is the connection result in the embodiment of the present application.

[0112] In the embodiment of the present application, the character probability data corresponding to the current time frame is obtained by inputting the coding feature corresponding to the current time frame and the language feature vector corresponding to the current time frame into the connector. Then, the character with the highest probability is selected from the character probability data as the predicted character of the current time frame. Finally, the predicted characters corresponding to each time frame are connected in chronological order to form a final output sequence. Through these steps, an efficient speech recognition process is achieved.

[0113] In one embodiment of the present application, the step of using the decoder to generate a language feature vector includes:

[0114] Using the decoder to generate a language feature vector corresponding to the current time frame;

[0115] The predicted characters corresponding to the current time frame are input into the decoder to generate a language feature vector corresponding to the next time frame.

[0116] In the speech recognition process, the decoder can predict the next word based on the recognized word, thereby predicting the current predicted character. Figure 4 As shown in the figure, the Decoder takes the predicted character ID as input based on the previous prediction, with a dimension of B×1, and its output will be used as the input of the Joiner, with a dimension of B×D d .

[0117] Specifically, Figure 4 Character Id t-1 (B) is the predicted character corresponding to the current time frame, and character Id is the predicted character corresponding to the next time frame.

[0118] In one embodiment of the present application, before concatenating the speech data corresponding to the speech recognition task to obtain batch features and batch cache, the method further includes:

[0119] Create a speech recognition task corresponding to the real-time audio stream, and place the speech recognition task in a task pool for management;

[0120] Extracting speech features from the real-time audio stream;

[0121] Creating a speech buffer for storing the coding feature, the language feature vector and the character probability data;

[0122] The speech feature and the speech buffer are determined as speech data corresponding to the speech recognition task.

[0123] First, you need to establish a speech recognition task, such as Figure 2 As shown, a corresponding speech recognition task will be created according to the real-time audio stream, and when the speech recognition task is established, the feature queue, result queue and speech buffer corresponding to the speech recognition task will also be created at the same time. Among them, the feature queue is used to store speech features.

[0124] Each time a speech recognition task is established, it will be placed in the task pool. The task pool is used to manage all speech recognition tasks to ensure that each task can be processed correctly.

[0125] like Figure 2 As shown in the figure, when a real-time audio stream is received, speech features are extracted from the real-time audio stream, and a feature queue, a result queue, and a speech cache are created. The speech cache is used to store the cache of coding features, language feature vectors, and character probability data; the feature queue is used to store the extracted speech features; the result queue is used to store the target connection results; the target connection results refer to the text content corresponding to the real-time audio stream.

[0126] Specifically, speech features are extracted from real-time audio streams. Usually, feature extraction algorithms (such as Mel-frequency cepstral coefficients MFCC, filter bank features, etc.) are used to convert audio signals into high-dimensional feature vectors. The extracted speech features will be stored in a feature queue for subsequent model inference.

[0127] In one embodiment of the present application, the speech recognition result includes a target speech cache and a target connection result, the speech recognition task has a result queue for storing the target connection result, and after splitting the batch recognition results to obtain the speech recognition result corresponding to the speech recognition task, the method further includes:

[0128] Using the target speech buffer corresponding to the speech recognition task to update the speech buffer corresponding to the speech recognition task;

[0129] The target connection result corresponding to the speech recognition task is stored in the result queue corresponding to the speech recognition task.

[0130] The speech recognition result in the embodiment of the present application includes a target speech buffer and a target connection result. The target speech buffer is an intermediate result generated during the model reasoning process and is used as input for the next model reasoning. The target connection result is the final recognized text sequence.

[0131] The speech recognition task has a result queue for storing target connection results.

[0132] After splitting the batch recognition results, the speech recognition results corresponding to each speech recognition task are obtained.

[0133] The target speech cache corresponding to the speech recognition task is used to update the speech cache corresponding to the speech recognition task, and the updated speech cache will be used for the next model inference.

[0134] The target connection result corresponding to the speech recognition task is stored in the result queue corresponding to the speech recognition task. The result queue is used to store the target connection result of each speech recognition task (the text sequence corresponding to the real-time audio stream) for subsequent processing and output.

[0135] In the embodiment of the present application, efficient speech recognition processing is achieved by updating the speech cache and storing the target connection results. The updated speech cache ensures that the intermediate results generated during the model reasoning process can be correctly stored and used, and will be used for the next model reasoning, ensuring that the model can continue to efficiently process subsequent speech data, further improving the recognition speed.

[0136] Reference Figure 5 , is a flow chart of a speech recognition task based on a speech recognition engine of the present application. Specifically, when the number of speech features in the feature queue is sufficient, the corresponding speech recognition task will be selected from the task pool and placed in the task queue. The speech recognition engine will monitor the status of the task queue in real time, and try to take out multiple speech recognition tasks from the task queue at one time. When the speech recognition engine takes out a batch of speech recognition tasks from the task queue, it will also take out the corresponding speech data (speech features and speech cache) from the feature queue to form batch data. Then, speech recognition is performed on the batch data (batch features and batch cache) to obtain batch recognition results. Finally, the batch recognition results are decomposed and pushed to the speech recognition task.

[0137] In the embodiments of the present application, the entire speech recognition process realizes efficient speech recognition task management, optimizes feature extraction, intermediate result storage and data preparation, and improves the overall efficiency and recognition accuracy of speech recognition.

[0138] The speech recognition method provided in the embodiment of the present application is suitable for various application scenarios, such as smart homes, vehicle-mounted systems, customer service robots, etc. Therefore, in order to deploy speech recognition capabilities in various application scenarios, the embodiment of the present application encapsulates a Web service based on the Websocket standard protocol for the speech recognition engine to form a speech recognition system to achieve remote access and interaction with the speech recognition engine.

[0139] Reference Figure 6 , is a flow chart of a speech recognition system of the present application. The system as a whole includes two parts, namely, a Web service based on the Websocket protocol and a speech recognition engine.

[0140] Specifically, the Web service is used to provide a standard Websocket interface for client calls, accept and parse client requests, and interact with the inference engine; the speech recognition engine is used to infer the audio data in the client request, obtain the recognition results, and then return them to the service.

[0141] The interaction between the Web service and the speech recognition engine mainly includes the following four operations:

[0142] 1) Establishing a speech recognition task: The Web service calls the speech recognition engine's task establishment interface. This operation will create a new speech recognition task in the speech recognition engine's task pool. The speech recognition task includes a feature queue, a result queue, and a speech cache.

[0143] 2) Real-time writing of audio stream: The Web service calls the real-time writing of audio stream interface of the speech recognition engine to write the audio data parsed from the client request into the speech recognition engine.

[0144] 3) Obtain recognition results in real time: The Web service monitors the result queue in the speech recognition task in real time. When the queue is not empty, it takes out the target connection result and returns it to the client.

[0145] 4) End the session: The Web service calls the end task interface of the speech recognition engine to release the resources of the feature queue, result queue and speech cache occupied by the current task.

[0146] The speech recognition engine constructed in the embodiment of the present application can be adapted to the Atlas 300IPro inference card. Atlas 300IPro is an AI inference card based on NPU and integrating "general processor, AI Core, and codec".

[0147] It should be noted that, for the method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present application are not limited by the described order of actions, because according to the embodiments of the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present application.

[0148] On the basis of the above embodiments, this embodiment further provides a speech recognition device, which is applied to electronic devices such as terminal devices and servers.

[0149] Reference Figure 7 , shows a structural block diagram of an embodiment of a speech recognition device of the present application, which may specifically include the following modules:

[0150] The speech recognition engine module 701 is used to construct a speech recognition engine for batch processing of speech recognition tasks; the speech recognition tasks have corresponding speech data;

[0151] The speech data module 702 is used to concatenate speech data corresponding to the speech recognition task to obtain batch features and batch buffers; the batch features and the batch buffers have corresponding dimensions respectively;

[0152] A dimension transposition module 703, configured to perform dimension transposition on the batch buffer using the speech recognition engine to obtain a transposed buffer whose dimension matches the dimension corresponding to the batch feature;

[0153] A batch recognition result module 704, used to obtain batch recognition results according to the transposed buffer and the batch features;

[0154] The speech recognition result module 705 is used to split the batch recognition results to obtain the speech recognition results corresponding to the speech recognition task.

[0155] The embodiment of the present application also provides a non-volatile readable storage medium, which stores one or more modules (programs). When the one or more modules are applied to a device, the device can execute instructions (instructions) of each method step in the embodiment of the present application.

[0156] The present application embodiment provides one or more machine-readable media on which instructions are stored, and when executed by one or more processors, an electronic device executes one or more of the methods described in the above embodiments. In the present application embodiment, the electronic device includes various types of devices such as terminal devices and servers (clusters).

[0157] The embodiments of the present disclosure may be implemented as a device configured as desired using any appropriate hardware, firmware, software, or any combination thereof, and the device may include electronic devices such as terminal devices and servers (clusters). Figure 8 An exemplary apparatus 800 that can be used to implement various embodiments described in this application is schematically illustrated.

[0158] For one embodiment, Figure 8An exemplary apparatus 800 is shown having one or more processors 802, a control module (chip set) 804 coupled to at least one of the processor(s) 802, a memory 806 coupled to the control module 804, a non-volatile memory (NVM) / storage device 808 coupled to the control module 804, one or more input / output devices 810 coupled to the control module 804, and a network interface 812 coupled to the control module 804.

[0159] The processor 802 may include one or more single-core or multi-core processors, and the processor 802 may include any combination of general-purpose processors or special-purpose processors (such as graphics processors, application processors, baseband processors, etc.). In some embodiments, the device 800 can be used as a terminal device, server (cluster), etc. described in the embodiments of the present application.

[0160] In some embodiments, the apparatus 800 may include one or more computer-readable media (e.g., memory 806 or NVM / storage device 808) having instructions 814 and one or more processors 802 combined with the one or more computer-readable media and configured to execute the instructions 814 to implement a module to perform the actions described in the present disclosure.

[0161] For one embodiment, the control module 804 may include any suitable interface controller to provide any suitable interface to at least one of the processor(s) 802 and / or any suitable device or component in communication with the control module 804 .

[0162] The control module 804 may include a memory controller module to provide an interface to the memory 806. The memory controller module may be a hardware module, a software module, and / or a firmware module.

[0163] The memory 806 may be used, for example, to load and store data and / or instructions 814 for the device 800. For one embodiment, the memory 806 may include any suitable volatile memory, such as a suitable DRAM. In some embodiments, the memory 806 may include double data rate type four synchronous dynamic random access memory (DDR4 SDRAM).

[0164] For one embodiment, control module 804 may include one or more input / output controllers to provide an interface to NVM / storage device 808 and input / output device(s) 810 .

[0165] For example, NVM / storage 808 may be used to store data and / or instructions 814. NVM / storage 808 may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable non-volatile storage device(s) (e.g., one or more hard disk drives (HDDs), one or more compact disk (CD) drives, and / or one or more digital versatile disk (DVD) drives).

[0166] NVM / storage device 808 may include storage resources that are physically part of the device on which apparatus 800 is installed, or it may be accessible to the device without being part of the device. For example, NVM / storage device 808 may be accessed via input / output device(s) 810 over a network.

[0167] (One or more) input / output devices 810 may provide an interface for the apparatus 800 to communicate with any other appropriate device, and the input / output device 810 may include a communication component, an audio component, a sensor component, etc. The network interface 812 may provide an interface for the apparatus 800 to communicate through one or more networks, and the apparatus 800 may wirelessly communicate with one or more components of a wireless network according to any of one or more wireless network standards and / or protocols, for example, accessing a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G, 5G, etc., or a combination thereof for wireless communication.

[0168] For one embodiment, at least one of the processor(s) 802 may be packaged together with the logic of one or more controllers (e.g., a memory controller module) of the control module 804. For one embodiment, at least one of the processor(s) 802 may be packaged together with the logic of one or more controllers of the control module 804 to form a system-in-package (SiP). For one embodiment, at least one of the processor(s) 802 may be integrated on the same die with the logic of one or more controllers of the control module 804. For one embodiment, at least one of the processor(s) 802 may be integrated on the same die with the logic of one or more controllers of the control module 804 to form a system-on-chip (SoC).

[0169] In various embodiments, the device 800 may be, but is not limited to, a terminal device such as a server, a desktop computing device, or a mobile computing device (e.g., a laptop computing device, a handheld computing device, a tablet computer, a netbook, etc.). In various embodiments, the device 800 may have more or fewer components and / or different architectures. For example, in some embodiments, the device 800 includes one or more cameras, a keyboard, a liquid crystal display (LCD) screen (including a touch screen display), a non-volatile memory port, multiple antennas, a graphics chip, an application-specific integrated circuit (ASIC), and a speaker.

[0170] Among them, the main control chip can be used as a processor or control module in the detection device, sensor data, location information, etc. are stored in a memory or NVM / storage device, the sensor group can be used as an input / output device, and the communication interface may include a network interface.

[0171] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0172] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0173] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable speech recognition terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable speech recognition terminal device generate instructions for implementing the processes in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0174] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable speech recognition terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0175] These computer program instructions can also be loaded onto a computer or other programmable speech recognition terminal device, so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions executed on the computer or other programmable terminal device for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0176] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0177] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0178] The above is a detailed introduction to a speech recognition method and device, an electronic device and a storage medium provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A speech recognition method, characterized in that: The method comprises: Constructing a speech recognition engine for batch processing of speech recognition tasks; the speech recognition tasks have corresponding speech data; Concatenating speech data corresponding to the speech recognition task to obtain batch features and batch buffers; the batch features and the batch buffers have corresponding dimensions respectively; After performing dimension transposition on the batch buffer using the speech recognition engine, a transposed buffer having a dimension matching a dimension corresponding to the batch feature is obtained; Obtaining batch recognition results according to the transposed buffer and the batch features; The batch recognition results are split to obtain speech recognition results corresponding to the speech recognition task.

2. The method according to claim 1, characterized in that The speech recognition engine includes an encoder, a decoder and a connector, and obtaining batch recognition results according to the transposed buffer and the batch features includes: Encoding the transposed buffer and the batch features using the encoder to obtain an encoded buffer and encoded features; generating a language feature vector using the decoder; Processing the coding feature and the language feature vector using the connector to obtain a connection result corresponding to the speech recognition task; The encoding cache and the connection result are used as the batch recognition result.

3. The method according to claim 2, characterized in that The speech recognition task supports multiple executions, and the encoder is used to encode the transposed buffer and the batch features to obtain the encoded buffer and the encoded features, including: Inputting the transposed buffer into the encoder to obtain the encoding buffer of the speech recognition task during the current execution process; Inputting the batch features into the encoder to obtain the encoded features; The encoding cache of the speech recognition task in the current execution process is used as the input of the encoder in the next execution process of the speech recognition task to obtain the encoding cache of the speech recognition task in the next execution process.

4. The method according to claim 2, characterized in that: The coding feature has a corresponding time frame, and the connector is used to process the coding feature and the language feature vector to obtain a connection result, including: Inputting the coding feature corresponding to the current time frame and the language feature vector corresponding to the current time frame into the connector to obtain character probability data corresponding to the current time frame; the character probability data includes probabilities corresponding to a plurality of characters respectively; Decoding the character probability data to obtain a predicted character corresponding to the current time frame; The predicted characters are connected according to the time frames corresponding to the predicted characters to obtain the connection result.

5. The method according to claim 4, characterized in that The step of using the decoder to generate a language feature vector comprises: Using the decoder to generate a language feature vector corresponding to the current time frame; The predicted characters corresponding to the current time frame are input into the decoder to generate a language feature vector corresponding to the next time frame.

6. The method according to claim 4, characterized in that Before concatenating the speech data corresponding to the speech recognition task to obtain batch features and batch cache, the method further includes: Create a speech recognition task corresponding to the real-time audio stream, and place the speech recognition task in a task pool for management; Extracting speech features from the real-time audio stream; Creating a speech buffer for storing the coding feature, the language feature vector and the character probability data; The speech feature and the speech buffer are determined as speech data corresponding to the speech recognition task.

7. The method according to claim 6, characterized in that The speech recognition result includes a target speech cache and a target connection result, and the speech recognition task has a result queue for storing the target connection result. After splitting the batch recognition results to obtain the speech recognition result corresponding to the speech recognition task, the method further includes: Using the target speech buffer corresponding to the speech recognition task to update the speech buffer corresponding to the speech recognition task; The target connection result corresponding to the speech recognition task is stored in the result queue corresponding to the speech recognition task.

8. A speech recognition device, characterized in that: The device comprises: A speech recognition engine module, used to construct a speech recognition engine for batch processing of speech recognition tasks; the speech recognition tasks have corresponding speech data; A speech data module, used for splicing speech data corresponding to the speech recognition task to obtain batch features and batch buffers; the batch features and the batch buffers have corresponding dimensions respectively; A dimension transposition module, configured to perform dimension transposition on the batch buffer using the speech recognition engine to obtain a transposed buffer whose dimension matches the dimension corresponding to the batch feature; A batch recognition result module, used for obtaining batch recognition results according to the transposed buffer and the batch features; The speech recognition result module is used to split the batch recognition results to obtain the speech recognition results corresponding to the speech recognition task.

9. An electronic device, characterized in that: include: processor; and A memory having executable codes stored thereon, which, when executed, enables the processor to execute the speech recognition method as claimed in any one of claims 1 to 7.

10. One or more machine-readable media having executable codes stored thereon, which, when executed, enable a processor to perform the speech recognition method according to any one of claims 1 to 7.

Images