A speech recognition method and a speech recognition device

Abstract

The embodiment of the present application provides a speech recognition method and a speech recognition device, the method comprising: performing windowing processing on a speech data stream to determine the speech data located in the window; performing object recognition processing on the speech data in the window, and adjusting the length of the window according to the result of the object recognition processing, and determining the speech data located in the adjusted window as a target speech segment; performing speech recognition processing on the target speech segment based on a recognition model to obtain a target recognition result. In this way, when performing windowing processing on the speech data stream, the window length is flexibly adjusted according to the result of the object recognition processing to obtain target speech segments of different sizes, which can take into account both the recognition speed and the recognition effect, thereby comprehensively improving the speech recognition performance of the end-to-end speech recognition scenario.

Description

Technical Field

[0001] The present application relates to the field of speech recognition technology, and in particular to a speech recognition method and a speech recognition device. Background Art

[0002] End-to-end speech recognition refers to a recognition technology whose input is a speech signal and whose output is a text signal. It is expected to directly realize speech input and decoding recognition without the need for complicated alignment work and pronunciation dictionary production work. It can save a lot of time and has application in various scenarios.

[0003] In the related art, a mainstream framework for end-to-end speech recognition is a neural network based on the self-attention model. However, this framework does not support streaming scenarios, that is, it does not support real-time recognition, which makes it difficult to apply it in practice. To address this problem, the speech stream is usually windowed in real-time speech recognition scenarios to obtain multiple speech segments (Chunks) so that streaming end-to-end speech recognition can be achieved using the Attention model. However, the results of windowing in the related art are not ideal, resulting in poor speech recognition performance. Summary of the invention

[0004] The embodiments of the present application provide a speech recognition method and a speech recognition device, which can take into account both recognition speed and recognition effect, thereby comprehensively improving the speech recognition performance of end-to-end speech recognition scenarios.

[0005] The technical solution of this application is implemented as follows:

[0006] In a first aspect, an embodiment of the present application provides a speech recognition method, the method comprising:

[0007] Performing windowing processing on the voice data stream to determine the voice data located within the window;

[0008] Performing object recognition processing on the speech data in the window, adjusting the length of the window according to the object recognition processing result, and determining the speech data in the adjusted window as the target speech segment;

[0009] The target speech segment is processed for speech recognition based on the recognition model to obtain the target recognition result.

[0010] In a second aspect, an embodiment of the present application provides a speech recognition device, the speech recognition device comprising a processing unit, an adjustment unit and a recognition unit; wherein:

[0011] A processing unit configured to perform windowing processing on the voice data stream and determine the voice data located within the window;

[0012] an adjustment unit configured to perform object recognition processing on the speech data in the window, adjust the length of the window according to the object recognition processing result, and determine the speech data in the adjusted window as the target speech segment;

[0013] The recognition unit is configured to perform speech recognition processing on the target speech segment based on the recognition model to obtain a target recognition result.

[0014] The embodiment of the present application provides a speech recognition method and a speech recognition device, which performs windowing processing on a speech data stream to determine the speech data within the window; performs object recognition processing on the speech data within the window, and adjusts the length of the window according to the result of the object recognition processing, and determines the speech data within the adjusted window as a target speech segment; performs speech recognition processing on the target speech segment based on a recognition model to obtain a target recognition result. In this way, when performing windowing processing on the speech data stream, the window length is flexibly adjusted according to the result of the object recognition processing to obtain target speech segments of different sizes, which can take into account both the recognition speed and the recognition effect, thereby comprehensively improving the speech recognition performance of the end-to-end speech recognition scenario. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A flowchart of a speech recognition method provided in an embodiment of the present application;

[0016] Figure 2 A schematic diagram of the structure of a speech recognition network provided in an embodiment of the present application;

[0017] Figure 3 A schematic diagram of the structure of another speech recognition network provided in an embodiment of the present application;

[0018] Figure 4 A schematic diagram of the structure of a speech recognition device provided in an embodiment of the present application;

[0019] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0020] Figure 6 A schematic diagram of the structure of another electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0021] The following will be combined with the accompanying drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. It is understood that the specific embodiments described herein are only used to explain the related applications, rather than to limit the applications. It should also be noted that, for the convenience of description, only the parts related to the related applications are shown in the accompanying drawings.

[0022] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0023] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0024] It should be pointed out that the terms "first\second\third" involved in the embodiments of the present application are only used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0025] It should be understood that end-to-end speech recognition is mainly divided into two frameworks, based on the RNN-T (Recurrent Neural Network Transducer) model and the self-attention Attention model, especially the Attention model. However, the Attention-based neural network (Transformer) needs to refer to context features, so it does not support streaming translation and cannot meet the requirements of actual application scenarios.

[0026] In order to solve this problem, it is generally necessary to perform windowing on the speech stream and divide the speech stream into multiple different speech segments (generally called Chunks) for input to construct context information. Here, the size of the Chunk is generally fixed. Studies have found that if the length of the Chunk is too large, it will cause a large delay, and if the Chunk is too small, the recognition result will be poor.

[0027] Based on this, the embodiment of the present application provides a speech recognition method, the basic idea of ​​which is: perform windowing processing on the speech data stream to determine the speech data located in the window; perform object recognition processing on the speech data in the window, and adjust the length of the window according to the object recognition processing result, and determine the speech data located in the adjusted window as the target speech segment; perform speech recognition processing on the target speech segment based on the recognition model to obtain the target recognition result. In this way, when performing windowing processing on the speech data stream, the window length is flexibly adjusted according to the object recognition processing result to obtain target speech segments of different sizes, which can take into account both the recognition speed and the recognition effect, thereby comprehensively improving the speech recognition performance of the end-to-end speech recognition scenario.

[0028] The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0029] In one embodiment of the present application, see Figure 1 , which shows a flow chart of a speech recognition method provided by an embodiment of the present application. Figure 1 As shown, the method may include:

[0030] S101: Perform windowing processing on the voice data stream to determine the voice data within the window.

[0031] It should be noted that the speech recognition method provided in the embodiments of the present application is applied to a variety of electronic devices, for example, the electronic devices may be computers, smart phones, tablet computers, laptop computers, PDAs, navigation devices, smart furniture, vehicle-mounted equipment, etc., and the embodiments of the present application do not specifically limit this.

[0032] In the embodiments of the present application, the speech recognition method is explained using end-to-end speech recognition as an application scenario, but this does not constitute a relevant limitation.

[0033] In end-to-end speech recognition, it is often necessary to perform windowing (or segmentation) on the speech data stream, so as to divide the continuous speech stream data into multiple speech segments (Chunks), and then perform speech recognition in chunks. For example, in the Attention-based speech recognition model, speech recognition needs to rely on context, but in the end-to-end speech scenario, the speech stream needs to be translated in real time, which lacks "context information"; through windowing, the real-time speech data stream is divided into different speech segments, and the speech segments before and after a certain speech segment can be regarded as the "context information" of the speech segment, so that the Attention-based speech recognition model can be applied to end-to-end speech recognition.

[0034] However, in the relevant embodiments, the size of the Chunk is fixed. The larger the Chunk, the higher the recognition accuracy, but the longer the delay; the smaller the Chunk, the faster the recognition speed, but the lower the accuracy. Therefore, how to reasonably set the size of the Chunk has become an urgent problem to be solved.

[0035] Therefore, an embodiment of the present application provides a speech recognition method that can flexibly adjust the size of the Chunk, thereby taking into account both the recognition speed and the recognition effect, thereby comprehensively improving the speech recognition performance of the end-to-end speech recognition scenario.

[0036] S102: performing object recognition processing on the speech data in the window, adjusting the length of the window according to the object recognition processing result, and determining the speech data in the adjusted window as the target speech segment.

[0037] It should be noted that when windowing the speech data stream, object recognition is performed on the speech data in the window, and the length of the window is adjusted based on the result of the object recognition to determine the target speech segment. Here, the length of the window refers to the number of speech frames contained in the window.

[0038] In the embodiments of the present application, the window is a virtual concept, which generally refers to the algorithm concept used to segment the voice data stream in the windowing process, and does not specify any entity concept. For example, the window can be a rectangular window (Rectwin), a Hamming window (Hamming), a Hanning window (Hann), etc.

[0039] It should be noted that object recognition processing refers to determining whether there is an identifiable object (or called the minimum identifiable unit, Token) in the voice data. It should be understood that a Chunk is only identifiable if there are one or more identifiable objects. In particular, the definition of Token is different depending on the usage scenario. For example, characters (Char), words (word), phonemes, etc. can be used as Tokens.

[0040] That is to say, in the speech recognition scenario, the length of the recognition object is uncertain and is affected by factors such as the speaker's pronunciation habits, word and voice characteristics, and the scene where the speech data stream is generated. However, for the speech recognition process, it is expected that each Chunk has at least one Token in order to achieve a better recognition effect and reduce the recognition delay.

[0041] In an embodiment of the present application, the length of the window is adjusted according to the object recognition processing result, and the voice data in the adjusted window is determined as a target Chunk, that is, the size of the Chunk can be flexibly adjusted according to the duration of the Token to ensure that each Chunk has at least one Token, thereby improving the effect of voice recognition.

[0042] In some embodiments, the window may be adjusted by extending the window. The adjusting the length of the window according to the object recognition processing result may include:

[0043] If the object recognition processing result indicates that the speech data in the window does not contain a recognition object, the length of the window is extended until the speech data in the adjusted window contains a recognition object;

[0044] If the object recognition processing result indicates that the speech data within the window contains a recognition object, the length of the window is kept unchanged.

[0045] It should be noted that if there is a Token in the voice data within the window, the voice data within the window can be determined as a Chunk and sent to the subsequent processing link; conversely, if there is no Token in the voice data within the window, the length of the window may need to be extended until there is an identification object in the voice data within the window. At this time, the voice data in the adjusted window is determined as a Chunk and sent to the subsequent processing link.

[0046] Furthermore, in some embodiments, the length of the extended window may include:

[0047] Adjust the length of the window to a preset multiple of the initial length.

[0048] It should be noted that when extending the window, the initial length can be extended each time. Taking the initial length of 5 frames as an example, if there is no token in the voice data in the window, the length of the window is increased by 5 frames, and the voice data in the window is re-detected. If there is still no token in the voice data in the window after the increase of 5 frames, the window after the increase of 5 frames will continue to increase by 5 frames... until there is a token in the voice data in the adjusted window.

[0049] In some other embodiments, the window may be adjusted by shortening or extending the window. The window length adjustment according to the object recognition processing result may include:

[0050] If the object recognition processing result indicates that the speech data within the window has a recognition object, the length of the window is shortened according to the position of the recognition object.

[0051] It should be noted that if there are tokens in the speech data within the window, the length of the window can be shortened according to the boundary position of the token, so as to obtain a chunk with less redundant data and improve the efficiency of speech recognition.

[0052] Furthermore, in some embodiments, after determining the target speech segment, the method may further include:

[0053] If the length of the adjusted window is different from the initial length of the window, the adjusted window is restored to its original length.

[0054] The window is slid on the voice data stream, and the step of determining the voice data located within the window is performed again.

[0055] It should be noted that after determining the target speech segment, if the window has been adjusted in length, the length of the window needs to be restored to the initial length; then the window continues to slide in the speech data stream to continue the windowing process.

[0056] Here, the window sliding distance is generally smaller than the window length, that is, different chunks may have overlapping speech frames, thus avoiding missing tokens.

[0057] It should also be noted that the criterion for judging whether there is an identification object in a piece of voice data can be determined according to the actual application scenario. For example, the end boundary of the Token can be used as the criterion for the existence of the Token, that is, as long as there is a boundary in the voice data at one end, it is determined that the Token exists in the voice data.

[0058] Therefore, in some embodiments, performing object recognition processing on the voice data in the window may include:

[0059] Perform frame-by-frame detection on the speech data within the window;

[0060] If it is detected that no object boundary exists in all frames, determining that the speech data in the object recognition processing result indication window does not have a recognized object;

[0061] If it is detected that at least one frame has an object boundary, it is determined that the object recognition processing result indicates that the speech data within the window has a recognized object.

[0062] It should be noted that in some other embodiments, the end boundary of two tokens may be used as the criterion for the existence of a token. In this case, only when at least two frames of the speech data in the window are object boundaries, will it be determined that the recognized object is detected.

[0063] It should be understood that object recognition processing can be implemented by a variety of models, such as gate networks, CTC models, Scotnet boundary models, Token duration models, etc.

[0064] To sum up, the embodiments of the present application provide a method that can divide a continuous data stream into data segments (Chunks) of different sizes. This method can be applied to the field of end-to-end speech recognition, as well as to the fields of image processing, signal processing, etc. Technicians can apply it to any scenario involving data segmentation according to the description of the embodiments of the present application, and these scenarios should be regarded as the protection scope of the embodiments of the present application.

[0065] Taking speech recognition as an example, the embodiments of the present application further provide a variety of different methods for adjusting the size of data segments, such as window extension, window shortening, window extension + window shortening, and several different examples are given below.

[0066] Example 1:

[0067] Set a window with a smaller initial length, for example, 5 frames;

[0068] The window is used to perform windowing processing on the voice data stream, and the voice data in the window is subjected to object recognition processing frame by frame;

[0069] If a certain speech frame is detected as an object boundary, the speech data frame located in the window is determined as a Chunk;

[0070] If all speech frames are not object boundaries, the length of the window is increased by 5 frames, and the process returns to the step of "performing object recognition processing on the speech data in the window frame by frame" until a speech frame is detected as an object boundary, and the speech data in the window at this time (i.e., the adjusted window) is determined as a Chunk.

[0071] Example 2:

[0072] Set a larger initial window length, for example, 15 frames;

[0073] The window is used to perform windowing processing on the voice data stream, and for the voice data located in the window, object recognition processing is performed frame by frame from the starting point (or the ending point);

[0074] If a speech frame is detected as an object boundary, the end edge of the window is shortened to the speech frame, and the speech data frame in the window is determined as a Chunk.

[0075] Example 3:

[0076] Set a window with a moderate initial length, for example, 10 frames;

[0077] The window is used to perform windowing processing on the voice data stream, and for the voice data located in the window, object recognition processing is performed frame by frame from the starting point (or the ending point);

[0078] If a speech frame is detected as an object boundary, the end edge of the window is shortened to the speech frame, and the speech data frame in the window is determined as a Chunk;

[0079] If all speech frames are not object boundaries, the length of the window is increased by 5 frames, and the process returns to the step of "performing object recognition processing on the speech data in the window frame by frame".

[0080] In this way, through the above methods, Chunks can be determined in the voice data stream, and the size of the Chunks can be flexibly changed with the length of the Token, so that each Chunk includes at least one Token, thereby improving the efficiency and accuracy of voice recognition.

[0081] S103: Perform speech recognition processing on the target speech segment based on the recognition model to obtain a target recognition result.

[0082] It should be noted that after obtaining the target speech segment, the target speech segment is sent to a subsequent recognition model (or called a speech recognition model) to obtain a target recognition result.

[0083] It should be understood that in the scenario of end-to-end speech recognition, the speech data stream will be windowed into multiple speech segments, and these speech segments will continuously enter the recognition model. In some embodiments, the recognition model will recognize the target speech segment in combination with the speech segments before and after the target speech segment.

[0084] Furthermore, in some embodiments, the recognition model includes an encoder and a decoder; performing speech recognition processing on the target speech segment based on the recognition model to obtain a target recognition result may include:

[0085] The target speech segment is encoded by using an encoder to obtain feature vector information;

[0086] The decoder is used to decode the feature vector information to obtain the target recognition result.

[0087] It should be noted that the encoder and decoder are the most common neural network architectures. Specifically, the encoder is used to determine the feature vector information of the target speech segment, and the decoder is used to perform decoding processing to determine the target recognition result.

[0088] Here, since the data fed into the encoder is in units of Chunks, the decoder can only decode in units of Chunks. However, in the field of speech recognition, a more efficient and stable decoding unit is Token, that is, the recognition model will recognize the target Token in combination with the tokens before and after the target Token. Therefore, in some embodiments, the recognition model also includes at least one position recognition model, and the decoder is used to decode the feature vector information to obtain the target recognition result, including:

[0089] Performing position recognition processing on the target speech segment using at least one position recognition model to obtain at least one boundary position information;

[0090] Performing confidence screening on at least one boundary position information to determine target boundary position information;

[0091] The target boundary position information and feature vector information are input into the decoder to obtain the target recognition result.

[0092] It should be noted that the boundary position information is used to indicate the specific position of the Token in each Chunk. In this way, the decoder can decode in units of Tokens according to the specific position of the Token, thereby improving the efficiency and accuracy of recognition.

[0093] At present, there are many location recognition models based on different principles for determining the specific location of the token in the voice data, but different location recognition models have their own advantages and disadvantages, and using only one location recognition model will inevitably reduce the accuracy. Therefore, in the embodiment of the present application, multiple location recognition models can also be used for location recognition processing, and then the result with the highest confidence is selected to determine the target boundary information, which can improve the accuracy of the boundary location information, and thus improve the accuracy of the recognition result.

[0094] Of course, in speed-first application scenarios, a location recognition model can also be used for location recognition processing. In this case, the confidence screening process can be skipped, and the result of the location recognition model is used to determine the target boundary location information.

[0095] Furthermore, the position recognition model includes at least one of the following: a connection time classification model, a boundary recognition model and a duration recognition model; using at least one position recognition model to perform position recognition processing on the target speech segment to obtain at least one boundary position information, including at least:

[0096] Using the connection time series classification model to perform object quantity analysis on the feature vector information to determine the first boundary position information;

[0097] Performing boundary position analysis on the target speech segment using a boundary recognition model to determine second boundary position information;

[0098] Using the duration recognition model to perform object duration analysis on the target speech segment to determine the third boundary position information;

[0099] The first boundary position information, the second boundary position information, and the third boundary position information are determined as at least one boundary position information.

[0100] It should be noted that the connectionist temporal classification model can be a CTC (Connectionist Temporal Classification) network, which can count the number of target speech segments and output confidence information; the boundary recognition model can be a Scotnet network, which can determine the boundary of a word (each word is regarded as a token) and output confidence information; the duration recognition model can be a Token duration network, which can analyze the duration of each token in the target speech segment and perform confidence scoring. In addition, the CTC network can also mark each frame of the target speech segment to indicate whether the current frame is blank, thereby determining the range of the token.

[0101] In other words, the connection time series classification model, boundary recognition model and duration recognition model can all provide boundary position information. Therefore, the confidence of the output results of the connection time series classification model, boundary recognition model and duration recognition model is compared, and the model with the highest confidence is selected to determine the final target boundary information, thereby improving the recognition accuracy.

[0102] In addition, the above position recognition model can also be selectively used for the object recognition process in step S102.

[0103] The embodiment of the present application provides a speech recognition method, which performs windowing processing on the speech data stream to determine the speech data located in the window; performs object recognition processing on the speech data in the window, and adjusts the length of the window according to the object recognition processing result, and determines the speech data located in the adjusted window as the target speech segment; performs speech recognition processing on the target speech segment based on the recognition model to obtain the target recognition result. In this way, on the one hand, the speech recognition method provided by the embodiment of the present application can determine the size of the Chunk according to the Token duration, taking into account the processing speed and recognition accuracy; on the other hand, the recognition model includes multiple position recognition models, and adopts a multi-level judgment method to determine the target boundary position information, thereby avoiding the boundary position deviation caused by a single judgment standard, and further improving the recognition accuracy.

[0104] In one embodiment of the present application, see Figure 2 , which shows a schematic diagram of the structure of a speech recognition network 20 provided in an embodiment of the present application. The speech recognition network 20 can execute the speech recognition method as described above.

[0105] like Figure 2 As shown, the speech recognition network 20 at least partially includes a speech input module 201, a chunk setting module 202, a gate network module 203, a chunk confirmation module 204, an encoder 205, a connection time series classification model 206, a decoder 207 and a recognition result output module 208.

[0106] The voice input module 201 is used to collect voice signals from terminal devices such as mobile phones and laptop computers to form a voice data stream.

[0107] The chunk setting module 202 is used to intercept a smaller chunk from the voice data stream according to the initial window length. For example, the preset length may be 5 frames.

[0108] The gate network module 203 is used to perform object recognition processing on the chunk output by the chunk setting module 202 to determine whether there is a recognition object in the current chunk; if there is no recognition object in the current chunk, the current chunk is increased by 5 frames and the object recognition processing is performed again until the recognition object is detected.

[0109] It should be noted that the gate network module 203 can be constructed by a variety of network models. Exemplarily, the gate network module 203 can be composed of several fully connected layers or convolutional neural networks (CNN). At this time, the gate network module 203 inputs a segment of voice data signal, and the gate network module 203 judges the voice data signal frame by frame, and outputs the judgment result of whether the current frame is the boundary of the Token.

[0110] The Chunk confirmation module 204 is used to confirm the size of the Chunk according to the output result of the gate network.

[0111] Here, if the output result of the gate network module 203 is that the current frame is not the boundary of a Token, the judgment of the next frame is looped. If all frames of the current Chunk do not judge the boundary of a Token, the length of the Chunk is increased to 2 times, 3 times, 4 times... and the object recognition process is continued after the length of the Chunk is increased; on the contrary, if the current frame is the boundary of a Token, the current Chunk is determined as the target Chunk and input into the encoder 205.

[0112] The encoder module 205 is used to encode the Chunk and output the hidden layer node, that is, the feature vector information of the Chunk.

[0113] Here, the encoder module 205 can be an encoder based on Conformer (a coding method), for example, the encoder can include a downsampling module, a position encoding module / / a self-attention module, a feed-forward module and a CNN module.

[0114] The connection time series classification model 206 is used to identify the specific location of the Token in the Chunk, that is, to determine the target boundary location information.

[0115] Here, the essence of the connection time series classification model 206 is a position recognition model, which can be implemented by the CTC network. The CTC network can obtain frame-level alignment information, and the CTC network can model the mapping relationship between two sequences of unequal lengths, because the CTC network adds a blank symbol (Blank) to the annotation symbol set, which means that this frame has no predicted value output, so in the prediction output based on the CTC network, it may contain many blank symbols, and finally perform blank removal and de-compression. This feature is suitable for sequence modeling problems in speech recognition research. The character sequence or phoneme sequence of each speech segment is known in advance, but the alignment relationship between them and the input feature sequence is uncertain, and the length of the general character sequence or phoneme sequence is much smaller than the length of the input frame sequence, so the CTC network can be used for speech sequence prediction.

[0116] The decoder 207 is used to decode the feature vector information according to the target boundary information to determine a decoding result.

[0117] The recognition result output module 208 is used to determine the target recognition result according to the decoding result.

[0118] Here, the decoder 207 may be a Decoder, and the recognition result output module 208 may be a scoring algorithm (eg, a Softmax algorithm).

[0119] In particular, the decoder and the recognition result output module can also be logically regarded as a whole. Exemplarily, the concept of the decoder in the aforementioned embodiment is actually the whole of the decoder 207 and the recognition result output module 208. In other words, it can be considered that the input of the decoder is feature vector information and target boundary information, and the output is the target recognition result.

[0120] It should also be noted that the decoder 207 can adopt an Attention-based architecture, and in this case, it is also necessary to refer to the context information of the Token to better determine the prediction result. Exemplarily, the decoder 207 can be composed of two Self-Attention modules, the first Self-Attention module input is a historical predicted Token sequence (i.e., the context information of the target Token), and the output is a vector representing the features of the next Token predicted based on the context; the second Self-Attention module input is the output of the first Self-Attention module and the output of the Encoder, and the Token label is predicted through Softmax to determine the final prediction result.

[0121] In summary, the speech recognition network 20 involves two major parts: Chunk determination and speech recognition.

[0122] On the one hand, the Chunk determination part includes a voice input module 201, a Chunk setting module 202, a gate network module 203, and a Chunk confirmation module 204, which are mainly used to flexibly determine the Chunk duration according to the Token duration, taking into account both the recognition speed and the recognition efficiency.

[0123] On the other hand, the speech recognition part (equivalent to the aforementioned recognition model) includes an encoder 205, a connection time series classification model 206, a decoder 207 and a recognition result output module 208. The encoder 205 and the decoder 207 are respectively composed of a multi-head self-attention module (Multi-Head Attention). After the input speech is processed by windowing to extract features, it enters the encoder 205 to generate feature vector information, and uses the CTC network to obtain a frame-based classification result (the classification result indicates whether each frame is an object boundary). When the CTC network outputs a non-Blank Token, the current frame node is input to the decoder 207 for decoding to predict the label of the next Token.

[0124] It should be understood that in the related art, dynamically sized chunks are often used when training neural networks. After training, the appropriate chunk size needs to be determined based on actual application requirements, mainly based on the degree of delay acceptance. For example, in an offline scenario, the chunk can be larger; in real-time, the chunk needs to be reduced as needed. However, the above method has the following disadvantages: when the chunk size is fixed during the actual processing of the voice data stream to be processed, no adjustment is made. If the chunk size is too large, it will cause a waste of resources, and if the chunk size is too small, the recognition performance will be reduced.

[0125] Therefore, in the embodiment of the present application, the modeling unit (including the Chunk setting module 202, the gate network module 203, and the Chunk confirmation module 204) is first used to estimate the Token duration. According to the length of the Token, different Chunks are used for Tokens of different durations, and different Chunks are used for decoding, which can reduce unnecessary resource waste and improve recognition accuracy.

[0126] Specifically, in an embodiment of the present application, a smaller fixed Chunk is used for input, and the Chunk is input into the gate network to predict the duration of the Token (Token can be a Word). According to the size of the Token (Word) duration, the size of the Chunk is adjusted to obtain the target Chunk, and then the target Chunk is sent to the encoder for feature extraction, and then the hidden layer information output by the encoder (equivalent to the feature vector information) is input into the Decoder to predict the current Token.

[0127] The embodiment of the present application provides a speech recognition method. The specific implementation method of the aforementioned embodiment is elaborated in detail through this embodiment. It can be seen that when the speech data stream is windowed, the window length is flexibly adjusted according to the result of the object recognition processing to obtain target speech segments of different sizes, that is, the size of the Chunk is determined according to the Token duration, which can take into account both the recognition speed and the recognition effect, thereby comprehensively improving the speech recognition performance of the end-to-end speech recognition scenario.

[0128] In yet another embodiment, see Figure 3 , which shows a schematic diagram of the structure of a speech recognition network 30 provided in an embodiment of the present application. The speech recognition network 30 can execute the speech recognition method as described above.

[0129] like Figure 3 As shown, the speech recognition model 30 at least partially includes a speech input module 301, a chunk setting module 302, a boundary recognition model 303, a duration recognition model 304, an encoder 305, a connection timing classification model 306, a confidence judgment module 307, a decoder 308 and a recognition result output module 309.

[0130] The voice input module 301 is used to collect voice signals from terminal devices such as mobile phones and notebooks to form a voice data stream.

[0131] The chunk setting module 302 is used to obtain multiple chunks by performing windowing processing on the voice data stream.

[0132] The boundary recognition model 303 is used to perform boundary recognition processing on each Chunk to obtain a boundary recognition result and a first confidence level.

[0133] Here, the boundary recognition model 303 can use a Scotnet network, which can be composed of several fully connected layers or CNN networks. The Scotnet network can judge the voice data in the Chunk frame by frame to determine which frame is the boundary of the Word; if the current Chunk does not have a Word boundary, the next Chunk is judged. In the embodiment of the present application, a Word is a Token.

[0134] The duration recognition model 304 is used to perform Token duration recognition processing on the target Chunk to obtain a duration recognition result and a second confidence level.

[0135] Here, the duration recognition model 304 can adopt a Token duration network, which can be composed of several layers of fully connected layers or CNN networks. The input is a speech signal, and the output is the Token duration in the speech signal and the confidence score of the Token duration.

[0136] The encoder (Encoder) 306 is used to encode the Chunk and output the information as a hidden layer node, that is, the feature vector of the Chunk.

[0137] Here, the encoder 305 may be an encoder based on Conformer (a coding method), for example, the encoder may include a downsampling module, a position encoding module, a self-attention module, a feed-forward module, and a CNN module.

[0138] The connection time series classification model 306 is used to perform token quantity statistics processing on each Chunk to obtain quantity statistics results and a third confidence level.

[0139] Here, the connection time series classification model 306 may use a CTC network. Here, unlike the Scotnet network and the Token duration network, the input of the CTC network preferentially uses the feature vector information of the Chunk.

[0140] The confidence judgment module 307 is used to compare the first confidence, the second confidence and the third confidence to determine the target boundary information.

[0141] Here, the Scotnet network, the Token Duration Network, and the CTC network can all be considered as location recognition models, which are used to determine the location of the Token in each Chunk. Therefore, for a specific Chunk, the confidence comparison can determine which of the above networks has the most accurate location recognition, and the final target boundary information can be determined based on the network with the highest confidence.

[0142] The decoder (Decoder) 308 is used to decode the feature vector information according to the target boundary information to determine the decoding result.

[0143] The recognition result output module 309 is used to determine the target recognition result according to the decoding result.

[0144] To summarize, in a specific embodiment, the peak of the CTC curve can be used to obtain the number of tokens, the Scotnet network can be used to determine the word boundary, the token duration network can be used to determine the token duration, and the target boundary information can be determined based on the network with the highest confidence, thereby improving the accuracy of the recognition result.

[0145] In the related art, according to the output of the encoder, the number of tokens, or the boundary of the word, or the duration of the token is generally used for the output of the encoder, and then input into the decoder for decoding. However, when a judgment criterion is used for judgment, the boundary judgment may be inaccurate, which directly affects the performance of the decoder and causes inaccurate recognition results.

[0146] In the embodiment of the present application, the number of tokens and word boundaries are judged by confidence, the results with high confidence are used for output, and the results of the encoder are confirmed according to the duration of the token. In this way, a multi-level hybrid method can be used to obtain more accurate encoder output information, thereby improving the recognition results.

[0147] The embodiment of the present application provides a speech recognition method. The specific implementation method of the aforementioned embodiment is elaborated in detail through this embodiment. It can be seen that the recognition model includes multiple position recognition models, and a multi-level judgment method is used to determine the target boundary position information, thereby avoiding boundary position deviation caused by a single judgment standard and further improving the recognition accuracy.

[0148] In yet another embodiment of the present application, see Figure 4 , which shows a schematic diagram of the structure of a speech recognition device 40 provided in an embodiment of the present application. Figure 4 As shown, the speech recognition device 40 includes a processing unit 401, an adjustment unit 402 and a recognition unit 403; wherein,

[0149] The processing unit 401 is configured to perform windowing processing on the voice data stream and determine the voice data located within the window;

[0150] An adjustment unit 402 is configured to perform object recognition processing on the speech data in the window, adjust the length of the window according to the object recognition processing result, and determine the speech data in the adjusted window as the target speech segment;

[0151] The recognition unit 403 is configured to perform speech recognition processing on the target speech segment based on the recognition model to obtain a target recognition result.

[0152] In some embodiments, the adjustment unit 402 is specifically configured to extend the length of the window until a recognition object exists in the voice data within the adjusted window if the object recognition processing result indicates that there is no recognition object in the voice data within the window; and to keep the length of the window unchanged if the object recognition processing result indicates that there is a recognition object in the voice data within the window.

[0153] In some embodiments, the adjustment unit 402 is further configured to shorten the length of the window according to the position of the recognized object if the object recognition processing result indicates that there is a recognized object in the voice data within the window.

[0154] In some embodiments, the adjusting unit 402 is further configured to perform a length recovery process on the adjusted window to obtain a window of the initial length if the length of the adjusted window is different from the initial length of the window;

[0155] Correspondingly, the processing unit 401 is further configured to slide the window on the voice data stream and execute the step of determining the voice data located in the window again.

[0156] In some embodiments, the adjustment unit 402 is further configured to perform frame-by-frame detection on the voice data within the window; if it is detected that no object boundaries exist in all frames, it is determined that the object recognition processing result indicates that there is no recognized object in the voice data within the window; if it is detected that there is an object boundary in at least one frame, it is determined that the object recognition processing result indicates that there is a recognized object in the voice data within the window.

[0157] In some embodiments, the adjusting unit 402 is further configured to adjust the length of the window to a preset multiple of the initial length.

[0158] In some embodiments, the recognition model includes an encoder and a decoder; the recognition unit 403 is specifically configured to use the encoder to encode the target speech segment to obtain feature vector information;

[0159] The decoder is used to decode the feature vector information to obtain the target recognition result.

[0160] In some embodiments, the recognition model also includes at least one position recognition model, and the recognition unit 403 is further configured to use at least one position recognition model to perform position recognition processing on the target speech segment to obtain at least one boundary position information; perform confidence screening on at least one boundary position information to determine the target boundary position information; input the target boundary position information and feature vector information into the decoder to obtain a target recognition result.

[0161] In some embodiments, the position recognition model includes at least one of the following: a connection timing classification model, a boundary recognition model and a duration recognition model; the recognition unit 403 is specifically configured to use the connection timing classification model to perform object quantity analysis on the feature vector information to determine the first boundary position information; use the boundary recognition model to perform boundary position analysis on the target speech segment to determine the second boundary position information; use the duration recognition model to perform object duration analysis on the target speech segment to determine the third boundary position information; determine the first boundary position information, the second boundary position information and the third boundary position information as at least one boundary position information.

[0162] It can be understood that in this embodiment, a "unit" can be a part of a circuit, a part of a processor, a part of a program or software, etc., and of course it can also be a module, or it can be non-modular. Moreover, the components in this embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional module.

[0163] If the integrated unit is implemented in the form of a software function module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment is essentially or the part that contributes to the prior art or the whole or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) or a processor to perform all or part of the steps of the method of this embodiment. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0164] Therefore, this embodiment provides a computer storage medium, which stores a computer program. When the computer program is executed by multiple processors, the steps of the method in any one of the above embodiments are implemented.

[0165] Based on the above-mentioned composition of a speech recognition device 40 and a computer storage medium, see Figure 5 , which shows a hardware structure diagram of an electronic device 50 provided in an embodiment of the present application. Figure 5 As shown, the electronic device 50 may include: a communication interface 501, a memory 502 and a processor 503; each component is coupled together via a bus device 504. It is understood that the bus device 504 is used to realize the connection and communication between these components. In addition to the data bus, the bus device 504 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, Figure 5 Various buses are marked as bus devices 504. Among them, the communication interface 501 is used to receive and send signals during the process of sending and receiving information between other external network elements;

[0166] A memory 502, used to store a computer program that can be run on the processor 503;

[0167] The processor 503 is used to execute, when running the computer program:

[0168] Performing windowing processing on the voice data stream to determine the voice data located within the window;

[0169] Performing object recognition processing on the speech data in the window, adjusting the length of the window according to the object recognition processing result, and determining the speech data in the adjusted window as the target speech segment;

[0170] The target speech segment is subjected to speech recognition processing based on the recognition model to obtain a target recognition result.

[0171] It can be understood that the memory 502 in the embodiment of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DRRAM). The memory 502 of the apparatus and method described in the present application is intended to include, but is not limited to, these and any other suitable types of memory.

[0172] The processor 503 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 503. The above processor 503 may be a general processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor to execute, or the hardware and software modules in the decoding processor are combined and executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 502, and the processor 503 reads the information in the memory 502 and completes the steps of the above method in combination with its hardware.

[0173] It is understood that the embodiments described in the present application can be implemented by hardware, software, firmware, middleware, microcode or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions of the present application or a combination thereof.

[0174] For software implementation, the technology of the present application can be implemented by a module (e.g., a process, a function, etc.) that performs the functions of the present application. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0175] Optionally, as another embodiment, the processor 503 is further configured to execute the steps of the method in any one of the aforementioned embodiments when running the computer program.

[0176] In yet another embodiment of the present application, based on the composition diagram of the above-mentioned speech recognition device 40, see Figure 6 , which shows a schematic diagram of the composition structure of another electronic device 50 provided in an embodiment of the present application. Figure 6 As shown, the electronic device 50 at least includes the speech recognition device 40 of any one of the aforementioned embodiments.

[0177] For the electronic device 50, since it includes a speech recognition device 40, when performing windowing processing on the speech data stream, the window length is flexibly adjusted according to the result of the object recognition processing to obtain target speech segments of different sizes, which can take into account both the recognition speed and the recognition effect, thereby comprehensively improving the speech recognition performance of the end-to-end speech recognition scenario.

[0178] The above are only preferred embodiments of the present application and are not intended to limit the protection scope of the present application.

[0179] It should be noted that, in this application, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0180] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0181] The methods disclosed in several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0182] The features disclosed in several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0183] The features disclosed in several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0184] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A speech recognition method, the method comprising: Performing windowing processing on the voice data stream to determine the voice data located within the window; Performing object recognition processing on the speech data in the window, adjusting the length of the window according to the object recognition processing result, and determining the speech data in the adjusted window as the target speech segment; the object recognition processing is to determine whether there is a recognition object in the speech data in the window; The target speech segment is subjected to speech recognition processing based on the recognition model to obtain a target recognition result.

2. The speech recognition method according to claim 1, wherein the step of adjusting the length of the window according to the object recognition processing result comprises: If the object recognition processing result indicates that the voice data in the window does not contain a recognition object, extending the length of the window until the voice data in the adjusted window contains a recognition object; If the object recognition processing result indicates that the speech data within the window contains a recognition object, the length of the window is kept unchanged.

3. The speech recognition method according to claim 1, wherein the step of adjusting the length of the window according to the object recognition processing result further comprises: If the object recognition processing result indicates that there is a recognition object in the voice data within the window, the length of the window is shortened according to the position of the recognition object.

4. The speech recognition method according to claim 2 or 3, after determining the target speech segment, the method further comprises: If the length of the adjusted window is different from the initial length of the window, performing length recovery processing on the adjusted window to obtain the window of the initial length; The window is slid on the voice data stream, and the step of determining the voice data within the window is performed again.

5. The speech recognition method according to claim 2, wherein the performing object recognition processing on the speech data in the window comprises: Performing frame-by-frame detection on the speech data within the window; If it is detected that no object boundary exists in all frames, determining that the object recognition processing result indicates that the speech data in the window does not have a recognized object; If it is detected that at least one frame has an object boundary, it is determined that the object recognition processing result indicates that the speech data within the window has a recognized object.

6. The speech recognition method according to claim 2, wherein the extending the length of the window comprises: The length of the window is adjusted to a preset multiple of the initial length.

7. The speech recognition method according to claim 1, wherein the recognition model comprises an encoder and a decoder; and performing speech recognition processing on the target speech segment based on the recognition model to obtain a target recognition result comprises: Using the encoder to encode the target speech segment to obtain feature vector information; The decoder is used to decode the feature vector information to obtain the target recognition result.

8. The speech recognition method according to claim 7, wherein the recognition model further comprises at least one position recognition model, and the decoding process of the feature vector information by the decoder to obtain the target recognition result comprises at least: Performing position recognition processing on the target speech segment using the at least one position recognition model to obtain at least one boundary position information; Performing confidence screening on the at least one boundary position information to determine target boundary position information; The target boundary position information and the feature vector information are input into the decoder to obtain the target recognition result.

9. The speech recognition method according to claim 8, wherein the position recognition model comprises at least one of the following: a connection time classification model, a boundary recognition model and a duration recognition model; wherein the at least one position recognition model is used to perform position recognition processing on the target speech segment to obtain at least one boundary position information, comprising at least: Performing object quantity analysis on the feature vector information using a connection temporal classification model to determine first boundary position information; Performing boundary position analysis on the target speech segment using the boundary recognition model to determine second boundary position information; Performing object duration analysis on the target speech segment using the duration recognition model to determine third boundary position information; The first boundary position information, the second boundary position information and the third boundary position information are determined as the at least one boundary position information.

10. A speech recognition device, comprising a processing unit, an adjustment unit and a recognition unit; wherein: The processing unit is configured to perform windowing processing on the voice data stream to determine the voice data located within the window; The adjustment unit is configured to perform object recognition processing on the voice data in the window, and adjust the length of the window according to the object recognition processing result, and determine the voice data in the adjusted window as the target voice segment; the object recognition processing is to determine whether there is a recognition object in the voice data in the window; The recognition unit is configured to perform speech recognition processing on the target speech segment based on a recognition model to obtain a target recognition result.

Images