Speech recognition method, device, equipment and storage medium

Abstract

Disclosed are a speech recognition method, apparatus, device and storage medium. The method comprises: obtaining acoustic features of a speech signal, wherein the speech signal comprises a plurality of speech frames; inputting the acoustic features into a convolutional network to obtain convolutional features of the acoustic features; inputting the convolutional features into a long short-term memory (LSTM) recurrent neural network to obtain associated features of a plurality of speech frames output by the LSTM recurrent neural network; and obtaining a phoneme recognition result of a first speech frame before the set time length according to the associated features of a plurality of speech frames within a set time length.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a speech recognition method, apparatus, device and storage medium. Background Art

[0002] Speech recognition is currently widely used in many fields, such as smart speakers, smart cars, smart customer service, mobile phone voice assistants and other life service fields. In addition, speech recognition is also used in human-computer interaction and virtual human driving, such as virtual anchors and virtual idols, which all use related technologies.

[0003] Most scenarios involving speech, such as speech keyword detection, speech transcription, and speech signal driving, require algorithms to achieve streaming input / output. On the one hand, streaming input / output is consistent with the human brain's understanding process, and it is also a rigid demand for some real-time application scenarios. For example, in the process of speech transcription, along with the input of speech signals, readers hope to see real-time text output, rather than dozens or even hundreds of words appearing together after a piece of speech data is input.

[0004] Therefore, how to better achieve real-time speech recognition is an issue that needs to be actively studied. Summary of the invention

[0005] The embodiment of the present disclosure provides a speech recognition solution.

[0006] According to a first aspect of the present disclosure, a speech recognition method is provided, the method comprising: acquiring acoustic features of a speech signal, wherein the speech signal comprises a plurality of speech frames; inputting the acoustic features into a convolutional network to obtain convolutional features of the acoustic features; inputting the convolutional features into a long short-term memory (LSTM) recurrent neural network to obtain associated features of a plurality of speech frames output by the LSTM recurrent neural network; and obtaining a phoneme recognition result of the first speech frame before the set time length based on the associated features of a plurality of speech frames within a set time length.

[0007] In some embodiments, the method of obtaining the phoneme recognition result of the first speech frame before the set time length based on the associated features of multiple speech frames within the set time length includes: inputting the associated features of multiple speech frames output by the LSTM recurrent neural network into a feature queue in real time, wherein the maximum capacity of the feature queue is determined according to the set time length; when the number of associated features in the feature queue reaches the maximum capacity, in response to the entry of associated features, outputting the associated features of the speech frame that first enters the feature queue, wherein the speech frame that first enters the feature queue is the first speech frame before the set time length; and determining the phoneme recognition result of the first speech frame based on the associated features of multiple speech frames in the feature queue.

[0008] In some embodiments, the method further includes: after inputting the associated features of multiple speech frames output by the LSTM recurrent neural network into the feature queue in real time, inputting a set number of speech frames with zero features into the feature queue, wherein the set number is equal to the maximum capacity of the feature queue.

[0009] In some embodiments, determining the phoneme recognition result of the first speech frame based on the associated features of the multiple speech frames in the feature queue includes: inputting the associated features of the first speech frame and the associated features of the multiple speech frames in the feature queue into the prediction network at the same time, the prediction network predicting the target features of the first speech frame based on the associated features of the multiple speech frames in the feature queue; performing a full connection operation on the target features to obtain the phoneme posterior probability of the first speech frame.

[0010] In some embodiments, the speech recognition method is performed using a sound coding network, which includes at least the LSTM recurrent sound coding network, the convolutional network and the prediction network, and the sound coding network is trained using triphone samples.

[0011] In some embodiments, the method of obtaining the phoneme recognition result of the first speech frame before the set time length based on the associated features of multiple speech frames within the set time length includes: obtaining the triphoneme recognition result of the first speech frame before the set time length based on the associated features of multiple speech frames within the set time length; obtaining the monophoneme recognition result of the first speech frame based on the triphoneme recognition result of the first speech frame.

[0012] In some implementations, the sound coding network further includes a normalization module, and the normalization module is used to perform a normalization operation on the acoustic features of the speech signal before obtaining the convolution features of the acoustic features.

[0013] In some embodiments, the sound coding network further includes an activation module, which is used to perform an activation operation on the convolutional features before inputting the convolutional features into the long short-term memory recurrent neural network.

[0014] In some implementations, the method further includes: after obtaining the phoneme recognition result of the first speech frame before the set time length, clearing the associated features before the set time length.

[0015] According to a second aspect of the present disclosure, a speech recognition device is provided, the device comprising: a first acquisition unit, used to acquire acoustic features of a speech signal, wherein the speech signal comprises a plurality of speech frames; a second acquisition unit, used to input the acoustic features into a convolutional network to obtain convolutional features of the acoustic features; a third acquisition unit, used to input the convolutional features into a long short-term memory (LSTM) recurrent neural network to obtain associated features of a plurality of speech frames output by the LSTM recurrent neural network; and a recognition unit, used to obtain a phoneme recognition result of the first speech frame before a set time length according to the associated features of a plurality of speech frames within the set time length.

[0016] In some embodiments, the recognition unit is specifically used to: input the associated features of multiple speech frames output by the LSTM recurrent neural network into a feature queue in real time, and the maximum capacity of the feature queue is determined according to the set time length; when the number of associated features in the feature queue reaches the maximum capacity, in response to the entry of associated features, output the associated features of the speech frame that first enters the feature queue, wherein the speech frame that first enters the feature queue is the first speech frame before the set time length; determine the phoneme recognition result of the first speech frame according to the associated features of multiple speech frames in the feature queue.

[0017] In some embodiments, the device also includes an output unit for inputting a set number of speech frames with zero features into the feature queue after the associated features of multiple speech frames output by the LSTM recurrent neural network are input into the feature queue in real time, wherein the set number is equal to the maximum capacity of the feature queue.

[0018] In some embodiments, the recognition unit is specifically used to: input the associated features of the first speech frame and the associated features of multiple speech frames in the feature queue into the prediction network at the same time, the prediction network predicts the target features of the first speech frame based on the associated features of the multiple speech frames in the feature queue; perform a full connection operation on the target features to obtain the phoneme posterior probability of the first speech frame.

[0019] In some embodiments, the speech recognition device is applied to a sound coding network, which includes at least the LSTM recurrent sound coding network, the convolutional network and the prediction network, and the sound coding network is trained using triphone samples.

[0020] In some embodiments, the recognition unit is specifically used to: obtain a triphone recognition result of the first speech frame before the set time length based on the associated features of multiple speech frames within the set time length; and obtain a monophone recognition result of the first speech frame based on the triphone recognition result of the first speech frame.

[0021] In some implementations, the sound coding network further includes a normalization module, and the normalization module is used to perform a normalization operation on the acoustic features of the speech signal before obtaining the convolution features of the acoustic features.

[0022] In some embodiments, the sound coding network further includes an activation module, which is used to perform an activation operation on the convolutional features before inputting the convolutional features into the long short-term memory recurrent neural network.

[0023] In some embodiments, the device further includes a clearing unit for clearing associated features before the set time length after obtaining the phoneme recognition result of the first speech frame before the set time length.

[0024] According to a third aspect of the present disclosure, an electronic device is provided, the device comprising a memory and a processor, the memory being used to store computer instructions executable on the processor, the processor being used to implement the speech recognition method described in any embodiment provided by the present disclosure when executing the computer instructions.

[0025] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the speech recognition method described in any embodiment provided by the present disclosure is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate one or more embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in one or more embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0027] Figure 1 is a flow chart of a speech recognition method proposed by at least one embodiment of the present disclosure;

[0028] Figure 2A is a schematic diagram of a speech recognition process proposed by at least one embodiment of the present disclosure;

[0029] Figure 2B is a structural diagram of a sound coding network shown in at least one embodiment of the present disclosure;

[0030] Figure 3 is a structural schematic diagram of a speech recognition device proposed in at least one embodiment of the present disclosure;

[0031] Figure 4It is a schematic diagram of the structure of an electronic device proposed in at least one embodiment of the present disclosure. DETAILED DESCRIPTION

[0032] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0033] The term "and / or" herein is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can represent including any one or more elements selected from the set consisting of A, B, and C.

[0034] At least one embodiment of the present disclosure provides a speech recognition method, which can be executed by an electronic device such as a terminal device or a server. The terminal device can be a fixed terminal or a mobile terminal, such as a mobile phone, a tablet computer, a game console, a desktop computer, an advertising machine, an all-in-one machine, a vehicle-mounted terminal, etc. The server includes a local server or a cloud server, etc. The method can also be implemented by a processor calling computer-readable instructions stored in a memory.

[0035] Figure 1 A flow chart of a speech recognition method according to at least one embodiment of the present disclosure is shown. Figure 1 As shown, the method includes steps 101 to 104.

[0036] In step 101, the acoustic features of a speech signal are acquired.

[0037] The speech signal includes a plurality of speech frames, and the plurality of speech frames are time-sequential.

[0038] The acoustic features may be Mel Frequency Cofficient (MFCC), Fbank features, and the like.

[0039] In the disclosed embodiment, the acoustic features of each speech frame may be acquired by means of a sliding window.

[0040] For example, for a 300 ms speech signal, a sliding window is performed with a step size of 10 ms, and 30 frames of acoustic features can be obtained, and the acoustic features are, for example, 80-dimensional Fbank features.

[0041] In step 102, the acoustic features are input into a convolutional network to obtain convolutional features of the acoustic features.

[0042] See also Figure 2A The acoustic features are streamed into the sound coding network 20, specifically, into the convolutional network 201 in the sound coding network 20, that is, the convolutional features of each speech frame are obtained by convolution operation on the acoustic features of the speech frame.

[0043] The convolution network 201 may be a one-dimensional convolution layer (conv1d), which may be used to perform high-dimensional feature extraction on the acoustic features.

[0044] In step 103, the convolutional features are input into the long short-term memory (LSTM) recurrent neural network 202 in the sound coding network 20 to obtain the associated features of multiple speech frames output by the LSTM recurrent neural network 202.

[0045] Since the LSTM recurrent neural network is a multiple copy of the same neural network, the neural network at each moment will pass information to the next moment. Therefore, the LSTM recurrent neural network can learn the historical information of the input convolution features, and the output associated feature information includes the impact of historical information on current information.

[0046] In step 104, based on the associated features of multiple speech frames before the set time length, the phoneme recognition result of the first speech frame before the set time length is obtained.

[0047] The set duration can be the duration of a time period calculated from the current moment of receiving the voice signal, or the duration of a time period calculated from other moments before the current moment. The associated features of the multiple voice frames before the set duration are the associated features of the voice frames received in this time period; the first voice frame before the set duration is the most recently received voice frame before this time period.

[0048] According to the correlation features of the multiple speech frames before the set time length, the phoneme category of the first speech frame before the set time length can be more accurately predicted and judged, thereby obtaining a more accurate phoneme recognition result. In addition, for the speech signal acquired in real time, the phoneme recognition result of the first speech frame before the set time length is output, which is equivalent to outputting the phoneme recognition result of each speech frame with a set delay, that is, streaming output with a set delay, so as to realize real-time processing of the speech signal.

[0049] In the disclosed embodiment, the acoustic features of the speech signal are first acquired, and then the convolutional features of the acoustic features are obtained by using a convolutional network, and the convolutional features are input into a long short-term memory (LSTM) recurrent neural network to obtain the associated features of multiple speech frames contained in the speech signal. Finally, based on the associated features of multiple speech frames before a set time length, the phoneme recognition result of the first speech frame before the set time length is obtained, which can improve the accuracy of the phoneme recognition result while realizing real-time processing of the speech signal.

[0050] In some embodiments, the feature queue can be used to operate the associated features of multiple speech frames output by the LSTM recurrent neural network to achieve a set time delay, and obtain the associated features of multiple speech frames before the set time length, as well as the phoneme recognition result of the first speech frame before the set time length. Figure 2A Schematic diagram of the speech recognition method shown.

[0051] The acoustic features are streamed into the convolutional network 201, the convolutional features output by the convolutional network 201 are input into the LSTM recurrent neural network 202, and the LSTM recurrent neural network 202 streams out the associated features of the speech frames.

[0052] The associated features of the speech frames output by the LSTM recurrent neural network 202 are sequentially entered into the feature queue 20. That is, the associated features of each speech frame output by the LSTM recurrent neural network 202 enter the feature queue 20 in real time.

[0053] The maximum capacity of the feature queue 20 is determined according to the set duration. For example, when the set duration is 300ms and the duration of each frame is 10ms, the maximum capacity of the feature queue can be set to 30. That is, the feature queue can accommodate the associated features of 30 voice frames at most.

[0054] The feature queue can be set to a first-in-first-out format. When the associated features of the speech frames in the feature queue have reached the maximum capacity, for example, there are already 30 associated features, when another associated feature of a speech frame enters the feature queue, the associated features of the speech frame that first enters the feature queue are dequeued. The associated features of the speech frame that first enters the feature queue are the first speech frame before the set duration. As the associated features of new speech frames continue to enter the feature queue, the associated features of the first speech frame at the current moment are continuously dequeued, thereby realizing the streaming output of the associated features of the speech frame with the set duration as a delay. After subsequent recognition processing of the associated features, the streaming output of the speech recognition results is realized.

[0055] The phoneme recognition result of the first speech frame can be determined based on the associated features of the multiple speech frames in the feature queue.

[0056] In the embodiment of the present disclosure, by utilizing the delay of the feature queue setting time length, the associated features of multiple speech frames of the set time length and the associated features of the first speech frame before the set time length can be obtained simultaneously, and the real-time output of the associated features of the speech frames can be achieved.

[0057] In one example, after the associated features of multiple speech frames output by the LSTM recurrent neural network are input into the feature queue in real time, a set number of speech frames with zero features are input into the feature queue, wherein the set number is equal to the maximum capacity of the feature queue.

[0058] Taking the acoustic feature as an 80-dimensional Fbank feature and the maximum capacity of the feature queue as N as an example, N*80 zeros are added after the input associated feature.

[0059] By supplementing speech frames with zero features as many as the maximum capacity after the associated features of multiple input speech frames, the last associated feature in the feature queue can also be dequeued, thereby making the speech recognition result more complete.

[0060] In some embodiments, an advance prediction module may be used to predict the phoneme recognition result of the first speech frame based on the associated features of the multiple speech frames in the feature queue. Figure 2A As shown in the schematic diagram of the speech recognition method, the sound coding network 20 also includes a prediction network 203 and a fully connected network 204.

[0061] The associated features of the first speech frame output from the feature queue 20 and the associated features of multiple speech frames in the feature queue are simultaneously input to the prediction network 203. That is, when the number of associated features in the feature queue reaches the maximum capacity, when another associated feature of a speech frame enters the feature queue, the first speech frame output from the feature queue is directly input to the prediction network 203, and the associated features of all speech frames in the feature queue are simultaneously input to the prediction network. The prediction network predicts the target features of the first speech frame of the output queue based on the associated features of all speech frames in the received feature queue, and outputs the target features.

[0062] Next, the target feature is input into the fully connected network 204 for full connection operation to obtain the phoneme posterior probability of the first speech frame, that is, to determine the phoneme category of the first speech frame, wherein the phoneme category indicates which phoneme in the phoneme table the phoneme belongs to.

[0063] In the embodiments of the present disclosure, by combining the LSTM recurrent neural network with the prediction network, under the premise of ensuring that past information is not forgotten, future information with a set delay is obtained to predict the current phoneme posterior probability, which can improve the recognition accuracy compared to the conventional unidirectional LSTM recurrent neural network, and improve the ability to output phoneme recognition results in real time compared to the bidirectional LSTM recurrent neural network.

[0064] In some embodiments, the sound coding network 20 can be trained using triphone samples. In this case, the sound coding network 20 outputs the triphone recognition result of the first speech frame before the set time length, and then obtains the final monophone recognition result based on the triphone recognition result.

[0065] Compared with monophones, triphones take into account the context. That is, compared with the monophone classification results, the state of triphones not only represents the state of the current frame, but also the states of the previous and next frames. When the recognition result is triphones, the phoneme detection effect is improved.

[0066] In some embodiments, the triphone samples may be frame-level phonemes obtained based on the Kaldi-MFA phoneme recognition method. The accuracy of the frame-level phonemes obtained in this way is relatively high, which can further improve the training effect of the sound coding network.

[0067] Among them, the three-phoneme sample can be selected from the P2FA phoneme table. On the one hand, this type of phoneme table has a small number of phonemes, which can improve training efficiency, and the phoneme table has better effects in some application scenarios, such as in the expression driving of digital humans, which can obtain a more delicate driving effect.

[0068] In some embodiments, the sound coding network may further include a normalization module. Figure 2B The structural diagram of the sound coding network shown in the figure may also include a normalization module 201A before the convolution network 201, and the normalization module 201A is used to normalize the acoustic features of the speech signal before obtaining the convolution features of the acoustic features.

[0069] In some embodiments, the sound coding network may further include an activation module. Figure 2B The structural diagram of the sound coding network shown in the figure may also include an activation module 201B after the convolution network 201, and the activation module 201B is used to activate the convolution feature before inputting the convolution feature into the long short-term memory recurrent neural network.

[0070] In some embodiments, after obtaining the phoneme recognition result of the first speech frame before the set time length, the associated features before the set time length are cleared. Figure 2AAs shown in the schematic diagram of the sound recognition method, the associated features of the speech frames streamed out of the LSTM recurrent neural network 202 and the associated features of the speech frames outside the feature queue, such as the associated features of each speech frame that has exited the feature queue, can be deleted from the storage area to ensure that the data that has been used is not accumulated.

[0071] The present disclosure also proposes a method for engineering implementation of a sound coding network. In the process of engineering implementation, the sound coding network is divided into three modules for implementation. The first module includes a convolutional network and an LSTM recurrent neural network; the second module includes a prediction network; and the third module includes a fully connected network. By splitting the sound coding network into three parts for operational implementation, real-time control of the associated features of the speech frame in and out of the queue is achieved, meeting the requirements for real-time, low-latency input and output of speech recognition.

[0072] In addition, for the Fbank feature, since there are differences in data precision between the python toolkit and C++, the present disclosure implements a speech recognition process based on Fbank by unifying the calculation logic and decimal precision of Fbank in python and C++.

[0073] Figure 3 is a schematic diagram of the structure of a speech recognition device according to at least one embodiment of the present disclosure, such as Figure 3 As shown, the device may include: a first acquisition unit 301, used to acquire acoustic features of a speech signal, wherein the speech signal includes multiple speech frames; a second acquisition unit 302, used to input the acoustic features into a convolutional network to obtain convolutional features of the acoustic features; a third acquisition unit 303, used to input the convolutional features into a long short-term memory (LSTM) recurrent neural network to obtain associated features of multiple speech frames output by the LSTM recurrent neural network; and a recognition unit 304, used to obtain a phoneme recognition result of the first speech frame before the set time length based on the associated features of multiple speech frames within the set time length.

[0074] In some embodiments, the recognition unit is specifically used to: input the associated features of multiple speech frames output by the LSTM recurrent neural network into a feature queue in real time, and the maximum capacity of the feature queue is determined according to the set time length; when the number of associated features in the feature queue reaches the maximum capacity, in response to the entry of associated features, output the associated features of the speech frame that first enters the feature queue, wherein the speech frame that first enters the feature queue is the first speech frame before the set time length; determine the phoneme recognition result of the first speech frame according to the associated features of multiple speech frames in the feature queue.

[0075] In some embodiments, the device also includes an output unit for inputting a set number of speech frames with zero features into the feature queue after the associated features of multiple speech frames output by the LSTM recurrent neural network are input into the feature queue in real time, wherein the set number is equal to the maximum capacity of the feature queue.

[0076] In some embodiments, the recognition unit is specifically used to: input the associated features of the first speech frame and the associated features of multiple speech frames in the feature queue into the prediction network at the same time, the prediction network predicts the target features of the first speech frame based on the associated features of the multiple speech frames in the feature queue; perform a full connection operation on the target features to obtain the phoneme posterior probability of the first speech frame.

[0077] In some embodiments, the speech recognition device is applied to a sound coding network, which includes at least the LSTM recurrent sound coding network, the convolutional network and the prediction network, and the sound coding network is trained using triphone samples.

[0078] In some embodiments, the recognition unit is specifically used to: obtain a triphone recognition result of the first speech frame before the set time length based on the associated features of multiple speech frames within the set time length; and obtain a monophone recognition result of the first speech frame based on the triphone recognition result of the first speech frame.

[0079] In some implementations, the sound coding network further includes a normalization module, and the normalization module is used to perform a normalization operation on the acoustic features of the speech signal before obtaining the convolution features of the acoustic features.

[0080] In some embodiments, the sound coding network further includes an activation module, which is used to perform an activation operation on the convolutional features before inputting the convolutional features into the long short-term memory recurrent neural network.

[0081] In some embodiments, the device further includes a clearing unit for clearing associated features before the set time length after obtaining the phoneme recognition result of the first speech frame before the set time length.

[0082] At least one embodiment of the present disclosure further provides an electronic device, such as Figure 4 As shown, the device includes a memory and a processor, the memory is used to store computer instructions that can be executed on the processor, and the processor is used to implement the speech recognition method described in any embodiment of the present disclosure when executing the computer instructions.

[0083] At least one embodiment of the present disclosure further provides a computer-readable storage medium having a computer program stored thereon, and when the program is executed by a processor, the speech recognition method described in any embodiment of the present disclosure is implemented.

[0084] At least one embodiment of the present disclosure further provides a computer program product, including a computer program, which, when executed by a processor, implements the speech recognition method described in any embodiment of the present disclosure.

[0085] It should be understood by those skilled in the art that one or more embodiments of the present specification may be provided as a method, system or computer program product. Therefore, one or more embodiments of the present specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, one or more embodiments of the present specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0086] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the data processing device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0087] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0088] The embodiments of the subject matter and functional operations described in this specification may be implemented in the following: digital electronic circuits, tangibly embodied computer software or firmware, computer hardware including the structures disclosed in this specification and their structural equivalents, or a combination of one or more of them. The embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules in computer program instructions encoded on a tangible non-temporary program carrier to be executed by a data processing device or to control the operation of the data processing device. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagation signal, such as a machine-generated electrical, optical or electromagnetic signal, which is generated to encode information and transmit it to a suitable receiver device for execution by a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

[0089] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform corresponding functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuits, such as FPGAs (field programmable gate arrays) or ASICs (application-specific integrated circuits), and the apparatus can also be implemented as special purpose logic circuits.

[0090] Computers suitable for executing computer programs include, for example, general and / or special microprocessors, or any other type of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory and / or a random access memory. The basic components of a computer include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, the computer will also include one or more large-capacity storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or the computer will be operably coupled to this large-capacity storage device to receive data from it or to transmit data to it, or both. However, the computer does not necessarily have such a device. In addition, the computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few.

[0091] Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CDROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0092] Although this specification includes many specific implementation details, these should not be interpreted as limiting the scope of any invention or the scope of protection claimed, but are mainly used to describe the features of the specific embodiments of specific inventions. Certain features described in multiple embodiments in this specification may also be implemented in combination in a single embodiment. On the other hand, the various features described in a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. In addition, although features may work in certain combinations as described above and even initially claim protection, one or more features from the claimed combination may be removed from the combination in some cases, and the claimed combination may point to a sub-combination or a variation of a sub-combination.

[0093] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that these operations be performed in the particular order shown or performed sequentially, or requiring that all illustrated operations be performed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product, or packaged into multiple software products.

[0094] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the particular order or sequential order shown to achieve the desired results. In some implementations, multitasking and parallel processing may be advantageous.

[0095] The above description is merely a preferred embodiment of one or more embodiments of the present specification and is not intended to limit one or more embodiments of the present specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present specification shall be included in the scope of protection of one or more embodiments of the present specification.

Claims

1. A speech recognition method, characterized in that: The method comprises: Acquiring acoustic features of a speech signal, wherein the speech signal includes a plurality of speech frames; Input the acoustic features of each speech frame into the convolutional network to obtain the convolutional features of each speech frame; Inputting the convolution feature of each speech frame into the long short-term memory (LSTM) recurrent neural network, obtaining the associated features of each speech frame output by the LSTM recurrent neural network, wherein the associated features of each speech frame include the influence of the historical information corresponding to the convolution feature of the speech frame on the current information; According to the correlation features of multiple speech frames within the set time length, the phoneme recognition result of the first speech frame before the set time length is obtained, including: Inputting the associated features of the plurality of speech frames output by the LSTM recurrent neural network into a feature queue in real time, wherein the maximum capacity of the feature queue is determined according to the set duration; When the number of associated features in the feature queue reaches the maximum capacity, in response to the entry of associated features, outputting the associated features of the speech frame that first enters the feature queue, wherein the speech frame that first enters the feature queue is the first speech frame before the set time length; The phoneme recognition result of the first speech frame is determined according to the associated features of the first speech frame and the associated features of multiple speech frames in the feature queue.

2. The method according to claim 1, characterized in that The method further comprises: After the associated features of multiple speech frames output by the LSTM recurrent neural network are input into the feature queue in real time, a set number of speech frames with zero features are input into the feature queue, wherein the set number is equal to the maximum capacity of the feature queue.

3. The method according to claim 2, characterized in that The determining the phoneme recognition result of the first speech frame according to the associated features of the plurality of speech frames in the feature queue comprises: The associated features of the first speech frame and the associated features of the multiple speech frames in the feature queue are simultaneously input into the prediction network, and the prediction network predicts the target features of the first speech frame according to the associated features of the multiple speech frames in the feature queue; A full connection operation is performed on the target feature to obtain the phoneme posterior probability of the first speech frame.

4. The method according to claim 3, characterized in that The speech recognition method is performed using a sound coding network, which at least includes the LSTM cycle sound coding network, the convolutional network and the prediction network, and the sound coding network is trained using triphone samples.

5. The method according to claim 1, characterized in that The step of obtaining the phoneme recognition result of the first speech frame before the set time length according to the correlation features of the plurality of speech frames within the set time length comprises: According to the correlation features of multiple speech frames within the set time length, a triphone recognition result of the first speech frame before the set time length is obtained; According to the triphone recognition result of the first speech frame, a monophone recognition result of the first speech frame is obtained.

6. The method according to claim 4, characterized in that The sound coding network also includes a normalization module, which is used to perform a normalization operation on the acoustic features of the speech signal before obtaining the convolution features of the acoustic features.

7. The method according to any one of claims 4 to 6, characterized in that: The sound coding network also includes an activation module, which is used to activate the convolutional features before inputting the convolutional features into the long short-term memory recurrent neural network.

8. The method according to claim 1, characterized in that The method further comprises: After obtaining the phoneme recognition result of the first speech frame before the set time length, the associated features before the set time length are cleared.

9. A speech recognition device, characterized in that: The device comprises: A first acquisition unit, configured to acquire acoustic features of a speech signal, wherein the speech signal includes a plurality of speech frames; A second acquisition unit is used to input the acoustic features of each speech frame into the convolutional network to obtain the convolutional features of each speech frame; A third acquisition unit is used to input the convolution feature of each speech frame into a long short-term memory (LSTM) recurrent neural network to obtain the associated features of each speech frame output by the LSTM recurrent neural network, wherein the associated features of each speech frame include the influence of the historical information corresponding to the convolution feature of the speech frame on the current information; A recognition unit, configured to obtain a phoneme recognition result of a first speech frame before the set time length according to associated features of a plurality of speech frames within the set time length; The identification unit is specifically used for: Inputting the associated features of the plurality of speech frames output by the LSTM recurrent neural network into a feature queue in real time, wherein the maximum capacity of the feature queue is determined according to the set duration; When the number of associated features in the feature queue reaches the maximum capacity, in response to the entry of associated features, outputting the associated features of the speech frame that first enters the feature queue, wherein the speech frame that first enters the feature queue is the first speech frame before the set time length; The phoneme recognition result of the first speech frame is determined according to the associated features of the first speech frame and the associated features of multiple speech frames in the feature queue.

10. An electronic device, characterized in that: The device comprises a memory and a processor, wherein the memory is used to store computer instructions executable on the processor, and the processor is used to implement the method according to any one of claims 1 to 8 when executing the computer instructions.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images