Human voice detection voice recognition method and system based on low-resource scene, and storage medium

Abstract

The invention discloses a human voice detection voice recognition method based on a low-resource scene. An acoustic model is divided into a network shallow layer and a network deep layer; adding human voice detection neurons in the network shallow layer; voice data is judged to contain voice by using computing power resources of the network shallow layer and the voice detection neurons; when the phoneme duration time of the voice data is within a preset duration time range; and calling network deep computing power resources to identify the voice data. Therefore, when the voice data passes through the neural network, the non-stationary noise can be effectively recognized under the condition of low computing power resources, and meanwhile, the misrecognition of non-instruction voice can be greatly reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of speech recognition, and in particular to a human voice detection and speech recognition method, system and storage medium based on low-resource scenarios. Background Art

[0002] Speech recognition is also known as speech-to-text. Traditional speech recognition technology is generally composed of acoustic models and language models. The acoustic model is generally trained using a deep neural network. When deploying, you only need to send the speech into the trained neural network for forward reasoning, and then use Viterbi decoding to get the recognition result. However, in side-end speech recognition devices, such as smart speakers and smart desk lamps, there are limited computing resources and strict power consumption restrictions. The calculation of the acoustic model and language model of speech recognition will occupy most of the computing power and resources. Generally speaking, in actual application scenarios, the device is required to be online 24 hours a day. Therefore, the human voice detection algorithm will be used. After the human voice is detected, the acoustic model and decoder will start running. After the human voice ends, the decoding will stop and the recognition result will be returned, thereby saving unnecessary energy consumption and detection.

[0003] Human voice detection algorithms can basically be divided into two categories: 1. Based on traditional signal features. 2. Based on neural networks. In actual scenarios, traditional signal processing methods such as spectrum short-time energy threshold detection have the following shortcomings: a. It cannot distinguish non-stationary noise (keyboard tapping, door squeak, etc.) well; b. The distinction is not strong under long-distance and low signal-to-noise ratio conditions. c. Traditional signal processing algorithms make many simplifications and assumptions for the sake of calculation simplicity, such as the frequency band independence assumption. These assumptions are difficult to strictly meet in actual application scenarios. The neural network-based method solves the above shortcomings very well. However, the end-side device already has a deep neural network model for speech recognition. Deploying an additional trained human voice detection model will not only waste resources but also increase additional computing power overhead and reduce the real-time performance of the system. Summary of the invention

[0004] Based on the above situation, the present invention proposes a voice activity detection (VAD) speech recognition solution, which not only utilizes the recognition ability of the deep neural network model for non-stationary noise, but also does not increase additional computing power overhead, and is very suitable for recognition in resource-constrained scenarios.

[0005] The present invention provides a method for acoustic detection speech recognition based on low-resource scenarios, which is divided into the following two parts:

[0006] The first part is to use transfer learning to freeze the acoustic model in speech recognition, that is, the first third of the time-delay neural network TDNN. The neural network can be regarded as a feature extractor of the speech signal, which maps the sound signal to the distribution of the phoneme state through the softmax function. Due to the structural characteristics of the hierarchical series of the neural network, the neural network closer to the shallower layer usually extracts some low-level features, such as the local texture frequency structure of the Mel spectrogram, etc. These features are common to the same category of tasks, so they are transferable. The deep features are more condensed and abstract, closer to the specific task, and therefore specific. For the VAD-specific part, the approach taken is to add a human voice detection output layer with only 3 neurons on the basis of freezing the shallow layer. The 3 neurons represent 3 states, namely non-human voice state, human voice state, and possible human voice state.

[0007] The second part is to encode the human voice detection state into the speech recognition decoding graph through the combination algorithm of the weighted finite state machine (WFST). Since the essence of human voice detection is also the transition of state, it is well coupled with the weighted state machine algorithm. In the voice command recognition scenario, there will be cases of false awakening by human voice, which will bring a poor user experience. By encoding the duration of the voice command word into the state graph, the misrecognition of the voice command word can be well suppressed. For example, the duration of "Hello Xiaolan" is 1s-3s. The voice that is not within this pronunciation duration range will not be received by the state machine, so the misrecognition of non-command human voice (such as news broadcast) can be greatly reduced.

[0008] Some technical effects of the present disclosure are: by stratifying the neural network according to resource characteristics, adding human voice detection neurons and speech recognition neurons to the stratified neural network and training them, so that when the voice data passes through the neural network, non-stationary noise can be effectively recognized under low computing power resources, and the misrecognition of non-command human voice can be greatly reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] To better understand the technical solution of the present disclosure, you may refer to the following drawings for auxiliary explanation of the prior art or embodiments. These drawings will selectively display the products or methods involved in the prior art or some embodiments of the present disclosure. The basic information of these drawings is as follows:

[0010] Figure 1 It is a flowchart of an embodiment of a human voice detection and speech recognition method based on a low-resource scenario of the present invention;

[0011] Figure 2 It is a structural schematic diagram of an embodiment of a human voice detection and speech recognition system based on a low-resource scenario of the present invention. DETAILED DESCRIPTION

[0012] The technical means or technical effects involved in the present disclosure will be further described below. Obviously, the embodiments provided are only some of the embodiments of the present disclosure, not all of them. Based on the embodiments in the present disclosure and the explicit or implicit descriptions of the figures and texts, all other embodiments that can be obtained by those skilled in the art without making creative efforts will be within the scope of protection of the present disclosure.

[0013] like Figure 1 As shown, the method in this embodiment includes the steps of:

[0014] S1. Divide the acoustic model into the shallow layer and the deep layer of the network; add human voice detection neurons in the shallow layer of the network.

[0015] The voice detection module and acoustic model module of the speech recognition model are mainly trained using transfer learning. The acoustic model in this example uses a time-delay neural network TDNN. The algorithm is not limited to the type of neural network and can be any deep neural network such as recursive neural network, convolutional neural network, Transformer, etc. The offline steps used by the neural network TDNN algorithm are: 1. Training of the speech recognition acoustic model, adjusted to the optimal error rate, the trained acoustic model is as follows Figure 1 As shown in . The rectangular box represents a 9-layer TDNN neural network. The layers are connected in series. The extracted features are called hidden layer features. Because they are inside the network and do not directly correspond to the input or output, they are called "hidden layer features". The last layer is the phoneme state output layer of the acoustic model. If it is a speech recognition task, the output layer outputs the probability of the phoneme state. If it is a VAD task, the output layer outputs the probability of each VAD state. The first three layers of the 9 layers are used as the shallow layer of the network, which contain the common acoustic data of the same category of tasks and are used to extract common acoustic data. These common acoustic data contain network layer parameters and have less computing power resources. The network layer parameters are added with 3 human voice detection neurons to form a human voice detection layer in the acoustic model. The 3 human voice detection neurons detect non-human voice state, human voice state, and possible human voice state respectively. The last six layers of the 9 layers are used as the deep layer of the network. They are more condensed and abstract, closer to specific tasks, and have more computing power resources. Then add an output layer with 3 speech recognition neurons to the deep layer of the network and perform fine-tuning training for 1 round. This is because more fine-grained speech phoneme recognition is required, so deep modeling capabilities are needed. The confidence level of the results obtained by using the deep network is higher, and it is less likely to miss speech when performing false wake-up suppression.

[0016] S2: The voice data first uses the shallow computing resources of the network and the voice detection neurons to determine whether it contains human voice; if so, enter S3; if not, enter S1.

[0017] Through the output of these three voice detection neurons, three state probability values ​​are trained for one round of fine-tuning. The network parameters of the shallow layer of the TDNN corresponding to the network are fixed unchanged. During the training, only the state probability values ​​of the three voice detection neurons are updated. Through training, the shallow layer of the network can distinguish between human voice and background noise. The maximum output probability of the three voice detection neurons corresponds to the "human voice state", which means that the human voice is detected.

[0018] The fine-tuning training of both the acoustic model and the human voice detection acoustic model adopts the following discriminative training criteria:

[0019]

[0020] Where W represents the word sequence of the sentence, θ represents the maximum likelihood estimate of the neural network parameters, 0 represents the observation sequence. θ (O|W) represents the acoustic probability model, P(W) represents the linguistic model. The above formula can be simplified as learning a neural network probability model on a speech data set D, and finding the parameter θ that maximizes the prediction probability. The general expression is:

[0021]

[0022] By freezing the network layer parameters in the shallow layer of the network for voice detection, the aim is to use the speech recognition knowledge θ to learn a low-resource human voice detection probability model represented by θ0 The human voice detection model for false wakeup suppression represented by θ1

[0023] Using the pre-trained model θ, by freezing the bottom layer of θ and fine-tuning 3 neurons, we get the human voice detection model θ1. This step aims to use the speech recognition knowledge θ to learn a low-resource human voice detection probability model represented by θ0. After the training is completed, when inferring θ1 on the device side, max(P(s0|θ1), P(s1|θ1), P(s2|θ1)) is used to determine whether the human voice is detected by selecting whether the maximum probability value corresponds to s1 in the above manner, where max represents the state corresponding to the maximum value of the three probabilities.

[0024] When the terminal device is inferring, in the default environment, only one-third (i.e., the shallow layer of the network) of the acoustic model θ1 is run in the background for voice detection. The voice detection neuron with the largest probability value in the voice detection output layer is selected. If it corresponds to a non-voice state, the currently processed voice frame is non-voice, and the subsequent forward calculation and decoding calculation of the neural network are not performed, thus saving a lot of resources and computing power. If in the preliminary judgment, the three voice detection neurons with the highest output probability correspond to the "voice state", it means that a human voice has been detected.

[0025] S3. Determine whether the phoneme duration of the speech data is within a preset duration range; if so, call the computing power resources deep in the network to recognize the speech data.

[0026] Since recognizing human voices without a purpose will also cause a lot of computing power resources to be wasted, it is necessary to suppress false voice wake-up of voice data. Since the output state of the deep network and the phoneme state output layer both use the softmax function (where e represents a natural number) is converted into a probability value and then used as the probability input of the speech recognition decoding graph.

[0027] The step of determining whether the duration of the phoneme of the voice data is within the preset duration range. The present embodiment adopts the statistical command word duration range to form a state diagram and compile it into the decoding diagram. The method is as follows: when composing the diagram, the phoneme states of the corresponding phonemes of the command word, such as "nihaoxiaolan", are repeated to the specified duration. Assuming that the frame shift is 30ms, one repetition of the state means that the pronunciation duration of the repeated phoneme is 30ms (preset duration range). During detection, first, according to the output of 3 neurons of the initial layer of the network human voice detection, it is determined whether it belongs to which of the three states according to the maximum probability value. Since it is processed frame by frame in real time, a state stream will be generated. If the state stream does not conform to the state repetition method during composition, it means that it is not a command word. Therefore, the command words that are not within the preset duration range will not be received by the state machine, so as to suppress those false awakenings with similar pronunciations of phonemes but different durations.

[0028] After determining that the phoneme duration of the speech data is within the preset duration range, the remaining two-thirds of the acoustic model (deep layer of the network), namely θ2 and θ, are forward calculated, and the human voice detection probability output is obtained after the softmax function. Acoustic model output P θ ,make Send it to the integrated decoding graph for Viterbi decoding to get the recognition result: The probability of the speech acoustic model is P AC , the cumulative probability of VAD state is P VAD , the decoding graph probability is P HCLG ;P HCLG In the decoding HCLG graph, H indicates that the modeling unit of the phoneme is a three-state HMM, C indicates the phoneme context, L indicates the speech dictionary, and G indicates the language model. Since H, C, L, and G all describe the state transition, they can all be represented by WFST and recorded as the decoding HCLG graph. argmax indicates taking the path with the highest probability in the search space W. Represents the text prediction corresponding to a recognized speech segment.

[0029]

[0030] To prevent calculation overflow errors caused by multiplication. Convert the probability to the negative logarithmic domain. To distinguish, it is no longer called probability after conversion, but score. The acoustic model score is obtained by calculation, so it is variable. The score of the graph is determined after compilation, which is called the intrinsic score of the graph. Find the path that minimizes the decoding graph score, which is the text result of speech recognition, that is,

[0031]

[0032] The human voice detection VAD state is written as a weighted state transition machine according to the phoneme duration. The above HCLG graph and VAD graph together constitute the inherent score logP VHCLG , the calculation of acoustic model scoring is consistent with the conventional speech recognition method. The recognition result can be expressed as

[0033]

[0034] The weighted finite state machine decoding graph DHCLG is composed of a human voice detection speech command duration state machine V, a hidden Markov model state machine H, a dictionary state machine L and a grammar model state machine G, as described by the following formula, and is obtained through the graph combination algorithm, the graph determinization algorithm det(), and the graph minimization algorithm min().

[0035]

[0036] like Figure 2 As shown, this embodiment also includes a human voice detection and speech recognition system based on a low-resource scenario, including: an acoustic model, a human voice detection module and a speech recognition module, and each module is signal-connected.

[0037] The acoustic model is divided into a shallow network layer and a deep network layer, and human voice detection neurons are added to the shallow network layer; the human voice detection module is used to use the computing power resources of the shallow network layer to process the voice data and after the human voice detection neurons determine that the voice contains human voice, the voice data is sent to the voice recognition module; the voice recognition module determines that the phoneme duration of the voice data is within a preset duration range, and calls the computing power resources of the deep network layer to recognize the voice data.

[0038] An embodiment of a human voice detection and speech recognition system based on a low-resource scenario is based on a method embodiment, which will not be described repeatedly here.

[0039] It can be understood by those skilled in the art that all or part of the steps in the embodiment can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable medium, which can include a flash drive, a mobile hard disk, a read-only memory, a random access memory, a disk or an optical disk, and other media that can store program codes. In one embodiment, the present disclosure provides a computer-readable medium, in which a computer program is stored, and the computer program is loaded and executed by a processing module to implement a human voice detection and speech recognition method.

[0040] Within the scope of the knowledge and ability level of those skilled in the art, the various embodiments or technical features mentioned herein may be combined with each other as other optional embodiments without conflict. These limited number of optional embodiments, which are not listed one by one and are formed by combining a limited number of technical features, still fall within the technical scope disclosed in the present disclosure and can be understood or inferred by those skilled in the art in combination with the drawings and the above text.

[0041] In addition, the description of most embodiments is based on different focuses. For details not described in detail, please refer to the content of the prior art or other relevant descriptions in this article for understanding.

[0042] It is emphasized again that the embodiments listed above are typical and preferred embodiments of the present disclosure, which are only used to explain and interpret the technical solutions of the present disclosure in detail to facilitate the understanding of the readers, and are not intended to limit the protection scope or application of the present disclosure. Any technical solutions obtained by any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A human voice detection and speech recognition method based on low-resource scenarios, characterized by: S1. Divide the acoustic model into a shallow network layer and a deep network layer; add human voice detection neurons to the shallow network layer; S2: The voice data first uses the shallow computing resources of the network and the voice detection neurons to determine whether it contains human voice; if so, enter S3; if not, enter S1; S3. Determine whether the phoneme duration of the voice data is within a preset duration range; if so, call the computing power resources deep in the network to recognize the voice data.

2. The method for human voice detection and speech recognition according to claim 1, characterized in that: Adding human voice detection neurons in the shallow layer of the network is specifically as follows: adding three human voice detection neurons to the network layer parameters in the shallow layer of the network, the three human voice detection neurons respectively detecting non-human voice state S0, human voice state S1, and possible human voice state S2; when the maximum probability of the output of the three human voice detection neurons is the human voice state, it means that the human voice is detected.

3. The method for human voice detection and speech recognition according to claim 2, characterized in that: The specific steps of the three voice detection neural outputs having the maximum probability of being in the voice state are: establishing a voice detection model, Where W represents the word sequence of the sentence, θ represents the maximum likelihood estimate of the neural network parameters, and O represents the observation sequence; P θ (O|W) represents the acoustic probability model, P(W) represents the linguistic model, and the simplified expression is to use the neural network probability model to learn on a speech data set D, and find the parameter θ that maximizes the prediction probability. The general expression is Use speech recognition knowledge θ to learn a low-resource human voice detection probability model represented by θ0 When inferring θ1, the maximum probability value corresponding to S1 in max(P(s0|θ1), P(s1|θ1), P(s2|θ1)) is that a human voice is detected.

4. The method for human voice detection and speech recognition according to claim 3, characterized in that: The human voice detection model represented by θ1 is used to suppress false wake-up Perform forward calculation on θ2 and θ to obtain the probability output of human voice detection Acoustic model output P θ , let P θ =P AC , Send it to the integrated decoding graph for Viterbi decoding to get the recognition result: The probability of the speech acoustic model is P AC , the cumulative probability of VAD state is P VAD , the decoding graph probability is P HCLH ;P HCLG In the decoding HCLG graph, H indicates that the modeling unit of the phoneme is a three-state HMM, C indicates the phoneme context, L indicates the speech dictionary, and G indicates the language model. Since H, C, L, and G all describe the state transition, they can all be represented by WFST and recorded as the decoding HCLG graph. argmax indicates taking the path with the highest probability in the search space W. Represents the text prediction corresponding to the recognized speech.

5. A human voice detection and speech recognition system based on low-resource scenarios, characterized by: include: Acoustic model, human voice detection module and speech recognition module, and signal connection of each module; The acoustic model is divided into a shallow network layer and a deep network layer, and human voice detection neurons are added to the shallow network layer; the human voice detection module is used to use the computing power resources of the shallow network layer to process the voice data and after the human voice detection neurons determine that the voice contains human voice, the voice data is sent to the voice recognition module; the voice recognition module determines that the phoneme duration of the voice data is within a preset duration range, and calls the computing power resources of the deep network layer to recognize the voice data.

6. The human voice detection speech recognition system according to claim 5, characterized in that: Adding human voice detection neurons in the shallow layer of the network is specifically as follows: adding three human voice detection neurons to the network layer parameters in the shallow layer of the network, the three human voice detection neurons respectively detecting non-human voice state S0, human voice state S1, and possible human voice state S2; when the maximum probability of the output of the three human voice detection neurons is the human voice state, it means that the human voice is detected.

7. The human voice detection speech recognition system according to claim 6, characterized in that: The specific steps of the three voice detection neural outputs having the maximum probability of being in the voice state are: establishing a voice detection model, Where W represents the word sequence of the sentence, θ represents the maximum likelihood estimate of the neural network parameters, and O represents the observation sequence; P θ (O|W) represents the acoustic probability model, P(W) represents the linguistic model, and the simplified expression is to use the neural network probability model to learn on a speech data set D, and find the parameter θ that maximizes the prediction probability. The general expression is Use speech recognition knowledge θ to learn a low-resource human voice detection probability model represented by θ0 When inferring θ1, the maximum probability value corresponding to S1 in max(P(s0|θ1), P(s1|θ1), P(s2|θ1)) is that a human voice is detected.

8. The human voice detection speech recognition system according to claim 7, characterized in that: The human voice detection model represented by θ1 is used to suppress false wake-up Perform forward calculation on Θ2 and θ to obtain the probability output of human voice detection Acoustic model output P θ , let P θ =P AC Send it to the integrated decoding graph for Viterbi decoding to get the recognition result: The probability of the speech acoustic model is P AC , the cumulative probability of VAD state is P VAD , the decoding graph probability is P HCLG ;P HCLG In the decoding HCLG graph, H indicates that the modeling unit of the phoneme is a three-state HMM, C indicates the phoneme context, L indicates the speech dictionary, and G indicates the language model. Since H, C, L, and G all describe the state transition, they can all be represented by WFST and recorded as the decoding HCLG graph. argmax indicates taking the path with the highest probability in the search space W. Represents the text prediction corresponding to the recognized speech.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes the human voice detection and speech recognition method according to any one of claims 1 to 4.

Images