A low-resource end-to-end fuzzy command word recognition system

Abstract

The embodiment of the present application provides a low-resource end-to-end fuzzy command word modeling method and related method and device, the fuzzy command word modeling method comprises: providing a data expansion method, simulating fuzzy command word data, and improving the recognition rate of the model on the fuzzy command word. The present application simultaneously provides an end-to-end two-stage model training method, taking the class number 0 as the background voice and taking the class number c (1<=c<=C) as the command word category, wherein C represents the number of command word categories. The class number is used as a basic modeling unit, and the model is trained by using the cross-entropy criterion. The present application also provides an end-to-end decoding method, which can complete the command word segment detection without using a separate VAD model, and can complete the decoding of the command word without using a decoding network, thereby greatly reducing the memory consumption of the system.

Description

TECHNICAL FIELD

[0001] The present application relates to speech recognition technology, and in particular to a low-resource end-to-end fuzzy command word recognition system. BACKGROUND

[0002] Command word recognition technology is an artificial intelligence technology that allows machines to convert corresponding voice instructions into corresponding hardware control instructions by recognizing and understanding voice commands. Traditional command word systems only need to recognize fixed types of command words. However, users sometimes add their own habitual expressions before and after the command words or embed them in other sentences, which changes the originally designed command words into fuzzy command words. How to improve the accurate recognition of fuzzy command words has become a new challenge.

[0003] Traditional command word recognition technology includes a voice activity detection (VAD) module, an acoustic model module, a decoding network construction module, and a speech decoding module. The voice activity detection module, the decoding network construction module, and the speech decoding module consume a large amount of memory, which greatly limits the promotion of the command word system on memory-constrained products. At the same time, the decoding network construction needs to know the text content of the command word in advance. When the input voice is a fuzzy command word, the system cannot know the accurate text content of the fuzzy command word, so it cannot establish an accurate decoding network, which greatly reduces the recognition accuracy of the traditional command word technology on fuzzy command words. SUMMARY

[0004] The purpose of the present application is to provide a low-resource end-to-end fuzzy command word modeling method and related methods and devices, which can more accurately recognize fuzzy command words and reduce memory consumption.

[0005] The following presents a simplified summary of one or more aspects in order to provide a basic understanding of such aspects. This summary is not an extensive overview of all contemplated aspects, and is intended to neither identify key or critical elements of all aspects nor delineate the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form as a prelude to the more detailed description that is presented later.

[0006] According to a first aspect of the present application, a low-resource end-to-end fuzzy command word modeling method is provided, comprising: regarding all background audio (all audio except command words) as a category, and using category number 0 to represent. And the command word audio is represented by category number c (1≤c≤C) according to its command word category, where C represents the category of command words, and different categories of command words use different category numbers. The set {c|1≤c≤C} represents the command word category set, and the set {c|0≤c≤C} represents the command word plus background category set. Time alignment labeling is generated for the audio in the training set, including labeling all time points of the background audio as number 0 and labeling time points within a period of time (for example, 70ms) before and after the end of the command word in the command word audio as the category number corresponding to the command word. The time alignment labeling and the audio features extracted from the training set are input into a training tool for training, the category number is used as a basic modeling unit, the cross-entropy criterion is used as a training objective function, and a first-stage acoustic model is obtained.

[0007] In an embodiment, the command word audio includes fuzzy command word audio, and the fuzzy command word audio is obtained by splicing the background audio and the command word audio.

[0008] In an embodiment, the time point corresponding to the command word voice is the end time point of the command word and a plurality of frames before and after the end time point.

[0009] In an embodiment, the end time point of the command word is obtained by inputting the command word voice into a general voice recognition model to generate time alignment information.

[0010] In an embodiment, the time alignment labeling of the audio in the training set further comprises: labeling time points corresponding to non-command word voice in the command word audio as category number -1, and the time points corresponding to category number -1 do not participate in the calculation of the acoustic model objective function.

[0011] In an embodiment, the training objective function is as follows:

[0012] J=∑ n ∑ k log(p(n,k,y n，k )),

[0013] Wherein n represents the audio label, k represents the frame step, y n，k represents the category number corresponding to the time labeling vector y of the nth audio at the kth frame, and p(n, k, y n，k ) represents the probability of the nth audio being predicted to be classified as category number y n，k at the kth frame by the acoustic model after forward operation.

[0014] In an embodiment, the method further comprises:

[0015] performing forward operation on the audio features of the training set using the first-stage acoustic model to obtain a prediction probability matrix p(n, k, c), wherein n represents audio number, k represents frame step, and c represents category number;

[0016] For the command word audio, the prediction probability peak value of the number corresponding to the command word is calculated, the frame corresponding to the prediction probability peak value is relabeled as the category number corresponding to the command word, and the other frames are labeled as category number-1.

[0017] In an embodiment, the frame corresponding to the prediction probability peak value is a plurality of frames before and after the time corresponding to the prediction probability peak value.

[0018] In an embodiment, the method further comprises: for the background audio, calculating the prediction probability value of category number 0, selecting the frame to be optimized according to the prediction probability value, relabeling the frame to be optimized as the first category number 0, and labeling the other frames as category number-1.

[0019] In an embodiment, the selecting the frame to be optimized according to the prediction probability value comprises: judging whether the prediction probability is less than a first threshold value, and selecting the frame less than the first threshold value as the frame to be optimized.

[0020] In an embodiment, the method further comprises: taking the first-stage acoustic model as an initial model, inputting the relabeled time alignment label and the audio features extracted from the training set into an optimizer for training, adopting cross-entropy criterion as a training objective function, and obtaining a second-stage acoustic model.

[0021] According to a second aspect of the present application, a command word segment detection method is provided, comprising: obtaining audio, extracting audio features, inputting an acoustic model obtained by the acoustic model modeling method of the first aspect, checking the prediction probability of category number 0 frame by frame, considering the frame audio command as background audio when the prediction probability of category number 0 is greater than a second threshold value, and considering the frame audio command as command word audio when the prediction probability of category number 0 is less than the second threshold value.

[0022] According to a third aspect of the present application, a command word segment decoding method is provided, comprising: obtaining audio, extracting audio features, inputting an acoustic model obtained by the acoustic model modeling method of the first aspect, calculating the prediction probability of each category number of the command word, and finding the corresponding command word according to the optimal category number obtained by maximum likelihood estimation.

[0023] According to a fourth aspect of the present application, a command word recognition method is provided, comprising:

[0024] obtaining audio to be recognized, extracting audio features, and inputting an acoustic model obtained by the low-resource end-to-end fuzzy command word modeling method of the first aspect.

[0025] frame by frame, when the prediction probability of the category number 0 is greater than the second threshold value, the frame audio command is considered as background voice, and when the prediction probability of the category number 0 is less than the second threshold value, the frame audio command is considered as command word voice, and adjacent command word frames are combined to obtain a command word segment;

[0026] The prediction probability of the corresponding category number of the command word is calculated, and the corresponding command word text is found according to the optimal category number obtained by maximum likelihood estimation.

[0027] According to a fifth aspect of the present application, a device is provided, comprising a memory and a processor;

[0028] The memory is configured to store executable instructions.

[0029] The processor is configured to implement the low-resource end-to-end fuzzy command word modeling method according to the first aspect when executing the executable instructions.

[0030] According to a sixth aspect of the present application, a device is provided, comprising a memory and a processor;

[0031] The memory is configured to store executable instructions.

[0032] The processor is configured to implement the command word segment detection method according to the second aspect or the command word segment decoding method according to the third aspect or the command word recognition method according to the fourth aspect when executing the executable instructions.

[0033] According to a sixth aspect of the present application, a readable storage medium is provided, the storage medium stores a program, and when the program is executed by a processor, the low-resource end-to-end fuzzy command word modeling method according to the first aspect, the command word segment detection method according to the second aspect, the command word segment decoding method according to the third aspect or the command word recognition method according to the fourth aspect is implemented.

[0034] The beneficial effects of the embodiments of the present application include: by distinguishing the background audio and the command word audio in the training set with different category numbers and taking the category number as a basic modeling unit, the obtained acoustic model can complete the detection of the command word segment without a separate VAD model, and the decoding does not need to generate a decoding network and a state grid, thereby reducing the memory consumption. BRIEF DESCRIPTION OF DRAWINGS

[0035] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some of the embodiments of the present application, and therefore should not be regarded as limiting the scope. For those skilled in the art, other related drawings can also be obtained without creative labor by reading the following detailed description of the embodiments of the present application in conjunction with the following drawings.

[0036] The above features and advantages of the present application can be better understood after reading the detailed description of the embodiments of the present application in conjunction with the following drawings. In the drawings, the components are not necessarily drawn to scale, and the components having similar related properties or features can have the same or similar reference numerals.

[0037] Figure 1 is an acoustic model training flowchart provided by the embodiments of the present application;

[0038] Figure 2 is a speech recognition flowchart provided by the embodiments of the present application;

[0039] Figure 3 is a fuzzy command word time-frequency spectrogram provided by the embodiments of the present application;

[0040] Figure 4 is a time alignment annotation schematic diagram of a fuzzy command word audio generated in a first stage;

[0041] Figure 5 is a process schematic diagram of generating an annotation vector of a fuzzy command word audio in a second stage;

[0042] Figure 6 is a process schematic diagram of generating an annotation vector of a background audio in a second stage. DETAILED DESCRIPTION

[0043] The present application will be described in detail below in conjunction with the drawings and specific embodiments. It should be noted that the aspects described below in conjunction with the drawings and specific embodiments are only exemplary and should not be understood as limiting the scope of protection of the present application.

[0044] The embodiments of the present application provide an acoustic model modeling method, comprising the following steps:

[0045] S1. Generating time alignment annotations for audios in a training set, comprising:

[0046] ①annotating all time points of the background audio as class number 0;

[0047] ②annotating time points corresponding to command word audios in the command word audio as class numbers c corresponding to the command words (1≤c≤C);

[0048] The command word audio refers to audio containing command words, and the background audio refers to audio not containing command word words. The command word word set [w1, w2, w3,..., wN] can be established according to the words contained in the command words. N ] where w N represents the words contained in the command words. For example, assuming that the command words are only "open WeChat" and "return to the watch face", the command word word set is [open, WeChat, return, watch face]. The data not containing the command word word set is filtered from the annotated text of general recognition data as the background audio.

[0049] It should be noted that the background audio contains all non-command word audio, and regardless of the text corresponding to the audio, it is classified into one category and assigned the number 0 as the category number. The command word audio is classified according to the type of command word, and different types of command words are classified into one category and are respectively assigned a number as the category number. For example, if there are C types of command words, the category number 0 represents the background audio, the category number 1 represents the first type of command word, 2 represents the second type of command word,..., and C represents the Cth type of command word. At the same time, a special category number -1 is created, and the features corresponding to this category will not be involved in the calculation of the objective function.

[0050] S2. The time alignment label and the audio features extracted from the training set are input into the training tool for training, the category number is used as the basic modeling unit, the cross entropy criteria is used as the training objective function, and the first-stage acoustic model is obtained.

[0051] By distinguishing the background audio and different types of command word audio with different category numbers, and using these category numbers as the basic modeling unit for modeling, when detecting the command word segment later, only the probability p(k, 0) that the kth frame is category 0 (i.e. background audio) needs to be calculated, and whether the kth frame is background audio or command word audio can be determined according to whether p(k, 0) is higher than the threshold, so that the VAD model can be replaced to complete the detection of the command word segment, and the calculation resource consumption and memory consumption caused by the VAD model can be saved.

[0052] Further, since different types of command words use different category numbers, when performing speech recognition later, the corresponding command word can be directly found according to the category number, without the need to create a decoding network and generate a state grid as in the traditional command word system, so that the memory consumption of the command word decoding part can be greatly reduced.

[0053] Generally, the setting of voice command words is usually fixed by product designers according to product function design, such as the command words "back to watch" in a smart watch, "open WeChat", "start exercise" and the like. However, in use, users sometimes add their own habitual expressions before and after the command words, or embed them in other sentences, so that the originally determined command words by the designers become uncertain and fuzzy command words. For example, the actual voice corresponding to the command word "open WeChat" can be "I want to open WeChat", "open WeChat, I want to check the friend circle", "the first thing I do every day is to open WeChat". In recent years, with the rapid development of wearable devices, smart home and smart car, in addition to the stringent requirements of product designers on the computing resources and storage resources of the command word recognition system, the command word system is also required to accurately recognize fuzzy command words.

[0054] Therefore, in possible embodiments, the command word audio in the training set also includes fuzzy command word audio. In order to expand the training data of fuzzy command words, the background audio can be spliced with the head or tail of the command word audio, and the spliced audio is used for training of the model, so as to improve the recognition accuracy of the acoustic model for fuzzy command word audio.

[0055] In possible embodiments, the obtaining process of a complete training set is as follows:

[0056] Step 101: Determine the command word set [w1, w2, w3,..., wN] according to the words contained in the command words, wherein wn represents the words contained in the command words. From the annotated text of general recognition data, data not containing the command word set is selected as the background audio training set.

[0057] Step 102: Use a general speech recognition model to generate time alignment information for the background audio and the command word audio, and determine the start time and end time of the voice in each audio according to the time alignment information. Randomly extract a background audio and pair it with the current command word audio. Assume that the text of the background audio is s1, the start time of the voice part is t1 seconds, the end time is t2 seconds, and the audio duration is t3 seconds. Assume that the text of the command word audio is s1, the start time of the voice part is t4 seconds, the end time is t5 seconds, and the audio duration is t6 seconds.

[0058] Step 103: Assume that the signal of the command word audio is s1(t), the signal of the command word audio is s2(t), and l = min(t3-t2, t4, 0.1) is obtained. First, perform energy normalization on the background audio signal s1(t), and let the average energy of the normalized signal be equal to the command word audio signal s2(t). Then, the head of the background audio and the command word audio is spliced using the Hanning window to obtain the audio signal s3(t), and the specific formula is as follows:

[0059]

[0060]

[0061] The obtained spliced audio can be divided into three parts, the first part t < t2 is the signal of background audio, where t = t1 is the start time of background speech, and t = t2 is the end time of background speech. The second part t2≤t < t2+l is the superposition of background audio and the silent part of command word audio. In order to prevent the generation of spectral leakage, a Hanning window is used in this part. The third part t2+l≤t < t2-t4+l+t6 is the signal of command word audio. Where t = t2+l is the start time of command word speech, and t = t2-t4+l+t6 is the end time of spliced audio. The time-frequency spectrum of the obtained spliced audio signal is shown in FIG. 3. It can be seen that the interval length of background audio speech and command word speech in time is l, and when l is small enough, the obtained spliced audio can be approximately equal to the ambiguous command word speech. In addition to splicing the head part of background audio and command word, the tail part can also be selected for splicing. The spliced audio is mixed as the ambiguous command word audio, the command word audio, and the background audio, and is used as the training set of the final command word model. Figure 3

[0062] In possible embodiments, the time corresponding to the command word speech in S1 is the end time of the command word t and the time within the first time period (for example, 70 milliseconds) before and after the end time. The end time of the command word can be obtained by inputting the command word speech into a general speech recognition model to generate time alignment information.

[0063] In addition, when generating time alignment labels for the audio in the training set, the time corresponding to the non-command word speech in the command word audio can also be labeled as class number -1, and the time corresponding to the class number -1 does not participate in the calculation of the acoustic model objective function, so as to improve the accuracy of the calculation recognition.

[0064] In possible embodiments, a complete time alignment labeling process for the training set is as follows:

[0065] ​Step 201: Generate time alignment labels for the command word audio, the ambiguous command word audio and the background audio in the training set respectively. For the command word audio and the ambiguous command word audio, the time alignment label generation methods are the same. First, determine the end time t of the command word (ambiguous command word) audio according to step 102 (step 103), and label the time points corresponding to 70 milliseconds on the left and right of t as the class number c (1≤c≤C) corresponding to the command word, and label the remaining time points as class number -1. The class number c (1≤c≤C) represents the corresponding class of the command word, which corresponds to the command word text one by one. Assuming that the number of command words is C, the class numbers of all command words range from 1 to C, and the class number -1 represents uncertainty and will not be involved in subsequent model training. The obtained time alignment labels are as shown in Figure 4

[0066] Step 202: For the background audio, label all time points as class number 0. Class number 0 represents a non-command word, which is used to model all audio signals (including silence, noise, human voice, etc.) other than command words. Class number 0 will perform the function of VAD to distinguish non-command word signals from command word signals in the audio.

[0067] Step 203: Frame the time alignment labels in steps 201 and 202 according to 10 milliseconds as one frame, and convert the time alignment labels into time label vectors of command word numbers. For example, a certain command word audio has a duration of 200 milliseconds, and the command word termination time is 100 milliseconds (only an example, the actual command word audio duration is generally between 0.5 seconds and 2 seconds), and the command word corresponds to the number 5. According to step 201, the 30 milliseconds to 170 milliseconds part will be labeled as 5, and the remaining time points will be labeled as -1. After 10 milliseconds of framing, the final label vector is [-1, -1, -1, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, -1, -1, -1].

[0068] S2 specifically includes:

[0069] Step 204: After framing and windowing the audio in the training set, extract the mel-frequency cepstrum features, and send the mel-frequency cepstrum features and the time alignment labels obtained in steps 201 and 202 to the training tool for training. The model structure uses a one-dimensional convolutional network, and the model receptive field is set to 160. Use the class number as the basic modeling unit of the acoustic model. Assuming that the number of command word classes is C, the total number of modeling units is C+1, and the total number of acoustic model output layer classes is also C+1. Cross-entropy criterion is used as the acoustic model training objective function, and its formula is as follows:

[0070]

[0071] ​where n denotes audio index, k denotes frame step, y n，k denotes the time label vector y of the nth audio. p(n, k, y n，k ) denotes the probability of the nth audio being classified as the command word y n，k at the kth frame by the acoustic model. When y n，k is equal to -1, p(n, k, y n，k ) will be discarded and not involved in the calculation of the objective function J. After the objective function J is calculated, the gradient of the acoustic model parameters is calculated using the back propagation algorithm, and finally the acoustic model parameters are updated using the stochastic gradient algorithm. After training, the first stage acoustic model is obtained.

[0072] Further, in the training process of the above-mentioned first stage acoustic model, the label vector is obtained based on the general speech recognition model, and is not necessarily suitable for the training of the command word model taking the class number as the modeling unit. Therefore, the first stage acoustic model can be used as the basis to re-label and train to obtain a more accurate second stage acoustic model.

[0073] The second stage acoustic model training process includes:

[0074] Step 301: using the first stage acoustic model obtained in step 204 to perform forward operation on the training set mel-cepstral feature to obtain a prediction probability matrix p(n, k, c), where n denotes audio number, k denotes frame step, and c denotes class number.

[0075] When the audio n is a command word (ambiguous command word) audio, assuming the command word number is c n , the time k n，peak corresponding to the peak of the prediction probability is calculated according to the following formula:

[0076]

[0077] Then, the frames (for example, 5 frames) around k n，peak are labeled as the command word number c n , and the remaining frames are labeled as -1 to obtain a new time label vector of the audio for two-stage model training.

[0078] Figure 5 The above process is shown in the figure, and the position corresponding to the highest arrow in the figure is k n，peak , and the frames around k n，peak are labeled as c nThe rest of the time is marked as -1. Unlike the annotation vector generated in step 203, this step does not use the general speech recognition model, but uses the annotation vector generated by the command word model in step 204, so it is more suitable for the training of the command word model.

[0079] Preferably, the second stage of training further comprises step 302: when the audio n is background audio, the class number is 0, then calculate the time set k corresponding to p(n, k, 0) less than a threshold value e (for example, 0.5) according to the following formula low .

[0080] k low ={k|p(n, k, 0) < e}

[0081] Then mark the frames belonging to the set k low as 0, and mark the rest of the time as -1. Figure 6 The above process is shown, and the position corresponding to the arrow below the dashed line represents the set k low . The set k low represents frames with lower prediction probability, and marking these frames as 0 and the rest of the frames with higher prediction probability as -1 can target the optimization of frames with lower prediction probability in the second stage of training.

[0082] Step 303: The training set annotation vector obtained in steps 301 and 302 is sent to the optimizer together with the training set mel cepstrum features for training, using the one-stage acoustic model obtained in step 204 as the initial model, adjusting the training learning rate to one-tenth of that in step 204, and the objective function is still J, using back propagation to calculate the gradient of the acoustic model parameters, and finally using the stochastic gradient algorithm to update the acoustic model parameters. After training, the second-stage acoustic model can be obtained.

[0083] A complete acoustic model training modeling process from step 101 to step 303 is shown in Figure 1 .

[0084] Test the above acoustic model with a short audio:

[0085] After the short audio is framed and windowed, the mel-frequency cepstrum feature is extracted, and then sent to the second stage acoustic model obtained in step 303 for forward operation to obtain a prediction probability matrix p(k, c), where k represents a frame step, and c represents a class number. Since the model uses the class number as a modeling unit, and the class number 0 is trained using the text-rich background data, the VAD model can be replaced by p(k, 0). Tests show that when the input audio is background audio, the value of the vector p(k, 0) is higher than 0.5, and when the input audio is a command word (a fuzzy command word), the probability of p(k, 0) near the end of the command word is lower than 0.5, and the probability at other times is higher than 0.5. Based on this feature, p(k, 0) can be used to directly replace the VAD model to complete the detection of the command word segment under long audio, thereby saving the calculation resource consumption and memory consumption caused by the VAD model.

[0086] On this basis, the embodiment of the present application provides a command word segment detection method, comprising: obtaining audio, extracting audio features, inputting the acoustic model obtained by the acoustic model modeling method, checking the prediction probability of the class number 0 frame by frame, when the prediction probability of the class number 0 is greater than a second threshold, considering that the frame audio command is background audio, when the prediction probability of the class number 0 is less than the second threshold, considering that the frame audio command is a command word audio.

[0087] For example, given a long audio (which may contain multiple command words, and the command words may contain non-command word speech between them), the audio is framed and windowed to extract the mel-frequency cepstrum feature, and then sent to the acoustic model for forward operation to obtain a probability matrix p(k, c). Check p(k, 0) frame by frame, if p(k, 0) is less than 0.5, it is considered that the frame is a command word frame, if p(k, 0) is greater than 0.5, it is considered that the frame is a non-command word frame. Combining adjacent command word frames together can obtain a command word segment.

[0088] It is easy to understand that different command words can also be implemented by using one-to-one corresponding numbers for decoding. Therefore, the embodiment of the present application also provides a command word segment decoding method, comprising: obtaining audio, extracting audio features, inputting the acoustic model obtained by the acoustic model modeling method, calculating the prediction probability of the number corresponding to the command word, and finding the corresponding command word according to the number obtained by maximum likelihood estimation.

[0089] Suppose the command word segment obtained by the command word segment detection method is Ω = {m, m+1, m+2,..., m+L-1}, where L represents the length of the segment, m represents the frame step corresponding to the first frame of the command word segment, and Ω represents the set of all frames of the segment. The likelihood of each command word is calculated respectively, and the maximum likelihood estimation is performed using the following formula:

[0090]

[0091] wherein the symbol represents the class number obtained by maximum likelihood estimation, and the symbol is converted into corresponding command word text and output to the terminal. The above steps are repeated for the next command word frame set until all sets are processed. Since no decoding network needs to be created and no state grid needs to be generated in the decoding process as in the traditional command word system, the memory consumption of the command word decoding part is greatly reduced.

[0092] The command word segment detection method and the decoding method are combined to obtain a command word recognition method, as shown in Figure 2 which includes the following steps: step 401, input audio and extract features, input into an acoustic model, and perform forward operation; step 402, detect command word frames by the command word segment detection method, and combine adjacent command word frames to obtain command word segments; and step 403, determine corresponding command words by the command word segment decoding method.

[0093] It is easily understood that the embodiments of the present application also provide a device for acoustic model modeling, which includes a memory and a processor.

[0094] The memory can be used to store instructions, programs, codes, code sets or instruction sets. The memory can include a program storage area and a data storage area, wherein the program storage area can store instructions for implementing an operating system, instructions for at least one function, and instructions for implementing the above-mentioned acoustic model modeling method, etc.; and the data storage area can store data involved in the above-mentioned acoustic model modeling method, etc.

[0095] The processor can include one or more processing cores. The processor executes various functions and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory, and calling data stored in the memory. The processor can be at least one of an application specific integrated circuit, a digital signal processor, a digital signal processing device, a programmable logic device, a field programmable gate array, a central processing unit, a controller, a microcontroller and a microprocessor. It can be understood that for different devices, the electronic devices used to implement the functions of the above-mentioned processor can also be other devices, and the embodiments of the present application are not limited specifically.

[0096] The embodiments of the present application also provide a device, which can be a smart speaker, a vehicle-mounted terminal, a mobile terminal, etc. The device includes a memory and a processor. When the processor executes executable instructions in the memory, the above-mentioned command word segment detection method, or the command word segment decoding method, or the command word recognition method can be implemented.

[0097] The foregoing method of the embodiments of the present application, if realized in the form of a software function module and sold or used as an independent product, can also be stored in a computer readable storage medium. Based on such an understanding, the technical solutions of the embodiments of the present application can be embodied in the form of a software product in essence or the part that contributes to the prior art. The computer software product is stored in a storage medium, and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in the embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk, and various media that can store program codes. Thus, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0098] In summary, the embodiments of the present application provide a low-resource end-to-end fuzzy command word modeling method. The acoustic model obtained by the method does not need to construct two modules of speech endpoint detection and decoding network, and simplifies the complexity of the command word system. Since the decoding network and the state grid are not created, the memory consumption of the speech decoding part (excluding the acoustic model part) can be controlled within 10Kb, which greatly expands the promotion of the command word system on the memory limited product, and directly supports fuzzy command word recognition.

[0099] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0100] The foregoing description of the present disclosure has been provided for purposes of illustrating the principles of the present disclosure and its practical application. Various modifications to the present disclosure will be apparent to those skilled in the art, and the general principles defined herein can be applied to other variations without departing from the spirit or scope of the disclosure. Thus, the present disclosure is not intended to be limited to the examples described herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0101] The above is only a preferred embodiment of the present application, and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the scope of protection of the present application.

Claims

1. A low-resource end-to-end method for modeling ambiguous command words, the method comprising: The method comprises the following steps: Time alignment labels are generated for the audio in the training set, including labeling all time instances of the background audio as class number 0 and labeling the time instances corresponding to the command word speech in the command word audio as class number c ), where C represents the number of command word classes. The time alignment label and the audio features extracted from the training set are input into a training tool, a category number is used as a basic modeling unit, a cross-entropy criterion is used as a training target function, and a first-stage acoustic model is obtained; The background speech is indicated by using the category number 0, and the command word category is indicated by using the category number c( ). Different command words of different categories correspond to different category numbers, and a set of command word categories is indicated by The command word audio comprises ambiguous command word audio, and the ambiguous command word audio is obtained by splicing background audio and command word audio; The command word training label corresponds to a time point at the end of the command word and a time point in a time period before and after the end of the command word; A time point corresponding to non-command word speech in the command word audio is labeled as a category number-1, and a time point corresponding to the category number-1 is not involved in calculation of an acoustic model target function; Forward operation is performed on the audio features of the training set using the first-stage acoustic model to obtain a prediction probability matrix wherein n represents audio number, k represents frame step, and c represents category number.

2. The low-resource end-to-end fuzzy command word modeling method of claim 1, wherein: The end time of the command word is obtained by inputting the command word speech into a general speech recognition model to generate time alignment information.

3. The low-resource end-to-end fuzzy command word modeling method of claim 1, wherein, The training target function is as follows: wherein represents an audio number, represents a frame step, represents a time label vector corresponding to the nth audio represents a class number corresponding to the kth frame, represents a probability that the kth frame of the nth audio predicted by the acoustic model through forward operation is classified as the class number .

4. The low-resource end-to-end fuzzy command word modeling method according to claim 3, characterized in that, The method further comprises the following steps: For the command word audio n, assume its class number is , find the frame corresponding to the maximum value , re-label it and its left and right 5 frames with the class number , and label other frames as -1.

5. The low-resource end-to-end fuzzy command word modeling method of claim 4, wherein, The method further comprises the following steps: Forward operation is performed on the background audio features of the training set using the first-stage acoustic model to obtain a prediction probability matrix According to the prediction probability The frame to be optimized is selected and predicted, the frame to be optimized is relabeled as class number 0, and other frames are labeled as class number -1.

6. The low-resource end-to-end fuzzy command word modeling method of claim 5, wherein, The prediction probability The frame needing optimization is selected according to the prediction probability Whether the prediction probability is less than a first threshold value, and the frame less than the first threshold value is selected as the frame needing optimization.

7. The low-resource end-to-end method for modeling fuzzy command words of claim 6, wherein, The method further comprises the following steps: taking the first-stage acoustic model as an initial model, inputting the time alignment label after the re-labeling and the audio features extracted from the training set into an optimizer for training, using a cross-entropy criterion as a training target function, and obtaining a second-stage acoustic model.

8. A command phrase fragment detection method characterized by, The method comprises the following steps: Acquire audio, extract audio features, input the acoustic model obtained by the low-resource end-to-end fuzzy command word modeling method in any one of claims 1-7, and check the prediction probability of number 0 frame by frame When greater than the second threshold value, the frame audio is considered as background speech, and when less than the second threshold value, the frame audio is considered as a command word speech.

9. A command word fragment decoding method characterized by comprising: The method comprises the following steps: Audio is obtained, audio features are extracted, an acoustic model obtained by the low-resource end-to-end ambiguous command word modeling method in claim 1 is input, a prediction probability of a number corresponding to a command word is calculated, and a corresponding command word text is found according to a number obtained by maximum likelihood estimation.

10. A command word recognition method characterized by comprising: The method comprises the following steps: Audio to be recognized is obtained, audio features are extracted, and an acoustic model obtained by the low-resource end-to-end ambiguous command word modeling method in claim 2 is input; Frame-by-frame checking prediction probability of number 0 When greater than the second threshold value, the frame audio is considered as background voice, and when less than the second threshold value, the frame audio is considered as command word voice, and adjacent command word voice frames are combined to obtain a command word segment; A prediction probability of a number corresponding to a command word is calculated, and a corresponding command word text is found according to a number obtained by maximum likelihood estimation.

11. An apparatus, comprising: The device comprises a memory and a processor; The memory is configured to store executable instructions; The processor is configured to implement the low-resource end-to-end ambiguous command word modeling method in any one of claims 1 to 7 when the executable instructions are executed.

12. An apparatus, comprising: The device comprises a memory and a processor; The memory is configured to store executable instructions; The processor is configured to implement the command word segment detection method in claim 8 or the command word segment decoding method in claim 9 or the command word recognition method in claim 10 when the executable instructions are executed.

13. A readable storage medium characterized by: The storage medium stores a program, and when the program is executed by the processor, the low-resource end-to-end ambiguous command word modeling method in any one of claims 1 to 7, the command word segment detection method in claim 8, the command word segment decoding method in claim 9, or the command word recognition method in claim 10 is implemented.

Images